2024-09-13T14:03:42.512 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T14:03:42.541 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T14:03:42.648 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903558 branch: wip-aclamk-cbt-combined description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} email: akupczyk@redhat.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps - echo wait for minority of mons to upgrade - while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error; do sleep 2 ; done - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for majority of mons to upgrade - 'while ! ceph mon versions | grep $sha1 | egrep '': [23]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for all mons to upgrade - 'while ! ceph mon versions | grep $sha1 | grep '': 3'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for half of osds to upgrade - 'while ! ceph osd versions | grep $sha1 | egrep '': [45678]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch upgrade pause - ceph orch ps - ceph orch ps - ceph versions first-half-tasks: - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' first_in_suite: false job_id: '7903558' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run basic import/export cli tests for rbd ' name: akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSDMAP_FLAGS - PG_AVAILABILITY sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - ceph orch upgrade resume - sleep 60 - echo wait for upgrade to complete - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' seed: 6545 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: quincy clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 682/13280 suite: upgrade suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH2WsxwxvF6VL1Mb/1+PhXh9o8ufWa6zXLFbbBnscWKkObDmDNFaUdVuRA4sv2T60KMbf+KfrC+tMTXV8Ris+BA= smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLgxO4kXugsY56ZS4rvSOCyeBNTWaCsGcGlYrJ1e/2wo34qhPEWuyge1uZflJ3Y8pcWcyMGm5SuY8+TnmyaJFUA= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: branch: quincy exclude_packages: - ceph-volume - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_10:00:18 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T14:03:42.648 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T14:03:42.649 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T14:03:42.650 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T14:03:42.651 INFO:teuthology.task.internal:Checking packages... 2024-09-13T14:03:42.675 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T14:03:42.675 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:03:42.676 INFO:teuthology.packaging:ref: None 2024-09-13T14:03:42.676 INFO:teuthology.packaging:tag: None 2024-09-13T14:03:42.676 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T14:03:42.676 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:03:42.676 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T14:03:42.832 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T14:03:42.834 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T14:03:42.839 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T14:03:42.839 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T14:03:42.858 INFO:teuthology.task.internal:Saving configuration 2024-09-13T14:03:42.881 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T14:03:42.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T14:03:42.920 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903558', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:58:57.666947', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH2WsxwxvF6VL1Mb/1+PhXh9o8ufWa6zXLFbbBnscWKkObDmDNFaUdVuRA4sv2T60KMbf+KfrC+tMTXV8Ris+BA='} 2024-09-13T14:03:42.950 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903558', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:58:57.668927', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLgxO4kXugsY56ZS4rvSOCyeBNTWaCsGcGlYrJ1e/2wo34qhPEWuyge1uZflJ3Y8pcWcyMGm5SuY8+TnmyaJFUA='} 2024-09-13T14:03:42.950 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T14:03:42.956 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-13T14:03:42.956 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-13T14:03:42.956 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T14:03:43.037 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa322da7be0>, signals=[15]) 2024-09-13T14:03:43.037 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T14:03:43.042 INFO:teuthology.task.internal:Opening connections... 2024-09-13T14:03:43.042 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-09-13T14:03:43.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:03:43.149 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-09-13T14:03:43.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:03:43.235 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T14:03:43.243 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-09-13T14:03:43.261 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-09-13T14:03:43.261 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-09-13T14:03:43.318 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T14:03:43.319 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T14:03:43.320 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-09-13T14:03:43.355 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-09-13T14:03:43.373 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-09-13T14:03:43.374 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-09-13T14:03:43.431 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2024-09-13T14:03:43.432 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T14:03:43.433 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T14:03:43.433 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T14:03:43.433 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-09-13T14:03:43.463 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T14:03:43.472 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T14:03:43.483 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T14:03:43.483 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-09-13T14:03:43.486 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-09-13T14:03:43.503 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T14:03:43.508 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T14:03:43.509 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-09-13T14:03:43.543 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-09-13T14:03:43.573 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T14:03:43.647 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T14:03:43.662 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T14:03:43.662 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T14:03:43.662 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T14:03:43.663 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T14:03:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:03:43.680 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-09-13T14:03:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:03:43.681 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-09-13T14:03:43.737 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-503.el9.x86_64 2024-09-13T14:03:43.737 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-503.el9.x86_64 2024-09-13T14:03:43.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-09-13T14:03:43.739 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-503.el9.x86_64 2024-09-13T14:03:43.739 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-503.el9.x86_64 2024-09-13T14:03:43.739 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-09-13T14:03:45.983 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-13T14:03:46.190 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-13T14:03:49.625 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 9.0 MB/s | 20 MB 00:02 2024-09-13T14:03:55.092 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.5 MB 00:01 2024-09-13T14:03:56.725 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 19 kB 00:00 2024-09-13T14:03:58.505 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-13T14:04:04.110 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-13T14:04:05.932 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:18 2024-09-13T14:04:06.250 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T14:04:06.250 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T14:04:06.338 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:04:06.345 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:04:06.345 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-09-13T14:04:06.345 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:04:06.345 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:04:06.345 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T14:04:06.346 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:04:06.346 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T14:04:06.346 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T14:04:06.346 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T14:04:06.346 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:04:06.346 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:04:06.346 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:04:06.346 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-09-13T14:04:06.346 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:04:06.347 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 79 M 2024-09-13T14:04:06.347 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 126 M 2024-09-13T14:04:06.347 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:04:06.687 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 78 kB/s | 23 kB 00:00 2024-09-13T14:04:10.274 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-13T14:04:11.800 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-13T14:04:12.821 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-13T14:04:18.527 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-13T14:04:20.634 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T14:04:20.634 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T14:04:20.726 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:04:20.738 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:04:20.738 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-09-13T14:04:20.738 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:04:20.738 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:04:20.738 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T14:04:20.738 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 79 M 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 126 M 2024-09-13T14:04:20.739 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:04:21.120 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 180 kB/s | 23 kB 00:00 2024-09-13T14:04:22.429 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 24 MB/s | 35 MB 00:01 2024-09-13T14:04:22.612 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 9.7 MB/s | 16 MB 00:01 2024-09-13T14:04:22.869 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 973 kB/s | 16 MB 00:16 2024-09-13T14:04:23.637 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 11 MB/s | 29 MB 00:02 2024-09-13T14:04:23.638 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:04:23.638 INFO:teuthology.orchestra.run.smithi019.stdout:Total 27 MB/s | 79 MB 00:02 2024-09-13T14:04:24.352 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:04:24.717 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:04:24.718 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:04:25.741 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:04:25.741 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:04:28.331 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:04:28.998 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:04:29.080 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:04:31.849 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:04:31.974 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:04:35.293 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 1.0 MB/s | 29 MB 00:28 2024-09-13T14:04:35.601 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 1.2 MB/s | 35 MB 00:29 2024-09-13T14:04:35.601 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:04:35.601 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.7 MB/s | 79 MB 00:29 2024-09-13T14:04:36.308 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:04:36.679 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:04:36.679 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:04:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:04:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:04:37.649 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:04:37.650 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:04:40.232 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:04:40.905 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:04:40.986 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:04:42.288 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:04:43.794 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:04:43.922 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:04:49.043 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:04:49.089 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:04:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:05:21.545 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:05:21.586 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:05:21.949 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:05:21.949 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:05:21.949 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:05:21.949 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:05:22.775 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:05:22.775 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:05:22.775 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:05:22.776 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T14:05:22.776 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T14:05:22.776 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T14:05:22.776 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T14:05:22.776 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:05:22.776 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:05:23.127 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:05:23.219 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:05:23.220 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:05:23.220 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T14:05:23.221 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T14:05:23.221 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T14:05:23.221 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T14:05:23.221 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T14:05:23.221 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:05:23.280 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:05:23.280 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:05:23.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -qi grub2-tools 2024-09-13T14:05:23.316 INFO:teuthology.orchestra.run.smithi019.stdout:Name : grub2-tools 2024-09-13T14:05:23.316 INFO:teuthology.orchestra.run.smithi019.stdout:Epoch : 1 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Version : 2.06 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Release : 82.el9 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: x86_64 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Group : Unspecified 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Size : 8274670 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:License : GPLv3+ 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Packager : builder@centos.org 2024-09-13T14:05:23.317 INFO:teuthology.orchestra.run.smithi019.stdout:Vendor : CentOS 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout:Summary : Support tools for GRUB. 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout:Description : 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout:hardware devices. 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:05:23.318 INFO:teuthology.orchestra.run.smithi019.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T14:05:23.319 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T14:05:23.319 INFO:teuthology.task.kernel:Updating grub on smithi019 to boot 5.14.0-505.el9.x86_64 2024-09-13T14:05:23.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T14:05:23.963 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2024-09-13T14:05:25.317 INFO:teuthology.orchestra.run.smithi019.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T14:05:25.349 INFO:teuthology.orchestra.run.smithi019.stderr:done 2024-09-13T14:05:25.357 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-09-13T14:05:25.373 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.mi43cEfjPw 2024-09-13T14:05:25.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mi43cEfjPw 2024-09-13T14:05:25.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.mi43cEfjPw 2024-09-13T14:05:25.587 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.mi43cEfjPw is 6KB 2024-09-13T14:05:25.636 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.mi43cEfjPw 2024-09-13T14:05:25.651 DEBUG:teuthology.orchestra.run.smithi019:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T14:05:25.720 INFO:teuthology.orchestra.run.smithi019.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T14:05:25.720 INFO:teuthology.orchestra.run.smithi019.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T14:05:25.720 INFO:teuthology.orchestra.run.smithi019.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T14:05:25.720 INFO:teuthology.orchestra.run.smithi019.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T14:05:25.721 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T14:05:25.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2024-09-13T14:05:32.995 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:05:33.035 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:05:33.463 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:05:33.464 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:05:33.464 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:05:33.464 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:05:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:05:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:05:34.308 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:05:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T14:05:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T14:05:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T14:05:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T14:05:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:05:34.309 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:05:34.655 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:05:34.749 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:05:34.749 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:05:34.749 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T14:05:34.750 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T14:05:34.750 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T14:05:34.750 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T14:05:34.750 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T14:05:34.750 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:05:34.810 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:05:34.810 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:05:34.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2024-09-13T14:05:34.848 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2024-09-13T14:05:34.849 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 82.el9 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8274670 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T14:05:34.850 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:05:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T14:05:34.854 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T14:05:34.854 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-505.el9.x86_64 2024-09-13T14:05:34.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T14:05:35.539 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-09-13T14:05:36.860 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T14:05:36.889 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-09-13T14:05:36.901 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-09-13T14:05:36.916 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.iwWEV1gxSJ 2024-09-13T14:05:36.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iwWEV1gxSJ 2024-09-13T14:05:36.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.iwWEV1gxSJ 2024-09-13T14:05:37.118 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.iwWEV1gxSJ is 6KB 2024-09-13T14:05:37.166 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.iwWEV1gxSJ 2024-09-13T14:05:37.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T14:05:37.249 INFO:teuthology.orchestra.run.smithi003.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T14:05:37.250 INFO:teuthology.orchestra.run.smithi003.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T14:05:37.250 INFO:teuthology.orchestra.run.smithi003.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T14:05:37.250 INFO:teuthology.orchestra.run.smithi003.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T14:05:37.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T14:05:37.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-09-13T14:05:55.905 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T14:05:55.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-13T14:05:55.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:06:07.422 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T14:06:07.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-13T14:06:07.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:06:14.333 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2024-09-13T14:06:23.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-13T14:06:23.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:06:25.853 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-09-13T14:06:34.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-13T14:06:34.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:07:23.394 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T14:07:34.881 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T14:07:35.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-13T14:07:35.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:07:35.769 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-09-13T14:07:36.124 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-13T14:07:36.124 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T14:07:36.124 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-09-13T14:07:36.141 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-505.el9.x86_64 2024-09-13T14:07:36.142 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T14:07:36.142 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T14:07:36.142 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T14:07:37.143 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T14:07:37.143 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T14:07:37.210 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2024-09-13T14:07:37.231 DEBUG:teuthology.parallel:result is None 2024-09-13T14:07:46.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-13T14:07:46.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:07:47.276 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-09-13T14:07:47.631 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-13T14:07:47.631 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T14:07:47.631 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-09-13T14:07:47.648 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-505.el9.x86_64 2024-09-13T14:07:47.648 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T14:07:47.648 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T14:07:47.648 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T14:07:48.649 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T14:07:48.649 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T14:07:48.723 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-09-13T14:07:48.754 DEBUG:teuthology.parallel:result is None 2024-09-13T14:07:48.754 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T14:07:48.762 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T14:07:48.762 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T14:07:48.765 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T14:07:48.784 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T14:07:48.799 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T14:07:48.843 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T14:07:48.844 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T14:07:48.847 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T14:07:48.883 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T14:07:48.894 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T14:07:48.895 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T14:07:48.918 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T14:07:48.952 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:48.960 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:48.963 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:48.972 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:48.974 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T14:07:48.985 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T14:07:48.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T14:07:49.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T14:07:49.043 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T14:07:49.055 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T14:07:49.055 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T14:07:49.078 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T14:07:49.101 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:07:49.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:07:49.233 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:07:49.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:07:49.361 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:07:49.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T14:07:49.429 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:07:49.483 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:07:49.526 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:07:49.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:07:49.657 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:07:49.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T14:07:49.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-09-13T14:07:49.728 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-09-13T14:07:49.769 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T14:07:49.817 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T14:07:50.222 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T14:07:50.232 INFO:teuthology.task.internal:Starting timer... 2024-09-13T14:07:50.232 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T14:07:50.383 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T14:07:50.463 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-13T14:07:50.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-09-13T14:07:50.516 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-09-13T14:07:50.518 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-09-13T14:07:50.569 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-09-13T14:07:50.571 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T14:07:50.571 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-09-13T14:07:50.596 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-09-13T14:07:50.597 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-09-13T14:07:50.641 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-09-13T14:07:50.641 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi019.front.sepia.ceph.com': 'permissive'} 2024-09-13T14:07:50.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T14:07:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:07:50.670 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T14:07:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:07:50.710 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T14:07:50.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-09-13T14:07:50.742 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-09-13T14:07:50.790 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T14:07:50.801 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-13T14:07:50.804 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-13T14:07:50.919 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T14:07:50.930 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T14:07:50.931 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi019.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T14:11:44.616 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi019.front.sepia.ceph.com')] 2024-09-13T14:11:44.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-13T14:11:44.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:11:44.727 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-09-13T14:11:44.806 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-13T14:11:44.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-13T14:11:44.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:11:44.891 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-09-13T14:11:44.971 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-13T14:11:44.972 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T14:11:44.982 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T14:11:44.983 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T14:11:44.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T14:11:44.986 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T14:11:44.986 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T14:11:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T14:11:45.031 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T14:11:45.047 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T14:11:45.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-09-13T14:11:45.063 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T14:11:45.069 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-09-13T14:11:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T14:11:45.090 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-09-13T14:11:45.099 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T14:11:45.103 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-09-13T14:11:45.118 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T14:11:45.133 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T14:11:45.160 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-09-13T14:11:45.164 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T14:11:45.164 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-09-13T14:11:45.164 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:11:45.164 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:11:45.164 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:11:45.164 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:11:45.192 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-09-13T14:11:45.195 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T14:11:45.196 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-09-13T14:11:45.196 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:11:45.196 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:11:45.196 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:11:45.196 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:11:45.197 INFO:teuthology.run_tasks:Running task pexec... 2024-09-13T14:11:45.206 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-13T14:11:45.206 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi003.front.sepia.ceph.com 2024-09-13T14:11:45.206 DEBUG:teuthology.orchestra.run.smithi003:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T14:11:45.206 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi019.front.sepia.ceph.com 2024-09-13T14:11:45.207 DEBUG:teuthology.orchestra.run.smithi019:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T14:11:45.573 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:11:45.590 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:11:45.590 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-09-13T14:11:45.590 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:11:45.590 INFO:teuthology.orchestra.run.smithi003.stdout:Removing: 2024-09-13T14:11:45.590 INFO:teuthology.orchestra.run.smithi003.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T14:11:45.590 INFO:teuthology.orchestra.run.smithi003.stdout:Removing unused dependencies: 2024-09-13T14:11:45.590 INFO:teuthology.orchestra.run.smithi003.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T14:11:45.590 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:11:45.591 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:11:45.591 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:11:45.591 INFO:teuthology.orchestra.run.smithi003.stdout:Remove 2 Packages 2024-09-13T14:11:45.591 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:11:45.591 INFO:teuthology.orchestra.run.smithi003.stdout:Freed space: 5.5 M 2024-09-13T14:11:45.591 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:11:45.595 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:11:45.595 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:11:45.606 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:11:45.620 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:11:45.620 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:Removing: 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:Removing unused dependencies: 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:Remove 2 Packages 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:Freed space: 5.5 M 2024-09-13T14:11:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:11:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:11:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:11:45.654 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:11:45.655 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:11:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:11:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:11:45.802 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:11:45.827 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:11:45.876 INFO:teuthology.orchestra.run.smithi003.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T14:11:45.897 INFO:teuthology.orchestra.run.smithi019.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T14:11:45.932 INFO:teuthology.orchestra.run.smithi003.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:11:45.965 INFO:teuthology.orchestra.run.smithi019.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:11:46.422 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:11:46.422 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T14:11:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:11:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T14:11:46.664 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T14:11:46.664 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:11:46.664 INFO:teuthology.orchestra.run.smithi019.stdout:Removed: 2024-09-13T14:11:46.664 INFO:teuthology.orchestra.run.smithi019.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:11:46.664 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:11:46.664 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:11:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T14:11:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:11:46.686 INFO:teuthology.orchestra.run.smithi003.stdout:Removed: 2024-09-13T14:11:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:11:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:11:46.686 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:11:47.461 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:50 ago on Fri 13 Sep 2024 02:10:57 PM UTC. 2024-09-13T14:11:47.600 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T14:11:47.601 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T14:11:47.602 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:11:47.602 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:11:47.602 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:11:47.602 INFO:teuthology.orchestra.run.smithi019.stdout:Install 7 Packages 2024-09-13T14:11:47.602 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:11:47.611 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:51 ago on Fri 13 Sep 2024 02:10:56 PM UTC. 2024-09-13T14:11:47.617 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 5.2 M 2024-09-13T14:11:47.617 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 19 M 2024-09-13T14:11:47.617 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:11:47.748 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:11:47.749 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:11:47.749 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:11:47.749 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:11:47.749 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:11:47.749 INFO:teuthology.orchestra.run.smithi003.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T14:11:47.749 INFO:teuthology.orchestra.run.smithi003.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T14:11:47.749 INFO:teuthology.orchestra.run.smithi003.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout:Install 7 Packages 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:11:47.763 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 5.2 M 2024-09-13T14:11:47.763 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 19 M 2024-09-13T14:11:47.763 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:11:48.035 INFO:teuthology.orchestra.run.smithi019.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 384 kB/s | 44 kB 00:00 2024-09-13T14:11:48.069 INFO:teuthology.orchestra.run.smithi019.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 709 kB/s | 106 kB 00:00 2024-09-13T14:11:48.110 INFO:teuthology.orchestra.run.smithi019.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 953 kB/s | 72 kB 00:00 2024-09-13T14:11:48.144 INFO:teuthology.orchestra.run.smithi019.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.1 MB/s | 84 kB 00:00 2024-09-13T14:11:48.186 INFO:teuthology.orchestra.run.smithi019.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.3 MB/s | 906 kB 00:00 2024-09-13T14:11:48.253 INFO:teuthology.orchestra.run.smithi019.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 5.8 MB/s | 837 kB 00:00 2024-09-13T14:11:48.420 INFO:teuthology.orchestra.run.smithi019.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 11 MB/s | 3.2 MB 00:00 2024-09-13T14:11:48.420 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:48.420 INFO:teuthology.orchestra.run.smithi019.stdout:Total 6.4 MB/s | 5.2 MB 00:00 2024-09-13T14:11:48.534 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:11:48.550 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:11:48.550 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:11:48.686 INFO:teuthology.orchestra.run.smithi003.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 76 kB/s | 44 kB 00:00 2024-09-13T14:11:48.761 INFO:teuthology.orchestra.run.smithi003.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 160 kB/s | 106 kB 00:00 2024-09-13T14:11:48.871 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:11:48.872 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:11:49.020 INFO:teuthology.orchestra.run.smithi003.stdout:(3/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 983 kB/s | 906 kB 00:00 2024-09-13T14:11:49.355 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:11:49.386 INFO:teuthology.orchestra.run.smithi003.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 102 kB/s | 72 kB 00:00 2024-09-13T14:11:49.411 INFO:teuthology.orchestra.run.smithi003.stdout:(5/7): python3-kmod-0.9-32.el9.x86_64.rpm 129 kB/s | 84 kB 00:00 2024-09-13T14:11:49.440 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-13T14:11:49.474 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-13T14:11:49.503 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-13T14:11:49.717 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-13T14:11:49.750 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:11:49.912 INFO:teuthology.orchestra.run.smithi003.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 940 kB/s | 837 kB 00:00 2024-09-13T14:11:50.154 INFO:teuthology.orchestra.run.smithi003.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 4.1 MB/s | 3.2 MB 00:00 2024-09-13T14:11:50.154 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:50.154 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.2 MB/s | 5.2 MB 00:02 2024-09-13T14:11:50.223 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:11:50.253 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:11:50.275 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:11:50.292 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:11:50.292 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:11:50.376 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:11:50.442 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:11:50.632 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:11:50.633 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:11:51.115 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:11:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-13T14:11:51.241 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-13T14:11:51.269 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-13T14:11:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:11:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-13T14:11:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-13T14:11:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-13T14:11:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-13T14:11:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-13T14:11:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-13T14:11:51.493 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-13T14:11:51.534 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:11:51.797 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:11:51.797 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:11:51.797 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:11:51.797 INFO:teuthology.orchestra.run.smithi019.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:11:51.798 INFO:teuthology.orchestra.run.smithi019.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T14:11:51.798 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T14:11:51.798 INFO:teuthology.orchestra.run.smithi019.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-13T14:11:51.798 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:11:51.798 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:11:52.023 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:11:52.053 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:11:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:11:52.193 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:11:53.310 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:11:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-13T14:11:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-13T14:11:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-13T14:11:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-13T14:11:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-13T14:11:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-13T14:11:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:11:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:11:53.716 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:11:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:11:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T14:11:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T14:11:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-13T14:11:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:11:53.717 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:11:54.096 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:54.096 INFO:teuthology.run_tasks:Running task install... 2024-09-13T14:11:54.107 DEBUG:teuthology.task.install:project ceph 2024-09-13T14:11:54.107 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T14:11:54.107 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T14:11:54.107 INFO:teuthology.task.install:Using flavor: default 2024-09-13T14:11:54.113 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T14:11:54.113 INFO:teuthology.task.install:extra packages: [] 2024-09-13T14:11:54.113 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T14:11:54.113 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:11:54.114 INFO:teuthology.packaging:ref: None 2024-09-13T14:11:54.114 INFO:teuthology.packaging:tag: None 2024-09-13T14:11:54.114 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:11:54.114 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:11:54.114 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-13T14:11:54.116 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T14:11:54.117 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:11:54.117 INFO:teuthology.packaging:ref: None 2024-09-13T14:11:54.117 INFO:teuthology.packaging:tag: None 2024-09-13T14:11:54.117 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:11:54.117 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:11:54.117 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-13T14:11:54.342 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/ 2024-09-13T14:11:54.342 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1627.gbbdadfa3 2024-09-13T14:11:54.362 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/ 2024-09-13T14:11:54.362 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1627.gbbdadfa3 2024-09-13T14:11:54.461 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T14:11:54.461 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:11:54.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T14:11:54.477 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T14:11:54.478 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:11:54.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T14:11:54.494 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T14:11:54.494 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:11:54.494 INFO:teuthology.packaging:ref: None 2024-09-13T14:11:54.494 INFO:teuthology.packaging:tag: None 2024-09-13T14:11:54.494 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:11:54.494 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:11:54.494 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T14:11:54.506 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T14:11:54.506 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:11:54.506 INFO:teuthology.packaging:ref: None 2024-09-13T14:11:54.506 INFO:teuthology.packaging:tag: None 2024-09-13T14:11:54.506 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:11:54.506 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:11:54.507 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T14:11:54.564 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T14:11:54.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T14:11:54.644 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T14:11:54.657 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T14:11:54.711 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-09-13T14:11:54.713 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-09-13T14:11:54.722 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2024-09-13T14:11:54.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2024-09-13T14:11:54.947 INFO:teuthology.orchestra.run.smithi019.stdout:45 files removed 2024-09-13T14:11:54.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw 2024-09-13T14:11:54.979 INFO:teuthology.orchestra.run.smithi003.stdout:45 files removed 2024-09-13T14:11:55.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2024-09-13T14:11:55.543 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 288 kB/s | 77 kB 00:00 2024-09-13T14:11:55.585 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 285 kB/s | 77 kB 00:00 2024-09-13T14:11:55.827 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-13T14:11:55.877 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-13T14:11:56.111 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 7.3 kB/s | 1.8 kB 00:00 2024-09-13T14:11:56.144 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 7.8 kB/s | 1.8 kB 00:00 2024-09-13T14:11:57.652 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.2 MB 00:01 2024-09-13T14:11:57.702 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.2 MB 00:01 2024-09-13T14:11:59.753 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-13T14:11:59.920 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-13T14:12:05.321 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-09-13T14:12:05.404 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.5 MB 00:01 2024-09-13T14:12:06.964 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-13T14:12:07.705 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-13T14:12:08.968 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 12 MB/s | 23 MB 00:01 2024-09-13T14:12:09.417 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-13T14:12:14.707 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-13T14:12:15.039 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-13T14:12:17.180 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:12:17.182 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:12:17.182 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 14 M 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 3.1 M 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.9 M 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 5.0 M 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 22 M 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 25 k 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T14:12:17.183 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 637 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 457 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 4.9 M 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 45 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 120 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 156 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 320 k 2024-09-13T14:12:17.184 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 294 k 2024-09-13T14:12:17.185 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 99 k 2024-09-13T14:12:17.185 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T14:12:17.185 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:17.185 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:12:17.185 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:12:17.185 INFO:teuthology.orchestra.run.smithi019.stdout:Install 22 Packages 2024-09-13T14:12:17.185 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2024-09-13T14:12:17.185 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:17.186 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 58 M 2024-09-13T14:12:17.186 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:12:17.528 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:12:17.530 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:12:17.530 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:12:17.530 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:12:17.530 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:12:17.530 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 14 M 2024-09-13T14:12:17.530 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 3.1 M 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.9 M 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 5.0 M 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 22 M 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 25 k 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 637 k 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T14:12:17.531 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 457 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 4.9 M 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 45 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 120 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 156 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 320 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 294 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 99 k 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T14:12:17.532 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:17.533 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:12:17.533 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:12:17.533 INFO:teuthology.orchestra.run.smithi003.stdout:Install 22 Packages 2024-09-13T14:12:17.533 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2024-09-13T14:12:17.533 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:17.533 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 58 M 2024-09-13T14:12:17.533 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:12:17.775 INFO:teuthology.orchestra.run.smithi019.stdout:(1/24): ceph-base-17.2.7-1627.gbbdadfa3.el9.x86 9.2 MB/s | 5.0 MB 00:00 2024-09-13T14:12:17.842 INFO:teuthology.orchestra.run.smithi019.stdout:(2/24): ceph-selinux-17.2.7-1627.gbbdadfa3.el9. 378 kB/s | 25 kB 00:00 2024-09-13T14:12:17.950 INFO:teuthology.orchestra.run.smithi019.stdout:(3/24): libcephfs2-17.2.7-1627.gbbdadfa3.el9.x8 5.7 MB/s | 637 kB 00:00 2024-09-13T14:12:18.159 INFO:teuthology.orchestra.run.smithi019.stdout:(4/24): ceph-radosgw-17.2.7-1627.gbbdadfa3.el9. 16 MB/s | 14 MB 00:00 2024-09-13T14:12:18.184 INFO:teuthology.orchestra.run.smithi019.stdout:(5/24): libradosstriper1-17.2.7-1627.gbbdadfa3. 1.9 MB/s | 457 kB 00:00 2024-09-13T14:12:18.285 INFO:teuthology.orchestra.run.smithi019.stdout:(6/24): python3-ceph-argparse-17.2.7-1627.gbbda 451 kB/s | 45 kB 00:00 2024-09-13T14:12:18.385 INFO:teuthology.orchestra.run.smithi019.stdout:(7/24): python3-ceph-common-17.2.7-1627.gbbdadf 1.2 MB/s | 120 kB 00:00 2024-09-13T14:12:18.460 INFO:teuthology.orchestra.run.smithi019.stdout:(8/24): librgw2-17.2.7-1627.gbbdadfa3.el9.x86_6 16 MB/s | 4.9 MB 00:00 2024-09-13T14:12:18.485 INFO:teuthology.orchestra.run.smithi019.stdout:(9/24): python3-cephfs-17.2.7-1627.gbbdadfa3.el 1.5 MB/s | 156 kB 00:00 2024-09-13T14:12:18.519 INFO:teuthology.orchestra.run.smithi019.stdout:(10/24): python3-rados-17.2.7-1627.gbbdadfa3.el 5.4 MB/s | 320 kB 00:00 2024-09-13T14:12:18.544 INFO:teuthology.orchestra.run.smithi019.stdout:(11/24): python3-rbd-17.2.7-1627.gbbdadfa3.el9. 4.9 MB/s | 294 kB 00:00 2024-09-13T14:12:18.569 INFO:teuthology.orchestra.run.smithi019.stdout:(12/24): python3-rgw-17.2.7-1627.gbbdadfa3.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T14:12:18.657 INFO:teuthology.orchestra.run.smithi003.stdout:(1/24): ceph-base-17.2.7-1627.gbbdadfa3.el9.x86 6.1 MB/s | 5.0 MB 00:00 2024-09-13T14:12:18.707 INFO:teuthology.orchestra.run.smithi003.stdout:(2/24): ceph-selinux-17.2.7-1627.gbbdadfa3.el9. 506 kB/s | 25 kB 00:00 2024-09-13T14:12:18.795 INFO:teuthology.orchestra.run.smithi019.stdout:(13/24): ceph-common-17.2.7-1627.gbbdadfa3.el9. 14 MB/s | 22 MB 00:01 2024-09-13T14:12:18.858 INFO:teuthology.orchestra.run.smithi003.stdout:(3/24): libcephfs2-17.2.7-1627.gbbdadfa3.el9.x8 4.1 MB/s | 637 kB 00:00 2024-09-13T14:12:18.917 INFO:teuthology.orchestra.run.smithi003.stdout:(4/24): libradosstriper1-17.2.7-1627.gbbdadfa3. 7.7 MB/s | 457 kB 00:00 2024-09-13T14:12:19.275 INFO:teuthology.orchestra.run.smithi003.stdout:(5/24): librgw2-17.2.7-1627.gbbdadfa3.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-09-13T14:12:19.312 INFO:teuthology.orchestra.run.smithi019.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 61 kB/s | 45 kB 00:00 2024-09-13T14:12:19.425 INFO:teuthology.orchestra.run.smithi003.stdout:(6/24): ceph-radosgw-17.2.7-1627.gbbdadfa3.el9. 9.1 MB/s | 14 MB 00:01 2024-09-13T14:12:19.437 INFO:teuthology.orchestra.run.smithi019.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 179 kB/s | 160 kB 00:00 2024-09-13T14:12:19.450 INFO:teuthology.orchestra.run.smithi003.stdout:(7/24): python3-ceph-argparse-17.2.7-1627.gbbda 257 kB/s | 45 kB 00:00 2024-09-13T14:12:19.476 INFO:teuthology.orchestra.run.smithi003.stdout:(8/24): python3-ceph-common-17.2.7-1627.gbbdadf 2.3 MB/s | 120 kB 00:00 2024-09-13T14:12:19.509 INFO:teuthology.orchestra.run.smithi003.stdout:(9/24): python3-cephfs-17.2.7-1627.gbbdadfa3.el 2.6 MB/s | 156 kB 00:00 2024-09-13T14:12:19.535 INFO:teuthology.orchestra.run.smithi003.stdout:(10/24): python3-rados-17.2.7-1627.gbbdadfa3.el 5.4 MB/s | 320 kB 00:00 2024-09-13T14:12:19.560 INFO:teuthology.orchestra.run.smithi003.stdout:(11/24): python3-rbd-17.2.7-1627.gbbdadfa3.el9. 5.7 MB/s | 294 kB 00:00 2024-09-13T14:12:19.585 INFO:teuthology.orchestra.run.smithi003.stdout:(12/24): python3-rgw-17.2.7-1627.gbbdadfa3.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T14:12:19.712 INFO:teuthology.orchestra.run.smithi019.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 722 kB/s | 662 kB 00:00 2024-09-13T14:12:19.771 INFO:teuthology.orchestra.run.smithi019.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 5.2 MB/s | 308 kB 00:00 2024-09-13T14:12:19.796 INFO:teuthology.orchestra.run.smithi019.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-13T14:12:19.811 INFO:teuthology.orchestra.run.smithi003.stdout:(13/24): ceph-common-17.2.7-1627.gbbdadfa3.el9. 11 MB/s | 22 MB 00:01 2024-09-13T14:12:19.813 INFO:teuthology.orchestra.run.smithi019.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 4.0 MB/s | 67 kB 00:00 2024-09-13T14:12:19.869 INFO:teuthology.orchestra.run.smithi003.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 518 kB/s | 160 kB 00:00 2024-09-13T14:12:19.988 INFO:teuthology.orchestra.run.smithi019.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-09-13T14:12:19.994 INFO:teuthology.orchestra.run.smithi003.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-09-13T14:12:20.020 INFO:teuthology.orchestra.run.smithi003.stdout:(16/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 104 kB/s | 45 kB 00:00 2024-09-13T14:12:20.045 INFO:teuthology.orchestra.run.smithi003.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-09-13T14:12:20.047 INFO:teuthology.orchestra.run.smithi019.stdout:(21/24): python3-prettytable-0.7.2-27.el9.noarc 69 kB/s | 42 kB 00:00 2024-09-13T14:12:20.079 INFO:teuthology.orchestra.run.smithi003.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 504 kB/s | 42 kB 00:00 2024-09-13T14:12:20.139 INFO:teuthology.orchestra.run.smithi019.stdout:(22/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 354 kB/s | 292 kB 00:00 2024-09-13T14:12:20.153 INFO:teuthology.orchestra.run.smithi003.stdout:(19/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.3 MB/s | 308 kB 00:00 2024-09-13T14:12:20.179 INFO:teuthology.orchestra.run.smithi003.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 387 kB/s | 49 kB 00:00 2024-09-13T14:12:20.204 INFO:teuthology.orchestra.run.smithi003.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 586 kB/s | 67 kB 00:00 2024-09-13T14:12:20.264 INFO:teuthology.orchestra.run.smithi019.stdout:(23/24): librados2-17.2.7-1627.gbbdadfa3.el9.x8 11 MB/s | 3.1 MB 00:00 2024-09-13T14:12:20.271 INFO:teuthology.orchestra.run.smithi003.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-13T14:12:20.331 INFO:teuthology.orchestra.run.smithi019.stdout:(24/24): librbd1-17.2.7-1627.gbbdadfa3.el9.x86_ 10 MB/s | 2.9 MB 00:00 2024-09-13T14:12:20.336 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:12:20.336 INFO:teuthology.orchestra.run.smithi019.stdout:Total 18 MB/s | 58 MB 00:03 2024-09-13T14:12:20.403 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:12:20.421 INFO:teuthology.orchestra.run.smithi003.stdout:(23/24): librados2-17.2.7-1627.gbbdadfa3.el9.x8 13 MB/s | 3.1 MB 00:00 2024-09-13T14:12:20.442 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:12:20.442 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:12:20.479 INFO:teuthology.orchestra.run.smithi003.stdout:(24/24): librbd1-17.2.7-1627.gbbdadfa3.el9.x86_ 11 MB/s | 2.9 MB 00:00 2024-09-13T14:12:20.484 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:12:20.485 INFO:teuthology.orchestra.run.smithi003.stdout:Total 20 MB/s | 58 MB 00:02 2024-09-13T14:12:20.549 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:12:20.587 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:12:20.587 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:12:20.941 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:12:20.941 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:12:21.081 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:12:21.081 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:12:22.267 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:12:22.292 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:12:22.670 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-13T14:12:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-13T14:12:22.716 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:12:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:12:22.820 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:12:22.860 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:12:22.906 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/26 2024-09-13T14:12:22.947 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/26 2024-09-13T14:12:22.983 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:12:23.057 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:12:23.059 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:12:23.109 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-13T14:12:23.138 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-13T14:12:23.149 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:12:23.202 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-13T14:12:23.231 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-13T14:12:23.609 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-13T14:12:23.636 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 8/26 2024-09-13T14:12:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 8/26 2024-09-13T14:12:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-13T14:12:23.729 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 8/26 2024-09-13T14:12:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-13T14:12:23.779 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9. 10/26 2024-09-13T14:12:23.792 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 8/26 2024-09-13T14:12:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 11/26 2024-09-13T14:12:23.846 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-13T14:12:23.880 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 12/26 2024-09-13T14:12:23.889 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9. 10/26 2024-09-13T14:12:23.913 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 13/26 2024-09-13T14:12:23.926 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 11/26 2024-09-13T14:12:23.980 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 13/26 2024-09-13T14:12:23.990 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 12/26 2024-09-13T14:12:24.016 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 13/26 2024-09-13T14:12:24.019 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-13T14:12:24.072 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-13T14:12:24.082 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 13/26 2024-09-13T14:12:24.108 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-13T14:12:24.121 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-13T14:12:24.166 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-13T14:12:24.208 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-13T14:12:24.423 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-13T14:12:24.468 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 18/26 2024-09-13T14:12:24.516 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-13T14:12:24.558 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 18/26 2024-09-13T14:12:24.561 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 18/26 2024-09-13T14:12:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 18/26 2024-09-13T14:12:24.643 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 19/26 2024-09-13T14:12:24.678 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x8 20/26 2024-09-13T14:12:24.712 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 19/26 2024-09-13T14:12:24.863 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x8 20/26 2024-09-13T14:12:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:12:27.274 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:12:27.536 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:12:27.577 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:12:27.852 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:12:27.894 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 22/26 2024-09-13T14:12:27.999 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 22/26 2024-09-13T14:12:27.999 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T14:12:27.999 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T14:12:27.999 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:28.037 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:12:28.127 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:12:28.163 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 22/26 2024-09-13T14:12:28.264 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 22/26 2024-09-13T14:12:28.265 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T14:12:28.265 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T14:12:28.265 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:28.306 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:12:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:12:38.362 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2024-09-13T14:12:38.362 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2024-09-13T14:12:38.362 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2024-09-13T14:12:38.362 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2024-09-13T14:12:38.363 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2024-09-13T14:12:38.363 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2024-09-13T14:12:38.363 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2024-09-13T14:12:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:38.525 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:12:38.525 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-09-13T14:12:38.525 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-09-13T14:12:38.525 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-09-13T14:12:38.525 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-09-13T14:12:38.525 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-09-13T14:12:38.525 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2024-09-13T14:12:38.526 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-09-13T14:12:38.526 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:39.728 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 24/26 2024-09-13T14:12:39.808 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 24/26 2024-09-13T14:12:39.808 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:12:39.808 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T14:12:39.808 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:12:39.808 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:12:39.809 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:39.810 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-13T14:12:39.889 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-13T14:12:39.889 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:12:39.890 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 24/26 2024-09-13T14:12:39.970 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 24/26 2024-09-13T14:12:39.970 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:12:39.970 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T14:12:39.970 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:12:39.970 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:12:39.971 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:39.972 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-13T14:12:40.068 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-13T14:12:40.068 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:12:41.144 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:12:41.144 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 5/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 6/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 7/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9. 8/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x8 9/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 10/26 2024-09-13T14:12:41.145 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 11/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 12/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 13/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-13T14:12:41.146 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-13T14:12:41.147 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:12:41.147 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-13T14:12:41.147 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 25/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 5/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 6/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 7/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9. 8/26 2024-09-13T14:12:41.273 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x8 9/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 10/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 11/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 12/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 13/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-13T14:12:41.274 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-13T14:12:41.275 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:12:41.275 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-13T14:12:41.275 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 25/26 2024-09-13T14:12:42.134 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.135 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T14:12:42.136 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.137 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.137 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.137 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T14:12:42.137 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.137 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.137 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.137 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T14:12:42.137 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:42.137 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T14:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T14:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T14:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T14:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T14:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T14:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T14:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T14:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T14:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:12:42.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2024-09-13T14:12:42.676 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test 2024-09-13T14:12:43.075 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:28 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:12:43.219 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:12:43.220 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:12:43.220 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:12:43.220 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:12:43.220 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:12:43.220 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 48 M 2024-09-13T14:12:43.220 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:12:43.220 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 163 k 2024-09-13T14:12:43.221 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T14:12:43.221 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T14:12:43.221 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:43.221 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:12:43.221 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:12:43.221 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-09-13T14:12:43.221 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:43.221 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 48 M 2024-09-13T14:12:43.222 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 193 M 2024-09-13T14:12:43.222 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:12:43.271 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:12:43.410 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 48 M 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 163 k 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:43.411 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:12:43.412 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:12:43.412 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2024-09-13T14:12:43.412 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:43.412 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 48 M 2024-09-13T14:12:43.412 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 193 M 2024-09-13T14:12:43.412 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:12:43.469 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): libcephsqlite-17.2.7-1627.gbbdadfa3.el9. 723 kB/s | 163 kB 00:00 2024-09-13T14:12:43.495 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-13T14:12:43.603 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 477 kB/s | 64 kB 00:00 2024-09-13T14:12:43.612 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-13T14:12:43.654 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-13T14:12:43.705 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): libcephsqlite-17.2.7-1627.gbbdadfa3.el9. 670 kB/s | 163 kB 00:00 2024-09-13T14:12:45.245 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): ceph-test-17.2.7-1627.gbbdadfa3.el9.x86_ 24 MB/s | 48 MB 00:02 2024-09-13T14:12:45.247 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:12:45.248 INFO:teuthology.orchestra.run.smithi003.stdout:Total 24 MB/s | 48 MB 00:02 2024-09-13T14:12:45.276 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:12:45.297 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:12:45.297 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:12:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): ceph-test-17.2.7-1627.gbbdadfa3.el9.x86_ 22 MB/s | 48 MB 00:02 2024-09-13T14:12:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:12:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:Total 21 MB/s | 48 MB 00:02 2024-09-13T14:12:45.682 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:12:45.682 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:12:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:12:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:12:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:12:46.101 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:12:46.102 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:12:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:12:46.310 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T14:12:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T14:12:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/4 2024-09-13T14:12:46.655 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:12:46.729 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T14:12:46.781 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T14:12:46.807 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/4 2024-09-13T14:12:51.041 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/4 2024-09-13T14:12:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/4 2024-09-13T14:12:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/4 2024-09-13T14:12:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/4 2024-09-13T14:12:53.521 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/4 2024-09-13T14:12:53.521 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/4 2024-09-13T14:12:53.521 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/4 2024-09-13T14:12:53.521 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T14:12:53.626 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/4 2024-09-13T14:12:53.626 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/4 2024-09-13T14:12:53.626 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/4 2024-09-13T14:12:53.626 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T14:12:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T14:12:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:53.831 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:12:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T14:12:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T14:12:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:53.831 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:12:53.929 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T14:12:53.929 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:53.929 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:12:53.929 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:53.929 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:12:53.929 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T14:12:53.929 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T14:12:53.929 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:53.929 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:12:54.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2024-09-13T14:12:54.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph 2024-09-13T14:12:54.713 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:39 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:12:54.755 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:40 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:12:54.853 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:12:54.855 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 6.5 k 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.1 M 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 1.4 M 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 240 k 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 3.9 M 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 15 M 2024-09-13T14:12:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T14:12:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T14:12:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T14:12:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T14:12:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T14:12:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:54.859 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:12:54.859 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:12:54.859 INFO:teuthology.orchestra.run.smithi003.stdout:Install 34 Packages 2024-09-13T14:12:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:12:54.860 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 27 M 2024-09-13T14:12:54.860 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 96 M 2024-09-13T14:12:54.860 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:12:54.897 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:12:54.899 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:12:54.899 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 6.5 k 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.1 M 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 1.4 M 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 240 k 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 3.9 M 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 15 M 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T14:12:54.900 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T14:12:54.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T14:12:54.902 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T14:12:54.902 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T14:12:54.902 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T14:12:54.902 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T14:12:54.902 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T14:12:54.902 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T14:12:54.902 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T14:12:54.903 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T14:12:54.903 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T14:12:54.904 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T14:12:54.904 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T14:12:54.904 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:54.904 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:12:54.904 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:12:54.904 INFO:teuthology.orchestra.run.smithi019.stdout:Install 34 Packages 2024-09-13T14:12:54.904 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:12:54.905 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 27 M 2024-09-13T14:12:54.905 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 96 M 2024-09-13T14:12:54.905 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:12:55.321 INFO:teuthology.orchestra.run.smithi003.stdout:(1/34): ceph-17.2.7-1627.gbbdadfa3.el9.x86_64.r 47 kB/s | 6.5 kB 00:00 2024-09-13T14:12:55.496 INFO:teuthology.orchestra.run.smithi003.stdout:(2/34): ceph-mds-17.2.7-1627.gbbdadfa3.el9.x86_ 6.6 MB/s | 2.1 MB 00:00 2024-09-13T14:12:55.555 INFO:teuthology.orchestra.run.smithi003.stdout:(3/34): ceph-mgr-17.2.7-1627.gbbdadfa3.el9.x86_ 3.8 MB/s | 1.4 MB 00:00 2024-09-13T14:12:55.613 INFO:teuthology.orchestra.run.smithi003.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1627.gbbda 4.0 MB/s | 240 kB 00:00 2024-09-13T14:12:55.814 INFO:teuthology.orchestra.run.smithi003.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-13T14:12:55.842 INFO:teuthology.orchestra.run.smithi019.stdout:(1/34): ceph-17.2.7-1627.gbbdadfa3.el9.x86_64.r 27 kB/s | 6.5 kB 00:00 2024-09-13T14:12:55.981 INFO:teuthology.orchestra.run.smithi003.stdout:(6/34): ceph-osd-17.2.7-1627.gbbdadfa3.el9.x86_ 32 MB/s | 15 MB 00:00 2024-09-13T14:12:56.017 INFO:teuthology.orchestra.run.smithi019.stdout:(2/34): ceph-mds-17.2.7-1627.gbbdadfa3.el9.x86_ 4.9 MB/s | 2.1 MB 00:00 2024-09-13T14:12:56.048 INFO:teuthology.orchestra.run.smithi003.stdout:(7/34): ceph-mon-17.2.7-1627.gbbdadfa3.el9.x86_ 5.4 MB/s | 3.9 MB 00:00 2024-09-13T14:12:56.059 INFO:teuthology.orchestra.run.smithi019.stdout:(3/34): ceph-mgr-17.2.7-1627.gbbdadfa3.el9.x86_ 3.0 MB/s | 1.4 MB 00:00 2024-09-13T14:12:56.131 INFO:teuthology.orchestra.run.smithi003.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 3.9 MB/s | 1.2 MB 00:00 2024-09-13T14:12:56.151 INFO:teuthology.orchestra.run.smithi019.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1627.gbbda 2.6 MB/s | 240 kB 00:00 2024-09-13T14:12:56.190 INFO:teuthology.orchestra.run.smithi003.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-13T14:12:56.215 INFO:teuthology.orchestra.run.smithi003.stdout:(10/34): python3-ply-3.11-14.el9.noarch.rpm 455 kB/s | 106 kB 00:00 2024-09-13T14:12:56.235 INFO:teuthology.orchestra.run.smithi019.stdout:(5/34): ceph-mon-17.2.7-1627.gbbdadfa3.el9.x86_ 10 MB/s | 3.9 MB 00:00 2024-09-13T14:12:56.257 INFO:teuthology.orchestra.run.smithi003.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-13T14:12:56.290 INFO:teuthology.orchestra.run.smithi003.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 557 kB/s | 135 kB 00:00 2024-09-13T14:12:56.316 INFO:teuthology.orchestra.run.smithi003.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 595 kB/s | 35 kB 00:00 2024-09-13T14:12:56.341 INFO:teuthology.orchestra.run.smithi003.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T14:12:56.367 INFO:teuthology.orchestra.run.smithi003.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-13T14:12:56.392 INFO:teuthology.orchestra.run.smithi003.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 672 kB/s | 51 kB 00:00 2024-09-13T14:12:56.417 INFO:teuthology.orchestra.run.smithi003.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-13T14:12:56.467 INFO:teuthology.orchestra.run.smithi003.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-13T14:12:56.492 INFO:teuthology.orchestra.run.smithi003.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 470 kB/s | 11 kB 00:00 2024-09-13T14:12:56.518 INFO:teuthology.orchestra.run.smithi003.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T14:12:56.526 INFO:teuthology.orchestra.run.smithi019.stdout:(6/34): ceph-osd-17.2.7-1627.gbbdadfa3.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-13T14:12:56.543 INFO:teuthology.orchestra.run.smithi003.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-13T14:12:56.569 INFO:teuthology.orchestra.run.smithi003.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-13T14:12:56.605 INFO:teuthology.orchestra.run.smithi003.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 315 kB/s | 19 kB 00:00 2024-09-13T14:12:56.636 INFO:teuthology.orchestra.run.smithi003.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-09-13T14:12:56.661 INFO:teuthology.orchestra.run.smithi003.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 839 kB/s | 46 kB 00:00 2024-09-13T14:12:56.686 INFO:teuthology.orchestra.run.smithi003.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-13T14:12:56.711 INFO:teuthology.orchestra.run.smithi003.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-13T14:12:56.737 INFO:teuthology.orchestra.run.smithi003.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-13T14:12:56.762 INFO:teuthology.orchestra.run.smithi003.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-13T14:12:56.787 INFO:teuthology.orchestra.run.smithi003.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 717 kB/s | 36 kB 00:00 2024-09-13T14:12:56.813 INFO:teuthology.orchestra.run.smithi003.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-13T14:12:56.826 INFO:teuthology.orchestra.run.smithi019.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 375 kB/s | 253 kB 00:00 2024-09-13T14:12:56.838 INFO:teuthology.orchestra.run.smithi003.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-13T14:12:56.863 INFO:teuthology.orchestra.run.smithi003.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T14:12:56.897 INFO:teuthology.orchestra.run.smithi003.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-13T14:12:56.900 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:12:56.900 INFO:teuthology.orchestra.run.smithi003.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-13T14:12:56.944 INFO:teuthology.orchestra.run.smithi019.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-13T14:12:56.986 INFO:teuthology.orchestra.run.smithi019.stdout:(9/34): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-09-13T14:12:57.031 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:12:57.061 INFO:teuthology.orchestra.run.smithi019.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-13T14:12:57.072 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:12:57.072 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:12:57.161 INFO:teuthology.orchestra.run.smithi019.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-09-13T14:12:57.186 INFO:teuthology.orchestra.run.smithi019.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-13T14:12:57.465 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:12:57.465 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:12:57.764 INFO:teuthology.orchestra.run.smithi019.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 58 kB/s | 35 kB 00:00 2024-09-13T14:12:57.806 INFO:teuthology.orchestra.run.smithi019.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 231 kB/s | 172 kB 00:00 2024-09-13T14:12:57.831 INFO:teuthology.orchestra.run.smithi019.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 120 kB/s | 77 kB 00:00 2024-09-13T14:12:57.881 INFO:teuthology.orchestra.run.smithi019.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 874 kB/s | 43 kB 00:00 2024-09-13T14:12:57.923 INFO:teuthology.orchestra.run.smithi019.stdout:(17/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-13T14:12:57.974 INFO:teuthology.orchestra.run.smithi019.stdout:(18/34): python3-cherrypy-18.6.1-2.el9.noarch.r 7.0 MB/s | 358 kB 00:00 2024-09-13T14:12:57.999 INFO:teuthology.orchestra.run.smithi019.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 432 kB/s | 11 kB 00:00 2024-09-13T14:12:58.033 INFO:teuthology.orchestra.run.smithi019.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 693 kB/s | 18 kB 00:00 2024-09-13T14:12:58.058 INFO:teuthology.orchestra.run.smithi019.stdout:(21/34): python3-jaraco-collections-3.0.0-8.el9 936 kB/s | 23 kB 00:00 2024-09-13T14:12:58.091 INFO:teuthology.orchestra.run.smithi019.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 773 kB/s | 19 kB 00:00 2024-09-13T14:12:58.117 INFO:teuthology.orchestra.run.smithi019.stdout:(23/34): python3-jaraco-text-3.2.0-6.el9.noarch 812 kB/s | 20 kB 00:00 2024-09-13T14:12:58.142 INFO:teuthology.orchestra.run.smithi019.stdout:(24/34): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-13T14:12:58.167 INFO:teuthology.orchestra.run.smithi019.stdout:(25/34): python3-more-itertools-8.12.0-2.el9.no 3.2 MB/s | 79 kB 00:00 2024-09-13T14:12:58.209 INFO:teuthology.orchestra.run.smithi019.stdout:(26/34): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-09-13T14:12:58.234 INFO:teuthology.orchestra.run.smithi019.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 661 kB/s | 16 kB 00:00 2024-09-13T14:12:58.260 INFO:teuthology.orchestra.run.smithi019.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-13T14:12:58.285 INFO:teuthology.orchestra.run.smithi019.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-13T14:12:58.327 INFO:teuthology.orchestra.run.smithi019.stdout:(30/34): python3-webob-1.8.8-2.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-09-13T14:12:58.377 INFO:teuthology.orchestra.run.smithi019.stdout:(31/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.3 MB/s | 427 kB 00:00 2024-09-13T14:12:58.403 INFO:teuthology.orchestra.run.smithi019.stdout:(32/34): python3-pytz-2021.1-5.el9.noarch.rpm 80 kB/s | 51 kB 00:00 2024-09-13T14:12:58.419 INFO:teuthology.orchestra.run.smithi019.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-09-13T14:12:58.436 INFO:teuthology.orchestra.run.smithi019.stdout:(34/34): python3-toml-0.10.2-6.el9.noarch.rpm 66 kB/s | 42 kB 00:00 2024-09-13T14:12:58.441 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:12:58.441 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.7 MB/s | 27 MB 00:03 2024-09-13T14:12:58.578 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:12:58.619 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:12:58.619 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:12:58.845 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:12:58.908 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-13T14:12:58.957 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-13T14:12:58.990 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-13T14:12:59.004 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:12:59.004 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:12:59.095 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-13T14:12:59.193 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-13T14:12:59.244 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-13T14:12:59.295 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-13T14:12:59.324 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-13T14:12:59.359 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-13T14:12:59.392 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-13T14:12:59.442 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-13T14:12:59.485 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-13T14:12:59.519 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-13T14:12:59.568 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-13T14:12:59.617 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-13T14:12:59.693 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-13T14:12:59.844 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-13T14:12:59.936 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-13T14:12:59.995 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-13T14:13:00.033 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-13T14:13:00.091 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-13T14:13:00.154 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-13T14:13:00.350 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-13T14:13:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-13T14:13:00.492 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-13T14:13:00.519 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:13:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-13T14:13:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-13T14:13:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-13T14:13:00.639 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-13T14:13:00.671 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-13T14:13:00.750 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-13T14:13:00.768 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-13T14:13:00.854 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-13T14:13:00.900 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-13T14:13:00.913 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9. 29/34 2024-09-13T14:13:00.943 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-13T14:13:00.945 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 30/34 2024-09-13T14:13:00.972 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-13T14:13:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 30/34 2024-09-13T14:13:00.973 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:00.973 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T14:13:00.973 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:13:00.973 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:13:00.973 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:01.007 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-13T14:13:01.040 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-13T14:13:01.081 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-13T14:13:01.124 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-13T14:13:01.158 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-13T14:13:01.208 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-13T14:13:01.247 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-13T14:13:01.323 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-13T14:13:01.476 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-13T14:13:01.558 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-13T14:13:01.609 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-13T14:13:01.648 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-13T14:13:01.705 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-13T14:13:01.769 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-13T14:13:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-13T14:13:02.018 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-13T14:13:02.091 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-13T14:13:02.188 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-13T14:13:02.224 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-13T14:13:02.346 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-13T14:13:02.371 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 31/34 2024-09-13T14:13:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 31/34 2024-09-13T14:13:02.400 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:02.400 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T14:13:02.400 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:13:02.400 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:13:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:02.502 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9. 29/34 2024-09-13T14:13:02.526 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 30/34 2024-09-13T14:13:02.553 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 30/34 2024-09-13T14:13:02.553 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:02.553 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T14:13:02.553 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:13:02.553 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:13:02.553 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:02.788 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 32/34 2024-09-13T14:13:02.814 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 32/34 2024-09-13T14:13:02.814 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:02.814 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T14:13:02.814 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:13:02.814 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:13:02.814 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:03.022 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 33/34 2024-09-13T14:13:03.047 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 33/34 2024-09-13T14:13:03.048 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:03.048 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T14:13:03.048 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:13:03.048 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:13:03.048 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:03.114 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 34/34 2024-09-13T14:13:03.969 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 31/34 2024-09-13T14:13:04.000 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 31/34 2024-09-13T14:13:04.000 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:04.001 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T14:13:04.001 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:13:04.001 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:13:04.001 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:04.387 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 32/34 2024-09-13T14:13:04.413 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 32/34 2024-09-13T14:13:04.413 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:04.413 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T14:13:04.413 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:13:04.413 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:13:04.413 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:04.621 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 33/34 2024-09-13T14:13:04.646 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 33/34 2024-09-13T14:13:04.646 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:04.646 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T14:13:04.647 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:13:04.647 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:13:04.647 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:04.712 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 34/34 2024-09-13T14:13:04.846 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 34/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 5/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9. 6/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-13T14:13:04.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-13T14:13:04.848 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-13T14:13:04.849 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-13T14:13:05.802 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-13T14:13:05.802 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:05.802 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:13:05.802 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:05.802 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:05.802 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:05.802 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T14:13:05.803 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T14:13:05.804 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T14:13:05.805 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T14:13:05.805 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T14:13:05.805 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T14:13:05.805 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T14:13:05.805 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T14:13:05.805 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:05.805 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:13:06.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2024-09-13T14:13:06.287 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 34/34 2024-09-13T14:13:06.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/34 2024-09-13T14:13:06.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/34 2024-09-13T14:13:06.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/34 2024-09-13T14:13:06.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/34 2024-09-13T14:13:06.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 5/34 2024-09-13T14:13:06.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9. 6/34 2024-09-13T14:13:06.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-13T14:13:06.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-13T14:13:06.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-13T14:13:06.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-13T14:13:06.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-13T14:13:06.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-13T14:13:06.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-13T14:13:06.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-13T14:13:06.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-13T14:13:06.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-13T14:13:06.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-13T14:13:06.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-13T14:13:06.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-13T14:13:06.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-13T14:13:06.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-13T14:13:06.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-13T14:13:06.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-13T14:13:06.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-13T14:13:06.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-13T14:13:06.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-13T14:13:06.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-13T14:13:06.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-13T14:13:06.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-13T14:13:06.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-13T14:13:06.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-13T14:13:06.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-13T14:13:06.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-13T14:13:06.766 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:51 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:13:06.861 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:13:06.906 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:13:06.907 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-09-13T14:13:06.908 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:13:06.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2024-09-13T14:13:07.269 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-13T14:13:07.269 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:07.270 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:13:07.270 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:07.270 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:07.270 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:07.270 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:07.271 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:07.271 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:07.271 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T14:13:07.271 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T14:13:07.271 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T14:13:07.271 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T14:13:07.272 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T14:13:07.272 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T14:13:07.272 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T14:13:07.272 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T14:13:07.272 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T14:13:07.273 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T14:13:07.273 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T14:13:07.273 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T14:13:07.273 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T14:13:07.273 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T14:13:07.273 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T14:13:07.274 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T14:13:07.274 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T14:13:07.274 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T14:13:07.274 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T14:13:07.274 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T14:13:07.274 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T14:13:07.275 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T14:13:07.275 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T14:13:07.275 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T14:13:07.275 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T14:13:07.275 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T14:13:07.276 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T14:13:07.276 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T14:13:07.276 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:07.276 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:13:07.574 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:13:07.715 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:13:07.715 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:07.715 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:13:07.715 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:07.716 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:13:07.716 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 83 k 2024-09-13T14:13:07.716 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:07.716 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:13:07.716 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:07.716 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-09-13T14:13:07.716 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:07.717 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 83 k 2024-09-13T14:13:07.717 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 375 k 2024-09-13T14:13:07.717 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:13:07.766 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base 2024-09-13T14:13:07.904 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-17.2.7-1627.gbbdadfa3.el9.noarch.rpm 441 kB/s | 83 kB 00:00 2024-09-13T14:13:07.905 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:07.905 INFO:teuthology.orchestra.run.smithi003.stdout:Total 437 kB/s | 83 kB 00:00 2024-09-13T14:13:07.905 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:13:07.911 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:13:07.911 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:13:07.913 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:13:07.913 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:13:07.983 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:13:08.378 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:13:08.456 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:13:08.471 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:13:08.515 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:13:08.516 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-13T14:13:08.516 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:13:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:13:08.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install cephadm 2024-09-13T14:13:08.913 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:13:09.175 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:13:09.189 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:13:09.189 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:09.189 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:13:09.189 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:09.189 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:09.189 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:13:09.314 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:13:09.315 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:09.315 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:13:09.315 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:09.315 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:13:09.315 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 83 k 2024-09-13T14:13:09.315 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:09.315 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:13:09.315 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:09.316 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-13T14:13:09.316 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:09.316 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 83 k 2024-09-13T14:13:09.316 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 375 k 2024-09-13T14:13:09.316 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:13:09.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T14:13:09.502 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm-17.2.7-1627.gbbdadfa3.el9.noarch.rpm 444 kB/s | 83 kB 00:00 2024-09-13T14:13:09.503 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:09.503 INFO:teuthology.orchestra.run.smithi019.stdout:Total 440 kB/s | 83 kB 00:00 2024-09-13T14:13:09.503 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:13:09.509 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:13:09.509 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:13:09.511 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:13:09.511 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:13:09.594 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:13:09.989 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:13:09.995 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:13:10.100 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:13:10.128 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 137 k 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:10.129 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 137 k 2024-09-13T14:13:10.130 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 379 k 2024-09-13T14:13:10.130 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:13:10.344 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-17.2.7-1627.gbbdadf 642 kB/s | 137 kB 00:00 2024-09-13T14:13:10.344 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:10.345 INFO:teuthology.orchestra.run.smithi003.stdout:Total 637 kB/s | 137 kB 00:00 2024-09-13T14:13:10.345 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:13:10.353 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:13:10.353 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:13:10.414 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:13:10.414 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:13:10.436 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:13:10.557 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:13:10.661 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:13:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:13:10.689 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:10.689 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T14:13:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:10.704 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:13:10.705 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:10.705 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:13:10.705 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:10.705 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:10.705 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:13:10.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T14:13:11.516 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:13:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:13:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:11.638 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:13:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:11.638 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:13:11.654 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:13:11.654 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:11.654 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-09-13T14:13:11.654 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:11.655 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:13:11.655 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 137 k 2024-09-13T14:13:11.655 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:11.655 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:13:11.655 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:11.655 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-13T14:13:11.655 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:11.655 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 137 k 2024-09-13T14:13:11.656 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 379 k 2024-09-13T14:13:11.656 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:13:11.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2024-09-13T14:13:11.849 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-17.2.7-1627.gbbdadf 710 kB/s | 137 kB 00:00 2024-09-13T14:13:11.850 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:11.850 INFO:teuthology.orchestra.run.smithi019.stdout:Total 704 kB/s | 137 kB 00:00 2024-09-13T14:13:11.851 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:13:11.858 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:13:11.859 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:13:11.920 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:13:11.920 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:13:12.069 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:13:12.126 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:13:12.152 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:13:12.152 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:13:12.152 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T14:13:12.152 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:12.436 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:13:12.530 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:13:12.576 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:13:12.577 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-09-13T14:13:12.577 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:13:12.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T14:13:13.071 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:13:13.071 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:13.071 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:13:13.071 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:13:13.071 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:13.071 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:13:13.220 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:13:13.278 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr 2024-09-13T14:13:13.356 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:13:13.357 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:13.357 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:13:13.357 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:13.357 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:13:13.357 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 3.4 M 2024-09-13T14:13:13.357 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:13:13.357 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 24 k 2024-09-13T14:13:13.357 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 15 k 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout:Install 7 Packages 2024-09-13T14:13:13.358 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:13.359 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.7 M 2024-09-13T14:13:13.359 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 80 M 2024-09-13T14:13:13.359 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:13:13.539 INFO:teuthology.orchestra.run.smithi003.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1627.gbbd 163 kB/s | 24 kB 00:00 2024-09-13T14:13:13.564 INFO:teuthology.orchestra.run.smithi003.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1627.gbbda 83 kB/s | 15 kB 00:00 2024-09-13T14:13:13.614 INFO:teuthology.orchestra.run.smithi003.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 120 kB/s | 9.0 kB 00:00 2024-09-13T14:13:13.648 INFO:teuthology.orchestra.run.smithi003.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 975 kB/s | 31 kB 00:00 2024-09-13T14:13:13.673 INFO:teuthology.orchestra.run.smithi003.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 376 kB/s | 41 kB 00:00 2024-09-13T14:13:13.698 INFO:teuthology.orchestra.run.smithi003.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-13T14:13:13.815 INFO:teuthology.orchestra.run.smithi003.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1627.gbbdadfa3 7.9 MB/s | 3.4 MB 00:00 2024-09-13T14:13:13.818 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:13.819 INFO:teuthology.orchestra.run.smithi003.stdout:Total 7.9 MB/s | 3.7 MB 00:00 2024-09-13T14:13:13.819 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:13:13.828 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:13:13.828 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:13:13.874 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:13:13.917 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:13:13.917 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:13:13.968 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:13:14.013 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:13:14.014 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-13T14:13:14.014 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:13:14.083 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T14:13:14.233 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:13:14.300 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-13T14:13:14.335 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-13T14:13:14.391 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-13T14:13:14.429 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-13T14:13:14.458 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.n 5/7 2024-09-13T14:13:14.677 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:13:14.816 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 3.4 M 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 24 k 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 15 k 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T14:13:14.817 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2024-09-13T14:13:14.818 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-13T14:13:14.818 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:14.818 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:13:14.818 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:14.818 INFO:teuthology.orchestra.run.smithi019.stdout:Install 7 Packages 2024-09-13T14:13:14.818 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:14.818 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.7 M 2024-09-13T14:13:14.819 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 80 M 2024-09-13T14:13:14.819 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:13:15.020 INFO:teuthology.orchestra.run.smithi019.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1627.gbbd 166 kB/s | 24 kB 00:00 2024-09-13T14:13:15.045 INFO:teuthology.orchestra.run.smithi019.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1627.gbbda 85 kB/s | 15 kB 00:00 2024-09-13T14:13:15.095 INFO:teuthology.orchestra.run.smithi019.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 120 kB/s | 9.0 kB 00:00 2024-09-13T14:13:15.129 INFO:teuthology.orchestra.run.smithi019.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 981 kB/s | 31 kB 00:00 2024-09-13T14:13:15.154 INFO:teuthology.orchestra.run.smithi019.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 376 kB/s | 41 kB 00:00 2024-09-13T14:13:15.187 INFO:teuthology.orchestra.run.smithi019.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-13T14:13:15.263 INFO:teuthology.orchestra.run.smithi019.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1627.gbbdadfa3 8.6 MB/s | 3.4 MB 00:00 2024-09-13T14:13:15.266 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:15.266 INFO:teuthology.orchestra.run.smithi019.stdout:Total 8.1 MB/s | 3.7 MB 00:00 2024-09-13T14:13:15.266 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:13:15.275 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:13:15.275 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:13:15.369 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:13:15.369 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:13:15.507 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9. 6/7 2024-09-13T14:13:15.545 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 7/7 2024-09-13T14:13:15.694 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:13:15.756 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-13T14:13:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-13T14:13:15.839 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-13T14:13:15.868 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-13T14:13:15.889 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.n 5/7 2024-09-13T14:13:16.603 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 7/7 2024-09-13T14:13:16.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9. 1/7 2024-09-13T14:13:16.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 2/7 2024-09-13T14:13:16.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.n 3/7 2024-09-13T14:13:16.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-13T14:13:16.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-13T14:13:16.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-13T14:13:16.904 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-13T14:13:16.904 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:16.904 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:13:16.904 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:16.904 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:16.904 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:16.905 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-13T14:13:16.905 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-13T14:13:16.905 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T14:13:16.905 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T14:13:16.905 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:16.905 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:13:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9. 6/7 2024-09-13T14:13:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 7/7 2024-09-13T14:13:17.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T14:13:17.709 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:02 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:13:17.847 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:13:17.848 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================= 2024-09-13T14:13:17.848 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:13:17.848 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================= 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 7.4 M 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T14:13:17.849 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T14:13:17.850 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T14:13:17.850 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:17.850 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:13:17.850 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================= 2024-09-13T14:13:17.850 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-09-13T14:13:17.850 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:17.850 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 38 M 2024-09-13T14:13:17.851 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 212 M 2024-09-13T14:13:17.851 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:13:17.991 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 7/7 2024-09-13T14:13:17.991 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9. 1/7 2024-09-13T14:13:17.991 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 2/7 2024-09-13T14:13:17.992 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.n 3/7 2024-09-13T14:13:17.992 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-13T14:13:17.992 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-13T14:13:17.992 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-13T14:13:18.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-13T14:13:18.288 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:18.288 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:13:18.288 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:18.288 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:18.289 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:18.289 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-13T14:13:18.289 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-13T14:13:18.289 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T14:13:18.289 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T14:13:18.289 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:18.290 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:13:18.290 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-13T14:13:18.348 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.0 MB/s | 801 kB 00:00 2024-09-13T14:13:18.506 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T14:13:18.615 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-13T14:13:18.640 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 93 kB/s | 33 kB 00:00 2024-09-13T14:13:18.841 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-13T14:13:18.900 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-13T14:13:19.096 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:05 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:13:19.234 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:13:19.235 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-09-13T14:13:19.235 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:13:19.235 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-09-13T14:13:19.235 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:13:19.235 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 7.4 M 2024-09-13T14:13:19.235 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:13:19.235 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T14:13:19.235 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:13:19.236 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-09-13T14:13:19.237 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-09-13T14:13:19.237 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:19.237 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 38 M 2024-09-13T14:13:19.237 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 212 M 2024-09-13T14:13:19.238 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:13:19.358 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 534 kB/s | 245 kB 00:00 2024-09-13T14:13:19.475 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-09-13T14:13:19.617 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-09-13T14:13:19.645 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-13T14:13:19.687 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.5 MB/s | 801 kB 00:00 2024-09-13T14:13:19.979 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-13T14:13:20.004 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 91 kB/s | 33 kB 00:00 2024-09-13T14:13:20.059 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.1 MB/s | 4.9 MB 00:01 2024-09-13T14:13:20.205 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-13T14:13:20.263 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-13T14:13:20.584 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-09-13T14:13:20.730 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-09-13T14:13:20.755 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 497 kB/s | 245 kB 00:00 2024-09-13T14:13:20.956 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 250 kB 00:00 2024-09-13T14:13:21.214 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.9 MB/s | 4.9 MB 00:01 2024-09-13T14:13:21.298 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.0 MB/s | 5.1 MB 00:00 2024-09-13T14:13:38.008 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 905 kB/s | 16 MB 00:18 2024-09-13T14:13:38.009 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:38.009 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.9 MB/s | 38 MB 00:20 2024-09-13T14:13:38.383 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:13:38.411 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:13:38.411 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:13:38.921 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 926 kB/s | 16 MB 00:17 2024-09-13T14:13:38.922 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:38.923 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.9 MB/s | 38 MB 00:19 2024-09-13T14:13:38.950 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:13:38.951 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:13:39.291 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:13:39.319 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:13:39.319 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:13:39.734 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:13:39.818 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T14:13:39.856 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:13:39.859 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:13:39.859 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:13:39.973 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:13:40.024 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T14:13:40.216 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T14:13:40.248 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T14:13:40.336 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T14:13:40.770 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:13:40.848 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T14:13:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:13:40.992 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:13:41.029 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T14:13:41.038 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T14:13:41.167 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T14:13:41.232 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T14:13:41.261 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T14:13:41.341 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T14:13:42.040 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T14:13:42.165 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T14:13:42.552 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:13:43.492 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:13:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T14:13:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 12/12 2024-09-13T14:13:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T14:13:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 12/12 2024-09-13T14:13:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 12/12 2024-09-13T14:13:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 1/12 2024-09-13T14:13:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:13:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:13:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T14:13:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T14:13:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T14:13:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T14:13:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T14:13:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T14:13:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:13:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T14:13:46.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T14:13:46.851 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:46.851 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:13:46.851 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:46.851 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T14:13:46.851 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T14:13:46.851 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:46.852 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:13:47.073 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 12/12 2024-09-13T14:13:47.073 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 1/12 2024-09-13T14:13:47.073 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:13:47.073 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:13:47.074 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T14:13:47.074 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T14:13:47.074 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T14:13:47.074 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T14:13:47.074 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T14:13:47.074 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T14:13:47.074 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:13:47.074 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T14:13:47.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2024-09-13T14:13:47.491 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T14:13:47.491 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:47.491 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:13:47.491 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:47.491 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T14:13:47.491 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:47.492 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:13:47.794 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:13:47.857 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-rook 2024-09-13T14:13:47.933 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:13:47.934 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:47.934 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 47 k 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T14:13:47.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T14:13:47.936 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T14:13:47.936 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T14:13:47.936 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T14:13:47.936 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:47.936 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:13:47.936 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:47.936 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2024-09-13T14:13:47.936 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:47.937 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.2 M 2024-09-13T14:13:47.937 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2024-09-13T14:13:47.937 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:13:48.463 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:13:48.604 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:13:48.605 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 47 k 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T14:13:48.606 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout:Install 13 Packages 2024-09-13T14:13:48.607 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:48.608 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.2 M 2024-09-13T14:13:48.608 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2024-09-13T14:13:48.608 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:13:48.638 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 345 kB/s | 26 kB 00:00 2024-09-13T14:13:48.671 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 178 kB/s | 19 kB 00:00 2024-09-13T14:13:48.755 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-13T14:13:48.780 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-13T14:13:48.822 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 279 kB 00:00 2024-09-13T14:13:48.848 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): ceph-mgr-rook-17.2.7-1627.gbbdadfa3.el9 167 kB/s | 47 kB 00:00 2024-09-13T14:13:48.873 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 583 kB/s | 54 kB 00:00 2024-09-13T14:13:48.924 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-09-13T14:13:48.948 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 141 kB/s | 14 kB 00:00 2024-09-13T14:13:48.973 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T14:13:48.999 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-13T14:13:49.024 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-13T14:13:49.074 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.9 MB/s | 1.0 MB 00:00 2024-09-13T14:13:49.078 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:49.078 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-13T14:13:49.153 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:13:49.170 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:13:49.170 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:13:49.270 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:13:49.271 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:13:49.303 INFO:teuthology.orchestra.run.smithi019.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 243 kB/s | 19 kB 00:00 2024-09-13T14:13:49.328 INFO:teuthology.orchestra.run.smithi019.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 246 kB/s | 26 kB 00:00 2024-09-13T14:13:49.520 INFO:teuthology.orchestra.run.smithi019.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.0 MB/s | 222 kB 00:00 2024-09-13T14:13:49.545 INFO:teuthology.orchestra.run.smithi019.stdout:(4/13): ceph-mgr-rook-17.2.7-1627.gbbdadfa3.el9 148 kB/s | 47 kB 00:00 2024-09-13T14:13:49.570 INFO:teuthology.orchestra.run.smithi019.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 656 kB/s | 159 kB 00:00 2024-09-13T14:13:49.612 INFO:teuthology.orchestra.run.smithi019.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.0 MB/s | 279 kB 00:00 2024-09-13T14:13:49.638 INFO:teuthology.orchestra.run.smithi019.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 799 kB/s | 54 kB 00:00 2024-09-13T14:13:49.671 INFO:teuthology.orchestra.run.smithi019.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-13T14:13:49.705 INFO:teuthology.orchestra.run.smithi019.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 156 kB/s | 14 kB 00:00 2024-09-13T14:13:49.739 INFO:teuthology.orchestra.run.smithi019.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-13T14:13:49.772 INFO:teuthology.orchestra.run.smithi019.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.7 MB/s | 90 kB 00:00 2024-09-13T14:13:49.797 INFO:teuthology.orchestra.run.smithi019.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.4 MB/s | 220 kB 00:00 2024-09-13T14:13:49.831 INFO:teuthology.orchestra.run.smithi019.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-13T14:13:49.833 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:49.834 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-09-13T14:13:49.900 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:13:49.916 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:13:49.916 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:13:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:13:50.012 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:13:50.012 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:13:50.114 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T14:13:50.169 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T14:13:50.225 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T14:13:50.260 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T14:13:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T14:13:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T14:13:50.531 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:13:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T14:13:50.587 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T14:13:50.711 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T14:13:50.776 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T14:13:50.839 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T14:13:50.875 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T14:13:50.917 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T14:13:51.032 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T14:13:51.141 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T14:13:51.194 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T14:13:51.354 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T14:13:51.409 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T14:13:51.450 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T14:13:51.497 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T14:13:51.519 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 13/13 2024-09-13T14:13:51.955 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T14:13:52.007 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T14:13:52.041 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T14:13:52.087 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T14:13:52.119 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 13/13 2024-09-13T14:13:52.315 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 13/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T14:13:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T14:13:52.317 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T14:13:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T14:13:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:52.650 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:13:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T14:13:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T14:13:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T14:13:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T14:13:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T14:13:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T14:13:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T14:13:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T14:13:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T14:13:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T14:13:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T14:13:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T14:13:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:52.652 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:13:52.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T14:13:52.911 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 13/13 2024-09-13T14:13:52.911 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T14:13:52.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T14:13:53.257 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T14:13:53.257 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:53.257 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:13:53.257 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:53.257 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T14:13:53.257 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T14:13:53.257 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T14:13:53.257 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:53.258 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:13:53.470 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:13:53.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T14:13:53.608 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:13:53.609 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:53.609 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:13:53.609 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:53.609 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:13:53.609 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 124 k 2024-09-13T14:13:53.609 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:13:53.609 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T14:13:53.609 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T14:13:53.610 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T14:13:53.610 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T14:13:53.610 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T14:13:53.610 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:53.610 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:13:53.610 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:53.610 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-09-13T14:13:53.610 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:53.610 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 7.0 M 2024-09-13T14:13:53.611 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 33 M 2024-09-13T14:13:53.611 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:13:53.884 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-13T14:13:53.942 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1627.gbbdadfa3.e 620 kB/s | 124 kB 00:00 2024-09-13T14:13:53.975 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-13T14:13:54.001 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.2 MB/s | 74 kB 00:00 2024-09-13T14:13:54.026 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 689 kB/s | 58 kB 00:00 2024-09-13T14:13:54.102 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:13:54.242 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:13:54.242 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:54.242 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:13:54.242 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 124 k 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2024-09-13T14:13:54.243 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:54.244 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 7.0 M 2024-09-13T14:13:54.244 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 33 M 2024-09-13T14:13:54.244 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:13:54.583 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-13T14:13:54.624 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1627.gbbdadfa3.e 653 kB/s | 124 kB 00:00 2024-09-13T14:13:54.750 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-09-13T14:13:54.766 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 407 kB/s | 58 kB 00:00 2024-09-13T14:13:54.800 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.5 MB/s | 548 kB 00:00 2024-09-13T14:13:54.825 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 992 kB/s | 74 kB 00:00 2024-09-13T14:13:54.828 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:54.829 INFO:teuthology.orchestra.run.smithi019.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-13T14:13:54.902 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:13:54.913 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:13:54.913 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:13:55.021 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:13:55.021 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:13:55.309 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.8 MB/s | 6.0 MB 00:01 2024-09-13T14:13:55.313 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:55.313 INFO:teuthology.orchestra.run.smithi003.stdout:Total 4.1 MB/s | 7.0 MB 00:01 2024-09-13T14:13:55.367 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:13:55.389 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:13:55.402 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:13:55.402 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:13:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T14:13:55.595 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T14:13:55.827 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:13:55.999 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T14:13:56.045 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T14:13:56.171 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T14:13:56.259 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T14:13:56.321 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T14:13:56.351 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 6/6 2024-09-13T14:13:56.597 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T14:13:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T14:13:56.746 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T14:13:56.803 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 6/6 2024-09-13T14:13:57.079 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 6/6 2024-09-13T14:13:57.079 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/6 2024-09-13T14:13:57.079 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T14:13:57.080 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T14:13:57.080 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T14:13:57.080 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T14:13:57.373 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T14:13:57.373 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:57.373 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:13:57.373 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:57.373 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T14:13:57.373 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T14:13:57.373 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T14:13:57.374 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T14:13:57.374 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T14:13:57.374 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:57.374 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:13:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 6/6 2024-09-13T14:13:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/6 2024-09-13T14:13:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T14:13:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T14:13:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T14:13:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T14:13:57.627 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T14:13:57.889 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:57.890 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:13:58.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2024-09-13T14:13:58.215 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:13:58.353 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:13:58.354 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:58.354 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-13T14:13:58.354 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:58.354 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:13:58.354 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 813 k 2024-09-13T14:13:58.354 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-13T14:13:58.354 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T14:13:58.355 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:58.355 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:13:58.355 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:13:58.355 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-09-13T14:13:58.355 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:13:58.355 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 893 k 2024-09-13T14:13:58.356 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.6 M 2024-09-13T14:13:58.356 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:13:58.511 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 601 kB/s | 80 kB 00:00 2024-09-13T14:13:58.677 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-fuse-17.2.7-1627.gbbdadfa3.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-13T14:13:58.678 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:58.678 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.7 MB/s | 893 kB 00:00 2024-09-13T14:13:58.690 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:13:58.713 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:13:58.713 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:13:58.721 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:13:58.799 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:13:58.799 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:13:58.859 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:13:58.860 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:58.860 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-09-13T14:13:58.860 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:58.860 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:13:58.860 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 813 k 2024-09-13T14:13:58.860 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-09-13T14:13:58.861 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T14:13:58.861 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:58.861 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:13:58.861 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:13:58.861 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-09-13T14:13:58.861 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:13:58.861 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 893 k 2024-09-13T14:13:58.861 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.6 M 2024-09-13T14:13:58.862 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:13:58.998 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:13:59.103 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 570 kB/s | 80 kB 00:00 2024-09-13T14:13:59.135 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T14:13:59.205 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/2 2024-09-13T14:13:59.245 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-fuse-17.2.7-1627.gbbdadfa3.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-13T14:13:59.245 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:13:59.245 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.3 MB/s | 893 kB 00:00 2024-09-13T14:13:59.257 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:13:59.279 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:13:59.280 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:13:59.365 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:13:59.365 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:13:59.560 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:13:59.703 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T14:13:59.772 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/2 2024-09-13T14:13:59.871 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/2 2024-09-13T14:13:59.871 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/2 2024-09-13T14:14:00.175 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T14:14:00.175 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:00.175 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:14:00.175 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T14:14:00.175 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:00.175 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:00.455 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/2 2024-09-13T14:14:00.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/2 2024-09-13T14:14:00.466 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel 2024-09-13T14:14:00.756 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T14:14:00.756 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:00.756 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:14:00.756 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T14:14:00.757 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:00.757 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:01.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2024-09-13T14:14:01.060 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:01.200 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:01.201 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:01.201 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-09-13T14:14:01.201 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:01.201 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:14:01.202 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 126 k 2024-09-13T14:14:01.202 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:01.202 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:14:01.202 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:01.202 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-13T14:14:01.202 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:01.202 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 126 k 2024-09-13T14:14:01.202 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 448 k 2024-09-13T14:14:01.202 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:14:01.431 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-17.2.7-1627.gbbdadfa3.el9.x86_64 550 kB/s | 126 kB 00:00 2024-09-13T14:14:01.432 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:01.432 INFO:teuthology.orchestra.run.smithi019.stdout:Total 547 kB/s | 126 kB 00:00 2024-09-13T14:14:01.432 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:14:01.439 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:14:01.439 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:14:01.506 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:14:01.506 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:14:01.637 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:01.663 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:14:01.765 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:01.778 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:01.778 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-09-13T14:14:01.778 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:01.779 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:14:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 126 k 2024-09-13T14:14:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:01.779 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:14:01.779 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:01.779 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-09-13T14:14:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:01.780 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 126 k 2024-09-13T14:14:01.780 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 448 k 2024-09-13T14:14:01.780 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:14:01.991 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-17.2.7-1627.gbbdadfa3.el9.x86_64 597 kB/s | 126 kB 00:00 2024-09-13T14:14:01.992 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:01.992 INFO:teuthology.orchestra.run.smithi003.stdout:Total 593 kB/s | 126 kB 00:00 2024-09-13T14:14:01.992 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:14:02.001 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:14:02.001 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:14:02.065 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:14:02.066 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:14:02.216 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:02.225 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:14:02.324 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:02.479 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:02.480 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:02.480 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:14:02.480 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:02.480 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:02.480 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:02.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2 2024-09-13T14:14:02.749 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:03.056 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:14:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:03.056 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:03.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2024-09-13T14:14:03.322 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:03.418 INFO:teuthology.orchestra.run.smithi019.stdout:Package libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:03.464 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:03.465 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-13T14:14:03.465 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:03.535 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs-devel 2024-09-13T14:14:03.887 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:03.981 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:04.028 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:04.029 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-09-13T14:14:04.029 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:04.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2024-09-13T14:14:04.121 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:04.260 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:04.261 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:04.261 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-09-13T14:14:04.261 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:04.261 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:14:04.262 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 25 k 2024-09-13T14:14:04.262 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:04.262 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:14:04.262 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:04.262 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-13T14:14:04.262 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:04.262 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 25 k 2024-09-13T14:14:04.262 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 110 k 2024-09-13T14:14:04.263 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:14:04.418 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs-devel-17.2.7-1627.gbbdadfa3.el9.x86_6 160 kB/s | 25 kB 00:00 2024-09-13T14:14:04.418 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:04.418 INFO:teuthology.orchestra.run.smithi019.stdout:Total 159 kB/s | 25 kB 00:00 2024-09-13T14:14:04.419 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:14:04.422 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:14:04.422 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:14:04.424 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:14:04.424 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:14:04.504 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:14:04.626 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:04.682 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:04.821 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:04.822 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:04.822 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-09-13T14:14:04.822 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:04.823 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:14:04.823 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 25 k 2024-09-13T14:14:04.823 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:04.823 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:14:04.823 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:04.824 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-09-13T14:14:04.824 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:04.824 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 25 k 2024-09-13T14:14:04.824 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 110 k 2024-09-13T14:14:04.824 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:14:04.970 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-17.2.7-1627.gbbdadfa3.el9.x86_6 169 kB/s | 25 kB 00:00 2024-09-13T14:14:04.971 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:04.971 INFO:teuthology.orchestra.run.smithi003.stdout:Total 168 kB/s | 25 kB 00:00 2024-09-13T14:14:04.971 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:14:04.974 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:14:04.974 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:14:04.976 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:14:04.976 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:14:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:14:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:05.186 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:05.353 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:05.353 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:05.353 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:14:05.353 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:05.353 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:05.354 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:05.540 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2 2024-09-13T14:14:05.579 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:05.821 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:05.821 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:05.821 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:14:05.821 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:05.821 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:05.821 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:06.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2024-09-13T14:14:06.130 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:06.223 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:06.268 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:06.269 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-13T14:14:06.269 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:06.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1 2024-09-13T14:14:06.597 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:06.690 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:06.736 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:06.738 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-09-13T14:14:06.738 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:06.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2024-09-13T14:14:06.937 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:07.031 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:07.077 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:07.078 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-13T14:14:07.078 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:07.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados 2024-09-13T14:14:07.394 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:07.487 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:07.533 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:07.534 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-09-13T14:14:07.534 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:07.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2024-09-13T14:14:07.745 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:07.838 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:07.884 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:07.885 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-13T14:14:07.885 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:07.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw 2024-09-13T14:14:08.196 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:08.289 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:08.335 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:08.336 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-09-13T14:14:08.336 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:08.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2024-09-13T14:14:08.563 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:54 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:08.656 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:08.702 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:08.703 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-13T14:14:08.703 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:08.777 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs 2024-09-13T14:14:08.990 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:09.083 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:09.129 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:09.130 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-09-13T14:14:09.130 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:09.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2024-09-13T14:14:09.365 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:55 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:09.458 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:09.503 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:09.504 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-13T14:14:09.504 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:09.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd 2024-09-13T14:14:09.791 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:54 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:09.885 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:09.931 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:09.932 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-09-13T14:14:09.932 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:10.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2024-09-13T14:14:10.166 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:56 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:10.259 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:10.305 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:10.306 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-13T14:14:10.307 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:10.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse 2024-09-13T14:14:10.590 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:55 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:10.683 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:14:10.729 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:10.730 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-09-13T14:14:10.730 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:10.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2024-09-13T14:14:10.963 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:56 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:11.101 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:11.102 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:11.102 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-09-13T14:14:11.102 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:11.102 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:14:11.102 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 82 k 2024-09-13T14:14:11.102 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:11.102 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:14:11.102 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:11.103 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-13T14:14:11.103 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:11.103 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 82 k 2024-09-13T14:14:11.103 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 222 k 2024-09-13T14:14:11.103 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:14:11.304 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 405 kB/s | 82 kB 00:00 2024-09-13T14:14:11.305 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:11.305 INFO:teuthology.orchestra.run.smithi019.stdout:Total 403 kB/s | 82 kB 00:00 2024-09-13T14:14:11.305 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:14:11.313 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:14:11.313 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:14:11.380 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:14:11.380 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:14:11.390 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:56 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:11.531 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:11.531 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 82 k 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 82 k 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 222 k 2024-09-13T14:14:11.532 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:14:11.533 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:14:11.655 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:11.723 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 429 kB/s | 82 kB 00:00 2024-09-13T14:14:11.724 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:11.724 INFO:teuthology.orchestra.run.smithi003.stdout:Total 426 kB/s | 82 kB 00:00 2024-09-13T14:14:11.724 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:14:11.731 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:14:11.732 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:14:11.799 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:14:11.799 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:14:11.960 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:14:12.064 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:12.311 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:12.312 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:12.312 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:14:12.312 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:12.312 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:12.312 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:12.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2024-09-13T14:14:12.748 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:12.748 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:12.748 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:14:12.749 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:12.749 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:12.749 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:13.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2024-09-13T14:14:13.104 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:59 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:13.242 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:13.242 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:13.242 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-09-13T14:14:13.242 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.9 M 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.9 M 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 12 M 2024-09-13T14:14:13.243 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:14:13.557 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 9.4 MB/s | 2.9 MB 00:00 2024-09-13T14:14:13.558 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:13.558 INFO:teuthology.orchestra.run.smithi019.stdout:Total 9.3 MB/s | 2.9 MB 00:00 2024-09-13T14:14:13.559 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:14:13.568 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:14:13.568 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:14:13.625 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:58 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:13.654 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:14:13.655 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:14:13.773 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:13.774 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-09-13T14:14:13.774 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:13.774 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:14:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.9 M 2024-09-13T14:14:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:13.775 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:14:13.775 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:13.775 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-09-13T14:14:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:13.775 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.9 M 2024-09-13T14:14:13.775 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 12 M 2024-09-13T14:14:13.775 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:14:14.090 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:14:14.101 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 9.0 MB/s | 2.9 MB 00:00 2024-09-13T14:14:14.102 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:14.102 INFO:teuthology.orchestra.run.smithi003.stdout:Total 9.0 MB/s | 2.9 MB 00:00 2024-09-13T14:14:14.102 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:14:14.112 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:14:14.112 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:14:14.150 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:14.176 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:14.176 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:14:14.176 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T14:14:14.176 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:14:14.176 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:14:14.177 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:14.197 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:14:14.197 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:14:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:14:14.714 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:14.740 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:14.741 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:14:14.741 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T14:14:14.741 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:14:14.741 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:14:14.741 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:15.304 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:15.304 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:15.304 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:14:15.304 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:15.304 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:15.304 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:15.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd 2024-09-13T14:14:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:15.837 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:14:15.837 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:15.837 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:15.837 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:16.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2024-09-13T14:14:16.106 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:02 ago on Fri 13 Sep 2024 02:12:14 PM UTC. 2024-09-13T14:14:16.248 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 165 k 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 165 k 2024-09-13T14:14:16.249 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 477 k 2024-09-13T14:14:16.250 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-13T14:14:16.453 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 814 kB/s | 165 kB 00:00 2024-09-13T14:14:16.453 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:16.453 INFO:teuthology.orchestra.run.smithi019.stdout:Total 808 kB/s | 165 kB 00:00 2024-09-13T14:14:16.454 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-13T14:14:16.462 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-13T14:14:16.463 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-13T14:14:16.531 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-13T14:14:16.531 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-13T14:14:16.686 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:01 ago on Fri 13 Sep 2024 02:12:15 PM UTC. 2024-09-13T14:14:16.710 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-13T14:14:16.804 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:16.825 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 165 k 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-09-13T14:14:16.826 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:16.827 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 165 k 2024-09-13T14:14:16.827 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 477 k 2024-09-13T14:14:16.827 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-09-13T14:14:17.038 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 782 kB/s | 165 kB 00:00 2024-09-13T14:14:17.039 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:17.039 INFO:teuthology.orchestra.run.smithi003.stdout:Total 776 kB/s | 165 kB 00:00 2024-09-13T14:14:17.039 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-09-13T14:14:17.047 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-09-13T14:14:17.047 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-09-13T14:14:17.115 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-09-13T14:14:17.115 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-09-13T14:14:17.189 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:17.283 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-09-13T14:14:17.389 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:17.464 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:17.464 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:17.464 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-13T14:14:17.464 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:17.464 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:14:17.464 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-13T14:14:17.667 DEBUG:teuthology.parallel:result is None 2024-09-13T14:14:17.823 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:18.111 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:14:18.111 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:18.111 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-09-13T14:14:18.111 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:14:18.112 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:18.112 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-09-13T14:14:18.351 DEBUG:teuthology.parallel:result is None 2024-09-13T14:14:18.352 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:14:18.352 INFO:teuthology.packaging:ref: None 2024-09-13T14:14:18.352 INFO:teuthology.packaging:tag: None 2024-09-13T14:14:18.352 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:14:18.352 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:14:18.352 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-13T14:14:18.570 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T14:14:18.593 INFO:teuthology.orchestra.run.smithi003.stdout:17.2.7-1627.gbbdadfa3.el9 2024-09-13T14:14:18.594 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1627.gbbdadfa3.el9 2024-09-13T14:14:18.594 INFO:teuthology.task.install:The correct ceph version 17.2.7-1627.gbbdadfa3 is installed. 2024-09-13T14:14:18.596 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:14:18.596 INFO:teuthology.packaging:ref: None 2024-09-13T14:14:18.596 INFO:teuthology.packaging:tag: None 2024-09-13T14:14:18.596 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:14:18.596 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:14:18.596 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-13T14:14:18.812 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T14:14:18.837 INFO:teuthology.orchestra.run.smithi019.stdout:17.2.7-1627.gbbdadfa3.el9 2024-09-13T14:14:18.838 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1627.gbbdadfa3.el9 2024-09-13T14:14:18.838 INFO:teuthology.task.install:The correct ceph version 17.2.7-1627.gbbdadfa3 is installed. 2024-09-13T14:14:18.839 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T14:14:18.840 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:14:18.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T14:14:18.868 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:14:18.868 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T14:14:18.908 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T14:14:18.908 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:14:18.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T14:14:18.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T14:14:19.003 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:14:19.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T14:14:19.032 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T14:14:19.100 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T14:14:19.100 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:14:19.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T14:14:19.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T14:14:19.195 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:14:19.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T14:14:19.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T14:14:19.290 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T14:14:19.290 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:14:19.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T14:14:19.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T14:14:19.385 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:14:19.385 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T14:14:19.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T14:14:19.480 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-13T14:14:19.581 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T14:14:19.581 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-09-13T14:14:19.581 INFO:tasks.cephadm:Cluster fsid is 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:14:19.581 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-13T14:14:19.582 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.3', 'mon.c': '[v2:172.21.15.3:3301,v1:172.21.15.3:6790]', 'mon.b': '172.21.15.19'} 2024-09-13T14:14:19.582 INFO:tasks.cephadm:First mon is mon.a on smithi003 2024-09-13T14:14:19.582 INFO:tasks.cephadm:First mgr is y 2024-09-13T14:14:19.582 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-13T14:14:19.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname $(hostname -s) 2024-09-13T14:14:19.612 DEBUG:teuthology.orchestra.run.smithi019:> sudo hostname $(hostname -s) 2024-09-13T14:14:19.640 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-09-13T14:14:19.640 DEBUG:teuthology.orchestra.run.smithi003:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T14:14:19.779 INFO:teuthology.orchestra.run.smithi003.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 13 14:14 /home/ubuntu/cephtest/cephadm 2024-09-13T14:14:19.779 DEBUG:teuthology.orchestra.run.smithi019:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T14:14:19.922 INFO:teuthology.orchestra.run.smithi019.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 13 14:14 /home/ubuntu/cephtest/cephadm 2024-09-13T14:14:19.922 DEBUG:teuthology.orchestra.run.smithi003:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T14:14:19.943 DEBUG:teuthology.orchestra.run.smithi019:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T14:14:19.971 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-09-13T14:14:19.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-13T14:14:19.986 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-13T14:14:21.532 INFO:teuthology.orchestra.run.smithi019.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-13T14:14:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-13T14:14:48.215 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:14:48.215 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)", 2024-09-13T14:14:48.215 INFO:teuthology.orchestra.run.smithi003.stdout: "image_id": "a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c", 2024-09-13T14:14:48.215 INFO:teuthology.orchestra.run.smithi003.stdout: "repo_digests": [ 2024-09-13T14:14:48.216 INFO:teuthology.orchestra.run.smithi003.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be" 2024-09-13T14:14:48.216 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:14:48.216 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:14:51.036 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:14:51.036 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_version": "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)", 2024-09-13T14:14:51.036 INFO:teuthology.orchestra.run.smithi019.stdout: "image_id": "a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c", 2024-09-13T14:14:51.036 INFO:teuthology.orchestra.run.smithi019.stdout: "repo_digests": [ 2024-09-13T14:14:51.036 INFO:teuthology.orchestra.run.smithi019.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be" 2024-09-13T14:14:51.036 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:14:51.036 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:14:51.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph 2024-09-13T14:14:51.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph 2024-09-13T14:14:51.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /etc/ceph 2024-09-13T14:14:51.161 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /etc/ceph 2024-09-13T14:14:51.187 INFO:tasks.cephadm:Writing seed config... 2024-09-13T14:14:51.188 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-13T14:14:51.189 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-13T14:14:51.190 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:14:51.190 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-13T14:14:51.219 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 77a3f016-71da-11ef-bceb-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-13T14:14:51.219 DEBUG:teuthology.orchestra.run.smithi003:mon.a> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a.service 2024-09-13T14:14:51.261 DEBUG:teuthology.orchestra.run.smithi003:mgr.y> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y.service 2024-09-13T14:14:51.305 INFO:tasks.cephadm:Bootstrapping... 2024-09-13T14:14:51.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 77a3f016-71da-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.3 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:51.515 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:14:51.515 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '77a3f016-71da-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.3', '--skip-admin-label'] 2024-09-13T14:14:51.539 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:14:51.540 INFO:teuthology.orchestra.run.smithi003.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-13T14:14:51.540 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying podman|docker is present... 2024-09-13T14:14:51.564 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:14:51.564 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying lvm2 is present... 2024-09-13T14:14:51.564 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying time synchronization is in place... 2024-09-13T14:14:51.572 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T14:14:51.572 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T14:14:51.579 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T14:14:51.579 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-09-13T14:14:51.586 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-09-13T14:14:51.593 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-09-13T14:14:51.593 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-09-13T14:14:51.593 INFO:teuthology.orchestra.run.smithi003.stdout:Repeating the final host check... 2024-09-13T14:14:51.616 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:14:51.616 INFO:teuthology.orchestra.run.smithi003.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-13T14:14:51.616 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl is present 2024-09-13T14:14:51.616 INFO:teuthology.orchestra.run.smithi003.stdout:lvcreate is present 2024-09-13T14:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T14:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T14:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T14:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-09-13T14:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-09-13T14:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-09-13T14:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-09-13T14:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout:Host looks OK 2024-09-13T14:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout:Cluster fsid: 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout:Acquiring lock 139990608855968 on /run/cephadm/77a3f016-71da-11ef-bceb-c7b262605968.lock 2024-09-13T14:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout:Lock 139990608855968 acquired on /run/cephadm/77a3f016-71da-11ef-bceb-c7b262605968.lock 2024-09-13T14:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 3300 ... 2024-09-13T14:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 6789 ... 2024-09-13T14:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout:Base mon IP(s) is [172.21.15.3:3300, 172.21.15.3:6789], mon addrv is [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-09-13T14:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.3 metric 100 2024-09-13T14:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.3 metric 100 2024-09-13T14:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-13T14:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-13T14:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1738sec hoplimit 64 pref medium 2024-09-13T14:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-13T14:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-13T14:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T14:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-13T14:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2024-09-13T14:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T14:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-09-13T14:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-09-13T14:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-13T14:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-13T14:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-13T14:14:52.125 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c 2024-09-13T14:14:52.125 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-13T14:14:52.125 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Getting image source signatures 2024-09-13T14:14:52.125 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-13T14:14:52.125 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:ca308b3f556da579f99cd79db684ebbf678db4c29a649b66aaf6f99bc8124cd6 2024-09-13T14:14:52.125 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying config sha256:a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c 2024-09-13T14:14:52.125 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-13T14:14:53.023 INFO:teuthology.orchestra.run.smithi003.stdout:ceph: stdout ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable) 2024-09-13T14:14:53.023 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph version: ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable) 2024-09-13T14:14:53.023 INFO:teuthology.orchestra.run.smithi003.stdout:Extracting ceph user uid/gid from container image... 2024-09-13T14:14:53.928 INFO:teuthology.orchestra.run.smithi003.stdout:stat: stdout 167 167 2024-09-13T14:14:53.928 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial keys... 2024-09-13T14:14:54.835 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQDeSORm+t/4DxAARNr49XIXSFV9kiJ2PIiNwg== 2024-09-13T14:14:55.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQDfSORmEBNxCxAApVLrMRxUX4T+Wm7/muwcfQ== 2024-09-13T14:14:56.961 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQDgSORm78EjDxAASA1qD/o5FC/1+cZqZpQIjw== 2024-09-13T14:14:56.962 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial monmap... 2024-09-13T14:14:57.875 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T14:14:57.876 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-13T14:14:57.876 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:14:57.876 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T14:14:57.876 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool for a [v2:172.21.15.3:3300,v1:172.21.15.3:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T14:14:57.876 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = octopus 2024-09-13T14:14:57.876 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: set fsid to 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:14:57.876 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T14:14:57.876 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:14:57.876 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mon... 2024-09-13T14:14:58.814 INFO:teuthology.orchestra.run.smithi003.stdout:create mon.a on 2024-09-13T14:14:59.040 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-13T14:14:59.191 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-13T14:14:59.343 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-77a3f016-71da-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-77a3f016-71da-11ef-bceb-c7b262605968.target. 2024-09-13T14:14:59.343 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-77a3f016-71da-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-77a3f016-71da-11ef-bceb-c7b262605968.target. 2024-09-13T14:14:59.521 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a 2024-09-13T14:14:59.522 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-13T14:14:59.690 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-77a3f016-71da-11ef-bceb-c7b262605968.target.wants/ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-77a3f016-71da-11ef-bceb-c7b262605968@.service. 2024-09-13T14:15:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:00 smithi003 podman[29130]: 2024-09-13 14:15:00.17007276 +0000 UTC m=+0.343351400 container start f9d27dd048901c43e80fda76f0381b363219fd907c948053289075837629421a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-09-13T14:15:00.279 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:15:00.279 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-09-13T14:15:00.287 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:15:00.287 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-09-13T14:15:00.287 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-09-13T14:15:00.287 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T14:15:00.287 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon to start... 2024-09-13T14:15:00.287 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon... 2024-09-13T14:15:00.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:00 smithi003 ceph-mon[29168]: mkfs 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:15:00.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:00 smithi003 ceph-mon[29168]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:15:00.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:00 smithi003 bash[29130]: f9d27dd048901c43e80fda76f0381b363219fd907c948053289075837629421a 2024-09-13T14:15:00.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:00 smithi003 systemd[1]: Started Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:15:01.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:01 smithi003 ceph-mon[29168]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:15:01.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:01 smithi003 ceph-mon[29168]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:15:01.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:01 smithi003 ceph-mon[29168]: fsmap 2024-09-13T14:15:01.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:01 smithi003 ceph-mon[29168]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T14:15:01.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:01 smithi003 ceph-mon[29168]: mgrmap e1: no daemons active 2024-09-13T14:15:01.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:01 smithi003 ceph-mon[29168]: from='client.? 172.21.15.3:0/2895684409' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-13T14:15:01.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout cluster: 2024-09-13T14:15:01.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout id: 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:15:01.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-13T14:15:01.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:01.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout services: 2024-09-13T14:15:01.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.569023s) 2024-09-13T14:15:01.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout data: 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pgs: 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:mon is available 2024-09-13T14:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:Assimilating anything we can from ceph.conf... 2024-09-13T14:15:02.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:02 smithi003 ceph-mon[29168]: from='client.? 172.21.15.3:0/470075376' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T14:15:02.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:02 smithi003 ceph-mon[29168]: from='client.? 172.21.15.3:0/470075376' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-13T14:15:02.472 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:02.472 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T14:15:02.472 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:15:02.472 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T14:15:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:Generating new minimal ceph.conf... 2024-09-13T14:15:03.461 INFO:teuthology.orchestra.run.smithi003.stdout:Restarting the monitor... 2024-09-13T14:15:03.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:03 smithi003 ceph-mon[29168]: from='client.? 172.21.15.3:0/528077283' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:03.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:03 smithi003 systemd[1]: Stopping Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:15:04.161 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:03 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[29145]: 2024-09-13T14:15:03.744+0000 7f34b7f7b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:15:04.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:03 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[29145]: 2024-09-13T14:15:03.744+0000 7f34b7f7b640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-13T14:15:04.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:03 smithi003 podman[29484]: 2024-09-13 14:15:03.899268919 +0000 UTC m=+0.261699278 container died f9d27dd048901c43e80fda76f0381b363219fd907c948053289075837629421a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, ceph=True, org.label-schema.schema-version=1.0) 2024-09-13T14:15:04.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:04 smithi003 podman[29484]: 2024-09-13 14:15:04.268375451 +0000 UTC m=+0.630805810 container cleanup f9d27dd048901c43e80fda76f0381b363219fd907c948053289075837629421a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=quincy-bbdadfa, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-13T14:15:04.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:04 smithi003 bash[29484]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a 2024-09-13T14:15:04.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:04 smithi003 podman[29502]: 2024-09-13 14:15:04.552730152 +0000 UTC m=+0.654056704 container remove f9d27dd048901c43e80fda76f0381b363219fd907c948053289075837629421a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-13T14:15:04.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:04 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T14:15:04.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:04 smithi003 systemd[1]: Stopped Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:15:04.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:04 smithi003 systemd[1]: Starting Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:15:05.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:04 smithi003 podman[29609]: 2024-09-13 14:15:04.831639262 +0000 UTC m=+0.016825963 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph:quincy 2024-09-13T14:15:05.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:04 smithi003 podman[29609]: 2024-09-13 14:15:04.961876729 +0000 UTC m=+0.147063436 container create 9c73445eb77ab5a00da5b9208f7059a9479328857246d1402b1f55f88a7bdef3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-13T14:15:05.244 INFO:teuthology.orchestra.run.smithi003.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-13T14:15:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 podman[29609]: 2024-09-13 14:15:05.172650861 +0000 UTC m=+0.357837560 container init 9c73445eb77ab5a00da5b9208f7059a9479328857246d1402b1f55f88a7bdef3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:15:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 podman[29609]: 2024-09-13 14:15:05.177205189 +0000 UTC m=+0.362391888 container start 9c73445eb77ab5a00da5b9208f7059a9479328857246d1402b1f55f88a7bdef3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-bbdadfa, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:15:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:15:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable), process ceph-mon, pid 7 2024-09-13T14:15:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: pidfile_write: ignore empty --pid-file 2024-09-13T14:15:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: load: jerasure load: lrc 2024-09-13T14:15:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: RocksDB version: 6.15.5 2024-09-13T14:15:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Compile date Sep 12 2024 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: DB SUMMARY 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: DB Session ID: 1ZGNAB9QOTWRNK164GIR 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83647 ; 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:15:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.env: 0x55ba15040440 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.fs: Posix File System 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.info_log: 0x55ba16c54ec0 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.statistics: (nil) 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.use_fsync: 0 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:15:05.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.db_log_dir: 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:15:05.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.write_buffer_manager: 0x55ba16d127e0 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.unordered_write: 0 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:15:05.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:15:05.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:15:05.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.row_cache: None 2024-09-13T14:15:05.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.wal_filter: None 2024-09-13T14:15:05.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:15:05.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:15:05.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.preserve_deletes: 0 2024-09-13T14:15:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:15:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:15:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:15:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:15:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:15:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:15:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:15:05.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_open_files: -1 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Compression algorithms supported: 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: kZSTD supported: 0 2024-09-13T14:15:05.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.merge_operator: 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_filter: None 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:15:05.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ba16bfba38) 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_type: 0 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_index_type: 0 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_shortening: 1 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: hash_index_allow_collision: 1 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: checksum: 1 2024-09-13T14:15:05.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: no_block_cache: 0 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache: 0x55ba16c311f0 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_options: 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: capacity : 536870912 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_shard_bits : 4 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: strict_capacity_limit : 0 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_compressed: (nil) 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: persistent_cache: (nil) 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size: 4096 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size_deviation: 10 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_restart_interval: 16 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_block_restart_interval: 1 2024-09-13T14:15:05.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_block_size: 4096 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: partition_filters: 0 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: use_delta_encoding: 1 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: whole_key_filtering: 1 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: verify_compression: 0 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: format_version: 4 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_index_compression: 1 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_align: 0 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compression: NoCompression 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:15:05.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.num_levels: 7 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:15:05.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:15:05.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.arena_block_size: 4194304 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:15:05.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:15:05.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.ttl: 2592000 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:15:05.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236905201650, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236905202425, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80220, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78462, "index_size": 220, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10130, "raw_average_key_size": 47, "raw_value_size": 72676, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726236905, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "e32e2c4f-7b67-41e7-a71f-fc3bac25accc", "db_session_id": "1ZGNAB9QOTWRNK164GIR"}} 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236905202948, "job": 1, "event": "recovery_finished"} 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55ba16c64700 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: DB pointer 0x55ba16c34000 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-13T14:15:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** DB Stats ** 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:15:05.452 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 2/0 80.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.9 0.00 0.00 1 0.000 0 0 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 80.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.9 0.00 0.00 1 0.000 0 0 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.9 0.00 0.00 1 0.000 0 0 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.9 0.00 0.00 1 0.000 0 0 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:15:05.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 22.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 22.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 2/0 80.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.9 0.00 0.00 1 0.000 0 0 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 80.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.9 0.00 0.00 1 0.000 0 0 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:15:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.9 0.00 0.00 1 0.000 0 0 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 22.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: starting mon.a rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:15:05.455 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mon.a@-1(???) e1 preinit fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mon.a@-1(???).mds e1 new map 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mon.a@-1(???).mds e1 print_map 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: e1 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: legacy client fscid: -1 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: No filesystems configured 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:15:05.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-13T14:15:05.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 bash[29609]: 9c73445eb77ab5a00da5b9208f7059a9479328857246d1402b1f55f88a7bdef3 2024-09-13T14:15:05.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 systemd[1]: Started Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:15:05.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:15:05.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:15:05.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: fsmap 2024-09-13T14:15:05.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T14:15:05.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:05 smithi003 ceph-mon[29646]: mgrmap e1: no daemons active 2024-09-13T14:15:06.307 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-13T14:15:06.308 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:15:06.308 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mgr... 2024-09-13T14:15:06.308 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-13T14:15:06.309 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-13T14:15:06.497 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y 2024-09-13T14:15:06.498 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-13T14:15:06.652 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-77a3f016-71da-11ef-bceb-c7b262605968.target.wants/ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-77a3f016-71da-11ef-bceb-c7b262605968@.service. 2024-09-13T14:15:06.785 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:06 smithi003 systemd[1]: Starting Ceph mgr.y for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:15:07.090 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:06 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2433785795' entity='client.admin' 2024-09-13T14:15:07.091 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:06 smithi003 podman[29882]: 2024-09-13 14:15:06.879921864 +0000 UTC m=+0.111337632 container create fe65bae0d85f36f80352cd9a95b3f221f37cdd513ce72f92f82acc3dfb3fd7ec (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-13T14:15:07.091 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:06 smithi003 podman[29882]: 2024-09-13 14:15:06.785298001 +0000 UTC m=+0.016713737 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph:quincy 2024-09-13T14:15:07.091 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 podman[29882]: 2024-09-13 14:15:07.09047677 +0000 UTC m=+0.321892496 container init fe65bae0d85f36f80352cd9a95b3f221f37cdd513ce72f92f82acc3dfb3fd7ec (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T14:15:07.213 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:15:07.214 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-09-13T14:15:07.221 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:15:07.221 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-09-13T14:15:07.221 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-09-13T14:15:07.221 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T14:15:07.230 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:15:07.231 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-09-13T14:15:07.238 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:15:07.239 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-09-13T14:15:07.239 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-09-13T14:15:07.239 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-13T14:15:07.239 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr to start... 2024-09-13T14:15:07.239 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr... 2024-09-13T14:15:07.343 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 podman[29882]: 2024-09-13 14:15:07.0961216 +0000 UTC m=+0.327537328 container start fe65bae0d85f36f80352cd9a95b3f221f37cdd513ce72f92f82acc3dfb3fd7ec (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:15:07.343 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 bash[29882]: fe65bae0d85f36f80352cd9a95b3f221f37cdd513ce72f92f82acc3dfb3fd7ec 2024-09-13T14:15:07.343 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 systemd[1]: Started Ceph mgr.y for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:15:07.343 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:07.212+0000 7effcab2bec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:15:07.343 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:07.273+0000 7effcab2bec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:15:07.343 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:07.340+0000 7effcab2bec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:15:07.619 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:07.470+0000 7effcab2bec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:15:07.619 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:07.617+0000 7effcab2bec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:15:07.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:07 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/4277617353' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T14:15:08.163 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:07.902+0000 7effcab2bec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:15:08.163 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:07.964+0000 7effcab2bec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:15:08.163 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:08 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:08.135+0000 7effcab2bec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "77a3f016-71da-11ef-bceb-c7b262605968", 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-13T14:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T14:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T14:15:08.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T14:15:08.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T14:15:08.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:08.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T14:15:08.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T14:15:08.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T14:15:08.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T14:15:08.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T14:15:08.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T14:15:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T14:15:00.205075+0000", 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T14:15:08.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:15:08.500 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (1/15)... 2024-09-13T14:15:09.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:09 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:09.024+0000 7effcab2bec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:15:09.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:09 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:09.091+0000 7effcab2bec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:15:10.145 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:09 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:09.841+0000 7effcab2bec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:15:10.146 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:09 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:09.957+0000 7effcab2bec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:15:10.146 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.082+0000 7effcab2bec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:15:10.146 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.143+0000 7effcab2bec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:15:10.406 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.274+0000 7effcab2bec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:15:10.406 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.344+0000 7effcab2bec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:15:10.406 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.404+0000 7effcab2bec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:15:10.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.550+0000 7effcab2bec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:15:10.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.616+0000 7effcab2bec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:15:11.000 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.684+0000 7effcab2bec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:15:11.000 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.772+0000 7effcab2bec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:15:11.000 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.832+0000 7effcab2bec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:15:11.000 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:10 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:10.892+0000 7effcab2bec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:15:11.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:11 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1648780613' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T14:15:11.653 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:11.653 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:15:11.653 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "77a3f016-71da-11ef-bceb-c7b262605968", 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-13T14:15:11.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T14:15:11.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:11.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-13T14:15:11.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T14:15:11.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:11.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T14:15:11.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T14:15:11.657 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T14:15:00.205075+0000", 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:15:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (2/15)... 2024-09-13T14:15:11.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:11 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:11.684+0000 7effcab2bec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: Activating manager daemon y 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: mgrmap e2: y(active, starting, since 0.00281888s) 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: Manager daemon y is now available 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:15:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' 2024-09-13T14:15:12.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' 2024-09-13T14:15:12.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:12 smithi003 ceph-mon[29646]: from='mgr.14100 172.21.15.3:0/3740396824' entity='mgr.y' 2024-09-13T14:15:14.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:13 smithi003 ceph-mon[29646]: mgrmap e3: y(active, since 1.00727s) 2024-09-13T14:15:15.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:14 smithi003 ceph-mon[29646]: mgrmap e4: y(active, since 2s) 2024-09-13T14:15:18.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:17 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/887338598' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T14:15:21.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:21.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:15:21.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "77a3f016-71da-11ef-bceb-c7b262605968", 2024-09-13T14:15:21.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T14:15:21.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T14:15:21.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T14:15:21.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T14:15:21.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:21.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T14:15:21.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:21.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T14:15:21.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T14:15:21.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T14:15:21.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T14:15:21.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T14:15:21.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T14:15:21.882 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:21.882 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T14:15:21.882 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:21.883 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T14:15:00.205075+0000", 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:15:21.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:15:21.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T14:15:21.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:15:21.885 INFO:teuthology.orchestra.run.smithi003.stdout:mgr is available 2024-09-13T14:15:22.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:22 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2325383159' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T14:15:23.081 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:23.081 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T14:15:23.081 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:15:23.081 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-13T14:15:23.081 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T14:15:23.082 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T14:15:23.083 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling cephadm module... 2024-09-13T14:15:24.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:23 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2587618793' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-13T14:15:24.014 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:23 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ignoring --setuser ceph since I am not root 2024-09-13T14:15:24.014 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:23 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ignoring --setgroup ceph since I am not root 2024-09-13T14:15:24.014 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:23 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:23.888+0000 7f2632c5fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:15:24.015 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:23 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:23.946+0000 7f2632c5fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:15:24.015 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:24.013+0000 7f2632c5fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:15:24.289 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:24.143+0000 7f2632c5fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:15:24.540 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:24.286+0000 7f2632c5fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:15:24.797 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:24 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2587618793' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-13T14:15:24.797 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:24 smithi003 ceph-mon[29646]: mgrmap e5: y(active, since 12s) 2024-09-13T14:15:24.797 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:24.559+0000 7f2632c5fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:15:24.797 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:24.622+0000 7f2632c5fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:15:25.078 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:24.796+0000 7f2632c5fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:15:25.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:15:25.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-13T14:15:25.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:15:25.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T14:15:25.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T14:15:25.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:15:25.712 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-09-13T14:15:25.712 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 5... 2024-09-13T14:15:25.958 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:25 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3548582547' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T14:15:25.958 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:25 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:25.710+0000 7f2632c5fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:15:25.958 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:25 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:25.783+0000 7f2632c5fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:15:26.841 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:26 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:26.536+0000 7f2632c5fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:15:26.841 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:26 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:26.653+0000 7f2632c5fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:15:26.841 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:26 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:26.778+0000 7f2632c5fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:15:26.841 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:26 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:26.840+0000 7f2632c5fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:15:27.102 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:26 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:26.971+0000 7f2632c5fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:15:27.102 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:27.041+0000 7f2632c5fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:15:27.102 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:27.101+0000 7f2632c5fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:15:27.376 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:27.241+0000 7f2632c5fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:15:27.377 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:27.307+0000 7f2632c5fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:15:27.377 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:27.375+0000 7f2632c5fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:15:27.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:27.459+0000 7f2632c5fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:15:27.683 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:27.519+0000 7f2632c5fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:15:27.683 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:27.578+0000 7f2632c5fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:15:28.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:28.358+0000 7f2632c5fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:15:28.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: Active manager daemon y restarted 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: Activating manager daemon y 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: osdmap e2: 0 total, 0 up, 0 in 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: mgrmap e6: y(active, starting, since 0.00321744s) 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: Manager daemon y is now available 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:28.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:28.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:15:28.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:28 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:15:29.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:29 smithi003 ceph-mon[29646]: Found migration_current of "None". Setting to last migration. 2024-09-13T14:15:29.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:29 smithi003 ceph-mon[29646]: mgrmap e7: y(active, since 1.00716s) 2024-09-13T14:15:29.915 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:15:29.915 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-13T14:15:29.916 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T14:15:29.916 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:15:29.916 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 5 is available 2024-09-13T14:15:29.916 INFO:teuthology.orchestra.run.smithi003.stdout:Setting orchestrator backend to cephadm... 2024-09-13T14:15:30.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:30 smithi003 ceph-mon[29646]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T14:15:30.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:30 smithi003 ceph-mon[29646]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T14:15:31.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:31 smithi003 ceph-mon[29646]: mgrmap e8: y(active, since 2s) 2024-09-13T14:15:31.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:31 smithi003 ceph-mon[29646]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:31.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:31 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:31.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:31 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:32.201 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-13T14:15:32.201 INFO:teuthology.orchestra.run.smithi003.stdout:Generating ssh key... 2024-09-13T14:15:32.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-mon[29646]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:32.502+0000 7f25e500e640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:15:32] ENGINE Error in HTTPServer.serve 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: Traceback (most recent call last): 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self._connections.run(self.expiration_interval) 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self._run(expiration_interval) 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:15:32.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:15:32.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: s = self.context.wrap_socket( 2024-09-13T14:15:32.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:15:32.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: return self.sslsocket_class._create( 2024-09-13T14:15:32.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:15:32.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self.do_handshake() 2024-09-13T14:15:32.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:15:32.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self._sslobj.do_handshake() 2024-09-13T14:15:32.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:15:32.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:33.506 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: Generating public/private rsa key pair. 2024-09-13T14:15:33.506 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: Your identification has been saved in /tmp/tmpf8fby5d4/key 2024-09-13T14:15:33.506 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: Your public key has been saved in /tmp/tmpf8fby5d4/key.pub 2024-09-13T14:15:33.506 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: The key fingerprint is: 2024-09-13T14:15:33.506 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: SHA256:8mF0FlEyAsMp8YeqtzGg8lJPi8d4AVeYr6oBtaX1MsE ceph-77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:15:33.506 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: The key's randomart image is: 2024-09-13T14:15:33.506 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: +---[RSA 3072]----+ 2024-09-13T14:15:33.506 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: | .=oo. =o. | 2024-09-13T14:15:33.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: | .+.+o . + | 2024-09-13T14:15:33.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: | . E+o o o | 2024-09-13T14:15:33.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: | ..=.+.o o | 2024-09-13T14:15:33.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: |. +o+.o S | 2024-09-13T14:15:33.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: |...o+o + . | 2024-09-13T14:15:33.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: |+..B+o . | 2024-09-13T14:15:33.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: |oo+.*+ | 2024-09-13T14:15:33.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: |.o.o. | 2024-09-13T14:15:33.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: +----[SHA256]-----+ 2024-09-13T14:15:33.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-mon[29646]: [13/Sep/2024:14:15:32] ENGINE Bus STARTING 2024-09-13T14:15:33.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-mon[29646]: [13/Sep/2024:14:15:32] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:15:33.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-mon[29646]: [13/Sep/2024:14:15:32] ENGINE Bus STARTED 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-mon[29646]: [13/Sep/2024:14:15:32] ENGINE Error in HTTPServer.serve 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:15:33.857 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:33.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:33 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:34.794 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:34 smithi003 ceph-mon[29646]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:34.794 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:34 smithi003 ceph-mon[29646]: Generating ssh key... 2024-09-13T14:15:35.038 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmQhIPHc6nflOoopCG6zLwcOoZQ7cz3IhM7EC5YVXAUn3GaoE/xGyuDM2OG1K+FAjiJeX10IhDDxu60NcYxzUrAjWAvpfttRVhjRXKa6MGpb5W/IsaTMIO1nKEAiH/M4UEK7i/xI82NU6pWBZXnqJJnp/NOnHZ2sJtmrVzd+DrFuxvR8fHtFsG9gh/q7j+KkwACZVHY1BfNPxfsbVxIyjCcNLQGK+JeK0wgnY/ErCnpPHhmoKPSPslRsQN+GdcILWQiHRUesWp+ETWKYtLpESvZty22j0c7npeg3gr3ovT8/kCtzmb8qeNaTkt1zpIZusPVMLZELjj03ugz9M8N8zCyYl+43geN4IqfDQTkJH+ap4gSKqfZYrGViOgH3fejR+q1V5Ch23MbIkSksfGbIaqKXw8crGxlccRHby8ILTfsopKhPtk9Hk5/ODYcelgDDB8ihMaxsj+GDR7sHFKY/cMyzMpd0haDKVhmx6X1a1uYCsD3N/GShey5Itxbzn/+8= ceph-77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:15:35.039 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-13T14:15:35.039 INFO:teuthology.orchestra.run.smithi003.stdout:Adding key to root@localhost authorized_keys... 2024-09-13T14:15:35.049 INFO:teuthology.orchestra.run.smithi003.stdout:Adding host smithi003... 2024-09-13T14:15:35.834 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:35 smithi003 ceph-mon[29646]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:36.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:36 smithi003 ceph-mon[29646]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:38.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:37 smithi003 ceph-mon[29646]: Deploying cephadm binary to smithi003 2024-09-13T14:15:39.538 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Added host 'smithi003' with addr '172.21.15.3' 2024-09-13T14:15:39.538 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mon service... 2024-09-13T14:15:40.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:40 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:40.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:40 smithi003 ceph-mon[29646]: Added host smithi003 2024-09-13T14:15:40.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:40 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:41.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:41 smithi003 ceph-mon[29646]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:41.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:41 smithi003 ceph-mon[29646]: Saving service mon spec with placement count:5 2024-09-13T14:15:41.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:41 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:41.851 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-13T14:15:41.852 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mgr service... 2024-09-13T14:15:42.788 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:42 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:42.788 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:42 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:43.050 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-13T14:15:43.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:43 smithi003 ceph-mon[29646]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:43.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:43 smithi003 ceph-mon[29646]: Saving service mgr spec with placement count:2 2024-09-13T14:15:43.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:43 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:44.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:44 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2428730318' entity='client.admin' 2024-09-13T14:15:45.781 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling the dashboard module... 2024-09-13T14:15:46.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:46 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/689215192' entity='client.admin' 2024-09-13T14:15:46.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:46 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:47.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:47 smithi003 ceph-mon[29646]: from='mgr.14118 172.21.15.3:0/1779828402' entity='mgr.y' 2024-09-13T14:15:47.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:47 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3678613337' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-13T14:15:47.779 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ignoring --setuser ceph since I am not root 2024-09-13T14:15:47.779 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ignoring --setgroup ceph since I am not root 2024-09-13T14:15:47.779 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:47.654+0000 7fdecbb27ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:15:47.779 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:47.711+0000 7fdecbb27ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:15:47.780 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:47.778+0000 7fdecbb27ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:15:48.052 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:47.908+0000 7fdecbb27ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:15:48.053 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:48.051+0000 7fdecbb27ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:15:48.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:48 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3678613337' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-13T14:15:48.681 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:48 smithi003 ceph-mon[29646]: mgrmap e9: y(active, since 19s) 2024-09-13T14:15:48.681 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:48.321+0000 7fdecbb27ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:15:48.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:48.387+0000 7fdecbb27ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:15:48.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:48.560+0000 7fdecbb27ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:15:49.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:49 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1071109796' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T14:15:49.519 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:49.449+0000 7fdecbb27ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:15:49.519 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:49.517+0000 7fdecbb27ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:15:49.567 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:15:49.567 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-13T14:15:49.567 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:15:49.567 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T14:15:49.567 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T14:15:49.568 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:15:49.568 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-09-13T14:15:49.568 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 9... 2024-09-13T14:15:50.569 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:50.266+0000 7fdecbb27ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:15:50.570 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:50.382+0000 7fdecbb27ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:15:50.570 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:50.507+0000 7fdecbb27ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:15:50.570 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:50.568+0000 7fdecbb27ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:15:50.829 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:50.699+0000 7fdecbb27ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:15:50.829 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:50.768+0000 7fdecbb27ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:15:50.829 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:50.828+0000 7fdecbb27ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:15:51.101 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:50.967+0000 7fdecbb27ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:15:51.102 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:51.033+0000 7fdecbb27ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:15:51.102 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:51.100+0000 7fdecbb27ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:15:51.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:51.184+0000 7fdecbb27ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:15:51.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:51.244+0000 7fdecbb27ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:15:51.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:51.302+0000 7fdecbb27ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:15:52.337 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:52.079+0000 7fdecbb27ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:15:52.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: Active manager daemon y restarted 2024-09-13T14:15:52.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: Activating manager daemon y 2024-09-13T14:15:52.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: osdmap e3: 0 total, 0 up, 0 in 2024-09-13T14:15:52.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: mgrmap e10: y(active, starting, since 0.00319614s) 2024-09-13T14:15:52.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:52.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:15:52.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:15:52.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:15:52.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:15:52.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: Manager daemon y is now available 2024-09-13T14:15:52.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:52.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:15:52.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:15:53.790 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:53.686+0000 7fde7c685640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:15:53] ENGINE Error in HTTPServer.serve 2024-09-13T14:15:53.790 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: Traceback (most recent call last): 2024-09-13T14:15:53.790 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:15:53.790 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self._connections.run(self.expiration_interval) 2024-09-13T14:15:53.790 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:15:53.790 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self._run(expiration_interval) 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: s = self.context.wrap_socket( 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: return self.sslsocket_class._create( 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self.do_handshake() 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self._sslobj.do_handshake() 2024-09-13T14:15:53.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:15:53.792 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:15:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:15:53.793 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:15:53.793 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-13T14:15:53.793 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T14:15:53.793 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:15:53.793 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 9 is available 2024-09-13T14:15:53.793 INFO:teuthology.orchestra.run.smithi003.stdout:Generating a dashboard self-signed certificate... 2024-09-13T14:15:54.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:54 smithi003 ceph-mon[29646]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T14:15:54.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:54 smithi003 ceph-mon[29646]: mgrmap e11: y(active, since 1.00527s) 2024-09-13T14:15:54.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:54 smithi003 ceph-mon[29646]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T14:15:54.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:15:55.441 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-13T14:15:55.441 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial admin user... 2024-09-13T14:15:55.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:55 smithi003 ceph-mon[29646]: [13/Sep/2024:14:15:53] ENGINE Bus STARTING 2024-09-13T14:15:55.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:55 smithi003 ceph-mon[29646]: [13/Sep/2024:14:15:53] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:15:55.667 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:55 smithi003 ceph-mon[29646]: [13/Sep/2024:14:15:53] ENGINE Bus STARTED 2024-09-13T14:15:55.667 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:55 smithi003 ceph-mon[29646]: [13/Sep/2024:14:15:53] ENGINE Error in HTTPServer.serve 2024-09-13T14:15:55.667 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:15:55.667 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:15:55.667 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:15:55.667 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:15:55.668 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:15:55.668 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:15:55.668 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:15:55.668 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:15:55.668 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:15:55.669 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:15:55.669 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:15:55.669 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:15:55.669 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:15:55.669 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:15:55.670 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-09-13T14:15:55.670 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:15:55.670 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:15:55.670 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:15:55.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:55 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:15:55.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:55 smithi003 ceph-mon[29646]: mgrmap e12: y(active, since 2s) 2024-09-13T14:15:55.671 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:55 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:15:55.671 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:55 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:15:55.671 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:55 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:15:56.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:56 smithi003 ceph-mon[29646]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:56.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:56 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:15:57.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:57 smithi003 ceph-mon[29646]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:57.652 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$mb4yYQ..mhgiCkXEoZlyrev5ROnF.JlGf8RKi.vo/iZfOwdfHQCZW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726236956, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-13T14:15:57.652 INFO:teuthology.orchestra.run.smithi003.stdout:Fetching dashboard port number... 2024-09-13T14:15:58.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:58 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3851282667' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-13T14:15:58.850 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 8443 2024-09-13T14:15:58.862 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:15:58.862 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-09-13T14:15:58.870 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:15:58.870 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-09-13T14:15:58.870 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-09-13T14:15:58.870 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-13T14:15:58.872 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph Dashboard is now available at: 2024-09-13T14:15:58.872 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:15:58.873 INFO:teuthology.orchestra.run.smithi003.stdout: URL: https://smithi003.front.sepia.ceph.com:8443/ 2024-09-13T14:15:58.873 INFO:teuthology.orchestra.run.smithi003.stdout: User: admin 2024-09-13T14:15:58.873 INFO:teuthology.orchestra.run.smithi003.stdout: Password: 7r0cmumcou 2024-09-13T14:15:58.873 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:15:58.873 INFO:teuthology.orchestra.run.smithi003.stdout:Saving cluster configuration to /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config directory 2024-09-13T14:15:58.873 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling autotune for osd_memory_target 2024-09-13T14:15:59.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:15:59 smithi003 ceph-mon[29646]: mgrmap e13: y(active, since 6s) 2024-09-13T14:16:01.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-13T14:16:01.262 INFO:teuthology.orchestra.run.smithi003.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout:Or, if you are only running a single cluster on this host: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: ceph telemetry on 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout:For more information see: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:01.263 INFO:teuthology.orchestra.run.smithi003.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-13T14:16:01.264 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:01.264 INFO:teuthology.orchestra.run.smithi003.stdout:Bootstrap complete. 2024-09-13T14:16:01.305 INFO:tasks.cephadm:Fetching config... 2024-09-13T14:16:01.305 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:16:01.305 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-13T14:16:01.324 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-13T14:16:01.324 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:16:01.324 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-13T14:16:01.381 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-13T14:16:01.381 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:16:01.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-13T14:16:01.448 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-13T14:16:01.448 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:16:01.448 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-13T14:16:01.505 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-13T14:16:01.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmQhIPHc6nflOoopCG6zLwcOoZQ7cz3IhM7EC5YVXAUn3GaoE/xGyuDM2OG1K+FAjiJeX10IhDDxu60NcYxzUrAjWAvpfttRVhjRXKa6MGpb5W/IsaTMIO1nKEAiH/M4UEK7i/xI82NU6pWBZXnqJJnp/NOnHZ2sJtmrVzd+DrFuxvR8fHtFsG9gh/q7j+KkwACZVHY1BfNPxfsbVxIyjCcNLQGK+JeK0wgnY/ErCnpPHhmoKPSPslRsQN+GdcILWQiHRUesWp+ETWKYtLpESvZty22j0c7npeg3gr3ovT8/kCtzmb8qeNaTkt1zpIZusPVMLZELjj03ugz9M8N8zCyYl+43geN4IqfDQTkJH+ap4gSKqfZYrGViOgH3fejR+q1V5Ch23MbIkSksfGbIaqKXw8crGxlccRHby8ILTfsopKhPtk9Hk5/ODYcelgDDB8ihMaxsj+GDR7sHFKY/cMyzMpd0haDKVhmx6X1a1uYCsD3N/GShey5Itxbzn/+8= ceph-77a3f016-71da-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T14:16:01.584 INFO:teuthology.orchestra.run.smithi003.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmQhIPHc6nflOoopCG6zLwcOoZQ7cz3IhM7EC5YVXAUn3GaoE/xGyuDM2OG1K+FAjiJeX10IhDDxu60NcYxzUrAjWAvpfttRVhjRXKa6MGpb5W/IsaTMIO1nKEAiH/M4UEK7i/xI82NU6pWBZXnqJJnp/NOnHZ2sJtmrVzd+DrFuxvR8fHtFsG9gh/q7j+KkwACZVHY1BfNPxfsbVxIyjCcNLQGK+JeK0wgnY/ErCnpPHhmoKPSPslRsQN+GdcILWQiHRUesWp+ETWKYtLpESvZty22j0c7npeg3gr3ovT8/kCtzmb8qeNaTkt1zpIZusPVMLZELjj03ugz9M8N8zCyYl+43geN4IqfDQTkJH+ap4gSKqfZYrGViOgH3fejR+q1V5Ch23MbIkSksfGbIaqKXw8crGxlccRHby8ILTfsopKhPtk9Hk5/ODYcelgDDB8ihMaxsj+GDR7sHFKY/cMyzMpd0haDKVhmx6X1a1uYCsD3N/GShey5Itxbzn/+8= ceph-77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:16:01.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmQhIPHc6nflOoopCG6zLwcOoZQ7cz3IhM7EC5YVXAUn3GaoE/xGyuDM2OG1K+FAjiJeX10IhDDxu60NcYxzUrAjWAvpfttRVhjRXKa6MGpb5W/IsaTMIO1nKEAiH/M4UEK7i/xI82NU6pWBZXnqJJnp/NOnHZ2sJtmrVzd+DrFuxvR8fHtFsG9gh/q7j+KkwACZVHY1BfNPxfsbVxIyjCcNLQGK+JeK0wgnY/ErCnpPHhmoKPSPslRsQN+GdcILWQiHRUesWp+ETWKYtLpESvZty22j0c7npeg3gr3ovT8/kCtzmb8qeNaTkt1zpIZusPVMLZELjj03ugz9M8N8zCyYl+43geN4IqfDQTkJH+ap4gSKqfZYrGViOgH3fejR+q1V5Ch23MbIkSksfGbIaqKXw8crGxlccRHby8ILTfsopKhPtk9Hk5/ODYcelgDDB8ihMaxsj+GDR7sHFKY/cMyzMpd0haDKVhmx6X1a1uYCsD3N/GShey5Itxbzn/+8= ceph-77a3f016-71da-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T14:16:01.650 INFO:teuthology.orchestra.run.smithi019.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmQhIPHc6nflOoopCG6zLwcOoZQ7cz3IhM7EC5YVXAUn3GaoE/xGyuDM2OG1K+FAjiJeX10IhDDxu60NcYxzUrAjWAvpfttRVhjRXKa6MGpb5W/IsaTMIO1nKEAiH/M4UEK7i/xI82NU6pWBZXnqJJnp/NOnHZ2sJtmrVzd+DrFuxvR8fHtFsG9gh/q7j+KkwACZVHY1BfNPxfsbVxIyjCcNLQGK+JeK0wgnY/ErCnpPHhmoKPSPslRsQN+GdcILWQiHRUesWp+ETWKYtLpESvZty22j0c7npeg3gr3ovT8/kCtzmb8qeNaTkt1zpIZusPVMLZELjj03ugz9M8N8zCyYl+43geN4IqfDQTkJH+ap4gSKqfZYrGViOgH3fejR+q1V5Ch23MbIkSksfGbIaqKXw8crGxlccRHby8ILTfsopKhPtk9Hk5/ODYcelgDDB8ihMaxsj+GDR7sHFKY/cMyzMpd0haDKVhmx6X1a1uYCsD3N/GShey5Itxbzn/+8= ceph-77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:16:01.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-13T14:16:01.917 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:16:01.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:01 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/165083254' entity='client.admin' 2024-09-13T14:16:05.332 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-13T14:16:05.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-13T14:16:05.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:05 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1338375927' entity='client.admin' 2024-09-13T14:16:06.023 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:16:07.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:07.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:07.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi003", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:07.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:07.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:07.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:08.451 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi019 2024-09-13T14:16:08.452 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:16:08.452 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/etc/ceph/ceph.conf 2024-09-13T14:16:08.472 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:16:08.472 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:16:08.530 INFO:tasks.cephadm:Adding host smithi019 to orchestrator... 2024-09-13T14:16:08.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch host add smithi019 2024-09-13T14:16:08.785 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:16:09.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:09.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:09.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:09.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:09.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:09.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:16:09.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:10.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:10 smithi003 ceph-mon[29646]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi019", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:13.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:13 smithi003 ceph-mon[29646]: Deploying cephadm binary to smithi019 2024-09-13T14:16:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:13 smithi003 ceph-mon[29646]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:13.912 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi019' with addr '172.21.15.19' 2024-09-13T14:16:14.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-13T14:16:14.784 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:16:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:14 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:14 smithi003 ceph-mon[29646]: Added host smithi019 2024-09-13T14:16:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:14 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:14 smithi003 ceph-mon[29646]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:14 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:14 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:16.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:16 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:16.532 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:16.532 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.19", "hostname": "smithi019", "labels": [], "status": ""}] 2024-09-13T14:16:17.126 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-13T14:16:17.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-13T14:16:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:16:17.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:17 smithi003 ceph-mon[29646]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:17.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:17 smithi003 ceph-mon[29646]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:16:19.353 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-09-13T14:16:19.646 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:19.646 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:19.646 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi019", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1646447297' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:19.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:19.942 INFO:tasks.cephadm:Adding mon.a on smithi003 2024-09-13T14:16:19.942 INFO:tasks.cephadm:Adding mon.c on smithi003 2024-09-13T14:16:19.942 INFO:tasks.cephadm:Adding mon.b on smithi019 2024-09-13T14:16:19.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi019:172.21.15.19=b' 2024-09-13T14:16:20.195 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:20.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:20 smithi003 ceph-mon[29646]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:20.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:20 smithi003 ceph-mon[29646]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:16:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:20 smithi003 ceph-mon[29646]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:16:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:20 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1646447297' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T14:16:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:20 smithi003 ceph-mon[29646]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:16:21.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:21 smithi003 ceph-mon[29646]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:21.924 INFO:teuthology.orchestra.run.smithi019.stdout:Scheduled mon update... 2024-09-13T14:16:22.645 DEBUG:teuthology.orchestra.run.smithi003:mon.c> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.c.service 2024-09-13T14:16:22.648 DEBUG:teuthology.orchestra.run.smithi019:mon.b> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.b.service 2024-09-13T14:16:22.650 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:16:22.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:16:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi019:172.21.15.19=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi019:172.21.15.19=b;count:3 2024-09-13T14:16:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: Deploying daemon mon.b on smithi019 2024-09-13T14:16:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:22 smithi003 ceph-mon[29646]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:25.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:25 smithi003 ceph-mon[29646]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:26.709 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:16:26.709 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":1,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","modified":"2024-09-13T14:14:57.332230Z","created":"2024-09-13T14:14:57.332230Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T14:16:26.709 INFO:teuthology.orchestra.run.smithi019.stderr:dumped monmap epoch 1 2024-09-13T14:16:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:26 smithi019 systemd[1]: Started Ceph mon.b for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:16:28.351 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:16:28.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:16:28.619 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:16:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:16:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: mon.a calling monitor election 2024-09-13T14:16:32.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:32.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: mon.b calling monitor election 2024-09-13T14:16:32.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:32.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: fsmap 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: mgrmap e13: y(active, since 39s) 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: overall HEALTH_OK 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:32.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:31 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:32.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:16:32.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: mon.a calling monitor election 2024-09-13T14:16:32.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:32.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: mon.b calling monitor election 2024-09-13T14:16:32.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: fsmap 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: mgrmap e13: y(active, since 39s) 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: overall HEALTH_OK 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:32.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:31 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:33.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:32 smithi019 ceph-mon[32895]: Deploying daemon mon.c on smithi003 2024-09-13T14:16:33.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:32 smithi019 ceph-mon[32895]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:32 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:32 smithi003 ceph-mon[29646]: Deploying daemon mon.c on smithi003 2024-09-13T14:16:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:32 smithi003 ceph-mon[29646]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:32 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:33.371 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:16:33.372 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":2,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","modified":"2024-09-13T14:16:26.785822Z","created":"2024-09-13T14:14:57.332230Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-13T14:16:33.372 INFO:teuthology.orchestra.run.smithi019.stderr:dumped monmap epoch 2 2024-09-13T14:16:33.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:33 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/19886310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:16:34.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:33 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/19886310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:16:34.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:34 smithi003 podman[41554]: 2024-09-13 14:16:34.802324449 +0000 UTC m=+0.126730561 container create a6373bc053ae585cf9f7c2cebbf38690d5872405a03302298ba5346edceb9188 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:16:35.000 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:16:35.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:16:35.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:34 smithi019 ceph-mon[32895]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:34 smithi003 ceph-mon[29646]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 podman[41554]: 2024-09-13 14:16:35.00529794 +0000 UTC m=+0.329704052 container init a6373bc053ae585cf9f7c2cebbf38690d5872405a03302298ba5346edceb9188 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-13T14:16:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 podman[41554]: 2024-09-13 14:16:35.010575422 +0000 UTC m=+0.334981534 container start a6373bc053ae585cf9f7c2cebbf38690d5872405a03302298ba5346edceb9188 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable), process ceph-mon, pid 7 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pidfile_write: ignore empty --pid-file 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: load: jerasure load: lrc 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: RocksDB version: 6.15.5 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Compile date Sep 12 2024 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: DB SUMMARY 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: DB Session ID: KKOJ0E0ONU10BF7NPI45 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-13T14:16:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.env: 0x55ef099f7440 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.fs: Posix File System 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.info_log: 0x55ef0c24eec0 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.statistics: (nil) 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.use_fsync: 0 2024-09-13T14:16:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:16:35.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:16:35.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:16:35.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.db_log_dir: 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:16:35.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.write_buffer_manager: 0x55ef0c30c7b0 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:16:35.188 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.unordered_write: 0 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.row_cache: None 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.wal_filter: None 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:16:35.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.preserve_deletes: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:16:35.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_open_files: -1 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:16:35.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Compression algorithms supported: 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: kZSTD supported: 0 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-13T14:16:35.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.merge_operator: 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_filter: None 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ef0c1f5a38) 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:16:35.193 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_type: 0 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_block_index_type: 0 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_shortening: 1 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: hash_index_allow_collision: 1 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: checksum: 1 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: no_block_cache: 0 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache: 0x55ef0c22b1f0 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_options: 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: capacity : 536870912 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: num_shard_bits : 4 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: strict_capacity_limit : 0 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_compressed: (nil) 2024-09-13T14:16:35.194 INFO:journalctl@ceph.mon.c.smithi003.stdout: persistent_cache: (nil) 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_size: 4096 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_size_deviation: 10 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_restart_interval: 16 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_block_restart_interval: 1 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: metadata_block_size: 4096 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: partition_filters: 0 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: use_delta_encoding: 1 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: whole_key_filtering: 1 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: verify_compression: 0 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: format_version: 4 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: enable_index_compression: 1 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_align: 0 2024-09-13T14:16:35.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compression: NoCompression 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.num_levels: 7 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:16:35.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:16:35.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.arena_block_size: 4194304 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-13T14:16:35.198 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:16:35.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.ttl: 2592000 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:16:35.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236995035180, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236995035595, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1696, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726236995, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1daf4819-06f4-4b08-948c-244f6581c9ad", "db_session_id": "KKOJ0E0ONU10BF7NPI45"}} 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236995035944, "job": 1, "event": "recovery_finished"} 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55ef0c25e700 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: DB pointer 0x55ef0c22e000 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-13T14:16:35.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** DB Stats ** 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2024-09-13T14:16:35.202 INFO:journalctl@ceph.mon.c.smithi003.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:16:35.203 INFO:journalctl@ceph.mon.c.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:16:35.204 INFO:journalctl@ceph.mon.c.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: using public_addrv [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2024-09-13T14:16:35.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: starting mon.c rank -1 at public addrs [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] at bind addrs [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(???) e0 preinit fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).mds e1 new map 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout: e1 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout: legacy client fscid: -1 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout: No filesystems configured 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-13T14:16:35.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi019", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1646447297' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:16:35.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1646447297' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi019:172.21.15.19=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi019:172.21.15.19=b;count:3 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: Deploying daemon mon.b on smithi019 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.a calling monitor election 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.b calling monitor election 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:16:35.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: fsmap 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mgrmap e13: y(active, since 39s) 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: overall HEALTH_OK 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: Deploying daemon mon.c on smithi003 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/19886310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:16:35.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:35.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 ceph-mon[41592]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-13T14:16:35.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 bash[41554]: a6373bc053ae585cf9f7c2cebbf38690d5872405a03302298ba5346edceb9188 2024-09-13T14:16:35.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:35 smithi003 systemd[1]: Started Ceph mon.c for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:16:35.263 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:16:40.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: mon.a calling monitor election 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: mon.b calling monitor election 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: mon.c calling monitor election 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],c=[v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: fsmap 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: mgrmap e13: y(active, since 47s) 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: overall HEALTH_OK 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: mon.a calling monitor election 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: mon.b calling monitor election 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: mon.c calling monitor election 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:40.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],c=[v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: fsmap 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: mgrmap e13: y(active, since 47s) 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: overall HEALTH_OK 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: mon.a calling monitor election 2024-09-13T14:16:40.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: mon.b calling monitor election 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: mon.c calling monitor election 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],c=[v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: fsmap 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:16:40.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: mgrmap e13: y(active, since 47s) 2024-09-13T14:16:40.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: overall HEALTH_OK 2024-09-13T14:16:40.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:40.453 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:16:40.453 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":3,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","modified":"2024-09-13T14:16:35.060163Z","created":"2024-09-13T14:14:57.332230Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3301","nonce":0},{"type":"v1","addr":"172.21.15.3:6790","nonce":0}]},"addr":"172.21.15.3:6790/0","public_addr":"172.21.15.3:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-13T14:16:40.453 INFO:teuthology.orchestra.run.smithi019.stderr:dumped monmap epoch 3 2024-09-13T14:16:40.940 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-13T14:16:40.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-13T14:16:41.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:41 smithi019 ceph-mon[32895]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:41.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:41 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:41 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/1638515729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:41 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:16:41.417 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:41 smithi003 ceph-mon[41592]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:41.417 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:41 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.417 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:41 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/1638515729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.417 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:41 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:41.417 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:41 smithi003 ceph-mon[29646]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:41.418 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:41 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.418 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:41 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/1638515729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.418 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:41 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:16:42.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:16:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:43.070 INFO:teuthology.orchestra.run.smithi003.stdout:# minimal ceph.conf for 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:16:43.070 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-09-13T14:16:43.070 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:16:43.070 INFO:teuthology.orchestra.run.smithi003.stdout: mon_host = [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2024-09-13T14:16:43.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:43 smithi019 ceph-mon[32895]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:16:43.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:43 smithi019 ceph-mon[32895]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:16:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:43 smithi019 ceph-mon[32895]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:43 smithi019 ceph-mon[32895]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:43 smithi019 ceph-mon[32895]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:43 smithi019 ceph-mon[32895]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:16:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:43 smithi019 ceph-mon[32895]: Reconfiguring daemon mon.a on smithi003 2024-09-13T14:16:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:43 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2201668749' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:43.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[29646]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:16:43.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[29646]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[29646]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[29646]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[29646]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[29646]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[29646]: Reconfiguring daemon mon.a on smithi003 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2201668749' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[41592]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[41592]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[41592]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[41592]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:16:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[41592]: Reconfiguring daemon mon.a on smithi003 2024-09-13T14:16:43.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:43 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2201668749' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:43.810 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-13T14:16:43.810 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:16:43.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T14:16:43.839 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:16:43.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:16:43.906 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:16:43.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T14:16:43.940 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:16:43.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:16:44.010 INFO:tasks.cephadm:Adding mgr.y on smithi003 2024-09-13T14:16:44.010 INFO:tasks.cephadm:Adding mgr.x on smithi019 2024-09-13T14:16:44.011 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi003=y;smithi019=x' 2024-09-13T14:16:44.308 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:16:45.812 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:45 smithi019 ceph-mon[32895]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:45.812 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:45 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:45.812 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:45 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:45.812 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:45 smithi019 ceph-mon[32895]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:16:45.813 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:45 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:45.813 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:45 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:16:45.813 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:45 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:45.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[29646]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:45.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:45.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:45.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[29646]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:16:45.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:45.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:16:45.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:45.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[41592]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:45.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:45.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:45.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[41592]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:16:45.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:45.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:16:45.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:45 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:46.001 INFO:teuthology.orchestra.run.smithi019.stdout:Scheduled mgr update... 2024-09-13T14:16:46.586 DEBUG:teuthology.orchestra.run.smithi019:mgr.x> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service 2024-09-13T14:16:46.588 DEBUG:tasks.cephadm:set 0 configs 2024-09-13T14:16:46.588 INFO:tasks.cephadm:Deploying OSDs... 2024-09-13T14:16:46.588 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:16:46.588 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T14:16:46.609 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T14:16:46.609 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-09-13T14:16:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T14:16:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:16:46.667 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-13T14:16:46.668 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:16:46.668 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:16:46.668 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-09-13 14:16:34.500796933 +0000 2024-09-13T14:16:46.668 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-09-13 14:11:52.142386229 +0000 2024-09-13T14:16:46.668 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-09-13 14:11:52.142386229 +0000 2024-09-13T14:16:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-09-13 14:11:52.142386229 +0000 2024-09-13T14:16:46.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T14:16:46.737 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-09-13T14:16:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-09-13T14:16:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000227442 s, 2.3 MB/s 2024-09-13T14:16:46.739 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T14:16:46.800 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-09-13T14:16:46.825 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:46 smithi019 ceph-mon[32895]: Reconfiguring daemon mon.c on smithi003 2024-09-13T14:16:46.825 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:46 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:46.825 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:46 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:46.825 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:46 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:46.825 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:46 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:46.825 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:46 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:16:46.826 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:46 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:46.858 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T14:16:46.859 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:16:46.859 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 972 Links: 1 2024-09-13T14:16:46.859 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:16:46.859 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:16:46.859 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-09-13 14:16:34.501796920 +0000 2024-09-13T14:16:46.859 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-09-13 14:11:52.142386229 +0000 2024-09-13T14:16:46.859 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-09-13 14:11:52.142386229 +0000 2024-09-13T14:16:46.859 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-09-13 14:11:52.142386229 +0000 2024-09-13T14:16:46.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T14:16:46.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[29646]: Reconfiguring daemon mon.c on smithi003 2024-09-13T14:16:46.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:46.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:46.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:46.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:46.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:16:46.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:46.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[41592]: Reconfiguring daemon mon.c on smithi003 2024-09-13T14:16:46.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:46.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:46.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:46.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:16:46.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:16:46.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:46 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:46.929 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-09-13T14:16:46.929 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-09-13T14:16:46.930 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000203402 s, 2.5 MB/s 2024-09-13T14:16:46.930 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T14:16:46.989 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-09-13T14:16:47.048 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T14:16:47.048 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:16:47.048 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 977 Links: 1 2024-09-13T14:16:47.049 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:16:47.049 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:16:47.049 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-09-13 14:16:34.501796920 +0000 2024-09-13T14:16:47.049 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-09-13 14:11:52.143386215 +0000 2024-09-13T14:16:47.049 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-09-13 14:11:52.143386215 +0000 2024-09-13T14:16:47.049 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-09-13 14:11:52.143386215 +0000 2024-09-13T14:16:47.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T14:16:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-09-13T14:16:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-09-13T14:16:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.00020937 s, 2.4 MB/s 2024-09-13T14:16:47.119 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T14:16:47.177 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-09-13T14:16:47.235 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T14:16:47.235 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:16:47.235 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-13T14:16:47.235 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:16:47.235 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:16:47.236 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-09-13 14:16:34.501796920 +0000 2024-09-13T14:16:47.236 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-09-13 14:11:52.143386215 +0000 2024-09-13T14:16:47.236 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-09-13 14:11:52.143386215 +0000 2024-09-13T14:16:47.236 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-09-13 14:11:52.143386215 +0000 2024-09-13T14:16:47.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T14:16:47.302 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-09-13T14:16:47.304 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-09-13T14:16:47.304 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000209309 s, 2.4 MB/s 2024-09-13T14:16:47.305 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T14:16:47.363 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:16:47.363 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T14:16:47.381 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T14:16:47.381 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-09-13T14:16:47.438 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T14:16:47.438 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:16:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 970 Links: 1 2024-09-13T14:16:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:16:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:16:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-09-13 14:16:25.633197289 +0000 2024-09-13T14:16:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-09-13 14:11:50.344812102 +0000 2024-09-13T14:16:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-09-13 14:11:50.344812102 +0000 2024-09-13T14:16:47.438 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-09-13 14:11:50.344812102 +0000 2024-09-13T14:16:47.439 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T14:16:47.504 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-09-13T14:16:47.505 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-09-13T14:16:47.505 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000211033 s, 2.4 MB/s 2024-09-13T14:16:47.506 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T14:16:47.564 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-09-13T14:16:47.620 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T14:16:47.620 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:16:47.620 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 968 Links: 1 2024-09-13T14:16:47.620 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:16:47.621 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:16:47.621 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-09-13 14:16:25.633197289 +0000 2024-09-13T14:16:47.621 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-09-13 14:11:50.344812102 +0000 2024-09-13T14:16:47.621 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-09-13 14:11:50.344812102 +0000 2024-09-13T14:16:47.621 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-09-13 14:11:50.344812102 +0000 2024-09-13T14:16:47.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T14:16:47.686 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-09-13T14:16:47.686 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-09-13T14:16:47.686 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000250505 s, 2.0 MB/s 2024-09-13T14:16:47.688 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T14:16:47.745 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-09-13T14:16:47.790 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:47 smithi019 ceph-mon[32895]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi003=y;smithi019=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:47.790 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:47 smithi019 ceph-mon[32895]: Saving service mgr spec with placement smithi003=y;smithi019=x;count:2 2024-09-13T14:16:47.790 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:47 smithi019 ceph-mon[32895]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:47.790 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:47 smithi019 ceph-mon[32895]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:16:47.790 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:47 smithi019 ceph-mon[32895]: Reconfiguring daemon mon.b on smithi019 2024-09-13T14:16:47.802 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T14:16:47.802 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:16:47.802 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-13T14:16:47.802 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:16:47.803 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:16:47.803 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-09-13 14:16:25.634197279 +0000 2024-09-13T14:16:47.803 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-09-13 14:11:50.347812414 +0000 2024-09-13T14:16:47.803 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-09-13 14:11:50.347812414 +0000 2024-09-13T14:16:47.803 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-09-13 14:11:50.347812414 +0000 2024-09-13T14:16:47.803 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T14:16:47.867 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-09-13T14:16:47.867 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-09-13T14:16:47.868 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000219282 s, 2.3 MB/s 2024-09-13T14:16:47.869 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T14:16:47.927 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-09-13T14:16:47.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[29646]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi003=y;smithi019=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:47.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[29646]: Saving service mgr spec with placement smithi003=y;smithi019=x;count:2 2024-09-13T14:16:47.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[29646]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:47.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[29646]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:16:47.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[29646]: Reconfiguring daemon mon.b on smithi019 2024-09-13T14:16:47.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[41592]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi003=y;smithi019=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:47.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[41592]: Saving service mgr spec with placement smithi003=y;smithi019=x;count:2 2024-09-13T14:16:47.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[41592]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:47.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[41592]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:16:47.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:47 smithi003 ceph-mon[41592]: Reconfiguring daemon mon.b on smithi019 2024-09-13T14:16:47.983 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T14:16:47.983 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:16:47.983 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-13T14:16:47.983 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:16:47.983 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:16:47.983 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-09-13 14:16:25.634197279 +0000 2024-09-13T14:16:47.983 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-09-13 14:11:50.345812206 +0000 2024-09-13T14:16:47.984 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-09-13 14:11:50.345812206 +0000 2024-09-13T14:16:47.984 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-09-13 14:11:50.345812206 +0000 2024-09-13T14:16:47.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T14:16:48.049 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-09-13T14:16:48.049 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-09-13T14:16:48.049 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000176405 s, 2.9 MB/s 2024-09-13T14:16:48.050 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T14:16:48.108 INFO:tasks.cephadm:Deploying osd.0 on smithi003 with /dev/vg_nvme/lv_4... 2024-09-13T14:16:48.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T14:16:48.366 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:16:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:16:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:16:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:16:49.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:49.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: Deploying daemon mgr.x on smithi019 2024-09-13T14:16:49.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:48 smithi019 ceph-mon[32895]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: Deploying daemon mgr.x on smithi019 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[29646]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:49.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:49.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:49.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:49.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:16:49.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:16:49.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:16:49.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:49.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: Deploying daemon mgr.x on smithi019 2024-09-13T14:16:49.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:48 smithi003 ceph-mon[41592]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:49.599 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:49 smithi019 systemd[1]: Starting Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:16:49.932 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:49 smithi019 podman[34946]: 2024-09-13 14:16:49.714305038 +0000 UTC m=+0.016207294 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be 2024-09-13T14:16:49.932 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:49 smithi019 podman[34946]: 2024-09-13 14:16:49.830078304 +0000 UTC m=+0.131980544 container create f4595aba35763e5274893a4b034ef1a9b36166d8d3d21dcbb122485db463a4fc (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T14:16:50.317 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 podman[34946]: 2024-09-13 14:16:50.049573309 +0000 UTC m=+0.351475550 container init f4595aba35763e5274893a4b034ef1a9b36166d8d3d21dcbb122485db463a4fc (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:16:50.317 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 podman[34946]: 2024-09-13 14:16:50.054775946 +0000 UTC m=+0.356678199 container start f4595aba35763e5274893a4b034ef1a9b36166d8d3d21dcbb122485db463a4fc (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa) 2024-09-13T14:16:50.317 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 ceph-mgr[34985]: -- 172.21.15.19:0/407906430 <== mon.1 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d45bf8c340 con 0x55d45bf98800 2024-09-13T14:16:50.317 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 bash[34946]: f4595aba35763e5274893a4b034ef1a9b36166d8d3d21dcbb122485db463a4fc 2024-09-13T14:16:50.317 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 systemd[1]: Started Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:16:50.318 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:50.176+0000 7fa139f9dec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:16:50.318 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:50.245+0000 7fa139f9dec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:16:50.318 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:50.314+0000 7fa139f9dec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:16:50.599 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:50.447+0000 7fa139f9dec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:16:50.781 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:16:50.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi003:vg_nvme/lv_4 2024-09-13T14:16:50.878 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:50.597+0000 7fa139f9dec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:16:51.051 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:16:51.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:50.876+0000 7fa139f9dec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:16:51.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:50.942+0000 7fa139f9dec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:16:51.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:51 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:51.115+0000 7fa139f9dec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:16:51.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[41592]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:51.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:51.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[29646]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:51.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:51 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:51.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:51 smithi019 ceph-mon[32895]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:51.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:51 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:51 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:51 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:51 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:51 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:52.381 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:52.009+0000 7fa139f9dec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:16:52.381 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:52.076+0000 7fa139f9dec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:16:52.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: Reconfiguring daemon mgr.y on smithi003 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.413 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:52.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:52.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:16:52.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:16:52.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:16:52.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:52.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: Reconfiguring daemon mgr.y on smithi003 2024-09-13T14:16:52.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:52 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:16:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:16:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:16:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: Reconfiguring daemon mgr.y on smithi003 2024-09-13T14:16:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:53.129 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:52.824+0000 7fa139f9dec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:16:53.129 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:52.940+0000 7fa139f9dec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:16:53.129 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.065+0000 7fa139f9dec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:16:53.129 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.127+0000 7fa139f9dec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:16:53.390 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.258+0000 7fa139f9dec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:16:53.390 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.327+0000 7fa139f9dec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:16:53.390 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.387+0000 7fa139f9dec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:16:53.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:53 smithi003 ceph-mon[29646]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:53.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:53 smithi003 ceph-mon[41592]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:53.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-mon[32895]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:53.666 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.529+0000 7fa139f9dec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:16:53.666 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.596+0000 7fa139f9dec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:16:53.666 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.663+0000 7fa139f9dec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:16:53.931 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.748+0000 7fa139f9dec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:16:53.931 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.807+0000 7fa139f9dec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:16:53.932 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:53 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:53.867+0000 7fa139f9dec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:16:54.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:16:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:16:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:54.931 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:16:54.647+0000 7fa139f9dec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:16:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:16:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:16:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:16:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:55.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[29646]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:55.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[29646]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[29646]: Standby manager daemon x started 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[41592]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[41592]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[41592]: Standby manager daemon x started 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:16:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:16:55.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:55 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:16:55.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:55 smithi019 ceph-mon[32895]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:55 smithi019 ceph-mon[32895]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:55 smithi019 ceph-mon[32895]: Standby manager daemon x started 2024-09-13T14:16:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:55 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:16:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:55 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:16:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:55 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:16:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:55 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3234299391' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:16:56.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[29646]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3172449032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a153ec29-6e01-4063-aa90-01d1de00e53d"}]: dispatch 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3172449032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a153ec29-6e01-4063-aa90-01d1de00e53d"}]': finished 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[29646]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[41592]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3172449032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a153ec29-6e01-4063-aa90-01d1de00e53d"}]: dispatch 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3172449032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a153ec29-6e01-4063-aa90-01d1de00e53d"}]': finished 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[41592]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:16:56.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:56 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:16:56.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:56 smithi019 ceph-mon[32895]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-13T14:16:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:56 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:16:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:56 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3172449032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a153ec29-6e01-4063-aa90-01d1de00e53d"}]: dispatch 2024-09-13T14:16:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:56 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3172449032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a153ec29-6e01-4063-aa90-01d1de00e53d"}]': finished 2024-09-13T14:16:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:56 smithi019 ceph-mon[32895]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:16:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:56 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:16:57.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:57 smithi019 ceph-mon[32895]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:57.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:57 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1978585107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:16:57.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:57 smithi003 ceph-mon[29646]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:57.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:57 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1978585107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:16:57.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:57 smithi003 ceph-mon[41592]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:57.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:57 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1978585107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:16:59.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:16:59 smithi003 ceph-mon[29646]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:59.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:16:59 smithi003 ceph-mon[41592]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:16:59 smithi019 ceph-mon[32895]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:01.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:01 smithi003 ceph-mon[29646]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:01 smithi003 ceph-mon[41592]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:01.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:01 smithi019 ceph-mon[32895]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:02.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:02 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:17:02.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:02 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:02.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:02 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:17:02.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:02 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:02.833 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:02 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:17:02.833 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:02 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:03.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:03 smithi003 ceph-mon[29646]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:03.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:03 smithi003 ceph-mon[29646]: Deploying daemon osd.0 on smithi003 2024-09-13T14:17:03.674 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:03 smithi003 ceph-mon[41592]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:03.674 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:03 smithi003 ceph-mon[41592]: Deploying daemon osd.0 on smithi003 2024-09-13T14:17:03.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:03 smithi019 ceph-mon[32895]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:03.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:03 smithi019 ceph-mon[32895]: Deploying daemon osd.0 on smithi003 2024-09-13T14:17:05.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:05 smithi003 ceph-mon[29646]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:05.827 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:05 smithi003 ceph-mon[41592]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:05.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:05 smithi019 ceph-mon[32895]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:07.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:07 smithi003 ceph-mon[29646]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:07.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:07 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:07.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:07 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:07.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:07 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:07.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:07 smithi003 ceph-mon[41592]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:07.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:07 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:07.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:07 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:07.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:07 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:07.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:07 smithi019 ceph-mon[32895]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:07.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:07 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:07 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:07 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:09.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:09 smithi003 ceph-mon[41592]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:09.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:09 smithi003 ceph-mon[29646]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:09.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:09 smithi019 ceph-mon[32895]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:10.213 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 0 on host 'smithi003' 2024-09-13T14:17:10.788 DEBUG:teuthology.orchestra.run.smithi003:osd.0> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.0.service 2024-09-13T14:17:10.794 INFO:tasks.cephadm:Deploying osd.1 on smithi003 with /dev/vg_nvme/lv_3... 2024-09-13T14:17:10.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T14:17:11.103 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:11.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[41592]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:11.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[41592]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:17:11.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[29646]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:11.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[29646]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:17:11.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:11 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:11 smithi019 ceph-mon[32895]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:11 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:11 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:11 smithi019 ceph-mon[32895]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:17:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:11 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:11 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:12.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-mon[41592]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:17:12.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-mon[41592]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:17:12.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-mon[41592]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:17:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-mon[29646]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:17:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-mon[29646]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:17:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-mon[29646]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:17:12.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[45851]: 2024-09-13T14:17:12.218+0000 7f35f4ec6640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T14:17:12.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:17:12 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[45851]: 2024-09-13T14:17:12.220+0000 7f35efcfd640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:17:12.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:12 smithi019 ceph-mon[32895]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:17:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:12 smithi019 ceph-mon[32895]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:17:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:12 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:12 smithi019 ceph-mon[32895]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:17:13.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[29646]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[29646]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:17:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[29646]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:17:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[41592]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[41592]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:17:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[41592]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:17:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:13.599 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:13 smithi019 ceph-mon[32895]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:13.599 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:13 smithi019 ceph-mon[32895]: from='osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:17:13.599 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:13 smithi019 ceph-mon[32895]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:17:13.599 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:13.599 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:14.230 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:17:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[29646]: purged_snaps scrub starts 2024-09-13T14:17:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[29646]: purged_snaps scrub ok 2024-09-13T14:17:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[29646]: osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307] boot 2024-09-13T14:17:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[29646]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:17:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:14.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[41592]: purged_snaps scrub starts 2024-09-13T14:17:14.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[41592]: purged_snaps scrub ok 2024-09-13T14:17:14.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[41592]: osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307] boot 2024-09-13T14:17:14.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[41592]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:17:14.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:14 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:14.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi003:vg_nvme/lv_3 2024-09-13T14:17:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:14 smithi019 ceph-mon[32895]: purged_snaps scrub starts 2024-09-13T14:17:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:14 smithi019 ceph-mon[32895]: purged_snaps scrub ok 2024-09-13T14:17:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:14 smithi019 ceph-mon[32895]: osd.0 [v2:172.21.15.3:6802/3028312307,v1:172.21.15.3:6803/3028312307] boot 2024-09-13T14:17:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:14 smithi019 ceph-mon[32895]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:17:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:14 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:15.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:15 smithi003 ceph-mon[29646]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:15.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:15 smithi003 ceph-mon[29646]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:17:15.532 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:15 smithi003 ceph-mon[41592]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:15.532 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:15 smithi003 ceph-mon[41592]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:17:15.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:15 smithi019 ceph-mon[32895]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:17:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:15 smithi019 ceph-mon[32895]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:17:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: Detected new or changed devices on smithi003 2024-09-13T14:17:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi003 to 16068M 2024-09-13T14:17:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:17.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:17.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:17.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:17.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:17.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: Detected new or changed devices on smithi003 2024-09-13T14:17:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi003 to 16068M 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: Detected new or changed devices on smithi003 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi003 to 16068M 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:18.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:18 smithi019 ceph-mon[32895]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:18.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:18 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:18.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:18 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:18.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:18 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:18.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:18 smithi003 ceph-mon[29646]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:18.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:18 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:18.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:18 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:18.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:18 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:18.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:18 smithi003 ceph-mon[41592]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:18.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:18 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:18.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:18 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:18.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:18 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:19.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:19 smithi019 ceph-mon[32895]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:19.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:19 smithi003 ceph-mon[29646]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:19 smithi003 ceph-mon[41592]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:20 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/4194971171' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70"}]: dispatch 2024-09-13T14:17:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:20 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70"}]: dispatch 2024-09-13T14:17:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:20 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70"}]': finished 2024-09-13T14:17:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:20 smithi019 ceph-mon[32895]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:17:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:20 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:20 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2104998764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:20.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/4194971171' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70"}]: dispatch 2024-09-13T14:17:20.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70"}]: dispatch 2024-09-13T14:17:20.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70"}]': finished 2024-09-13T14:17:20.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[29646]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:17:20.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:20.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2104998764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:20.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/4194971171' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70"}]: dispatch 2024-09-13T14:17:20.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70"}]: dispatch 2024-09-13T14:17:20.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70"}]': finished 2024-09-13T14:17:20.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[41592]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:17:20.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:20.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:20 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2104998764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:21.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:21 smithi019 ceph-mon[32895]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:21.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:21 smithi003 ceph-mon[29646]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:21.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:21 smithi003 ceph-mon[41592]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:23.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:23 smithi003 ceph-mon[29646]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:23.574 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:23 smithi003 ceph-mon[41592]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:23.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:23 smithi019 ceph-mon[32895]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:25.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:25 smithi003 ceph-mon[29646]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:25.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:25 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:17:25.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:25 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:25.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:25 smithi003 ceph-mon[41592]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:25.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:25 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:17:25.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:25 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:25.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:25 smithi019 ceph-mon[32895]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:25 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:17:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:25 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:26.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:26 smithi003 ceph-mon[29646]: Deploying daemon osd.1 on smithi003 2024-09-13T14:17:26.839 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:26 smithi003 ceph-mon[41592]: Deploying daemon osd.1 on smithi003 2024-09-13T14:17:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:26 smithi019 ceph-mon[32895]: Deploying daemon osd.1 on smithi003 2024-09-13T14:17:27.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:27 smithi003 ceph-mon[41592]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:27.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:27 smithi003 ceph-mon[29646]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:27.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:27 smithi019 ceph-mon[32895]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:29.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:29 smithi003 ceph-mon[29646]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:29.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:29 smithi003 ceph-mon[41592]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:29.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:29 smithi019 ceph-mon[32895]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:30.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:30 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:30.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:30 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:30.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:30 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:30.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:30 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:30.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:30 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:30.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:30 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:30.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:30 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:30.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:30 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:30.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:30 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:31.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:31 smithi003 ceph-mon[41592]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:31.805 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:31 smithi003 ceph-mon[29646]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:31.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:31 smithi019 ceph-mon[32895]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:33.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:33 smithi019 ceph-mon[32895]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:33 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:33 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:33 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:33 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:33 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:33 smithi019 ceph-mon[32895]: from='osd.1 [v2:172.21.15.3:6810/2008999734,v1:172.21.15.3:6811/2008999734]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:17:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:33 smithi019 ceph-mon[32895]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:17:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[29646]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[29646]: from='osd.1 [v2:172.21.15.3:6810/2008999734,v1:172.21.15.3:6811/2008999734]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:17:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[29646]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:17:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[41592]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[41592]: from='osd.1 [v2:172.21.15.3:6810/2008999734,v1:172.21.15.3:6811/2008999734]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:17:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:33 smithi003 ceph-mon[41592]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:17:34.013 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 1 on host 'smithi003' 2024-09-13T14:17:34.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:34.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:34.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:34.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[41592]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:17:34.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[41592]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:17:34.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:34.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[41592]: from='osd.1 [v2:172.21.15.3:6810/2008999734,v1:172.21.15.3:6811/2008999734]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:17:34.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[41592]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:17:34.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:34.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:34.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:34.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[29646]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:17:34.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[29646]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:17:34.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:34.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[29646]: from='osd.1 [v2:172.21.15.3:6810/2008999734,v1:172.21.15.3:6811/2008999734]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:17:34.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:34 smithi003 ceph-mon[29646]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:17:34.596 DEBUG:teuthology.orchestra.run.smithi003:osd.1> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.1.service 2024-09-13T14:17:34.599 INFO:tasks.cephadm:Deploying osd.2 on smithi003 with /dev/vg_nvme/lv_2... 2024-09-13T14:17:34.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T14:17:34.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:34 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:34 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:34 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:34 smithi019 ceph-mon[32895]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:17:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:34 smithi019 ceph-mon[32895]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:17:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:34 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:34 smithi019 ceph-mon[32895]: from='osd.1 [v2:172.21.15.3:6810/2008999734,v1:172.21.15.3:6811/2008999734]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:17:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:34 smithi019 ceph-mon[32895]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:17:35.350 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:35.525 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[51969]: 2024-09-13T14:17:35.228+0000 7fe3df678640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T14:17:35.525 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[51969]: 2024-09-13T14:17:35.231+0000 7fe3d6c86640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:17:35.912 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[41592]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:35.912 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[41592]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:17:35.912 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[41592]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:17:35.912 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:35.912 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:35.912 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[29646]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:35.912 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[29646]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:17:35.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[29646]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:17:35.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:35.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:35 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:35.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:35 smithi019 ceph-mon[32895]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:35.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:35 smithi019 ceph-mon[32895]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:17:35.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:35 smithi019 ceph-mon[32895]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:17:35.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:35 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:35 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:37.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[29646]: purged_snaps scrub starts 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[29646]: purged_snaps scrub ok 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[29646]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[29646]: osd.1 [v2:172.21.15.3:6810/2008999734,v1:172.21.15.3:6811/2008999734] boot 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[29646]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[41592]: purged_snaps scrub starts 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[41592]: purged_snaps scrub ok 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[41592]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[41592]: osd.1 [v2:172.21.15.3:6810/2008999734,v1:172.21.15.3:6811/2008999734] boot 2024-09-13T14:17:37.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[41592]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:17:37.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:37 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:37.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:37 smithi019 ceph-mon[32895]: purged_snaps scrub starts 2024-09-13T14:17:37.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:37 smithi019 ceph-mon[32895]: purged_snaps scrub ok 2024-09-13T14:17:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:37 smithi019 ceph-mon[32895]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:17:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:37 smithi019 ceph-mon[32895]: osd.1 [v2:172.21.15.3:6810/2008999734,v1:172.21.15.3:6811/2008999734] boot 2024-09-13T14:17:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:37 smithi019 ceph-mon[32895]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:17:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:37 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:38.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:38 smithi003 ceph-mon[29646]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:17:38.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:38 smithi003 ceph-mon[41592]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:17:38.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:38 smithi019 ceph-mon[32895]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:17:39.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: Detected new or changed devices on smithi003 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi003 to 8034M 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:39.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: Detected new or changed devices on smithi003 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi003 to 8034M 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:39.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:39 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.745 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:17:39.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi003:vg_nvme/lv_2 2024-09-13T14:17:39.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:39.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: Detected new or changed devices on smithi003 2024-09-13T14:17:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi003 to 8034M 2024-09-13T14:17:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:39 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:40.019 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:41.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:41 smithi003 ceph-mon[29646]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:41.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:41 smithi003 ceph-mon[41592]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:41.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:41 smithi019 ceph-mon[32895]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:42.880 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:42.880 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:42.880 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:42.880 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:42.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:42.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:42.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:42.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:42.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:43.823 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:43 smithi003 ceph-mon[41592]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:43.823 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:43 smithi003 ceph-mon[41592]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:43.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:43 smithi003 ceph-mon[29646]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:43.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:43 smithi003 ceph-mon[29646]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:43.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:43 smithi019 ceph-mon[32895]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:43.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:43 smithi019 ceph-mon[32895]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:44.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:44 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3613028045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "676ec1e5-071c-47ba-86ae-2193f448fe8d"}]: dispatch 2024-09-13T14:17:44.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:44 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3613028045' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "676ec1e5-071c-47ba-86ae-2193f448fe8d"}]': finished 2024-09-13T14:17:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:44 smithi003 ceph-mon[29646]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:17:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:44 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:17:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:44 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3613028045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "676ec1e5-071c-47ba-86ae-2193f448fe8d"}]: dispatch 2024-09-13T14:17:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:44 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3613028045' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "676ec1e5-071c-47ba-86ae-2193f448fe8d"}]': finished 2024-09-13T14:17:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:44 smithi003 ceph-mon[41592]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:17:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:44 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:17:44.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:44 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3613028045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "676ec1e5-071c-47ba-86ae-2193f448fe8d"}]: dispatch 2024-09-13T14:17:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:44 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3613028045' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "676ec1e5-071c-47ba-86ae-2193f448fe8d"}]': finished 2024-09-13T14:17:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:44 smithi019 ceph-mon[32895]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:17:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:44 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:17:45.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:45 smithi019 ceph-mon[32895]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:45.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:45 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3358272893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:45.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:45 smithi003 ceph-mon[29646]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:45.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:45 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3358272893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:45.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:45 smithi003 ceph-mon[41592]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:45.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:45 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3358272893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:47.664 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:47 smithi003 ceph-mon[29646]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:47.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:47 smithi003 ceph-mon[41592]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:47.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:47 smithi019 ceph-mon[32895]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:49 smithi003 ceph-mon[29646]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:49 smithi003 ceph-mon[41592]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:49.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:49 smithi019 ceph-mon[32895]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:50.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:50 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:17:50.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:50 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:50.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:50 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:17:50.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:50 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:50.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:50 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:17:50.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:50 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:51.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:51 smithi003 ceph-mon[41592]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:51.703 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:51 smithi003 ceph-mon[41592]: Deploying daemon osd.2 on smithi003 2024-09-13T14:17:51.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:51 smithi003 ceph-mon[29646]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:51.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:51 smithi003 ceph-mon[29646]: Deploying daemon osd.2 on smithi003 2024-09-13T14:17:51.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:51 smithi019 ceph-mon[32895]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:51.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:51 smithi019 ceph-mon[32895]: Deploying daemon osd.2 on smithi003 2024-09-13T14:17:53.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:53 smithi003 ceph-mon[29646]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:53 smithi003 ceph-mon[41592]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:53.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:53 smithi019 ceph-mon[32895]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:55.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:55 smithi003 ceph-mon[41592]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:55.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:55 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:55.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:55 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:55.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:55 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:55.778 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:55 smithi003 ceph-mon[29646]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:55.778 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:55 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:55.778 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:55 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:55.778 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:55 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:55 smithi019 ceph-mon[32895]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:55 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:55 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:55 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:57.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:57 smithi003 ceph-mon[29646]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:57.679 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:57 smithi003 ceph-mon[41592]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:57.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:57 smithi019 ceph-mon[32895]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:58.819 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 2 on host 'smithi003' 2024-09-13T14:17:59.395 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.395 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.395 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='osd.2 [v2:172.21.15.3:6818/2444149284,v1:172.21.15.3:6819/2444149284]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:59.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:59.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:59.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='osd.2 [v2:172.21.15.3:6818/2444149284,v1:172.21.15.3:6819/2444149284]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:17:59.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:17:59.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:59.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:17:59 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.420 DEBUG:teuthology.orchestra.run.smithi003:osd.2> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.2.service 2024-09-13T14:17:59.422 INFO:tasks.cephadm:Deploying osd.3 on smithi003 with /dev/vg_nvme/lv_1... 2024-09-13T14:17:59.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T14:17:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:17:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='osd.2 [v2:172.21.15.3:6818/2444149284,v1:172.21.15.3:6819/2444149284]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:17:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:17:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:17:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:17:59 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[41592]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:18:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[41592]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:18:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:00.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[41592]: from='osd.2 [v2:172.21.15.3:6818/2444149284,v1:172.21.15.3:6819/2444149284]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:18:00.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[41592]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:18:00.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[29646]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:18:00.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[29646]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:18:00.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:00.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[29646]: from='osd.2 [v2:172.21.15.3:6818/2444149284,v1:172.21.15.3:6819/2444149284]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:18:00.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-mon[29646]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:18:00.185 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[58420]: 2024-09-13T14:18:00.124+0000 7fbb7d79c640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T14:18:00.186 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:00 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[58420]: 2024-09-13T14:18:00.127+0000 7fbb77dd2640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:18:00.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:00 smithi019 ceph-mon[32895]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:18:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:00 smithi019 ceph-mon[32895]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:18:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:00 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:00 smithi019 ceph-mon[32895]: from='osd.2 [v2:172.21.15.3:6818/2444149284,v1:172.21.15.3:6819/2444149284]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:18:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:00 smithi019 ceph-mon[32895]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:18:00.538 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:01.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[29646]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:18:01.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[29646]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:18:01.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[29646]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:18:01.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:01.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:01.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:01.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[41592]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:18:01.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[41592]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:18:01.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[41592]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:18:01.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:01.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:01.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:01.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:01 smithi019 ceph-mon[32895]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:18:01.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:01 smithi019 ceph-mon[32895]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:18:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:01 smithi019 ceph-mon[32895]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:18:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:02.333 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[41592]: purged_snaps scrub starts 2024-09-13T14:18:02.333 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[41592]: purged_snaps scrub ok 2024-09-13T14:18:02.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[41592]: osd.2 [v2:172.21.15.3:6818/2444149284,v1:172.21.15.3:6819/2444149284] boot 2024-09-13T14:18:02.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[41592]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:18:02.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:02.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[29646]: purged_snaps scrub starts 2024-09-13T14:18:02.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[29646]: purged_snaps scrub ok 2024-09-13T14:18:02.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[29646]: osd.2 [v2:172.21.15.3:6818/2444149284,v1:172.21.15.3:6819/2444149284] boot 2024-09-13T14:18:02.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[29646]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:18:02.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:02 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:02.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:02 smithi019 ceph-mon[32895]: purged_snaps scrub starts 2024-09-13T14:18:02.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:02 smithi019 ceph-mon[32895]: purged_snaps scrub ok 2024-09-13T14:18:02.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:02 smithi019 ceph-mon[32895]: osd.2 [v2:172.21.15.3:6818/2444149284,v1:172.21.15.3:6819/2444149284] boot 2024-09-13T14:18:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:02 smithi019 ceph-mon[32895]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:18:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:02 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:18:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:18:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-13T14:18:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:03.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:03.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:03 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:18:03.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:18:03.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-13T14:18:03.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:03.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:03.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:03.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:03.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:03.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:18:03.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:18:03.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-13T14:18:03.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:03.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:03.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:03.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:03.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:03.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:03 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:03.880 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:18:03.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi003:vg_nvme/lv_1 2024-09-13T14:18:04.171 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:04.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[41592]: Detected new or changed devices on smithi003 2024-09-13T14:18:04.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi003 to 5356M 2024-09-13T14:18:04.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-13T14:18:04.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[41592]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:18:04.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:18:04.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[29646]: Detected new or changed devices on smithi003 2024-09-13T14:18:04.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi003 to 5356M 2024-09-13T14:18:04.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-13T14:18:04.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[29646]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:18:04.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:04 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:18:04.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:04 smithi019 ceph-mon[32895]: Detected new or changed devices on smithi003 2024-09-13T14:18:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:04 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi003 to 5356M 2024-09-13T14:18:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:04 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-13T14:18:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:04 smithi019 ceph-mon[32895]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:18:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:04 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:18:05.149 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:18:04 smithi003 sudo[62333]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:18:05.149 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:18:04 smithi003 sudo[62333]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:05.149 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:18:04 smithi003 sudo[62333]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:05.149 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:18:04 smithi003 sudo[62333]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:05 smithi019 ceph-mon[32895]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:05 smithi019 ceph-mon[32895]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:18:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:05 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:18:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:05 smithi019 ceph-mon[32895]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:18:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:05 smithi003 ceph-mon[41592]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:05 smithi003 ceph-mon[41592]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:18:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:05 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:18:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:05 smithi003 ceph-mon[41592]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:18:05.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:18:05 smithi003 sudo[62348]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:18:05.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:18:05 smithi003 sudo[62348]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:05.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:18:05 smithi003 sudo[62348]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:05.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:18:05 smithi003 sudo[62348]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:05 smithi003 ceph-mon[29646]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:05 smithi003 ceph-mon[29646]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:18:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:05 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:18:05.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:05 smithi003 ceph-mon[29646]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:18:06.156 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:18:05 smithi003 sudo[62397]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:18:06.156 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:18:05 smithi003 sudo[62397]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:06.156 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:18:05 smithi003 sudo[62397]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:06.156 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:18:05 smithi003 sudo[62397]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:06.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[41592]: osdmap e22: 3 total, 3 up, 3 in 2024-09-13T14:18:06.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:06.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:18:06.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:18:06.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:06.407 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:18:06 smithi003 sudo[62510]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:18:06.407 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:18:06 smithi003 sudo[62510]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:06.408 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:18:06 smithi003 sudo[62510]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:06.408 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:18:06 smithi003 sudo[62510]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:06.408 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[29646]: osdmap e22: 3 total, 3 up, 3 in 2024-09-13T14:18:06.408 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:06.408 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:18:06.408 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:18:06.408 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:06 smithi019 ceph-mon[32895]: osdmap e22: 3 total, 3 up, 3 in 2024-09-13T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:06 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:06 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:18:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:06 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:18:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:06 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:06.932 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:06 smithi003 sudo[62578]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:18:06.932 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:06 smithi003 sudo[62578]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:06.932 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:06 smithi003 sudo[62578]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:06.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:06 smithi003 sudo[62578]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:07.422 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[41592]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:07.423 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[41592]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:07.423 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[41592]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:18:07.423 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[41592]: Cluster is now healthy 2024-09-13T14:18:07.423 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[41592]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:18:07.423 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[29646]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:07.423 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[29646]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:07.423 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[29646]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:18:07.423 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[29646]: Cluster is now healthy 2024-09-13T14:18:07.423 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:07 smithi003 ceph-mon[29646]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:18:07.424 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:07 smithi003 sudo[62589]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:18:07.424 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:07 smithi003 sudo[62589]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:07.424 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:07 smithi003 sudo[62589]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:07.424 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:18:07 smithi003 sudo[62589]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:07.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:07 smithi019 ceph-mon[32895]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:07.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:07 smithi019 ceph-mon[32895]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:07.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:07 smithi019 ceph-mon[32895]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:18:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:07 smithi019 ceph-mon[32895]: Cluster is now healthy 2024-09-13T14:18:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:07 smithi019 ceph-mon[32895]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:18:07.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:07 smithi003 sudo[62613]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:18:07.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:07 smithi003 sudo[62613]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:07.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:07 smithi003 sudo[62613]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:07.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:07 smithi003 sudo[62613]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:08.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 sudo[62644]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:18:08.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 sudo[62644]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:08.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 sudo[62644]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:08.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 sudo[62644]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 sudo[35635]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:18:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 sudo[35635]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 sudo[35635]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 sudo[35635]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:09.235 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[41592]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:18:09.235 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[41592]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:18:09.235 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:09.235 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[29646]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[29646]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:09.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:09.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:08 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 sudo[35638]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 sudo[35638]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 sudo[35638]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 sudo[35638]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 ceph-mon[32895]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 ceph-mon[32895]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:08 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:09.581 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 sudo[62720]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:18:09.581 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 sudo[62720]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:09.581 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 sudo[62720]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:09.581 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 sudo[62720]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:10.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:18:10.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:10.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1978059049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ec0ad16-0df3-4319-b475-72cfaaa5edae"}]: dispatch 2024-09-13T14:18:10.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:18:10.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ec0ad16-0df3-4319-b475-72cfaaa5edae"}]: dispatch 2024-09-13T14:18:10.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ec0ad16-0df3-4319-b475-72cfaaa5edae"}]': finished 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2412967611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[29646]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:18:10.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 sudo[62770]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 sudo[62770]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 sudo[62770]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 sudo[62770]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1978059049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ec0ad16-0df3-4319-b475-72cfaaa5edae"}]: dispatch 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ec0ad16-0df3-4319-b475-72cfaaa5edae"}]: dispatch 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ec0ad16-0df3-4319-b475-72cfaaa5edae"}]': finished 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:18:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2412967611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:18:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:09 smithi003 ceph-mon[41592]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:18:10.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1978059049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ec0ad16-0df3-4319-b475-72cfaaa5edae"}]: dispatch 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ec0ad16-0df3-4319-b475-72cfaaa5edae"}]: dispatch 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ec0ad16-0df3-4319-b475-72cfaaa5edae"}]': finished 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:10.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:10.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:10.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2412967611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:18:10.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:09 smithi019 ceph-mon[32895]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:18:11.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:10 smithi019 ceph-mon[32895]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:11.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:10 smithi003 ceph-mon[29646]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:10 smithi003 ceph-mon[41592]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:13.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:13 smithi019 ceph-mon[32895]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-09-13T14:18:13.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:13 smithi003 ceph-mon[29646]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-09-13T14:18:13.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:13 smithi003 ceph-mon[41592]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-09-13T14:18:15.346 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:15 smithi003 ceph-mon[41592]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-13T14:18:15.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:15 smithi003 ceph-mon[29646]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-13T14:18:15.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:15 smithi019 ceph-mon[32895]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-13T14:18:16.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:16 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:18:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:16 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:16 smithi019 ceph-mon[32895]: Deploying daemon osd.3 on smithi003 2024-09-13T14:18:16.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:16 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:18:16.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:16 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:16.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:16 smithi003 ceph-mon[29646]: Deploying daemon osd.3 on smithi003 2024-09-13T14:18:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:16 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:18:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:16 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:16 smithi003 ceph-mon[41592]: Deploying daemon osd.3 on smithi003 2024-09-13T14:18:17.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:17 smithi019 ceph-mon[32895]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-13T14:18:17.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:17 smithi003 ceph-mon[29646]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-13T14:18:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:17 smithi003 ceph-mon[41592]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-13T14:18:19.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:19 smithi019 ceph-mon[32895]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-13T14:18:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:19 smithi003 ceph-mon[29646]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-13T14:18:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:19 smithi003 ceph-mon[41592]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-13T14:18:21.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:21 smithi019 ceph-mon[32895]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-09-13T14:18:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:21 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:21 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:21 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:21.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:21 smithi003 ceph-mon[29646]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-09-13T14:18:21.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:21 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:21 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:21 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:21 smithi003 ceph-mon[41592]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-09-13T14:18:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:21 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:21 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:21 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:23.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:23 smithi019 ceph-mon[32895]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-13T14:18:23.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:23 smithi003 ceph-mon[29646]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-13T14:18:23.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:23 smithi003 ceph-mon[41592]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-13T14:18:23.730 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 3 on host 'smithi003' 2024-09-13T14:18:24.291 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:24 smithi003 ceph-mon[41592]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:18:24.292 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:24 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:24.292 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:24 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:24.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:24 smithi003 ceph-mon[29646]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:18:24.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:24 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:24.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:24 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:24.294 DEBUG:teuthology.orchestra.run.smithi003:osd.3> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.3.service 2024-09-13T14:18:24.296 INFO:tasks.cephadm:Deploying osd.4 on smithi019 with /dev/vg_nvme/lv_4... 2024-09-13T14:18:24.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T14:18:24.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:24 smithi019 ceph-mon[32895]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:18:24.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:24 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:24.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:24 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:24.558 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:18:25.353 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:25 smithi019 ceph-mon[32895]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:25.353 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:25 smithi019 ceph-mon[32895]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:18:25.353 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:25 smithi019 ceph-mon[32895]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:18:25.353 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:25 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:25.353 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:25 smithi019 ceph-mon[32895]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:18:25.353 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:25 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:25.353 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:25 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:25.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[29646]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:25.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[29646]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:18:25.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[29646]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:18:25.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:25.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[29646]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:18:25.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:25.428 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:25.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[41592]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:25.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[41592]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:18:25.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[41592]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:18:25.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:25.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[41592]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-09-13T14:18:25.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:25.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:25.428 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[64478]: 2024-09-13T14:18:25.196+0000 7f90f28d7640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T14:18:25.429 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:18:25 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[64478]: 2024-09-13T14:18:25.204+0000 7f90ed70e640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:18:26.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:26 smithi019 ceph-mon[32895]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:18:26.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:26 smithi019 ceph-mon[32895]: osdmap e25: 4 total, 3 up, 4 in 2024-09-13T14:18:26.416 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:26 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:26.416 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:26 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:26.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:26 smithi003 ceph-mon[29646]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:18:26.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:26 smithi003 ceph-mon[29646]: osdmap e25: 4 total, 3 up, 4 in 2024-09-13T14:18:26.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:26 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:26.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:26 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:26 smithi003 ceph-mon[41592]: from='osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-09-13T14:18:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:26 smithi003 ceph-mon[41592]: osdmap e25: 4 total, 3 up, 4 in 2024-09-13T14:18:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:26 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:26 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:27.082 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:18:27.102 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi019:vg_nvme/lv_4 2024-09-13T14:18:27.352 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:18:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:27 smithi019 ceph-mon[32895]: purged_snaps scrub starts 2024-09-13T14:18:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:27 smithi019 ceph-mon[32895]: purged_snaps scrub ok 2024-09-13T14:18:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:27 smithi019 ceph-mon[32895]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:27 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:27 smithi019 ceph-mon[32895]: osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501] boot 2024-09-13T14:18:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:27 smithi019 ceph-mon[32895]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:18:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:27 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:27.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[29646]: purged_snaps scrub starts 2024-09-13T14:18:27.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[29646]: purged_snaps scrub ok 2024-09-13T14:18:27.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[29646]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:27.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:27.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[29646]: osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501] boot 2024-09-13T14:18:27.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[29646]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:18:27.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:27.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[41592]: purged_snaps scrub starts 2024-09-13T14:18:27.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[41592]: purged_snaps scrub ok 2024-09-13T14:18:27.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[41592]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:18:27.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:27.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[41592]: osd.3 [v2:172.21.15.3:6826/2595190501,v1:172.21.15.3:6827/2595190501] boot 2024-09-13T14:18:27.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[41592]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:18:27.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:27 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: osdmap e27: 4 total, 4 up, 4 in 2024-09-13T14:18:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:28.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:28.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:28.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:28.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:28.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: osdmap e27: 4 total, 4 up, 4 in 2024-09-13T14:18:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: osdmap e27: 4 total, 4 up, 4 in 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:28.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:29.573 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:29 smithi019 ceph-mon[32895]: Detected new or changed devices on smithi003 2024-09-13T14:18:29.573 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:29 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi003 to 4017M 2024-09-13T14:18:29.573 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:29 smithi019 ceph-mon[32895]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:29.573 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:29 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:18:29.573 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:29 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:18:29.573 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:29 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:29.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[29646]: Detected new or changed devices on smithi003 2024-09-13T14:18:29.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi003 to 4017M 2024-09-13T14:18:29.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[29646]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:29.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:18:29.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:18:29.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:29.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[41592]: Detected new or changed devices on smithi003 2024-09-13T14:18:29.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi003 to 4017M 2024-09-13T14:18:29.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[41592]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:29.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:18:29.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:18:29.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:29 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:30.409 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:30 smithi019 ceph-mon[32895]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:30.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:30 smithi003 ceph-mon[29646]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:30.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:30 smithi003 ceph-mon[41592]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:31 smithi019 ceph-mon[32895]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:31.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:31 smithi003 ceph-mon[29646]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:31.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:31 smithi003 ceph-mon[41592]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:32.584 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/3598770241' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ef54d09-4000-4695-942b-1b1cf9659983"}]: dispatch 2024-09-13T14:18:32.584 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ef54d09-4000-4695-942b-1b1cf9659983"}]: dispatch 2024-09-13T14:18:32.584 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ef54d09-4000-4695-942b-1b1cf9659983"}]': finished 2024-09-13T14:18:32.585 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[41592]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:18:32.585 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:32.585 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/3719054149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:18:32.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/3598770241' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ef54d09-4000-4695-942b-1b1cf9659983"}]: dispatch 2024-09-13T14:18:32.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ef54d09-4000-4695-942b-1b1cf9659983"}]: dispatch 2024-09-13T14:18:32.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ef54d09-4000-4695-942b-1b1cf9659983"}]': finished 2024-09-13T14:18:32.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[29646]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:18:32.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:32.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:32 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/3719054149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:18:32.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:32 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/3598770241' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ef54d09-4000-4695-942b-1b1cf9659983"}]: dispatch 2024-09-13T14:18:32.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:32 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ef54d09-4000-4695-942b-1b1cf9659983"}]: dispatch 2024-09-13T14:18:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:32 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ef54d09-4000-4695-942b-1b1cf9659983"}]': finished 2024-09-13T14:18:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:32 smithi019 ceph-mon[32895]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:18:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:32 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:32 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/3719054149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:18:33.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:33 smithi019 ceph-mon[32895]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:33.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:33 smithi003 ceph-mon[41592]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:33.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:33 smithi003 ceph-mon[29646]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:35 smithi019 ceph-mon[32895]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:35.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:35 smithi003 ceph-mon[29646]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:35.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:35 smithi003 ceph-mon[41592]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:37.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:37 smithi019 ceph-mon[32895]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:37 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:18:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:37 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:37.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:37 smithi003 ceph-mon[29646]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:37.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:37 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:18:37.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:37 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:37.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:37 smithi003 ceph-mon[41592]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:37 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:18:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:37 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:38 smithi019 ceph-mon[32895]: Deploying daemon osd.4 on smithi019 2024-09-13T14:18:38.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:38 smithi003 ceph-mon[29646]: Deploying daemon osd.4 on smithi019 2024-09-13T14:18:38.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:38 smithi003 ceph-mon[41592]: Deploying daemon osd.4 on smithi019 2024-09-13T14:18:39.451 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:39 smithi019 ceph-mon[32895]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:39.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:39 smithi003 ceph-mon[29646]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:39 smithi003 ceph-mon[41592]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:41 smithi019 ceph-mon[32895]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:41.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:41 smithi003 ceph-mon[41592]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:41.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:41 smithi003 ceph-mon[29646]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:42 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:42.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:42.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:42.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:42 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:42 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:43.670 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:43 smithi019 ceph-mon[32895]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:43.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:43 smithi003 ceph-mon[29646]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:43.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:43 smithi003 ceph-mon[41592]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:44 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:44 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:44 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:44 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:44 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:44 smithi019 ceph-mon[32895]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:45.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[29646]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:45.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:44 smithi003 ceph-mon[41592]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:45.576 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 4 on host 'smithi019' 2024-09-13T14:18:46.158 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:45 smithi019 ceph-mon[32895]: from='osd.4 [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:18:46.158 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:45 smithi019 ceph-mon[32895]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:18:46.158 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:45 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:46.158 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:45 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:46.158 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:45 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:46.182 DEBUG:teuthology.orchestra.run.smithi019:osd.4> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.4.service 2024-09-13T14:18:46.185 INFO:tasks.cephadm:Deploying osd.5 on smithi019 with /dev/vg_nvme/lv_3... 2024-09-13T14:18:46.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T14:18:46.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[29646]: from='osd.4 [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:18:46.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[29646]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:18:46.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:46.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:46.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[41592]: from='osd.4 [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:18:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[41592]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:18:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:45 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:46.942 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:18:47.064 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:46 smithi019 ceph-mon[32895]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:18:47.064 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:46 smithi019 ceph-mon[32895]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:18:47.064 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:46 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:47.064 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:46 smithi019 ceph-mon[32895]: from='osd.4 [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:18:47.064 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:46 smithi019 ceph-mon[32895]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:18:47.064 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:46 smithi019 ceph-mon[32895]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:47.065 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 14:18:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T14:18:46.986+0000 7f0c7a25c640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T14:18:47.065 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 14:18:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T14:18:46.990+0000 7f0c75093640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:18:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[41592]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:18:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[41592]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:18:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[41592]: from='osd.4 [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:18:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[41592]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:18:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[41592]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[29646]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:18:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[29646]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:18:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[29646]: from='osd.4 [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:18:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[29646]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:18:47.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:46 smithi003 ceph-mon[29646]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:48.306 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:47 smithi019 ceph-mon[32895]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:18:48.306 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:47 smithi019 ceph-mon[32895]: osdmap e30: 5 total, 4 up, 5 in 2024-09-13T14:18:48.306 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:47 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:48.306 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:47 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:48.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:47 smithi003 ceph-mon[29646]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:18:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:47 smithi003 ceph-mon[29646]: osdmap e30: 5 total, 4 up, 5 in 2024-09-13T14:18:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:47 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:47 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:47 smithi003 ceph-mon[41592]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:18:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:47 smithi003 ceph-mon[41592]: osdmap e30: 5 total, 4 up, 5 in 2024-09-13T14:18:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:47 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:47 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:49.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:49 smithi019 ceph-mon[32895]: purged_snaps scrub starts 2024-09-13T14:18:49.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:49 smithi019 ceph-mon[32895]: purged_snaps scrub ok 2024-09-13T14:18:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:49 smithi019 ceph-mon[32895]: osd.4 [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131] boot 2024-09-13T14:18:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:49 smithi019 ceph-mon[32895]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:18:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:49 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:49 smithi019 ceph-mon[32895]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:49.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[29646]: purged_snaps scrub starts 2024-09-13T14:18:49.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[29646]: purged_snaps scrub ok 2024-09-13T14:18:49.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[29646]: osd.4 [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131] boot 2024-09-13T14:18:49.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[29646]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:18:49.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:49.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[29646]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:49.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[41592]: purged_snaps scrub starts 2024-09-13T14:18:49.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[41592]: purged_snaps scrub ok 2024-09-13T14:18:49.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[41592]: osd.4 [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131] boot 2024-09-13T14:18:49.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[41592]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:18:49.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:49.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:49 smithi003 ceph-mon[41592]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:18:50.414 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:50 smithi019 ceph-mon[32895]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:18:50.414 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:50 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.414 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:50 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:50 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:50.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:50 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:50 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:50.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:50 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:50.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:50 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[29646]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:18:50.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[41592]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:50.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:50.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:50.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:50.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:50 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:18:51.166 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:51 smithi019 ceph-mon[32895]: Detected new or changed devices on smithi019 2024-09-13T14:18:51.167 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:51 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi019 to 17092M 2024-09-13T14:18:51.167 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:51 smithi019 ceph-mon[32895]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T14:18:51.167 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:51 smithi019 ceph-mon[32895]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:18:51.173 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:18:51.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi019:vg_nvme/lv_3 2024-09-13T14:18:51.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:51 smithi003 ceph-mon[29646]: Detected new or changed devices on smithi019 2024-09-13T14:18:51.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:51 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi019 to 17092M 2024-09-13T14:18:51.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:51 smithi003 ceph-mon[29646]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T14:18:51.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:51 smithi003 ceph-mon[29646]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:18:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:51 smithi003 ceph-mon[41592]: Detected new or changed devices on smithi019 2024-09-13T14:18:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:51 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi019 to 17092M 2024-09-13T14:18:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:51 smithi003 ceph-mon[41592]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T14:18:51.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:51 smithi003 ceph-mon[41592]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:18:51.446 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:18:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:53 smithi019 ceph-mon[32895]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-09-13T14:18:53.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:53 smithi003 ceph-mon[29646]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-09-13T14:18:53.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:53 smithi003 ceph-mon[41592]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-09-13T14:18:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:54 smithi019 ceph-mon[32895]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:18:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:18:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:54.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:54 smithi003 ceph-mon[41592]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:54.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:18:54.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:18:54.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:54.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:54 smithi003 ceph-mon[29646]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:54.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:18:54.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:18:54.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:55.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:55 smithi019 ceph-mon[32895]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-13T14:18:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:55 smithi003 ceph-mon[29646]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-13T14:18:55.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:55 smithi003 ceph-mon[41592]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-13T14:18:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:56 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/3383674606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6dfd8fce-4d3d-4a8b-90ac-556447045378"}]: dispatch 2024-09-13T14:18:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:56 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6dfd8fce-4d3d-4a8b-90ac-556447045378"}]: dispatch 2024-09-13T14:18:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:56 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6dfd8fce-4d3d-4a8b-90ac-556447045378"}]': finished 2024-09-13T14:18:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:56 smithi019 ceph-mon[32895]: osdmap e34: 6 total, 5 up, 6 in 2024-09-13T14:18:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:56 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:18:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:56 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/2342908565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:18:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/3383674606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6dfd8fce-4d3d-4a8b-90ac-556447045378"}]: dispatch 2024-09-13T14:18:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6dfd8fce-4d3d-4a8b-90ac-556447045378"}]: dispatch 2024-09-13T14:18:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6dfd8fce-4d3d-4a8b-90ac-556447045378"}]': finished 2024-09-13T14:18:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[29646]: osdmap e34: 6 total, 5 up, 6 in 2024-09-13T14:18:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:18:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/2342908565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:18:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/3383674606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6dfd8fce-4d3d-4a8b-90ac-556447045378"}]: dispatch 2024-09-13T14:18:56.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6dfd8fce-4d3d-4a8b-90ac-556447045378"}]: dispatch 2024-09-13T14:18:56.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6dfd8fce-4d3d-4a8b-90ac-556447045378"}]': finished 2024-09-13T14:18:56.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[41592]: osdmap e34: 6 total, 5 up, 6 in 2024-09-13T14:18:56.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:18:56.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:56 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/2342908565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:18:57.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:57 smithi019 ceph-mon[32895]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-13T14:18:57.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:57 smithi003 ceph-mon[41592]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-13T14:18:57.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:57 smithi003 ceph-mon[29646]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-13T14:18:59.674 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:18:59 smithi019 ceph-mon[32895]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-13T14:18:59.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:18:59 smithi003 ceph-mon[29646]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-13T14:18:59.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:18:59 smithi003 ceph-mon[41592]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-13T14:19:01.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:01 smithi019 ceph-mon[32895]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:19:01.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:19:01.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:01.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:01 smithi003 ceph-mon[29646]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:19:01.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:19:01.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:01.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:01 smithi003 ceph-mon[41592]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:19:01.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:19:01.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:02.697 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:02 smithi019 ceph-mon[32895]: Deploying daemon osd.5 on smithi019 2024-09-13T14:19:02.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:02 smithi003 ceph-mon[29646]: Deploying daemon osd.5 on smithi019 2024-09-13T14:19:02.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:02 smithi003 ceph-mon[41592]: Deploying daemon osd.5 on smithi019 2024-09-13T14:19:03.853 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:03 smithi019 ceph-mon[32895]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:03.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:03 smithi003 ceph-mon[29646]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:03.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:03 smithi003 ceph-mon[41592]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:05 smithi019 ceph-mon[32895]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:05.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:05 smithi003 ceph-mon[41592]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:05 smithi003 ceph-mon[29646]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:06.722 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:06 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:06.722 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:06 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:06.722 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:06 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:06 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:06 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:06 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:06 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:07.870 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:07 smithi019 ceph-mon[32895]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:07.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:07 smithi003 ceph-mon[29646]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:07.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:07 smithi003 ceph-mon[41592]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:09.457 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 5 on host 'smithi019' 2024-09-13T14:19:09.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:09.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='osd.5 [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:19:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:19:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:09 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='osd.5 [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='osd.5 [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:09.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:09 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:10.033 DEBUG:teuthology.orchestra.run.smithi019:osd.5> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.5.service 2024-09-13T14:19:10.035 INFO:tasks.cephadm:Deploying osd.6 on smithi019 with /dev/vg_nvme/lv_2... 2024-09-13T14:19:10.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T14:19:10.803 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:19:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:10 smithi019 ceph-mon[32895]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:19:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:10 smithi019 ceph-mon[32895]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:19:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:10 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:10 smithi019 ceph-mon[32895]: from='osd.5 [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:10 smithi019 ceph-mon[32895]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:10 smithi019 ceph-mon[32895]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:11.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 14:19:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T14:19:10.796+0000 7f51a3578640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T14:19:11.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 14:19:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T14:19:10.804+0000 7f519c3ab640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:19:11.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[41592]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[41592]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[41592]: from='osd.5 [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[41592]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[41592]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[29646]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[29646]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[29646]: from='osd.5 [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[29646]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:11.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:10 smithi003 ceph-mon[29646]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:19:12.035 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:11 smithi019 ceph-mon[32895]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:19:12.035 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:11 smithi019 ceph-mon[32895]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T14:19:12.035 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:11 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:12.035 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:11 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:12.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:11 smithi003 ceph-mon[41592]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:19:12.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:11 smithi003 ceph-mon[41592]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T14:19:12.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:11 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:12.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:11 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:12.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:11 smithi003 ceph-mon[29646]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:19:12.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:11 smithi003 ceph-mon[29646]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T14:19:12.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:11 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:12.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:11 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:13.108 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:12 smithi019 ceph-mon[32895]: purged_snaps scrub starts 2024-09-13T14:19:13.109 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:12 smithi019 ceph-mon[32895]: purged_snaps scrub ok 2024-09-13T14:19:13.109 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:12 smithi019 ceph-mon[32895]: osd.5 [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104] boot 2024-09-13T14:19:13.109 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:12 smithi019 ceph-mon[32895]: osdmap e37: 6 total, 6 up, 6 in 2024-09-13T14:19:13.109 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:12 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:13.109 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:12 smithi019 ceph-mon[32895]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[29646]: purged_snaps scrub starts 2024-09-13T14:19:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[29646]: purged_snaps scrub ok 2024-09-13T14:19:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[29646]: osd.5 [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104] boot 2024-09-13T14:19:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[29646]: osdmap e37: 6 total, 6 up, 6 in 2024-09-13T14:19:13.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:13.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[29646]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[41592]: purged_snaps scrub starts 2024-09-13T14:19:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[41592]: purged_snaps scrub ok 2024-09-13T14:19:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[41592]: osd.5 [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104] boot 2024-09-13T14:19:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[41592]: osdmap e37: 6 total, 6 up, 6 in 2024-09-13T14:19:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:19:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:12 smithi003 ceph-mon[41592]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:19:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: Detected new or changed devices on smithi019 2024-09-13T14:19:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:13.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:13.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:13.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:13.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi019 to 8546M 2024-09-13T14:19:13.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:13.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:13.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:13.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:14.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:19:14.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: Detected new or changed devices on smithi019 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi019 to 8546M 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: Detected new or changed devices on smithi019 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:14.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:14.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:14.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:14.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi019 to 8546M 2024-09-13T14:19:14.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:14.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:14.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:14.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:14.657 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:19:14.678 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi019:vg_nvme/lv_2 2024-09-13T14:19:14.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:14 smithi019 ceph-mon[32895]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T14:19:14.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:14 smithi019 ceph-mon[32895]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:14.933 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:19:15.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:14 smithi003 ceph-mon[29646]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T14:19:15.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:14 smithi003 ceph-mon[29646]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:15.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:14 smithi003 ceph-mon[41592]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T14:19:15.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:14 smithi003 ceph-mon[41592]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:17.279 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:17 smithi019 ceph-mon[32895]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:17.279 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:19:17.279 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:19:17.279 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:17 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:17.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:17 smithi003 ceph-mon[29646]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:17.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:19:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:19:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:17 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:17 smithi003 ceph-mon[41592]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:19:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:19:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:17 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:18.351 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:18 smithi019 ceph-mon[32895]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:18.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:18 smithi003 ceph-mon[29646]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:18.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:18 smithi003 ceph-mon[41592]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:19.396 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:19 smithi019 ceph-mon[32895]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:19.396 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:19 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/3937713201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17e27318-c0ce-413d-b660-d756731ea743"}]: dispatch 2024-09-13T14:19:19.396 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:19 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17e27318-c0ce-413d-b660-d756731ea743"}]: dispatch 2024-09-13T14:19:19.396 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:19 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17e27318-c0ce-413d-b660-d756731ea743"}]': finished 2024-09-13T14:19:19.396 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:19 smithi019 ceph-mon[32895]: osdmap e40: 7 total, 6 up, 7 in 2024-09-13T14:19:19.396 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:19 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:19.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[29646]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:19.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/3937713201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17e27318-c0ce-413d-b660-d756731ea743"}]: dispatch 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17e27318-c0ce-413d-b660-d756731ea743"}]: dispatch 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17e27318-c0ce-413d-b660-d756731ea743"}]': finished 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[29646]: osdmap e40: 7 total, 6 up, 7 in 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[41592]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/3937713201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17e27318-c0ce-413d-b660-d756731ea743"}]: dispatch 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17e27318-c0ce-413d-b660-d756731ea743"}]: dispatch 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17e27318-c0ce-413d-b660-d756731ea743"}]': finished 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[41592]: osdmap e40: 7 total, 6 up, 7 in 2024-09-13T14:19:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:19 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:20.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:20 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/1180488903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:19:20.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:20 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/1180488903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:19:20.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:20 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/1180488903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:19:21.541 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:21 smithi019 ceph-mon[32895]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:21.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:21 smithi003 ceph-mon[29646]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:21.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:21 smithi003 ceph-mon[41592]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:23.629 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:23 smithi019 ceph-mon[32895]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:23.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:23 smithi003 ceph-mon[29646]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:23.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:23 smithi003 ceph-mon[41592]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:25 smithi019 ceph-mon[32895]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:25 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:19:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:25 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:25.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:25 smithi003 ceph-mon[29646]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:25.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:25 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:19:25.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:25 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:25 smithi003 ceph-mon[41592]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:25 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:19:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:25 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:26.557 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:26 smithi019 ceph-mon[32895]: Deploying daemon osd.6 on smithi019 2024-09-13T14:19:26.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:26 smithi003 ceph-mon[29646]: Deploying daemon osd.6 on smithi019 2024-09-13T14:19:26.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:26 smithi003 ceph-mon[41592]: Deploying daemon osd.6 on smithi019 2024-09-13T14:19:27.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:27 smithi019 ceph-mon[32895]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:27.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:27 smithi003 ceph-mon[41592]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:27.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:27 smithi003 ceph-mon[29646]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:29.571 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:29 smithi019 ceph-mon[32895]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:29.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:29 smithi003 ceph-mon[41592]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:29.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:29 smithi003 ceph-mon[29646]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:30.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:30 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:30 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:30 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:30.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:30 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:30.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:30 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:30.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:30 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:30.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:30 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:30.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:30 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:30.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:30 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:31.506 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:31 smithi019 ceph-mon[32895]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:31.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:31 smithi003 ceph-mon[29646]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:31.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:31 smithi003 ceph-mon[41592]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:33.272 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:33 smithi019 ceph-mon[32895]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:33.298 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 6 on host 'smithi019' 2024-09-13T14:19:33.484 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:33 smithi003 ceph-mon[29646]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:33.485 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:33 smithi003 ceph-mon[41592]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:33.869 DEBUG:teuthology.orchestra.run.smithi019:osd.6> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.6.service 2024-09-13T14:19:33.872 INFO:tasks.cephadm:Deploying osd.7 on smithi019 with /dev/vg_nvme/lv_1... 2024-09-13T14:19:33.872 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T14:19:34.179 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:19:34.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:34 smithi019 ceph-mon[32895]: from='osd.6 [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:19:34.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:34 smithi019 ceph-mon[32895]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:19:34.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:34 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:34 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:34 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:34 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[29646]: from='osd.6 [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[29646]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[41592]: from='osd.6 [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[41592]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:34 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:35.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:35 smithi019 ceph-mon[32895]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:35.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:35 smithi019 ceph-mon[32895]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:19:35.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:35 smithi019 ceph-mon[32895]: osdmap e41: 7 total, 6 up, 7 in 2024-09-13T14:19:35.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:35 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:35.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:35 smithi019 ceph-mon[32895]: from='osd.6 [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:35.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:35 smithi019 ceph-mon[32895]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:35.348 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 14:19:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T14:19:35.231+0000 7fd955499640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T14:19:35.349 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 14:19:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T14:19:35.237+0000 7fd94e2cc640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:19:35.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[29646]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:35.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[29646]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[29646]: osdmap e41: 7 total, 6 up, 7 in 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[29646]: from='osd.6 [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[29646]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[41592]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[41592]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[41592]: osdmap e41: 7 total, 6 up, 7 in 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[41592]: from='osd.6 [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:35 smithi003 ceph-mon[41592]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:36.516 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:36 smithi019 ceph-mon[32895]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:19:36.516 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:36 smithi019 ceph-mon[32895]: osdmap e42: 7 total, 6 up, 7 in 2024-09-13T14:19:36.516 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:36 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:36.516 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:36 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:36.516 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:36 smithi019 ceph-mon[32895]: osdmap e43: 7 total, 6 up, 7 in 2024-09-13T14:19:36.517 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:36 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:36.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[41592]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[41592]: osdmap e42: 7 total, 6 up, 7 in 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[41592]: osdmap e43: 7 total, 6 up, 7 in 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[29646]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[29646]: osdmap e42: 7 total, 6 up, 7 in 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[29646]: osdmap e43: 7 total, 6 up, 7 in 2024-09-13T14:19:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:36 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-13T14:19:37.089 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi019:vg_nvme/lv_1 2024-09-13T14:19:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:37 smithi019 ceph-mon[32895]: purged_snaps scrub starts 2024-09-13T14:19:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:37 smithi019 ceph-mon[32895]: purged_snaps scrub ok 2024-09-13T14:19:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:37 smithi019 ceph-mon[32895]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:37 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:37 smithi019 ceph-mon[32895]: osd.6 [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215] boot 2024-09-13T14:19:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:37 smithi019 ceph-mon[32895]: osdmap e44: 7 total, 7 up, 7 in 2024-09-13T14:19:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:37 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:37.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[29646]: purged_snaps scrub starts 2024-09-13T14:19:37.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[29646]: purged_snaps scrub ok 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[29646]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[29646]: osd.6 [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215] boot 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[29646]: osdmap e44: 7 total, 7 up, 7 in 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[41592]: purged_snaps scrub starts 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[41592]: purged_snaps scrub ok 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[41592]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[41592]: osd.6 [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215] boot 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[41592]: osdmap e44: 7 total, 7 up, 7 in 2024-09-13T14:19:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:37 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:19:38.103 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:19:38.503 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:38 smithi019 ceph-mon[32895]: osdmap e45: 7 total, 7 up, 7 in 2024-09-13T14:19:38.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:38 smithi003 ceph-mon[29646]: osdmap e45: 7 total, 7 up, 7 in 2024-09-13T14:19:38.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:38 smithi003 ceph-mon[41592]: osdmap e45: 7 total, 7 up, 7 in 2024-09-13T14:19:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:39 smithi019 ceph-mon[32895]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:39 smithi019 ceph-mon[32895]: osdmap e46: 7 total, 7 up, 7 in 2024-09-13T14:19:39.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:39 smithi003 ceph-mon[29646]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:39.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:39 smithi003 ceph-mon[29646]: osdmap e46: 7 total, 7 up, 7 in 2024-09-13T14:19:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:39 smithi003 ceph-mon[41592]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:39 smithi003 ceph-mon[41592]: osdmap e46: 7 total, 7 up, 7 in 2024-09-13T14:19:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: Detected new or changed devices on smithi019 2024-09-13T14:19:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi019 to 5697M 2024-09-13T14:19:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:19:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:40 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: Detected new or changed devices on smithi019 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi019 to 5697M 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: Detected new or changed devices on smithi019 2024-09-13T14:19:41.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi019 to 5697M 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:41.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:41.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:41.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:41.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:19:41.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:19:41.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:40 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:43 smithi019 ceph-mon[32895]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:43 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/3129478539' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1edab2d1-dad5-45c1-977c-6911541c282e"}]: dispatch 2024-09-13T14:19:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:43 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1edab2d1-dad5-45c1-977c-6911541c282e"}]: dispatch 2024-09-13T14:19:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:43 smithi019 ceph-mon[32895]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1edab2d1-dad5-45c1-977c-6911541c282e"}]': finished 2024-09-13T14:19:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:43 smithi019 ceph-mon[32895]: osdmap e47: 8 total, 7 up, 8 in 2024-09-13T14:19:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:43 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:19:43.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[41592]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/3129478539' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1edab2d1-dad5-45c1-977c-6911541c282e"}]: dispatch 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1edab2d1-dad5-45c1-977c-6911541c282e"}]: dispatch 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[41592]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1edab2d1-dad5-45c1-977c-6911541c282e"}]': finished 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[41592]: osdmap e47: 8 total, 7 up, 8 in 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[29646]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/3129478539' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1edab2d1-dad5-45c1-977c-6911541c282e"}]: dispatch 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1edab2d1-dad5-45c1-977c-6911541c282e"}]: dispatch 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[29646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1edab2d1-dad5-45c1-977c-6911541c282e"}]': finished 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[29646]: osdmap e47: 8 total, 7 up, 8 in 2024-09-13T14:19:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:43 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:19:44.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:44 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/4051860402' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:19:44.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:44 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/4051860402' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:19:44.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:44 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/4051860402' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:19:45.574 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:45 smithi019 ceph-mon[32895]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:45.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:45 smithi003 ceph-mon[29646]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:45.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:45 smithi003 ceph-mon[41592]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:47 smithi019 ceph-mon[32895]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:47.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:47 smithi003 ceph-mon[29646]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:47.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:47 smithi003 ceph-mon[41592]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:49 smithi019 ceph-mon[32895]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:49 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:19:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:49 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:49.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:49 smithi003 ceph-mon[29646]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:49.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:49 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:19:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:49 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:49 smithi003 ceph-mon[41592]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:49 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:19:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:49 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:50.548 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:50 smithi019 ceph-mon[32895]: Deploying daemon osd.7 on smithi019 2024-09-13T14:19:50.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:50 smithi003 ceph-mon[29646]: Deploying daemon osd.7 on smithi019 2024-09-13T14:19:50.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:50 smithi003 ceph-mon[41592]: Deploying daemon osd.7 on smithi019 2024-09-13T14:19:51.654 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:51 smithi019 ceph-mon[32895]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:51.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:51 smithi003 ceph-mon[29646]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:51.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:51 smithi003 ceph-mon[41592]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:53.520 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:53 smithi019 ceph-mon[32895]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:53.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:53 smithi003 ceph-mon[29646]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:53.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:53 smithi003 ceph-mon[41592]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:54.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:54.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:54.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:54 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:54.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:54.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:54 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:54 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:55.509 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:55 smithi019 ceph-mon[32895]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:55.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:55 smithi003 ceph-mon[29646]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:55 smithi003 ceph-mon[41592]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:57.465 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:57 smithi019 ceph-mon[32895]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:57.465 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:57 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:57.465 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:57 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:57.465 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:57 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:57.465 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:57 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:57.465 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:57 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:57.505 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 7 on host 'smithi019' 2024-09-13T14:19:57.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[29646]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:57.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[41592]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:57 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:58.102 DEBUG:teuthology.orchestra.run.smithi019:osd.7> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.7.service 2024-09-13T14:19:58.105 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-13T14:19:58.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T14:19:58.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:58 smithi019 ceph-mon[32895]: from='osd.7 [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:19:58.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:58 smithi019 ceph-mon[32895]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:19:58.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:58 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:58.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:58 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:58.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:58 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:58.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[41592]: from='osd.7 [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:19:58.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[41592]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:19:58.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:58.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:58.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:58.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[29646]: from='osd.7 [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:19:58.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[29646]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:19:58.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:58.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:58.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:58 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:19:59.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:59 smithi019 ceph-mon[32895]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:59.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:59 smithi019 ceph-mon[32895]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:19:59.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:59 smithi019 ceph-mon[32895]: osdmap e48: 8 total, 7 up, 8 in 2024-09-13T14:19:59.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:59 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:19:59.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:59 smithi019 ceph-mon[32895]: from='osd.7 [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:59.358 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:19:59 smithi019 ceph-mon[32895]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:59.359 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 14:19:59 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T14:19:59.236+0000 7fd8244e3640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T14:19:59.359 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 14:19:59 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T14:19:59.244+0000 7fd81daf5640 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:19:59.549 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[41592]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:59.549 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[41592]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:19:59.549 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[41592]: osdmap e48: 8 total, 7 up, 8 in 2024-09-13T14:19:59.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:19:59.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[41592]: from='osd.7 [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:59.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[41592]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:59.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[29646]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:19:59.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[29646]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:19:59.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[29646]: osdmap e48: 8 total, 7 up, 8 in 2024-09-13T14:19:59.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:19:59.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[29646]: from='osd.7 [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:19:59.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:19:59 smithi003 ceph-mon[29646]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-09-13T14:20:00.141 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:20:00.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[41592]: purged_snaps scrub starts 2024-09-13T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[41592]: purged_snaps scrub ok 2024-09-13T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[41592]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[41592]: osdmap e49: 8 total, 7 up, 8 in 2024-09-13T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[41592]: overall HEALTH_OK 2024-09-13T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3478401922' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:20:00.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[29646]: purged_snaps scrub starts 2024-09-13T14:20:00.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[29646]: purged_snaps scrub ok 2024-09-13T14:20:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[29646]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:20:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[29646]: osdmap e49: 8 total, 7 up, 8 in 2024-09-13T14:20:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[29646]: overall HEALTH_OK 2024-09-13T14:20:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:00 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3478401922' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:20:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:00 smithi019 ceph-mon[32895]: purged_snaps scrub starts 2024-09-13T14:20:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:00 smithi019 ceph-mon[32895]: purged_snaps scrub ok 2024-09-13T14:20:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:00 smithi019 ceph-mon[32895]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-09-13T14:20:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:00 smithi019 ceph-mon[32895]: osdmap e49: 8 total, 7 up, 8 in 2024-09-13T14:20:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:00 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:00 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:00 smithi019 ceph-mon[32895]: overall HEALTH_OK 2024-09-13T14:20:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:00 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3478401922' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:20:00.843 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":49,"num_osds":8,"num_up_osds":7,"osd_up_since":1726237176,"num_in_osds":8,"osd_in_since":1726237182,"num_remapped_pgs":0} 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: osd.7 [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175] boot 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: osdmap e50: 8 total, 8 up, 8 in 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: osdmap e51: 8 total, 8 up, 8 in 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:01 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: osd.7 [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175] boot 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: osdmap e50: 8 total, 8 up, 8 in 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: osdmap e51: 8 total, 8 up, 8 in 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: osd.7 [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175] boot 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: osdmap e50: 8 total, 8 up, 8 in 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:01.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: osdmap e51: 8 total, 8 up, 8 in 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:01 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:01.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T14:20:02.112 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:02.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:02 smithi003 ceph-mon[29646]: Detected new or changed devices on smithi019 2024-09-13T14:20:02.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:02 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi019 to 4273M 2024-09-13T14:20:02.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:02 smithi003 ceph-mon[29646]: osdmap e52: 8 total, 8 up, 8 in 2024-09-13T14:20:02.560 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:02 smithi003 ceph-mon[41592]: Detected new or changed devices on smithi019 2024-09-13T14:20:02.560 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:02 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi019 to 4273M 2024-09-13T14:20:02.561 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:02 smithi003 ceph-mon[41592]: osdmap e52: 8 total, 8 up, 8 in 2024-09-13T14:20:02.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:02 smithi019 ceph-mon[32895]: Detected new or changed devices on smithi019 2024-09-13T14:20:02.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:02 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi019 to 4273M 2024-09-13T14:20:02.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:02 smithi019 ceph-mon[32895]: osdmap e52: 8 total, 8 up, 8 in 2024-09-13T14:20:03.388 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:03 smithi003 ceph-mon[29646]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:03.388 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:03 smithi003 ceph-mon[41592]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:03.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:03 smithi019 ceph-mon[32895]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:04.017 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:20:04.317 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:04 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2496635553' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:20:04.317 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:04 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2496635553' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:20:04.653 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":52,"num_osds":8,"num_up_osds":8,"osd_up_since":1726237200,"num_in_osds":8,"osd_in_since":1726237182,"num_remapped_pgs":0} 2024-09-13T14:20:04.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:20:04.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:04 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2496635553' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:20:04.910 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:05.345 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:05 smithi003 ceph-mon[41592]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:05.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:05 smithi003 ceph-mon[29646]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:05.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:05 smithi019 ceph-mon[32895]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:06.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:06 smithi003 ceph-mon[29646]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:20:06.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:06 smithi003 ceph-mon[29646]: Cluster is now healthy 2024-09-13T14:20:06.365 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:06 smithi003 ceph-mon[41592]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:20:06.365 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:06 smithi003 ceph-mon[41592]: Cluster is now healthy 2024-09-13T14:20:06.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:06 smithi019 ceph-mon[32895]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:20:06.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:06 smithi019 ceph-mon[32895]: Cluster is now healthy 2024-09-13T14:20:06.738 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:20:06.739 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":52,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:15:00.204452+0000","modified":"2024-09-13T14:20:01.250170+0000","last_up_change":"2024-09-13T14:20:00.222071+0000","last_in_change":"2024-09-13T14:19:42.900010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:18:02.574675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a153ec29-6e01-4063-aa90-01d1de00e53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6803","nonce":3028312307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6805","nonce":3028312307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6809","nonce":3028312307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6807","nonce":3028312307}]},"public_addr":"172.21.15.3:6803/3028312307","cluster_addr":"172.21.15.3:6805/3028312307","heartbeat_back_addr":"172.21.15.3:6809/3028312307","heartbeat_front_addr":"172.21.15.3:6807/3028312307","state":["exists","up"]},{"osd":1,"uuid":"9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6811","nonce":2008999734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6813","nonce":2008999734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6817","nonce":2008999734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6815","nonce":2008999734}]},"public_addr":"172.21.15.3:6811/2008999734","cluster_addr":"172.21.15.3:6813/2008999734","heartbeat_back_addr":"172.21.15.3:6817/2008999734","heartbeat_front_addr":"172.21.15.3:6815/2008999734","state":["exists","up"]},{"osd":2,"uuid":"676ec1e5-071c-47ba-86ae-2193f448fe8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6819","nonce":2444149284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6821","nonce":2444149284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6825","nonce":2444149284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6823","nonce":2444149284}]},"public_addr":"172.21.15.3:6819/2444149284","cluster_addr":"172.21.15.3:6821/2444149284","heartbeat_back_addr":"172.21.15.3:6825/2444149284","heartbeat_front_addr":"172.21.15.3:6823/2444149284","state":["exists","up"]},{"osd":3,"uuid":"7ec0ad16-0df3-4319-b475-72cfaaa5edae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6827","nonce":2595190501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6829","nonce":2595190501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6833","nonce":2595190501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6831","nonce":2595190501}]},"public_addr":"172.21.15.3:6827/2595190501","cluster_addr":"172.21.15.3:6829/2595190501","heartbeat_back_addr":"172.21.15.3:6833/2595190501","heartbeat_front_addr":"172.21.15.3:6831/2595190501","state":["exists","up"]},{"osd":4,"uuid":"0ef54d09-4000-4695-942b-1b1cf9659983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6801","nonce":4204660131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6803","nonce":4204660131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6807","nonce":4204660131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6805","nonce":4204660131}]},"public_addr":"172.21.15.19:6801/4204660131","cluster_addr":"172.21.15.19:6803/4204660131","heartbeat_back_addr":"172.21.15.19:6807/4204660131","heartbeat_front_addr":"172.21.15.19:6805/4204660131","state":["exists","up"]},{"osd":5,"uuid":"6dfd8fce-4d3d-4a8b-90ac-556447045378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6809","nonce":3591244104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6811","nonce":3591244104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6815","nonce":3591244104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6813","nonce":3591244104}]},"public_addr":"172.21.15.19:6809/3591244104","cluster_addr":"172.21.15.19:6811/3591244104","heartbeat_back_addr":"172.21.15.19:6815/3591244104","heartbeat_front_addr":"172.21.15.19:6813/3591244104","state":["exists","up"]},{"osd":6,"uuid":"17e27318-c0ce-413d-b660-d756731ea743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6817","nonce":2162145215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6819","nonce":2162145215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6823","nonce":2162145215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6821","nonce":2162145215}]},"public_addr":"172.21.15.19:6817/2162145215","cluster_addr":"172.21.15.19:6819/2162145215","heartbeat_back_addr":"172.21.15.19:6823/2162145215","heartbeat_front_addr":"172.21.15.19:6821/2162145215","state":["exists","up"]},{"osd":7,"uuid":"1edab2d1-dad5-45c1-977c-6911541c282e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6825","nonce":2814206175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6827","nonce":2814206175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6831","nonce":2814206175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6829","nonce":2814206175}]},"public_addr":"172.21.15.19:6825/2814206175","cluster_addr":"172.21.15.19:6827/2814206175","heartbeat_back_addr":"172.21.15.19:6831/2814206175","heartbeat_front_addr":"172.21.15.19:6829/2814206175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:11.323986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:34.429745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:59.523592+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:24.423713+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:46.452261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:10.483064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:34.315912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:58.203878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/3631040762":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2723658197":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/253916983":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6800/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6800/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/3505363388":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2497820386":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/738145194":"2024-09-14T14:15:28.360446+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:20:07.411 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T14:18:02.574675+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-13T14:20:07.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-13T14:20:07.667 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:07.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:07 smithi019 ceph-mon[32895]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:20:07.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:07 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3424892278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:20:07.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:07 smithi003 ceph-mon[29646]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:20:07.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:07 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3424892278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:20:07.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:07 smithi003 ceph-mon[41592]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:20:07.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:07 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3424892278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:20:09.453 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-09-13T14:20:09.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:09 smithi019 ceph-mon[32895]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-13T14:20:09.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:09 smithi003 ceph-mon[29646]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-13T14:20:09.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:09 smithi003 ceph-mon[41592]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-13T14:20:10.069 INFO:tasks.cephadm:Adding prometheus.a on smithi019 2024-09-13T14:20:10.069 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi019=a' 2024-09-13T14:20:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:20:10.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:10 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1563988323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:20:10.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:10 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1563988323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:20:10.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:10 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1563988323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:20:11.545 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:11 smithi019 ceph-mon[32895]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:20:11.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:11 smithi003 ceph-mon[29646]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:20:11.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:11 smithi003 ceph-mon[41592]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:20:12.266 INFO:teuthology.orchestra.run.smithi019.stdout:Scheduled prometheus update... 2024-09-13T14:20:12.878 DEBUG:teuthology.orchestra.run.smithi019:prometheus.a> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-13T14:20:12.880 INFO:tasks.cephadm:Adding node-exporter.a on smithi003 2024-09-13T14:20:12.880 INFO:tasks.cephadm:Adding node-exporter.b on smithi019 2024-09-13T14:20:12.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi003=a;smithi019=b' 2024-09-13T14:20:13.180 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:20:13.562 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:13 smithi019 ceph-mon[32895]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-13T14:20:13.563 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:13 smithi019 ceph-mon[32895]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi019=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:13.563 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:13 smithi019 ceph-mon[32895]: Saving service prometheus spec with placement smithi019=a;count:1 2024-09-13T14:20:13.563 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:13.563 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:13.563 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:13.563 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:13.563 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:13 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:13.563 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:13 smithi019 ceph-mon[32895]: Deploying daemon prometheus.a on smithi019 2024-09-13T14:20:13.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[41592]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-13T14:20:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[41592]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi019=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[41592]: Saving service prometheus spec with placement smithi019=a;count:1 2024-09-13T14:20:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[41592]: Deploying daemon prometheus.a on smithi019 2024-09-13T14:20:13.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[29646]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-13T14:20:13.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[29646]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi019=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:13.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[29646]: Saving service prometheus spec with placement smithi019=a;count:1 2024-09-13T14:20:13.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:13.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:13.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:13.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:13.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:13.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:13 smithi003 ceph-mon[29646]: Deploying daemon prometheus.a on smithi019 2024-09-13T14:20:15.619 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:15 smithi019 ceph-mon[32895]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:20:15.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:15 smithi003 ceph-mon[41592]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:20:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:15 smithi003 ceph-mon[29646]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:20:15.724 INFO:teuthology.orchestra.run.smithi019.stdout:Scheduled node-exporter update... 2024-09-13T14:20:16.727 DEBUG:teuthology.orchestra.run.smithi003:node-exporter.a> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-13T14:20:16.730 DEBUG:teuthology.orchestra.run.smithi019:node-exporter.b> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-13T14:20:16.732 INFO:tasks.cephadm:Adding alertmanager.a on smithi003 2024-09-13T14:20:16.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi003=a' 2024-09-13T14:20:17.027 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:20:17.149 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:16 smithi019 ceph-mon[32895]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi003=a;smithi019=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:17.149 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:16 smithi019 ceph-mon[32895]: Saving service node-exporter spec with placement smithi003=a;smithi019=b;count:2 2024-09-13T14:20:17.149 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:16 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:17.149 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:16 smithi019 ceph-mon[32895]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:20:17.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:16 smithi003 ceph-mon[29646]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi003=a;smithi019=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:17.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:16 smithi003 ceph-mon[29646]: Saving service node-exporter spec with placement smithi003=a;smithi019=b;count:2 2024-09-13T14:20:17.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:16 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:16 smithi003 ceph-mon[29646]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:20:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:16 smithi003 ceph-mon[41592]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi003=a;smithi019=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:16 smithi003 ceph-mon[41592]: Saving service node-exporter spec with placement smithi003=a;smithi019=b;count:2 2024-09-13T14:20:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:16 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:16 smithi003 ceph-mon[41592]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:20:19.455 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:19 smithi019 ceph-mon[32895]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:19.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:19 smithi003 ceph-mon[29646]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:19.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:19 smithi003 ceph-mon[41592]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:19.765 INFO:teuthology.orchestra.run.smithi019.stdout:Scheduled alertmanager update... 2024-09-13T14:20:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:20 smithi019 ceph-mon[32895]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:20 smithi019 ceph-mon[32895]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-09-13T14:20:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:20 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:20 smithi019 ceph-mon[32895]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:21.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:20 smithi003 ceph-mon[29646]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:21.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:20 smithi003 ceph-mon[29646]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-09-13T14:20:21.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:20 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:20 smithi003 ceph-mon[29646]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:20 smithi003 ceph-mon[41592]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:20 smithi003 ceph-mon[41592]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-09-13T14:20:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:20 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:20 smithi003 ceph-mon[41592]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:21.539 DEBUG:teuthology.orchestra.run.smithi003:alertmanager.a> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-13T14:20:21.542 INFO:tasks.cephadm:Adding grafana.a on smithi019 2024-09-13T14:20:21.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi019=a' 2024-09-13T14:20:21.914 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:20:23.297 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:23 smithi019 ceph-mon[32895]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:23.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:23 smithi003 ceph-mon[41592]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:23.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:23 smithi003 ceph-mon[29646]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:25.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:25 smithi019 ceph-mon[32895]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:25.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:25 smithi003 ceph-mon[29646]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:25 smithi003 ceph-mon[41592]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:26.511 INFO:teuthology.orchestra.run.smithi019.stdout:Scheduled grafana update... 2024-09-13T14:20:27.162 DEBUG:teuthology.orchestra.run.smithi019:grafana.a> sudo journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@grafana.a.service 2024-09-13T14:20:27.164 INFO:tasks.cephadm:Setting up client nodes... 2024-09-13T14:20:27.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T14:20:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:27.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 podman[61857]: 2024-09-13 14:20:27.326038892 +0000 UTC m=+0.123387342 container create 819b2e3729e53a38b2efdc35443b776e25a0a176844cef5ef378aa6e2b4a07dd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:20:27.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:27 smithi003 ceph-mon[41592]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:27.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:27 smithi003 ceph-mon[41592]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi019=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:27.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:27 smithi003 ceph-mon[41592]: Saving service grafana spec with placement smithi019=a;count:1 2024-09-13T14:20:27.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:27 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:27.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:27 smithi003 ceph-mon[29646]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:27.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:27 smithi003 ceph-mon[29646]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi019=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:27.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:27 smithi003 ceph-mon[29646]: Saving service grafana spec with placement smithi019=a;count:1 2024-09-13T14:20:27.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:27 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:27.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-mon[32895]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:27.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-mon[32895]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi019=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-mon[32895]: Saving service grafana spec with placement smithi019=a;count:1 2024-09-13T14:20:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:27.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 podman[61857]: 2024-09-13 14:20:27.54479984 +0000 UTC m=+0.342148250 container init 819b2e3729e53a38b2efdc35443b776e25a0a176844cef5ef378aa6e2b4a07dd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:20:27.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 podman[61857]: 2024-09-13 14:20:27.549622873 +0000 UTC m=+0.346971281 container start 819b2e3729e53a38b2efdc35443b776e25a0a176844cef5ef378aa6e2b4a07dd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:20:27.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.575Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.575Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.575Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi019 (none))" 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.575Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.575Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.576Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.577Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.578Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.578Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.580Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.580Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.528µs 2024-09-13T14:20:27.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.580Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.581Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.581Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=35.177µs wal_replay_duration=533.372µs wbl_replay_duration=89ns total_replay_duration=653.363µs 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.581Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.581Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.581Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.596Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.970627ms db_storage=734ns remote_storage=1.505µs web_handler=463ns query_engine=561ns scrape=445.244µs scrape_sd=17.605µs notify=705ns notify_sd=1.494µs rules=14.302016ms tracing=5.204µs 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.596Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:20:27.596Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 bash[61857]: 819b2e3729e53a38b2efdc35443b776e25a0a176844cef5ef378aa6e2b4a07dd 2024-09-13T14:20:27.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:20:27 smithi019 systemd[1]: Started Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:20:28.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:28.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:28.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:28.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[29646]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ignoring --setuser ceph since I am not root 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ignoring --setgroup ceph since I am not root 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mgr[29920]: -- 172.21.15.3:0/417284653 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55798c140340 con 0x55798c14d000 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:28.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:20:28.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-mon[41592]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:28.989 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:28.989 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:28.989 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' 2024-09-13T14:20:28.989 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:20:28.989 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-mon[32895]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:28.989 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ignoring --setuser ceph since I am not root 2024-09-13T14:20:28.990 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ignoring --setgroup ceph since I am not root 2024-09-13T14:20:28.990 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-mgr[34985]: -- 172.21.15.19:0/4263812046 <== mon.2 v2:172.21.15.3:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55573034c340 con 0x555730358c00 2024-09-13T14:20:28.990 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:28.863+0000 7f4a70379ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:20:28.990 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:28.920+0000 7f4a70379ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:20:28.990 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:28.987+0000 7f4a70379ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:20:29.158 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:28.859+0000 7fded6fa7ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:20:29.158 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:28.923+0000 7fded6fa7ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:20:29.158 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:28.991+0000 7fded6fa7ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:20:29.260 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:29 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:29.117+0000 7f4a70379ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:20:29.261 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:29 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:29.259+0000 7f4a70379ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:20:29.340 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-09-13T14:20:29.340 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAtSuRm79sHFBAAN5DK9Fdc8rFdMvMlHUI6ag== 2024-09-13T14:20:29.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:29.156+0000 7fded6fa7ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:20:29.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:29.299+0000 7fded6fa7ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:20:29.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:29 smithi019 ceph-mon[32895]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:20:29.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:29 smithi019 ceph-mon[32895]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:20:29.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:29 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/122134979' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:20:29.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:29 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/122134979' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:20:29.932 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:29 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:29.527+0000 7f4a70379ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:20:29.932 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:29 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:29.590+0000 7f4a70379ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:20:29.932 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:29 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:29.760+0000 7f4a70379ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:20:29.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-mon[41592]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:20:29.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-mon[41592]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:20:29.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/122134979' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:20:29.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/122134979' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-mon[29646]: from='mgr.14150 172.21.15.3:0/24544502' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-mon[29646]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/122134979' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/122134979' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:20:29.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:29.571+0000 7fded6fa7ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:20:29.934 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:29.633+0000 7fded6fa7ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:20:29.934 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:29.803+0000 7fded6fa7ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:20:30.043 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T14:20:30.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-13T14:20:30.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T14:20:30.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T14:20:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:20:30.724 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:30.670+0000 7f4a70379ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:20:31.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:30.743+0000 7f4a70379ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:20:31.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:30 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:30.687+0000 7fded6fa7ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:20:31.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:30 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:30.753+0000 7fded6fa7ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:20:31.610 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:31.493+0000 7f4a70379ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:20:31.794 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:31.490+0000 7fded6fa7ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:20:31.794 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:31.606+0000 7fded6fa7ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:20:31.795 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:31.731+0000 7fded6fa7ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:20:31.795 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:31.793+0000 7fded6fa7ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:20:31.898 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:31.611+0000 7f4a70379ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:20:31.898 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:31.744+0000 7f4a70379ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:20:31.899 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:31.808+0000 7f4a70379ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:20:32.055 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:31.924+0000 7fded6fa7ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:20:32.055 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:31.994+0000 7fded6fa7ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:20:32.055 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:32.054+0000 7fded6fa7ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:20:32.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:31.946+0000 7f4a70379ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:20:32.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:32.017+0000 7f4a70379ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:20:32.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:32.079+0000 7f4a70379ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:20:32.328 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:32.194+0000 7fded6fa7ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:20:32.329 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:32.260+0000 7fded6fa7ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:20:32.329 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:32.327+0000 7fded6fa7ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:20:32.393 INFO:teuthology.orchestra.run.smithi019.stdout:[client.1] 2024-09-13T14:20:32.393 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQAwSuRmoWExFxAAC8UmbKFe0wBTecpINlBpAA== 2024-09-13T14:20:32.451 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-mon[32895]: from='client.? 172.21.15.19:0/3119950872' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:20:32.451 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-mon[32895]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:20:32.451 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-mon[32895]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:20:32.452 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:32.224+0000 7f4a70379ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:20:32.452 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:32.292+0000 7f4a70379ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:20:32.452 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:32.360+0000 7f4a70379ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:20:32.452 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:32.450+0000 7f4a70379ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:20:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-mon[41592]: from='client.? 172.21.15.19:0/3119950872' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:20:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-mon[41592]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:20:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-mon[41592]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:20:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-mon[29646]: from='client.? 172.21.15.19:0/3119950872' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:20:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-mon[29646]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:20:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-mon[29646]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:20:32.683 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:32.413+0000 7fded6fa7ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:20:32.683 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:32.475+0000 7fded6fa7ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:20:32.683 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:32.534+0000 7fded6fa7ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:20:32.751 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:32.510+0000 7f4a70379ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:20:32.751 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:32.571+0000 7f4a70379ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:20:33.079 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-13T14:20:33.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-13T14:20:33.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T14:20:33.117 INFO:tasks.cephadm:Creating RBD pool 2024-09-13T14:20:33.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create rbd 8 2024-09-13T14:20:33.394 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:33.402 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:33.327+0000 7fded6fa7ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:20:33.658 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:20:33] ENGINE Bus STARTING 2024-09-13T14:20:33.658 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: CherryPy Checker: 2024-09-13T14:20:33.658 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: The Application mounted at '' has an empty config. 2024-09-13T14:20:33.658 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:33.658 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:20:33] ENGINE Serving on http://:::9283 2024-09-13T14:20:33.658 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:20:33] ENGINE Bus STARTED 2024-09-13T14:20:33.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: Active manager daemon y restarted 2024-09-13T14:20:33.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: Activating manager daemon y 2024-09-13T14:20:33.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: osdmap e53: 8 total, 8 up, 8 in 2024-09-13T14:20:33.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: mgrmap e17: y(active, starting, since 0.0084117s), standbys: x 2024-09-13T14:20:33.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:20:33.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:20:33.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:20:33.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:20:33.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: Manager daemon y is now available 2024-09-13T14:20:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: Standby manager daemon x restarted 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: Standby manager daemon x started 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:20:33.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:20:33.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: Active manager daemon y restarted 2024-09-13T14:20:33.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: Activating manager daemon y 2024-09-13T14:20:33.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: osdmap e53: 8 total, 8 up, 8 in 2024-09-13T14:20:33.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: mgrmap e17: y(active, starting, since 0.0084117s), standbys: x 2024-09-13T14:20:33.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:20:33.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:20:33.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:20:33.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: Manager daemon y is now available 2024-09-13T14:20:33.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: Standby manager daemon x restarted 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: Standby manager daemon x started 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:20:33.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:20:33.665 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:33.366+0000 7f4a70379ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:20:33.665 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: [13/Sep/2024:14:20:33] ENGINE Bus STARTING 2024-09-13T14:20:33.665 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: CherryPy Checker: 2024-09-13T14:20:33.665 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: The Application mounted at '' has an empty config. 2024-09-13T14:20:33.665 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: 2024-09-13T14:20:33.665 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: [13/Sep/2024:14:20:33] ENGINE Serving on http://:::9283 2024-09-13T14:20:33.665 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: [13/Sep/2024:14:20:33] ENGINE Bus STARTED 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: Active manager daemon y restarted 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: Activating manager daemon y 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: osdmap e53: 8 total, 8 up, 8 in 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: mgrmap e17: y(active, starting, since 0.0084117s), standbys: x 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:20:33.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: Manager daemon y is now available 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: Standby manager daemon x restarted 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: Standby manager daemon x started 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:20:33.667 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:20:33.668 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:33.668 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:20:33.668 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3141369160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:20:33.668 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:33.668 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:33.668 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:20:33.668 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:20:35.262 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:35.085+0000 7fde79d6a640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:20:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: Traceback (most recent call last): 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self._connections.run(self.expiration_interval) 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self._run(expiration_interval) 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: s = self.context.wrap_socket( 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:20:35.263 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: return self.sslsocket_class._create( 2024-09-13T14:20:35.264 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:20:35.264 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self.do_handshake() 2024-09-13T14:20:35.264 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:20:35.264 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: self._sslobj.do_handshake() 2024-09-13T14:20:35.264 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:20:35.264 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:20:35.400 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:35 smithi019 ceph-mon[32895]: mgrmap e18: y(active, since 1.01181s), standbys: x 2024-09-13T14:20:35.400 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:35 smithi019 ceph-mon[32895]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:35.400 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:35 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:35.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-mon[41592]: mgrmap e18: y(active, since 1.01181s), standbys: x 2024-09-13T14:20:35.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-mon[41592]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:35.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:35.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-mon[29646]: mgrmap e18: y(active, since 1.01181s), standbys: x 2024-09-13T14:20:35.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-mon[29646]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:35.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:35 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.352 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'rbd' created 2024-09-13T14:20:36.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: [13/Sep/2024:14:20:34] ENGINE Bus STARTING 2024-09-13T14:20:36.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: [13/Sep/2024:14:20:35] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: [13/Sep/2024:14:20:35] ENGINE Bus STARTED 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: [13/Sep/2024:14:20:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: Traceback (most recent call last): 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._run(expiration_interval) 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:20:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: s = self.context.wrap_socket( 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: return self.sslsocket_class._create( 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: self.do_handshake() 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._sslobj.do_handshake() 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1722746903' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:20:36.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:36 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: [13/Sep/2024:14:20:34] ENGINE Bus STARTING 2024-09-13T14:20:36.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: [13/Sep/2024:14:20:35] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: [13/Sep/2024:14:20:35] ENGINE Bus STARTED 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: [13/Sep/2024:14:20:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:20:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout: self.do_handshake() 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1722746903' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:20:36.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: [13/Sep/2024:14:20:34] ENGINE Bus STARTING 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: [13/Sep/2024:14:20:35] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: [13/Sep/2024:14:20:35] ENGINE Bus STARTED 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: [13/Sep/2024:14:20:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:20:36.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1722746903' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:20:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:20:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:20:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:36 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:36.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- rbd pool init rbd 2024-09-13T14:20:37.304 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:37.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:20:37.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:20:37.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi019 to 4017M 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:37.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi019 to 4017M 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:37.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:37.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:20:37.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:20:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:20:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi019 to 4017M 2024-09-13T14:20:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:37.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:37.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:38.480 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:20:38.480 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:20:38.480 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:20:38.480 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:20:38.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: pgmap v6: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:38.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:20:38.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:20:38.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:20:38.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:20:38.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:20:38.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:20:38.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[41592]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: pgmap v6: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:20:38.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:38 smithi003 ceph-mon[29646]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:20:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:20:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:20:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:20:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:20:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: pgmap v6: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:38.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:20:38.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:20:38.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:20:38.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:20:38.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:20:38.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:38.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:20:38.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:38 smithi019 ceph-mon[32895]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:20:39.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:39 smithi019 ceph-mon[32895]: Deploying daemon alertmanager.a on smithi003 2024-09-13T14:20:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:39 smithi019 ceph-mon[32895]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:20:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:39 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2234682664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:20:39.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:39 smithi003 ceph-mon[41592]: Deploying daemon alertmanager.a on smithi003 2024-09-13T14:20:39.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:39 smithi003 ceph-mon[41592]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:20:39.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:39 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2234682664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:20:39.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:39 smithi003 ceph-mon[29646]: Deploying daemon alertmanager.a on smithi003 2024-09-13T14:20:39.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:39 smithi003 ceph-mon[29646]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:20:39.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:39 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2234682664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:20:40.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:40 smithi019 ceph-mon[32895]: pgmap v9: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:40 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2234682664' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:20:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:40 smithi019 ceph-mon[32895]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:20:40.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:40 smithi003 ceph-mon[41592]: pgmap v9: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:40.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:40 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2234682664' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:20:40.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:40 smithi003 ceph-mon[41592]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:20:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:40 smithi003 ceph-mon[29646]: pgmap v9: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:40 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2234682664' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:20:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:40 smithi003 ceph-mon[29646]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:20:41.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:20:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:20:41.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:41 smithi019 ceph-mon[32895]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:20:41.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:41 smithi003 ceph-mon[41592]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:20:41.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:41 smithi003 ceph-mon[29646]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:20:42.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-13T14:20:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:42 smithi019 ceph-mon[32895]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-13T14:20:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:42 smithi019 ceph-mon[32895]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:20:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:42 smithi019 ceph-mon[32895]: Cluster is now healthy 2024-09-13T14:20:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:42 smithi019 ceph-mon[32895]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:20:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:42 smithi003 ceph-mon[41592]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-13T14:20:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:42 smithi003 ceph-mon[41592]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:20:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:42 smithi003 ceph-mon[41592]: Cluster is now healthy 2024-09-13T14:20:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:42 smithi003 ceph-mon[41592]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:20:42.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:42 smithi003 ceph-mon[29646]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-13T14:20:42.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:42 smithi003 ceph-mon[29646]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:20:42.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:42 smithi003 ceph-mon[29646]: Cluster is now healthy 2024-09-13T14:20:42.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:42 smithi003 ceph-mon[29646]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:20:42.897 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:44.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:44 smithi003 ceph-mon[41592]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-13T14:20:44.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:44 smithi003 ceph-mon[29646]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-13T14:20:44.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:44 smithi019 ceph-mon[32895]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-13T14:20:46.584 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:46 smithi003 ceph-mon[41592]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 341 B/s wr, 12 op/s 2024-09-13T14:20:46.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:46 smithi003 ceph-mon[29646]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 341 B/s wr, 12 op/s 2024-09-13T14:20:46.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:46 smithi019 ceph-mon[32895]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 341 B/s wr, 12 op/s 2024-09-13T14:20:47.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 podman[72104]: 2024-09-13 14:20:47.539080045 +0000 UTC m=+0.404206714 container create aec662fe2e6f0604749fedc78ca6f07aac65fae2b6fc671b40b3866a2d55bccf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:20:48.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:20:48] ENGINE Bus STOPPING 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 podman[72104]: 2024-09-13 14:20:47.858350065 +0000 UTC m=+0.723476693 container init aec662fe2e6f0604749fedc78ca6f07aac65fae2b6fc671b40b3866a2d55bccf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 podman[72104]: 2024-09-13 14:20:47.863518194 +0000 UTC m=+0.728644821 container start aec662fe2e6f0604749fedc78ca6f07aac65fae2b6fc671b40b3866a2d55bccf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:47.877Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:47.877Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:47.878Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.3 port=9094 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:47.879Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:47.902Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:47.903Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:47.905Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:47.905Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 bash[72104]: aec662fe2e6f0604749fedc78ca6f07aac65fae2b6fc671b40b3866a2d55bccf 2024-09-13T14:20:48.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:47 smithi003 systemd[1]: Started Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:20:48.350 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:20:48.553 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 256 B/s wr, 0 op/s 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2694681422' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 256 B/s wr, 0 op/s 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.554 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:20:48.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2694681422' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:20:48.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 256 B/s wr, 0 op/s 2024-09-13T14:20:48.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:20:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:48 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2694681422' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:20:48.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:20:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:20:48.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:20:48] ENGINE Bus STOPPED 2024-09-13T14:20:48.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:20:48] ENGINE Bus STARTING 2024-09-13T14:20:48.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:20:48] ENGINE Serving on http://:::9283 2024-09-13T14:20:48.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:20:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:20:48] ENGINE Bus STARTED 2024-09-13T14:20:48.989 INFO:teuthology.orchestra.run.smithi003.stdout:[{"version":19,"timestamp":"2024-09-13T14:20:48.061494+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/GRAFANA_API_SSL_VERIFY","new_value":"false"}]}] 2024-09-13T14:20:48.990 INFO:tasks.ceph_manager:config epoch is 19 2024-09-13T14:20:48.990 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:20:48.990 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:20:48.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:20:49.247 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:49.650 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:49 smithi003 ceph-mon[41592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:20:49.650 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:49 smithi003 ceph-mon[41592]: Deploying daemon grafana.a on smithi019 2024-09-13T14:20:49.650 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:49 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:49.650 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:49 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:49.650 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:49 smithi003 ceph-mon[29646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:20:49.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:49 smithi003 ceph-mon[29646]: Deploying daemon grafana.a on smithi019 2024-09-13T14:20:49.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:49 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:49.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:49 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:49.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:49 smithi019 ceph-mon[32895]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:20:49.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:49 smithi019 ceph-mon[32895]: Deploying daemon grafana.a on smithi019 2024-09-13T14:20:49.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:49 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:49.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:49 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:49.932 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:49.880Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001049572s 2024-09-13T14:20:50.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:50 smithi003 ceph-mon[41592]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 228 B/s wr, 0 op/s 2024-09-13T14:20:50.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:50 smithi003 ceph-mon[41592]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:20:50.664 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:50 smithi003 ceph-mon[29646]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 228 B/s wr, 0 op/s 2024-09-13T14:20:50.664 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:50 smithi003 ceph-mon[29646]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:20:50.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:50 smithi019 ceph-mon[32895]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 228 B/s wr, 0 op/s 2024-09-13T14:20:50.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:50 smithi019 ceph-mon[32895]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:20:51.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:20:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:20:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:20:51.228 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:20:51.588 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:51 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2090495673' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:20:51.588 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:51 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2090495673' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:20:51.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:51 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2090495673' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:20:51.896 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"flags":0,"active_gid":14439,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":70918773},{"type":"v1","addr":"172.21.15.3:6801","nonce":70918773}]},"active_addr":"172.21.15.3:6801/70918773","active_change":"2024-09-13T14:20:33.329993+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14442,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":53,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":1118901319}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3529413292}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2893689034}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":1863609487}]}]} 2024-09-13T14:20:51.900 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:20:51.900 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:20:51.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:20:52.155 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:52.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:52 smithi003 ceph-mon[41592]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s wr, 0 op/s 2024-09-13T14:20:52.605 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:52 smithi003 ceph-mon[29646]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s wr, 0 op/s 2024-09-13T14:20:52.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:52 smithi019 ceph-mon[32895]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s wr, 0 op/s 2024-09-13T14:20:54.018 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:20:54.018 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":60,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:15:00.204452+0000","modified":"2024-09-13T14:20:50.254579+0000","last_up_change":"2024-09-13T14:20:00.222071+0000","last_in_change":"2024-09-13T14:19:42.900010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:18:02.574675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:20:35.870236+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a153ec29-6e01-4063-aa90-01d1de00e53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6803","nonce":3028312307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6805","nonce":3028312307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6809","nonce":3028312307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6807","nonce":3028312307}]},"public_addr":"172.21.15.3:6803/3028312307","cluster_addr":"172.21.15.3:6805/3028312307","heartbeat_back_addr":"172.21.15.3:6809/3028312307","heartbeat_front_addr":"172.21.15.3:6807/3028312307","state":["exists","up"]},{"osd":1,"uuid":"9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6811","nonce":2008999734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6813","nonce":2008999734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6817","nonce":2008999734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6815","nonce":2008999734}]},"public_addr":"172.21.15.3:6811/2008999734","cluster_addr":"172.21.15.3:6813/2008999734","heartbeat_back_addr":"172.21.15.3:6817/2008999734","heartbeat_front_addr":"172.21.15.3:6815/2008999734","state":["exists","up"]},{"osd":2,"uuid":"676ec1e5-071c-47ba-86ae-2193f448fe8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6819","nonce":2444149284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6821","nonce":2444149284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6825","nonce":2444149284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6823","nonce":2444149284}]},"public_addr":"172.21.15.3:6819/2444149284","cluster_addr":"172.21.15.3:6821/2444149284","heartbeat_back_addr":"172.21.15.3:6825/2444149284","heartbeat_front_addr":"172.21.15.3:6823/2444149284","state":["exists","up"]},{"osd":3,"uuid":"7ec0ad16-0df3-4319-b475-72cfaaa5edae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6827","nonce":2595190501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6829","nonce":2595190501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6833","nonce":2595190501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6831","nonce":2595190501}]},"public_addr":"172.21.15.3:6827/2595190501","cluster_addr":"172.21.15.3:6829/2595190501","heartbeat_back_addr":"172.21.15.3:6833/2595190501","heartbeat_front_addr":"172.21.15.3:6831/2595190501","state":["exists","up"]},{"osd":4,"uuid":"0ef54d09-4000-4695-942b-1b1cf9659983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6801","nonce":4204660131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6803","nonce":4204660131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6807","nonce":4204660131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6805","nonce":4204660131}]},"public_addr":"172.21.15.19:6801/4204660131","cluster_addr":"172.21.15.19:6803/4204660131","heartbeat_back_addr":"172.21.15.19:6807/4204660131","heartbeat_front_addr":"172.21.15.19:6805/4204660131","state":["exists","up"]},{"osd":5,"uuid":"6dfd8fce-4d3d-4a8b-90ac-556447045378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6809","nonce":3591244104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6811","nonce":3591244104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6815","nonce":3591244104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6813","nonce":3591244104}]},"public_addr":"172.21.15.19:6809/3591244104","cluster_addr":"172.21.15.19:6811/3591244104","heartbeat_back_addr":"172.21.15.19:6815/3591244104","heartbeat_front_addr":"172.21.15.19:6813/3591244104","state":["exists","up"]},{"osd":6,"uuid":"17e27318-c0ce-413d-b660-d756731ea743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6817","nonce":2162145215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6819","nonce":2162145215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6823","nonce":2162145215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6821","nonce":2162145215}]},"public_addr":"172.21.15.19:6817/2162145215","cluster_addr":"172.21.15.19:6819/2162145215","heartbeat_back_addr":"172.21.15.19:6823/2162145215","heartbeat_front_addr":"172.21.15.19:6821/2162145215","state":["exists","up"]},{"osd":7,"uuid":"1edab2d1-dad5-45c1-977c-6911541c282e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6825","nonce":2814206175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6827","nonce":2814206175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6831","nonce":2814206175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6829","nonce":2814206175}]},"public_addr":"172.21.15.19:6825/2814206175","cluster_addr":"172.21.15.19:6827/2814206175","heartbeat_back_addr":"172.21.15.19:6831/2814206175","heartbeat_front_addr":"172.21.15.19:6829/2814206175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:11.323986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:34.429745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:59.523592+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:24.423713+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:46.452261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:10.483064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:34.315912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:58.203878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6800/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/1688297948":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/738145194":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2497820386":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/3505363388":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6800/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6801/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/253916983":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/1018276975":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6801/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2723658197":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2346548909":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/3631040762":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6800/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/3615653824":"2024-09-14T14:20:33.329796+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:20:54.607 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:54 smithi003 ceph-mon[41592]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:20:54.608 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:54 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:54.608 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:54 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/4051511266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:20:54.608 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:54 smithi003 ceph-mon[29646]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:20:54.608 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:54 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:54.608 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:54 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/4051511266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:20:54.633 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:20:54.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:20:54.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:54 smithi019 ceph-mon[32895]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:20:54.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:54 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:20:54.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:54 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/4051511266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:20:54.888 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:56.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:56 smithi003 ceph-mon[29646]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:56.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:56 smithi003 ceph-mon[41592]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:56 smithi019 ceph-mon[32895]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:56.721 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:20:56.721 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":60,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:15:00.204452+0000","modified":"2024-09-13T14:20:50.254579+0000","last_up_change":"2024-09-13T14:20:00.222071+0000","last_in_change":"2024-09-13T14:19:42.900010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:18:02.574675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:20:35.870236+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a153ec29-6e01-4063-aa90-01d1de00e53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6803","nonce":3028312307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6805","nonce":3028312307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6809","nonce":3028312307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6807","nonce":3028312307}]},"public_addr":"172.21.15.3:6803/3028312307","cluster_addr":"172.21.15.3:6805/3028312307","heartbeat_back_addr":"172.21.15.3:6809/3028312307","heartbeat_front_addr":"172.21.15.3:6807/3028312307","state":["exists","up"]},{"osd":1,"uuid":"9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6811","nonce":2008999734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6813","nonce":2008999734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6817","nonce":2008999734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6815","nonce":2008999734}]},"public_addr":"172.21.15.3:6811/2008999734","cluster_addr":"172.21.15.3:6813/2008999734","heartbeat_back_addr":"172.21.15.3:6817/2008999734","heartbeat_front_addr":"172.21.15.3:6815/2008999734","state":["exists","up"]},{"osd":2,"uuid":"676ec1e5-071c-47ba-86ae-2193f448fe8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6819","nonce":2444149284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6821","nonce":2444149284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6825","nonce":2444149284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6823","nonce":2444149284}]},"public_addr":"172.21.15.3:6819/2444149284","cluster_addr":"172.21.15.3:6821/2444149284","heartbeat_back_addr":"172.21.15.3:6825/2444149284","heartbeat_front_addr":"172.21.15.3:6823/2444149284","state":["exists","up"]},{"osd":3,"uuid":"7ec0ad16-0df3-4319-b475-72cfaaa5edae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6827","nonce":2595190501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6829","nonce":2595190501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6833","nonce":2595190501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6831","nonce":2595190501}]},"public_addr":"172.21.15.3:6827/2595190501","cluster_addr":"172.21.15.3:6829/2595190501","heartbeat_back_addr":"172.21.15.3:6833/2595190501","heartbeat_front_addr":"172.21.15.3:6831/2595190501","state":["exists","up"]},{"osd":4,"uuid":"0ef54d09-4000-4695-942b-1b1cf9659983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6801","nonce":4204660131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6803","nonce":4204660131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6807","nonce":4204660131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6805","nonce":4204660131}]},"public_addr":"172.21.15.19:6801/4204660131","cluster_addr":"172.21.15.19:6803/4204660131","heartbeat_back_addr":"172.21.15.19:6807/4204660131","heartbeat_front_addr":"172.21.15.19:6805/4204660131","state":["exists","up"]},{"osd":5,"uuid":"6dfd8fce-4d3d-4a8b-90ac-556447045378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6809","nonce":3591244104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6811","nonce":3591244104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6815","nonce":3591244104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6813","nonce":3591244104}]},"public_addr":"172.21.15.19:6809/3591244104","cluster_addr":"172.21.15.19:6811/3591244104","heartbeat_back_addr":"172.21.15.19:6815/3591244104","heartbeat_front_addr":"172.21.15.19:6813/3591244104","state":["exists","up"]},{"osd":6,"uuid":"17e27318-c0ce-413d-b660-d756731ea743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6817","nonce":2162145215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6819","nonce":2162145215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6823","nonce":2162145215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6821","nonce":2162145215}]},"public_addr":"172.21.15.19:6817/2162145215","cluster_addr":"172.21.15.19:6819/2162145215","heartbeat_back_addr":"172.21.15.19:6823/2162145215","heartbeat_front_addr":"172.21.15.19:6821/2162145215","state":["exists","up"]},{"osd":7,"uuid":"1edab2d1-dad5-45c1-977c-6911541c282e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6825","nonce":2814206175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6827","nonce":2814206175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6831","nonce":2814206175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6829","nonce":2814206175}]},"public_addr":"172.21.15.19:6825/2814206175","cluster_addr":"172.21.15.19:6827/2814206175","heartbeat_back_addr":"172.21.15.19:6831/2814206175","heartbeat_front_addr":"172.21.15.19:6829/2814206175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:11.323986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:34.429745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:59.523592+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:24.423713+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:46.452261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:10.483064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:34.315912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:58.203878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6800/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/1688297948":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/738145194":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2497820386":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/3505363388":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6800/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6801/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/253916983":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/1018276975":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6801/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2723658197":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2346548909":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/3631040762":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6800/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/3615653824":"2024-09-14T14:20:33.329796+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:20:57.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:20:57.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:20:57.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:20:57.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:20:57.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:20:57.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:20:57.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:20:57.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:20:57.674 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:57 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3096121222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:20:57.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:57 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3096121222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:20:57.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:57 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3096121222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:20:57.820 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:57.857 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:57.859 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:57.860 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:57.861 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:58.041 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:58.041 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:58.041 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:58.162 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:20:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:20:57.883Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004438191s 2024-09-13T14:20:58.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:20:58 smithi019 ceph-mon[32895]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:58.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:20:58 smithi003 ceph-mon[41592]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:58.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:20:58 smithi003 ceph-mon[29646]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:00.423 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 podman[63953]: 2024-09-13 14:21:00.321492734 +0000 UTC m=+0.120991214 container create dc631f7cf889e8943d8dde5e015a29858b440bbed0b09215a3f0a632fecd8c60 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, distribution-scope=public, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.expose-services=, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, release=239.1651231664) 2024-09-13T14:21:00.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-mon[32895]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:00.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 podman[63953]: 2024-09-13 14:21:00.540569385 +0000 UTC m=+0.340067823 container init dc631f7cf889e8943d8dde5e015a29858b440bbed0b09215a3f0a632fecd8c60 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.openshift.expose-services=, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-09-13T14:21:00.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 podman[63953]: 2024-09-13 14:21:00.545480548 +0000 UTC m=+0.344978987 container start dc631f7cf889e8943d8dde5e015a29858b440bbed0b09215a3f0a632fecd8c60 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, build-date=2022-04-29T11:48:11.055897, name=ubi8, version=8.5, vcs-type=git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-09-13T14:21:00.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 bash[63953]: dc631f7cf889e8943d8dde5e015a29858b440bbed0b09215a3f0a632fecd8c60 2024-09-13T14:21:00.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 systemd[1]: Started Ceph grafana.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:21:00.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:00 smithi003 ceph-mon[29646]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:00.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:00 smithi003 ceph-mon[41592]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:00.960 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:21:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:00.961 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710204452Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-13T14:21:00.961 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710399074Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-13T14:21:00.961 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710426794Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T14:21:00.961 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710441355Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T14:21:00.961 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710453848Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T14:21:00.961 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710466018Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710477138Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710488287Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710500385Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710512108Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710522956Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710533869Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710545772Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=settings t=2024-09-13T14:21:00.710557768Z level=info msg="App mode production" 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=sqlstore t=2024-09-13T14:21:00.710629607Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=sqlstore t=2024-09-13T14:21:00.710658123Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.711858697Z level=info msg="Starting DB migrations" 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.71196437Z level=info msg="Executing migration" id="create migration_log table" 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.712530519Z level=info msg="Executing migration" id="create user table" 2024-09-13T14:21:00.962 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.713024635Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.715891475Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.718927578Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.721526136Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.721988014Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.725022109Z level=info msg="Executing migration" id="create user table v2" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.727831971Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.730474448Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.730987582Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.733564749Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.73650538Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.739356238Z level=info msg="Executing migration" id="Update user table charset" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.742157371Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.745009024Z level=info msg="Executing migration" id="Add missing user data" 2024-09-13T14:21:00.963 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.74767964Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-13T14:21:00.964 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.750519761Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-13T14:21:00.964 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.753040495Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-13T14:21:00.964 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.755829274Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-13T14:21:00.964 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.758712483Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-13T14:21:00.965 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.761404429Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-13T14:21:00.965 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.764080279Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-13T14:21:00.965 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.766807204Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-13T14:21:00.965 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.76954777Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-13T14:21:00.965 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.772213346Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-13T14:21:00.965 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.774986762Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-13T14:21:00.965 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.775411778Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.776453533Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.776956669Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.777369021Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.778325982Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.778740852Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.779849096Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.78062771Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.78151554Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.781944359Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.782868247Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-13T14:21:00.966 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.783757957Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.784843928Z level=info msg="Executing migration" id="create star table" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.785581375Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.786511292Z level=info msg="Executing migration" id="create org table v1" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.787446688Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.787874254Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.788269904Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.788688755Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.789123363Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.789534381Z level=info msg="Executing migration" id="Update org table charset" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.789804769Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.790113471Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.790442851Z level=info msg="Executing migration" id="create dashboard table" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.790815084Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-13T14:21:00.967 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.791237544Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.791649339Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.792046513Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.794795244Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.797496138Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.800150371Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.802925597Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.805678554Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.808448389Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.811031018Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.811487599Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.811795326Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.814160089Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-13T14:21:00.968 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.817003595Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.819543111Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.820371941Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.821273932Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.822045791Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.822847691Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.823704447Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.824127962Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.824691634Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.825527578Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.826379393Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.827214079Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.828058956Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.828956167Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-13T14:21:00.969 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.829881448Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.83065123Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.831544622Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.832362208Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.833330216Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.834186833Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.835182014Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.835904785Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.836859123Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.83759868Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.838577681Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.839333066Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.840211623Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.841084061Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.842092672Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-13T14:21:00.970 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.842804371Z level=info msg="Executing migration" id="create data_source table" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.843797837Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.844350667Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.844792312Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.845632926Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.846636772Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.847810904Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.848620801Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.849587032Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.850315569Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.85127481Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.853858756Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.856907047Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.859949013Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-13T14:21:00.971 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.862974156Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.865750036Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.868428385Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.871210475Z level=info msg="Executing migration" id="Add uid column" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.874217529Z level=info msg="Executing migration" id="Update uid value" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.877105769Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.879916647Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.882706248Z level=info msg="Executing migration" id="create api_key table" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.885665827Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.888678643Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.891487925Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.894214096Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.897135716Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-13T14:21:00.972 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.899730494Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.902667301Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.905444311Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.905877768Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.908854032Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.911329308Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.914193468Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.916709394Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.917173016Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.919921101Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.922748591Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.925209625Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.928037108Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.930846076Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-13T14:21:00.973 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.933413222Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.936213036Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.939066992Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.941730578Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.944373154Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.947018282Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.947863426Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.948861171Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.949535337Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.950316752Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.951152723Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.952100073Z level=info msg="Executing migration" id="create quota table v1" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.952773203Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.953765371Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-13T14:21:00.974 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.954442653Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.954864402Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.95570712Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.956737717Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.957451967Z level=info msg="Executing migration" id="create session table" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.957933474Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.958281658Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.958581Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.959000025Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.959964095Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.960592523Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.960929678Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.961280205Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.961667023Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-13T14:21:00.975 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.962116042Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-13T14:21:00.976 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.962946919Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-13T14:21:00.976 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.963753413Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-13T14:21:00.976 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.964158966Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-13T14:21:01.214 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.964947534Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.965781937Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.966594497Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.967132608Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.968025022Z level=info msg="Executing migration" id="create alert table v1" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.968835466Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.969698691Z level=info msg="Executing migration" id="add index alert state" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.970234283Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.971159287Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.972091215Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.97296965Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.973810232Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.975333861Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-13T14:21:01.215 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.97624235Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.977097609Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.978184702Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.978938323Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.979843074Z level=info msg="Executing migration" id="Add column is_default" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.980791971Z level=info msg="Executing migration" id="Add column frequency" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.981785293Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.982593155Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.983548923Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.984600123Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.98554904Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.986502128Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.987297251Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-13T14:21:01.216 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.988227795Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.989137783Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.990216882Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.99099196Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.991943227Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.99299802Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.993790324Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.994970125Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.995658756Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:00.998715711Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.001620952Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-13T14:21:01.217 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.004401671Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-13T14:21:01.218 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.007515103Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-13T14:21:01.218 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.010613583Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.013455536Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.016492229Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.019473216Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.022613822Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.025553401Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.028539955Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.031231879Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.034186792Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.037070357Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.040233675Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.043050785Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.046030413Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.04882964Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-13T14:21:01.219 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.05156847Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.054290758Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.056984219Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.060030144Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.062859782Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.065654439Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.068300804Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.071031616Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.073927519Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.076731321Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.079793767Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.0823855Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.085288772Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.088136083Z level=info msg="Executing migration" id="Move region to single row" 2024-09-13T14:21:01.220 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.090873246Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.093644283Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.096331634Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.097183527Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.098075879Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.099051193Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.099882044Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.100939878Z level=info msg="Executing migration" id="create test_data table" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.101615399Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.10261063Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.103389085Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.104417621Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.105093036Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.106114526Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-13T14:21:01.221 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.106931159Z level=info msg="Executing migration" id="create team table" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.107930361Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.108606725Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.109441418Z level=info msg="Executing migration" id="create team member table" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.110319529Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.111222043Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.112105678Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.113039706Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.113934581Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.114865269Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.115736954Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.116803283Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.117647572Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.118594837Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-13T14:21:01.222 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.119530627Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.120438329Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.121385571Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.122280444Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.122756647Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.123672895Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.124534907Z level=info msg="Executing migration" id="create tag table" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.125486647Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.125972307Z level=info msg="Executing migration" id="create login attempt table" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.128970216Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.132001134Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.134977289Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.136923556Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.140069792Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-13T14:21:01.223 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.14309055Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.146074775Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.146524965Z level=info msg="Executing migration" id="create user auth table" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.149424611Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.149883835Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.152459521Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.155406858Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.156180798Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.159197027Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.162318609Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.165196078Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.166006424Z level=info msg="Executing migration" id="create server_lock table" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.16900984Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.171544426Z level=info msg="Executing migration" id="create user auth token table" 2024-09-13T14:21:01.224 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.174339259Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.17497117Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.177967463Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.180840181Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.183482377Z level=info msg="Executing migration" id="create cache_data table" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.184017995Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.187005845Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.189838907Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.19237455Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.192716834Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.195525448Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.198351815Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.201074239Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.201545654Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-13T14:21:01.225 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.201847036Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.202305868Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.202717646Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.203182183Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.203633977Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.20446064Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.204901229Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.205231201Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.205666965Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.206129223Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.206571719Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.206864839Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.207321323Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.207930195Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.208384247Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-13T14:21:01.226 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.208833491Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-13T14:21:01.227 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.209653231Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-13T14:21:01.227 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.210106999Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-13T14:21:01.227 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.210549026Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-13T14:21:01.227 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.214060546Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-13T14:21:01.227 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.217545613Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.218126324Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.218580785Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.219455435Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.21990985Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.22037661Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.220826692Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.221321021Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.221619032Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.222466428Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-13T14:21:01.470 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.223313524Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.224160504Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.224623801Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.225090396Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.225959914Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.226817078Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.227310117Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.228168265Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.229003531Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.229368945Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.229874414Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.23042142Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.230911489Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-13T14:21:01.471 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.231255173Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.232149192Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.233005022Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.233880003Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.234776512Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.235653883Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.23598777Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.236415622Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.237296778Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.237647904Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.238599683Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.241441487Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.244241474Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-13T14:21:01.472 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.247179131Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-13T14:21:01.473 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.247675291Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.250637227Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.2536394Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.256238035Z level=info msg="Executing migration" id="create alert_image table" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.256685712Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.259659946Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.262589053Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.263152765Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.263620461Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.263710249Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.266626427Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.267192719Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-13T14:21:01.474 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.269412409Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.269550526Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.269678639Z level=info msg="alerts found to migrate" alerts=0 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.272263346Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.27278144Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.275786855Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.27822385Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.281106461Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.281661435Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.284371641Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.284800092Z level=info msg="Executing migration" id="create data_keys table" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.287187607Z level=info msg="Executing migration" id="create secrets table" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.289979858Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.294383109Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.297179694Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-13T14:21:01.475 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.299968034Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.304335873Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.308681375Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.309591212Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.310431719Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.311188686Z level=info msg="Executing migration" id="create permission table" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.311637878Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.312420142Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.313410102Z level=info msg="Executing migration" id="create role table" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.314112871Z level=info msg="Executing migration" id="add column display_name" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.315104933Z level=info msg="Executing migration" id="add column group_name" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.31606174Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.316917095Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-13T14:21:01.476 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.317785468Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.31826944Z level=info msg="Executing migration" id="create team role table" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.319074034Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.319899294Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.320724748Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.321219422Z level=info msg="Executing migration" id="create user role table" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.322033695Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.322876194Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.323723586Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.324198762Z level=info msg="Executing migration" id="create builtin role table" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.325083867Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.325948275Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.326790108Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.327803834Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-13T14:21:01.477 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.328357745Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.329125655Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.329611217Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.330124673Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.330915716Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.331438946Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.332477015Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.332929154Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.333433555Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.333739423Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.334203456Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.334720555Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.335162214Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.335528401Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-13T14:21:01.478 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.335914607Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.336377717Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.336879313Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.337917892Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.338288643Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.338778947Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.339285411Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.339782492Z level=info msg="Executing migration" id="add correlation config column" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.340824789Z level=info msg="Executing migration" id="create entity_events table" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.341324963Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.341801218Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.341922947Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.342281924Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:21:01.479 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.342384443Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.342697116Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.34315636Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.343611367Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.344109054Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.346969654Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.349692734Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.352308645Z level=info msg="Executing migration" id="Drop public config table" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.352768654Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.353753708Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.354488011Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.35528147Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.355784224Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-13T14:21:01.480 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.360007799Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.361068161Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.362129142Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.363222363Z level=info msg="Executing migration" id="add share column" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.364325092Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.365241709Z level=info msg="Executing migration" id="create file table" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.366160702Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.367028584Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.367553984Z level=info msg="Executing migration" id="create file_meta table" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.368537759Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.369260747Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.370123931Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.370571663Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.371393003Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-13T14:21:01.481 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.371900255Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.372967034Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.374022642Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.374520677Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.375447357Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.375823721Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.376401844Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.37746779Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.378542694Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.379461558Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.389510714Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.390428279Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.390929894Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.394438388Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-13T14:21:01.482 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.395321126Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-13T14:21:01.483 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.396278501Z level=info msg="Executing migration" id="create folder table" 2024-09-13T14:21:01.483 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.39673612Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-13T14:21:01.483 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.399877879Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-13T14:21:01.483 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.402781683Z level=info msg="Executing migration" id="Update folder title length" 2024-09-13T14:21:01.483 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.405709937Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-13T14:21:01.483 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=migrator t=2024-09-13T14:21:01.406290895Z level=info msg="migrations completed" performed=485 skipped=0 duration=694.356157ms 2024-09-13T14:21:01.483 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=sqlstore t=2024-09-13T14:21:01.406721353Z level=info msg="Created default organization" 2024-09-13T14:21:01.483 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=plugin.loader t=2024-09-13T14:21:01.469329981Z level=info msg="Plugin registered" pluginID=input 2024-09-13T14:21:01.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-mon[32895]: Deploying daemon node-exporter.a on smithi003 2024-09-13T14:21:01.932 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=plugin.loader t=2024-09-13T14:21:01.505663846Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-13T14:21:01.932 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=plugin.loader t=2024-09-13T14:21:01.505685169Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-13T14:21:01.932 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=secrets t=2024-09-13T14:21:01.505748114Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T14:21:01.932 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=query_data t=2024-09-13T14:21:01.507484102Z level=info msg="Query Service initialization" 2024-09-13T14:21:01.932 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=live.push_http t=2024-09-13T14:21:01.509785631Z level=info msg="Live Push Gateway initialization" 2024-09-13T14:21:01.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[41592]: Deploying daemon node-exporter.a on smithi003 2024-09-13T14:21:01.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:01.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:01 smithi003 ceph-mon[29646]: Deploying daemon node-exporter.a on smithi003 2024-09-13T14:21:02.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=infra.usagestats.collector t=2024-09-13T14:21:02.277949023Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T14:21:02.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=provisioning.datasources t=2024-09-13T14:21:02.278244127Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-13T14:21:02.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=provisioning.datasources t=2024-09-13T14:21:02.309650149Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T14:21:02.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=provisioning.datasources t=2024-09-13T14:21:02.342274627Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-13T14:21:02.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=provisioning.alerting t=2024-09-13T14:21:02.36453289Z level=info msg="starting to provision alerting" 2024-09-13T14:21:02.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=provisioning.alerting t=2024-09-13T14:21:02.364553335Z level=info msg="finished to provision alerting" 2024-09-13T14:21:02.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=grafanaStorageLogger t=2024-09-13T14:21:02.364687378Z level=info msg="storage starting" 2024-09-13T14:21:02.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=ngalert.state.manager t=2024-09-13T14:21:02.364735419Z level=info msg="Warming state cache for startup" 2024-09-13T14:21:02.683 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=http.server t=2024-09-13T14:21:02.369982826Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T14:21:02.683 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=sqlstore.transactions t=2024-09-13T14:21:02.375013677Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:21:02.683 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=ngalert.state.manager t=2024-09-13T14:21:02.375208493Z level=info msg="State cache has been initialized" states=0 duration=10.472634ms 2024-09-13T14:21:02.683 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=ticker t=2024-09-13T14:21:02.375263452Z level=info msg=starting first_tick=2024-09-13T14:21:10Z 2024-09-13T14:21:02.683 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=ngalert.multiorg.alertmanager t=2024-09-13T14:21:02.375273413Z level=info msg="starting MultiOrg Alertmanager" 2024-09-13T14:21:02.683 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[63968]: logger=sqlstore.transactions t=2024-09-13T14:21:02.42104119Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:21:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:02 smithi003 ceph-mon[41592]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:02.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:02 smithi003 ceph-mon[29646]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:02 smithi019 ceph-mon[32895]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:03.390 INFO:teuthology.orchestra.run.smithi003.stdout:188978561044 2024-09-13T14:21:03.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:21:03.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:03 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:03.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:03 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:03.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:03 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:03.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:03 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:03 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:03 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:04 smithi003 ceph-mon[41592]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:04.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:04 smithi003 ceph-mon[29646]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:05.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:04 smithi019 ceph-mon[32895]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:06 smithi003 ceph-mon[29646]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:06 smithi003 ceph-mon[41592]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:07.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:06 smithi019 ceph-mon[32895]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:08.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:08 smithi003 ceph-mon[41592]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:08.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:08 smithi003 ceph-mon[29646]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:09.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:08 smithi019 ceph-mon[32895]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:10.633 INFO:teuthology.orchestra.run.smithi003.stdout:111669149732 2024-09-13T14:21:10.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:21:10.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:10 smithi003 ceph-mon[41592]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:10.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:10 smithi003 ceph-mon[29646]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:10 smithi019 ceph-mon[32895]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:11.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:21:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:21:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:11.902 INFO:teuthology.orchestra.run.smithi003.stdout:55834574893 2024-09-13T14:21:11.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:21:12.339 INFO:teuthology.orchestra.run.smithi003.stdout:214748364816 2024-09-13T14:21:12.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:21:12.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:12 smithi003 ceph-mon[29646]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:12.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:12 smithi003 ceph-mon[41592]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:12.991 INFO:teuthology.orchestra.run.smithi003.stdout:133143986206 2024-09-13T14:21:12.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:21:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:13.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:12 smithi019 ceph-mon[32895]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:13.657 INFO:teuthology.orchestra.run.smithi003.stdout:77309411369 2024-09-13T14:21:13.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:21:14.768 INFO:teuthology.orchestra.run.smithi003.stdout:34359738418 2024-09-13T14:21:14.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:21:14.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:14 smithi019 ceph-mon[32895]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:14.994 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:14 smithi003 ceph-mon[41592]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:14.994 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:14 smithi003 ceph-mon[29646]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:16.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:16 smithi003 ceph-mon[41592]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:16 smithi003 ceph-mon[29646]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:16.946 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:16.947 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:16.947 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:16.980 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:16.983 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:16.983 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:16 smithi019 ceph-mon[32895]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:18.000 INFO:teuthology.orchestra.run.smithi003.stdout:188978561047 2024-09-13T14:21:18.334 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:18 smithi003 bash[74767]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-13T14:21:18.334 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:18 smithi003 bash[74767]: Writing manifest to image destination 2024-09-13T14:21:18.932 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:18 smithi003 podman[74767]: 2024-09-13 14:21:18.835162271 +0000 UTC m=+5.861267371 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-13T14:21:18.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:18 smithi003 ceph-mon[29646]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:18.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:18 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3352359335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:21:18.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:18 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:18 smithi003 ceph-mon[41592]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:18 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3352359335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:21:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:18 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:18 smithi019 ceph-mon[32895]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:18 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3352359335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:21:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:18 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:20.391 INFO:teuthology.orchestra.run.smithi003.stdout:158913789979 2024-09-13T14:21:20.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:21:20.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:20 smithi003 ceph-mon[29646]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:20.991 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:20 smithi003 podman[74767]: 2024-09-13 14:21:20.795999443 +0000 UTC m=+7.822104542 container create a786edb7f0b5897fa412ad4fa9aa841e3e40b2df9900f65dacdf5e6b32239700 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:20.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:20 smithi003 ceph-mon[41592]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:21.014 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561044 got 188978561047 for osd.6 2024-09-13T14:21:21.014 DEBUG:teuthology.parallel:result is None 2024-09-13T14:21:21.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:21:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:21:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:20 smithi019 ceph-mon[32895]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:21.336 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 podman[74767]: 2024-09-13 14:21:21.090277858 +0000 UTC m=+8.116382957 container init a786edb7f0b5897fa412ad4fa9aa841e3e40b2df9900f65dacdf5e6b32239700 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:21.336 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 podman[74767]: 2024-09-13 14:21:21.09541086 +0000 UTC m=+8.121515962 container start a786edb7f0b5897fa412ad4fa9aa841e3e40b2df9900f65dacdf5e6b32239700 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:21.336 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.099Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-13T14:21:21.336 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.099Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-13T14:21:21.336 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.100Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:21:21.336 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.100Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.100Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.100Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.100Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:21:21.337 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:21:21.338 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:21:21.339 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[75560]: ts=2024-09-13T14:21:21.101Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 bash[74767]: a786edb7f0b5897fa412ad4fa9aa841e3e40b2df9900f65dacdf5e6b32239700 2024-09-13T14:21:21.340 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:21:21 smithi003 systemd[1]: Started Ceph node-exporter.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:21:22.248 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:22 smithi019 bash[64250]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-13T14:21:22.504 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:22 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:22.504 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:22 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:22.504 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:22 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:22.504 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:22 smithi019 ceph-mon[32895]: Deploying daemon node-exporter.b on smithi019 2024-09-13T14:21:22.504 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:22 smithi019 ceph-mon[32895]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:22.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:22.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:22.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:22.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[41592]: Deploying daemon node-exporter.b on smithi019 2024-09-13T14:21:22.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[41592]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:22.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:22.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:22.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:22.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[29646]: Deploying daemon node-exporter.b on smithi019 2024-09-13T14:21:22.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:22 smithi003 ceph-mon[29646]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:22.931 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:22 smithi019 bash[64250]: Getting image source signatures 2024-09-13T14:21:22.931 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:22 smithi019 bash[64250]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-13T14:21:22.931 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:22 smithi019 bash[64250]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-13T14:21:22.932 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:22 smithi019 bash[64250]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-13T14:21:24.181 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:23 smithi019 bash[64250]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-13T14:21:24.182 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:23 smithi019 bash[64250]: Writing manifest to image destination 2024-09-13T14:21:24.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:24 smithi003 ceph-mon[41592]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:24.552 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:24 smithi003 ceph-mon[29646]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:24.623 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-mon[32895]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:24.623 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 podman[64250]: 2024-09-13 14:21:24.255321501 +0000 UTC m=+2.186422208 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-13T14:21:24.623 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 podman[64250]: 2024-09-13 14:21:24.353234228 +0000 UTC m=+2.284334931 container create e22916e85bdc446b204f9cc8828482adca626c8d757b245a70a3ab7a571db547 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:21:24.932 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 podman[64250]: 2024-09-13 14:21:24.622703954 +0000 UTC m=+2.553804649 container init e22916e85bdc446b204f9cc8828482adca626c8d757b245a70a3ab7a571db547 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:21:24.932 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 podman[64250]: 2024-09-13 14:21:24.628455218 +0000 UTC m=+2.559555921 container start e22916e85bdc446b204f9cc8828482adca626c8d757b245a70a3ab7a571db547 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:21:24.932 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.632Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-13T14:21:24.932 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.632Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-13T14:21:24.932 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.633Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:21:24.932 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.633Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.633Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.633Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:21:24.933 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:21:24.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:21:24.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:21:24.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:21:24.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:21:24.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[64326]: ts=2024-09-13T14:21:24.634Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:21:24.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 bash[64250]: e22916e85bdc446b204f9cc8828482adca626c8d757b245a70a3ab7a571db547 2024-09-13T14:21:24.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:21:24 smithi019 systemd[1]: Started Ceph node-exporter.b for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:21:25.557 INFO:teuthology.orchestra.run.smithi003.stdout:55834574895 2024-09-13T14:21:26.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:21:26.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1278895757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:21:26.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:21:26.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:25 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1278895757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:21:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:25 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:25 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:25 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:25 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:25 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:25 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:21:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:25 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1278895757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:21:27.048 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:26 smithi003 ceph-mon[41592]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:27.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:26 smithi003 ceph-mon[29646]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:27.050 INFO:teuthology.orchestra.run.smithi003.stdout:34359738421 2024-09-13T14:21:27.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:26 smithi019 ceph-mon[32895]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:27.829 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:27 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1059115089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:21:27.829 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:27 smithi003 ceph-mon[41592]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:27.829 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:27 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1059115089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:21:27.829 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:27 smithi003 ceph-mon[29646]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:27.830 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738421 for osd.0 2024-09-13T14:21:27.830 DEBUG:teuthology.parallel:result is None 2024-09-13T14:21:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:27 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1059115089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:21:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:27 smithi019 ceph-mon[32895]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:28.215 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574895 for osd.1 2024-09-13T14:21:28.216 DEBUG:teuthology.parallel:result is None 2024-09-13T14:21:28.557 INFO:teuthology.orchestra.run.smithi003.stdout:77309411372 2024-09-13T14:21:29.323 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411372 for osd.2 2024-09-13T14:21:29.323 DEBUG:teuthology.parallel:result is None 2024-09-13T14:21:29.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:29 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:29.591 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:29 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2250170069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:21:29.591 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:29 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:29.591 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:29 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2250170069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:21:29.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:29 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:29 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2250170069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:21:29.845 INFO:teuthology.orchestra.run.smithi003.stdout:214748364820 2024-09-13T14:21:30.529 INFO:teuthology.orchestra.run.smithi003.stdout:133143986210 2024-09-13T14:21:30.529 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:30 smithi003 ceph-mon[41592]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:30.529 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:30 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2407835234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:21:30.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:30 smithi003 ceph-mon[29646]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:30.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:30 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2407835234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:21:30.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:30 smithi019 ceph-mon[32895]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:30.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:30 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2407835234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:21:30.756 INFO:teuthology.orchestra.run.smithi003.stdout:111669149736 2024-09-13T14:21:31.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:21:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:21:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:31.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364816 got 214748364820 for osd.7 2024-09-13T14:21:31.305 DEBUG:teuthology.parallel:result is None 2024-09-13T14:21:31.645 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:31 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2627260272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:21:31.645 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:31 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2048954763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:21:31.646 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:31 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2627260272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:21:31.646 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:31 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2048954763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:21:31.680 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986206 got 133143986210 for osd.4 2024-09-13T14:21:31.681 DEBUG:teuthology.parallel:result is None 2024-09-13T14:21:31.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:31 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2627260272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:21:31.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:31 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2048954763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:21:31.682 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:32.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:32 smithi019 ceph-mon[32895]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:32 smithi003 ceph-mon[41592]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:32.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:32 smithi003 ceph-mon[29646]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:33.271 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149732 got 111669149736 for osd.3 2024-09-13T14:21:33.271 DEBUG:teuthology.parallel:result is None 2024-09-13T14:21:33.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:33.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:33.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:34.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:34 smithi019 ceph-mon[32895]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:34.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:34 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:34.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:34 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:34.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:34 smithi003 ceph-mon[41592]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:34.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:34 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:34.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:34 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:34.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:34 smithi003 ceph-mon[29646]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:34 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:34 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:34.829 INFO:teuthology.orchestra.run.smithi003.stdout:158913789982 2024-09-13T14:21:35.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:35 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3741510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:21:35.602 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:35 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3741510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:21:35.627 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789979 got 158913789982 for osd.5 2024-09-13T14:21:35.627 DEBUG:teuthology.parallel:result is None 2024-09-13T14:21:35.627 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:21:35.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:21:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:35 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3741510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:21:36.122 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:36.566 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:36 smithi003 ceph-mon[41592]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:36.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:36 smithi003 ceph-mon[29646]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:36.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:36 smithi019 ceph-mon[32895]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:38.464 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:38.465 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:38.465 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[29646]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:38.465 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:38.465 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:21:38.465 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:38.465 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[29646]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:21:38.465 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[29646]: Reconfiguring daemon alertmanager.a on smithi003 2024-09-13T14:21:38.465 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:38.465 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:38.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[41592]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:38.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:38.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:21:38.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:38.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[41592]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:21:38.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:38 smithi003 ceph-mon[41592]: Reconfiguring daemon alertmanager.a on smithi003 2024-09-13T14:21:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:38 smithi019 ceph-mon[32895]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:21:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:38 smithi019 ceph-mon[32895]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:21:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:38 smithi019 ceph-mon[32895]: Reconfiguring daemon alertmanager.a on smithi003 2024-09-13T14:21:40.325 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:21:40.326 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:21:40.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:40 smithi019 ceph-mon[32895]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:40.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:40 smithi003 ceph-mon[41592]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:40.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:40 smithi003 ceph-mon[29646]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:41.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:21:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:21:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:41.412 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:41 smithi003 systemd[1]: Stopping Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:21:41.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:41 smithi019 ceph-mon[32895]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:21:41.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:41 smithi003 ceph-mon[29646]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:21:41.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:41 smithi003 ceph-mon[41592]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:21:41.805 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-13T14:21:39.353241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385256,"kb_used_data":6408,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575960,"statfs":{"total":767960285184,"available":765517783040,"internally_reserved":0,"allocated":6561792,"data_stored":3777158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001616"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.257173+0000","last_change":"2024-09-13T14:20:41.394322+0000","last_active":"2024-09-13T14:20:50.257173+0000","last_peered":"2024-09-13T14:20:50.257173+0000","last_clean":"2024-09-13T14:20:50.257173+0000","last_became_active":"2024-09-13T14:20:37.356874+0000","last_became_peered":"2024-09-13T14:20:37.356874+0000","last_unstale":"2024-09-13T14:20:50.257173+0000","last_undegraded":"2024-09-13T14:20:50.257173+0000","last_fullsized":"2024-09-13T14:20:50.257173+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:05.446185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590447+0000","last_change":"2024-09-13T14:20:41.389271+0000","last_active":"2024-09-13T14:20:50.590447+0000","last_peered":"2024-09-13T14:20:50.590447+0000","last_clean":"2024-09-13T14:20:50.590447+0000","last_became_active":"2024-09-13T14:20:37.355416+0000","last_became_peered":"2024-09-13T14:20:37.355416+0000","last_unstale":"2024-09-13T14:20:50.590447+0000","last_undegraded":"2024-09-13T14:20:50.590447+0000","last_fullsized":"2024-09-13T14:20:50.590447+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:47.155583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026697500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590491+0000","last_change":"2024-09-13T14:20:41.646302+0000","last_active":"2024-09-13T14:20:50.590491+0000","last_peered":"2024-09-13T14:20:50.590491+0000","last_clean":"2024-09-13T14:20:50.590491+0000","last_became_active":"2024-09-13T14:20:37.356803+0000","last_became_peered":"2024-09-13T14:20:37.356803+0000","last_unstale":"2024-09-13T14:20:50.590491+0000","last_undegraded":"2024-09-13T14:20:50.590491+0000","last_fullsized":"2024-09-13T14:20:50.590491+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:41.431722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590342+0000","last_change":"2024-09-13T14:20:41.389300+0000","last_active":"2024-09-13T14:20:50.590342+0000","last_peered":"2024-09-13T14:20:50.590342+0000","last_clean":"2024-09-13T14:20:50.590342+0000","last_became_active":"2024-09-13T14:20:37.356625+0000","last_became_peered":"2024-09-13T14:20:37.356625+0000","last_unstale":"2024-09-13T14:20:50.590342+0000","last_undegraded":"2024-09-13T14:20:50.590342+0000","last_fullsized":"2024-09-13T14:20:50.590342+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:28.847614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":20,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590613+0000","last_change":"2024-09-13T14:20:41.393368+0000","last_active":"2024-09-13T14:20:50.590613+0000","last_peered":"2024-09-13T14:20:50.590613+0000","last_clean":"2024-09-13T14:20:50.590613+0000","last_became_active":"2024-09-13T14:20:37.357807+0000","last_became_peered":"2024-09-13T14:20:37.357807+0000","last_unstale":"2024-09-13T14:20:50.590613+0000","last_undegraded":"2024-09-13T14:20:50.590613+0000","last_fullsized":"2024-09-13T14:20:50.590613+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:17.845640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.257759+0000","last_change":"2024-09-13T14:20:41.645840+0000","last_active":"2024-09-13T14:20:50.257759+0000","last_peered":"2024-09-13T14:20:50.257759+0000","last_clean":"2024-09-13T14:20:50.257759+0000","last_became_active":"2024-09-13T14:20:37.388195+0000","last_became_peered":"2024-09-13T14:20:37.388195+0000","last_unstale":"2024-09-13T14:20:50.257759+0000","last_undegraded":"2024-09-13T14:20:50.257759+0000","last_fullsized":"2024-09-13T14:20:50.257759+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:39.037088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590493+0000","last_change":"2024-09-13T14:20:41.646316+0000","last_active":"2024-09-13T14:20:50.590493+0000","last_peered":"2024-09-13T14:20:50.590493+0000","last_clean":"2024-09-13T14:20:50.590493+0000","last_became_active":"2024-09-13T14:20:37.358143+0000","last_became_peered":"2024-09-13T14:20:37.358143+0000","last_unstale":"2024-09-13T14:20:50.590493+0000","last_undegraded":"2024-09-13T14:20:50.590493+0000","last_fullsized":"2024-09-13T14:20:50.590493+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:37.774534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":19,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590627+0000","last_change":"2024-09-13T14:20:41.392754+0000","last_active":"2024-09-13T14:20:50.590627+0000","last_peered":"2024-09-13T14:20:50.590627+0000","last_clean":"2024-09-13T14:20:50.590627+0000","last_became_active":"2024-09-13T14:20:37.356130+0000","last_became_peered":"2024-09-13T14:20:37.356130+0000","last_unstale":"2024-09-13T14:20:50.590627+0000","last_undegraded":"2024-09-13T14:20:50.590627+0000","last_fullsized":"2024-09-13T14:20:50.590627+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:50.167135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"53'79","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-13T14:20:41.645995+0000","last_change":"2024-09-13T14:20:01.261809+0000","last_active":"2024-09-13T14:20:41.645995+0000","last_peered":"2024-09-13T14:20:41.645995+0000","last_clean":"2024-09-13T14:20:41.645995+0000","last_became_active":"2024-09-13T14:20:01.256039+0000","last_became_peered":"2024-09-13T14:20:01.256039+0000","last_unstale":"2024-09-13T14:20:41.645995+0000","last_undegraded":"2024-09-13T14:20:41.645995+0000","last_fullsized":"2024-09-13T14:20:41.645995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:01.420572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298516,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446636,"statfs":{"total":95995035648,"available":95689355264,"internally_reserved":0,"allocated":1187840,"data_stored":840995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.63900000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.57899999999999996}]},{"osd":3,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.623},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65100000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561052,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298520,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446632,"statfs":{"total":95995035648,"available":95689351168,"internally_reserved":0,"allocated":1191936,"data_stored":840964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.51500000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.57499999999999996}]},{"osd":3,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.71099999999999997}]},{"osd":5,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.73799999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.64600000000000002}]}]},{"osd":5,"up_from":37,"seq":158913789984,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297936,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447216,"statfs":{"total":95995035648,"available":95689949184,"internally_reserved":0,"allocated":593920,"data_stored":250352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.50800000000000001,"15min":0.51700000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.69399999999999995,"5min":0.752,"15min":0.752},"last":0.64000000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76200000000000001,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.38,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.623}]},{"osd":3,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.497,"15min":0.497},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70999999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.35299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.65300000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.40999999999999998,"15min":0.39400000000000002},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.54300000000000004}]},{"osd":6,"last update":"Fri Sep 13 14:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.71699999999999997}]},{"osd":7,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.5},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.69999999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":250892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68100000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54400000000000004,"15min":0.54300000000000004},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.622}]},{"osd":1,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.66000000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71499999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.36699999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.68400000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:21:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.40400000000000003,"15min":0.39200000000000002},"min":{"1min":0.30099999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.74099999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.317,"5min":0.247,"15min":0.247},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.73299999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.57099999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:21:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.76500000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":250892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.36699999999999999,"15min":0.35499999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.50900000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.379},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.39500000000000002,"15min":0.40300000000000002},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.55300000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.27200000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.41299999999999998,"15min":0.42199999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.48699999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.435,"15min":0.437},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59299999999999997,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.32500000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73699999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.438,"5min":0.433,"15min":0.433},"max":{"1min":0.79700000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79700000000000004,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78500000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61499999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58999999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":250892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35299999999999998,"15min":0.34200000000000003},"min":{"1min":0.23000000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.497,"5min":0.623,"15min":0.623},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.36699999999999999,"15min":0.35099999999999998},"min":{"1min":0.26200000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.60299999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38800000000000001,"15min":0.378},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.39200000000000002},"min":{"1min":0.27800000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.59799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.59699999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.435,"15min":0.434},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.63900000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.435,"15min":0.43099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.66500000000000004}]},{"osd":4,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76600000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.745}]},{"osd":5,"last update":"Fri Sep 13 14:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.59699999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81699999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.68899999999999995}]},{"osd":6,"last update":"Fri Sep 13 14:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.68100000000000005}]},{"osd":7,"last update":"Fri Sep 13 14:21:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.755}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297944,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447208,"statfs":{"total":95995035648,"available":95689940992,"internally_reserved":0,"allocated":602112,"data_stored":250911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.35999999999999999,"15min":0.34100000000000003},"min":{"1min":0.224,"5min":0.216,"15min":0.216},"max":{"1min":0.51800000000000002,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.35899999999999999,"15min":0.33100000000000002},"min":{"1min":0.30299999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.40400000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:21:07 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.40300000000000002,"15min":0.38500000000000001},"min":{"1min":0.29799999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.373},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.371}]},{"osd":3,"last update":"Fri Sep 13 14:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33700000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.436}]},{"osd":4,"last update":"Fri Sep 13 14:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.62,"15min":0.626},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77400000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51100000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.61599999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.70099999999999996}]},{"osd":7,"last update":"Fri Sep 13 14:21:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298520,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446632,"statfs":{"total":95995035648,"available":95689351168,"internally_reserved":0,"allocated":1191936,"data_stored":841260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:20:44 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.34599999999999997,"15min":0.32900000000000001},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.315},{"interface":"front","average":{"1min":0.435,"5min":0.38800000000000001,"15min":0.36199999999999999},"min":{"1min":0.28199999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.29399999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41799999999999998,"15min":0.42399999999999999},"min":{"1min":0.253,"5min":0.246,"15min":0.246},"max":{"1min":0.51600000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.437,"15min":0.441},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.59999999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.35599999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.47499999999999998,"15min":0.48999999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.55100000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.44700000000000001,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.625,"5min":0.629,"15min":0.629},"last":0.26900000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59399999999999997,"15min":0.58999999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76100000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47299999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:21:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.56299999999999994,"15min":0.55000000000000004},"min":{"1min":0.44,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.433},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.41699999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.40500000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.44900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:21:41.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:21:42.378 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:42 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[72140]: ts=2024-09-13T14:21:42.045Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:21:42.379 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:42 smithi003 podman[77546]: 2024-09-13 14:21:42.076671021 +0000 UTC m=+0.132101531 container died aec662fe2e6f0604749fedc78ca6f07aac65fae2b6fc671b40b3866a2d55bccf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:42 smithi019 ceph-mon[32895]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:42.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:42 smithi003 ceph-mon[29646]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:42.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:42 smithi003 ceph-mon[41592]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:42.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:42 smithi003 podman[77546]: 2024-09-13 14:21:42.468274111 +0000 UTC m=+0.523704612 container cleanup aec662fe2e6f0604749fedc78ca6f07aac65fae2b6fc671b40b3866a2d55bccf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:42.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:42 smithi003 bash[77546]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T14:21:42.783 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:43.053 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:42 smithi003 podman[77588]: 2024-09-13 14:21:42.685943314 +0000 UTC m=+0.608488500 container remove aec662fe2e6f0604749fedc78ca6f07aac65fae2b6fc671b40b3866a2d55bccf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:43.053 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:42 smithi003 podman[77588]: 2024-09-13 14:21:42.761648622 +0000 UTC m=+0.684193807 volume remove 11f5b6dad205db9d2faae2174f0cbd5f5d75c96bcd4dbb7899bb8add71692b6e 2024-09-13T14:21:43.432 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:43 smithi003 podman[77639]: 2024-09-13 14:21:43.28999572 +0000 UTC m=+0.504693134 container died 6ba531afe2282d24e8604da97ec1f45fd004165bce433eb3fedbd8cd3bf8bb9a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=blissful_feistel, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-13T14:21:43.932 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:43 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T14:21:43.933 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:43 smithi003 systemd[1]: Stopped Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:21:43.933 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:43 smithi003 systemd[1]: Starting Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:21:44.336 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 podman[77762]: 2024-09-13 14:21:44.102409299 +0000 UTC m=+0.101413348 volume create d04f150da32cf5bc66dbad61cddf02c4da282417af68e97f2813a5c1ce8dcad3 2024-09-13T14:21:44.336 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 podman[77762]: 2024-09-13 14:21:44.02224338 +0000 UTC m=+0.021247375 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-13T14:21:44.336 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 podman[77762]: 2024-09-13 14:21:44.185996362 +0000 UTC m=+0.185000347 container create 9994b1ee34d8e3ba211c04279e16b33b7e0e98bfae609e48654a2000f73b6791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:44.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:44 smithi019 ceph-mon[32895]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:44.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-mon[41592]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:44.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-mon[29646]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:44.682 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 podman[77762]: 2024-09-13 14:21:44.472614246 +0000 UTC m=+0.471618230 container init 9994b1ee34d8e3ba211c04279e16b33b7e0e98bfae609e48654a2000f73b6791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 podman[77762]: 2024-09-13 14:21:44.477909574 +0000 UTC m=+0.476913559 container start 9994b1ee34d8e3ba211c04279e16b33b7e0e98bfae609e48654a2000f73b6791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:44.491Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:44.491Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:44.492Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.3 port=9094 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:44.493Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:44.517Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:44.517Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:44.520Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:44.520Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 bash[77762]: 9994b1ee34d8e3ba211c04279e16b33b7e0e98bfae609e48654a2000f73b6791 2024-09-13T14:21:44.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:44 smithi003 systemd[1]: Started Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:21:45.289 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:21:45.290 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:21:45.930 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:45 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:45.930 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:45 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:45.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:45 smithi003 ceph-mon[41592]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:21:45.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:45 smithi003 ceph-mon[41592]: Reconfiguring daemon prometheus.a on smithi019 2024-09-13T14:21:45.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:45 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:45.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:45 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:45.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:45 smithi003 ceph-mon[29646]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:21:45.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:45 smithi003 ceph-mon[29646]: Reconfiguring daemon prometheus.a on smithi019 2024-09-13T14:21:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:45 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:45 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:45 smithi019 ceph-mon[32895]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:21:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:45 smithi019 ceph-mon[32895]: Reconfiguring daemon prometheus.a on smithi019 2024-09-13T14:21:45.956 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-13T14:21:43.354161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385256,"kb_used_data":6408,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575960,"statfs":{"total":767960285184,"available":765517783040,"internally_reserved":0,"allocated":6561792,"data_stored":3777158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001973"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.257173+0000","last_change":"2024-09-13T14:20:41.394322+0000","last_active":"2024-09-13T14:20:50.257173+0000","last_peered":"2024-09-13T14:20:50.257173+0000","last_clean":"2024-09-13T14:20:50.257173+0000","last_became_active":"2024-09-13T14:20:37.356874+0000","last_became_peered":"2024-09-13T14:20:37.356874+0000","last_unstale":"2024-09-13T14:20:50.257173+0000","last_undegraded":"2024-09-13T14:20:50.257173+0000","last_fullsized":"2024-09-13T14:20:50.257173+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:05.446185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590447+0000","last_change":"2024-09-13T14:20:41.389271+0000","last_active":"2024-09-13T14:20:50.590447+0000","last_peered":"2024-09-13T14:20:50.590447+0000","last_clean":"2024-09-13T14:20:50.590447+0000","last_became_active":"2024-09-13T14:20:37.355416+0000","last_became_peered":"2024-09-13T14:20:37.355416+0000","last_unstale":"2024-09-13T14:20:50.590447+0000","last_undegraded":"2024-09-13T14:20:50.590447+0000","last_fullsized":"2024-09-13T14:20:50.590447+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:47.155583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026697500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590491+0000","last_change":"2024-09-13T14:20:41.646302+0000","last_active":"2024-09-13T14:20:50.590491+0000","last_peered":"2024-09-13T14:20:50.590491+0000","last_clean":"2024-09-13T14:20:50.590491+0000","last_became_active":"2024-09-13T14:20:37.356803+0000","last_became_peered":"2024-09-13T14:20:37.356803+0000","last_unstale":"2024-09-13T14:20:50.590491+0000","last_undegraded":"2024-09-13T14:20:50.590491+0000","last_fullsized":"2024-09-13T14:20:50.590491+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:41.431722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590342+0000","last_change":"2024-09-13T14:20:41.389300+0000","last_active":"2024-09-13T14:20:50.590342+0000","last_peered":"2024-09-13T14:20:50.590342+0000","last_clean":"2024-09-13T14:20:50.590342+0000","last_became_active":"2024-09-13T14:20:37.356625+0000","last_became_peered":"2024-09-13T14:20:37.356625+0000","last_unstale":"2024-09-13T14:20:50.590342+0000","last_undegraded":"2024-09-13T14:20:50.590342+0000","last_fullsized":"2024-09-13T14:20:50.590342+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:28.847614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":20,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590613+0000","last_change":"2024-09-13T14:20:41.393368+0000","last_active":"2024-09-13T14:20:50.590613+0000","last_peered":"2024-09-13T14:20:50.590613+0000","last_clean":"2024-09-13T14:20:50.590613+0000","last_became_active":"2024-09-13T14:20:37.357807+0000","last_became_peered":"2024-09-13T14:20:37.357807+0000","last_unstale":"2024-09-13T14:20:50.590613+0000","last_undegraded":"2024-09-13T14:20:50.590613+0000","last_fullsized":"2024-09-13T14:20:50.590613+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:17.845640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.257759+0000","last_change":"2024-09-13T14:20:41.645840+0000","last_active":"2024-09-13T14:20:50.257759+0000","last_peered":"2024-09-13T14:20:50.257759+0000","last_clean":"2024-09-13T14:20:50.257759+0000","last_became_active":"2024-09-13T14:20:37.388195+0000","last_became_peered":"2024-09-13T14:20:37.388195+0000","last_unstale":"2024-09-13T14:20:50.257759+0000","last_undegraded":"2024-09-13T14:20:50.257759+0000","last_fullsized":"2024-09-13T14:20:50.257759+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:39.037088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590493+0000","last_change":"2024-09-13T14:20:41.646316+0000","last_active":"2024-09-13T14:20:50.590493+0000","last_peered":"2024-09-13T14:20:50.590493+0000","last_clean":"2024-09-13T14:20:50.590493+0000","last_became_active":"2024-09-13T14:20:37.358143+0000","last_became_peered":"2024-09-13T14:20:37.358143+0000","last_unstale":"2024-09-13T14:20:50.590493+0000","last_undegraded":"2024-09-13T14:20:50.590493+0000","last_fullsized":"2024-09-13T14:20:50.590493+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:37.774534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":19,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T14:20:50.590627+0000","last_change":"2024-09-13T14:20:41.392754+0000","last_active":"2024-09-13T14:20:50.590627+0000","last_peered":"2024-09-13T14:20:50.590627+0000","last_clean":"2024-09-13T14:20:50.590627+0000","last_became_active":"2024-09-13T14:20:37.356130+0000","last_became_peered":"2024-09-13T14:20:37.356130+0000","last_unstale":"2024-09-13T14:20:50.590627+0000","last_undegraded":"2024-09-13T14:20:50.590627+0000","last_fullsized":"2024-09-13T14:20:50.590627+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:50.167135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"53'79","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-13T14:20:41.645995+0000","last_change":"2024-09-13T14:20:01.261809+0000","last_active":"2024-09-13T14:20:41.645995+0000","last_peered":"2024-09-13T14:20:41.645995+0000","last_clean":"2024-09-13T14:20:41.645995+0000","last_became_active":"2024-09-13T14:20:01.256039+0000","last_became_peered":"2024-09-13T14:20:01.256039+0000","last_unstale":"2024-09-13T14:20:41.645995+0000","last_undegraded":"2024-09-13T14:20:41.645995+0000","last_fullsized":"2024-09-13T14:20:41.645995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:01.420572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298516,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446636,"statfs":{"total":95995035648,"available":95689355264,"internally_reserved":0,"allocated":1187840,"data_stored":840995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.63900000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.57899999999999996}]},{"osd":3,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.623},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65100000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561052,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298520,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446632,"statfs":{"total":95995035648,"available":95689351168,"internally_reserved":0,"allocated":1191936,"data_stored":840964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.51500000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.57499999999999996}]},{"osd":3,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.71099999999999997}]},{"osd":5,"last update":"Fri Sep 13 14:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.73799999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.64600000000000002}]}]},{"osd":5,"up_from":37,"seq":158913789984,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297936,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447216,"statfs":{"total":95995035648,"available":95689949184,"internally_reserved":0,"allocated":593920,"data_stored":250352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.50800000000000001,"15min":0.51700000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.69399999999999995,"5min":0.752,"15min":0.752},"last":0.64000000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76200000000000001,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.38,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.623}]},{"osd":3,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.497,"15min":0.497},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70999999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Sep 13 14:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.35299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.65300000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.40999999999999998,"15min":0.39400000000000002},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.54300000000000004}]},{"osd":6,"last update":"Fri Sep 13 14:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.71699999999999997}]},{"osd":7,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.5},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.69999999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":250892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68100000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54400000000000004,"15min":0.54300000000000004},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.622}]},{"osd":1,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.66000000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71499999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.36699999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.68400000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:21:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.40400000000000003,"15min":0.39200000000000002},"min":{"1min":0.30099999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.74099999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.317,"5min":0.247,"15min":0.247},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.73299999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.57099999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:21:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.76500000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":250892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.36699999999999999,"15min":0.35499999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.50900000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.379},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.39500000000000002,"15min":0.40300000000000002},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.55300000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.27200000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.41299999999999998,"15min":0.42199999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.48699999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.435,"15min":0.437},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59299999999999997,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.32500000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73699999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.438,"5min":0.433,"15min":0.433},"max":{"1min":0.79700000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79700000000000004,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78500000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61499999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58999999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":250892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35299999999999998,"15min":0.34200000000000003},"min":{"1min":0.23000000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.497,"5min":0.623,"15min":0.623},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.36699999999999999,"15min":0.35099999999999998},"min":{"1min":0.26200000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.60299999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38800000000000001,"15min":0.378},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.39200000000000002},"min":{"1min":0.27800000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.59799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.59699999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.435,"15min":0.434},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.63900000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.435,"15min":0.43099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.66500000000000004}]},{"osd":4,"last update":"Fri Sep 13 14:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76600000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.745}]},{"osd":5,"last update":"Fri Sep 13 14:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.59699999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81699999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.68899999999999995}]},{"osd":6,"last update":"Fri Sep 13 14:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.68100000000000005}]},{"osd":7,"last update":"Fri Sep 13 14:21:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.755}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297944,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447208,"statfs":{"total":95995035648,"available":95689940992,"internally_reserved":0,"allocated":602112,"data_stored":250911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.35999999999999999,"15min":0.34100000000000003},"min":{"1min":0.224,"5min":0.216,"15min":0.216},"max":{"1min":0.51800000000000002,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.35899999999999999,"15min":0.33100000000000002},"min":{"1min":0.30299999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.40400000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:21:07 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.40300000000000002,"15min":0.38500000000000001},"min":{"1min":0.29799999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.373},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.371}]},{"osd":3,"last update":"Fri Sep 13 14:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33700000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.436}]},{"osd":4,"last update":"Fri Sep 13 14:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.62,"15min":0.626},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77400000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51100000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.61599999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.70099999999999996}]},{"osd":7,"last update":"Fri Sep 13 14:21:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298520,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446632,"statfs":{"total":95995035648,"available":95689351168,"internally_reserved":0,"allocated":1191936,"data_stored":841260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:20:44 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.34599999999999997,"15min":0.32900000000000001},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.315},{"interface":"front","average":{"1min":0.435,"5min":0.38800000000000001,"15min":0.36199999999999999},"min":{"1min":0.28199999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.29399999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41799999999999998,"15min":0.42399999999999999},"min":{"1min":0.253,"5min":0.246,"15min":0.246},"max":{"1min":0.51600000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.437,"15min":0.441},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.59999999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.35599999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.47499999999999998,"15min":0.48999999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.55100000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.44700000000000001,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.625,"5min":0.629,"15min":0.629},"last":0.26900000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59399999999999997,"15min":0.58999999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76100000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47299999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:21:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.56299999999999994,"15min":0.55000000000000004},"min":{"1min":0.44,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.433},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.41699999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.40500000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.44900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:21:45.958 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:21:45.958 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:21:45.958 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:21:45.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:21:46.217 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:46.615 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:46 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:46.494Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000606529s 2024-09-13T14:21:46.818 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:46 smithi019 ceph-mon[32895]: from='client.24515 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:21:46.818 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:46 smithi019 ceph-mon[32895]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:46.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:46 smithi003 ceph-mon[41592]: from='client.24515 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:21:46.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:46 smithi003 ceph-mon[41592]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:46.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:46 smithi003 ceph-mon[29646]: from='client.24515 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:21:46.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:46 smithi003 ceph-mon[29646]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:47.931 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:47 smithi019 systemd[1]: Stopping Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:21:48.153 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:21:48.153 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:21:48.376 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.069Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:21:48.376 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.069Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.069Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.069Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.069Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.069Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.069Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.069Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.069Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.070Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.070Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[61873]: ts=2024-09-13T14:21:48.070Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:21:48.377 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 podman[65207]: 2024-09-13 14:21:48.079642985 +0000 UTC m=+0.132005412 container died 819b2e3729e53a38b2efdc35443b776e25a0a176844cef5ef378aa6e2b4a07dd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:48.637 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 podman[65207]: 2024-09-13 14:21:48.608354086 +0000 UTC m=+0.660716556 container cleanup 819b2e3729e53a38b2efdc35443b776e25a0a176844cef5ef378aa6e2b4a07dd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:48.637 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 bash[65207]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:21:48.760 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:48 smithi003 ceph-mon[41592]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:48.761 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:48 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/554016633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:21:48.761 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:48.761 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:48 smithi003 ceph-mon[29646]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:48.761 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:48 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/554016633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:21:48.761 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:48.787 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:21:48.788 INFO:tasks.cephadm:Setup complete, yielding 2024-09-13T14:21:48.788 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-13T14:21:48.798 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-09-13T14:21:48.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-09-13T14:21:48.931 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:48 smithi019 podman[65226]: 2024-09-13 14:21:48.8183934 +0000 UTC m=+0.740220018 container remove 819b2e3729e53a38b2efdc35443b776e25a0a176844cef5ef378aa6e2b4a07dd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-mon[32895]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/554016633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:21:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:49.053 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:49.358 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:21:49.359 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 systemd[1]: Stopped Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:21:49.359 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.156s CPU time. 2024-09-13T14:21:49.359 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 systemd[1]: Starting Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:21:49.359 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 podman[65318]: 2024-09-13 14:21:49.252187636 +0000 UTC m=+0.107654163 container create f9fc99d6a298947e7da34345e801bcb2a80a86b830aa9505f10b496c458f3cc5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:49.359 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 podman[65318]: 2024-09-13 14:21:49.1614576 +0000 UTC m=+0.016924139 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-13T14:21:49.724 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:49] ENGINE Bus STOPPING 2024-09-13T14:21:49.724 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:21:49.724 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:49] ENGINE Bus STOPPED 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 podman[65318]: 2024-09-13 14:21:49.479526725 +0000 UTC m=+0.334993263 container init f9fc99d6a298947e7da34345e801bcb2a80a86b830aa9505f10b496c458f3cc5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 podman[65318]: 2024-09-13 14:21:49.484248803 +0000 UTC m=+0.339715341 container start f9fc99d6a298947e7da34345e801bcb2a80a86b830aa9505f10b496c458f3cc5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.508Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.508Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.508Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi019 (none))" 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.508Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.508Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.510Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.510Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:21:49.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.512Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.512Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.513Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.513Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.564µs 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.513Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.514Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.514Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.514Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=22.651µs wal_replay_duration=469.968µs wbl_replay_duration=94ns total_replay_duration=518.856µs 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.515Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.515Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.515Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:21:49.933 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.530Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.331619ms db_storage=783ns remote_storage=1.137µs web_handler=428ns query_engine=530ns scrape=549.89µs scrape_sd=32.219µs notify=19.727µs notify_sd=5.906µs rules=14.498601ms tracing=5.75µs 2024-09-13T14:21:49.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.530Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:21:49.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:21:49.530Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:21:49.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 bash[65318]: f9fc99d6a298947e7da34345e801bcb2a80a86b830aa9505f10b496c458f3cc5 2024-09-13T14:21:49.934 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:21:49 smithi019 systemd[1]: Started Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:21:50.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:49] ENGINE Bus STARTING 2024-09-13T14:21:50.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:49] ENGINE Serving on http://:::9283 2024-09-13T14:21:50.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:49] ENGINE Bus STARTED 2024-09-13T14:21:50.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:49] ENGINE Bus STOPPING 2024-09-13T14:21:50.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:21:50.592 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:50] ENGINE Bus STOPPED 2024-09-13T14:21:50.592 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:50] ENGINE Bus STARTING 2024-09-13T14:21:50.592 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:50] ENGINE Serving on http://:::9283 2024-09-13T14:21:50.592 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:50] ENGINE Bus STARTED 2024-09-13T14:21:50.592 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:50] ENGINE Bus STOPPING 2024-09-13T14:21:50.592 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:50.592 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.592 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:21:50.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:21:50.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:21:50.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:21:50.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:21:50.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:21:50.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:21:50.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:21:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:21:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:21:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:21:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:50 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:21:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:21:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:21:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:21:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:21:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:21:50.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:50.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:21:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:21:50.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:21:50.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:50.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:21:51.233 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:21:51.233 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:50] ENGINE Bus STOPPED 2024-09-13T14:21:51.233 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:50 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:50] ENGINE Bus STARTING 2024-09-13T14:21:51.233 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:51] ENGINE Serving on http://:::9283 2024-09-13T14:21:51.233 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:21:51] ENGINE Bus STARTED 2024-09-13T14:21:51.598 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:51 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:21:51.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:51 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:21:51.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:51 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:21:52.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:52 smithi019 ceph-mon[32895]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:52.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:52 smithi019 ceph-mon[32895]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:52.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:52 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:21:52.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:52 smithi019 ceph-mon[32895]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:21:52.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:52 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:21:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[41592]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[41592]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:21:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[41592]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:21:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:21:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[29646]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[29646]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:21:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[29646]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:21:52.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:21:52.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[29623]: 2024-09-13T14:21:52.613+0000 7efdd695f640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:21:53.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-13T14:21:53.671 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:21:53.671 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:21:53.671 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:21:53.671 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:21:53.671 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:21:53.671 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:21:53.672 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:21:53.672 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: fsmap foo:0 2024-09-13T14:21:53.672 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: Saving service mds.foo spec with placement count:2 2024-09-13T14:21:53.672 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:53.672 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:53.672 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:53.672 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:21:53.672 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: fsmap foo:0 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: Saving service mds.foo spec with placement count:2 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:53.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:53 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:53.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: fsmap foo:0 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: Saving service mds.foo spec with placement count:2 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:53 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.104 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:54.931 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:21:54.498Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004677171s 2024-09-13T14:21:54.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: pgmap v53: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:21:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:21:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.zmhiel", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:21:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.zmhiel", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: pgmap v53: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.zmhiel", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:21:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.zmhiel", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:54 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: pgmap v53: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.zmhiel", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.zmhiel", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:21:54.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:54 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:55.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:55 smithi003 ceph-mon[41592]: Deploying daemon mds.foo.smithi003.zmhiel on smithi003 2024-09-13T14:21:55.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:55 smithi003 ceph-mon[41592]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:21:55.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:55 smithi003 ceph-mon[29646]: Deploying daemon mds.foo.smithi003.zmhiel on smithi003 2024-09-13T14:21:55.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:55 smithi003 ceph-mon[29646]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:21:55.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:55 smithi019 ceph-mon[32895]: Deploying daemon mds.foo.smithi003.zmhiel on smithi003 2024-09-13T14:21:55.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:55 smithi019 ceph-mon[32895]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:21:56.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-13T14:21:56.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:56 smithi019 ceph-mon[32895]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:56.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:56 smithi003 ceph-mon[41592]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:56 smithi003 ceph-mon[29646]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:57.370 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:21:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:21:57] "GET /metrics HTTP/1.1" 200 203791 "" "Prometheus/2.43.0" 2024-09-13T14:21:57.471 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:58.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:21:58 smithi003 ceph-mon[41592]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:58.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:21:58 smithi003 ceph-mon[29646]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:58.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:21:58 smithi019 ceph-mon[32895]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:00.568 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-13T14:22:00.577 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:22:00.577 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:22:00.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:22:00.840 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:00.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:00.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi019.ydtwbg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi019.ydtwbg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: Deploying daemon mds.foo.smithi019.ydtwbg on smithi019 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: mds.? [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] up:boot 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: daemon mds.foo.smithi003.zmhiel assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: Cluster is now healthy 2024-09-13T14:22:00.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: fsmap foo:0 1 up:standby 2024-09-13T14:22:00.909 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"}]: dispatch 2024-09-13T14:22:00.909 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:creating} 2024-09-13T14:22:00.909 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-mon[32895]: daemon mds.foo.smithi003.zmhiel is now active in filesystem foo as rank 0 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi019.ydtwbg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi019.ydtwbg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: Deploying daemon mds.foo.smithi019.ydtwbg on smithi019 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: mds.? [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] up:boot 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: daemon mds.foo.smithi003.zmhiel assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:22:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: Cluster is now healthy 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: fsmap foo:0 1 up:standby 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"}]: dispatch 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:creating} 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[41592]: daemon mds.foo.smithi003.zmhiel is now active in filesystem foo as rank 0 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:00.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi019.ydtwbg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi019.ydtwbg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: Deploying daemon mds.foo.smithi019.ydtwbg on smithi019 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: mds.? [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] up:boot 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: daemon mds.foo.smithi003.zmhiel assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: Cluster is now healthy 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: fsmap foo:0 1 up:standby 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"}]: dispatch 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:creating} 2024-09-13T14:22:00.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:00 smithi003 ceph-mon[29646]: daemon mds.foo.smithi003.zmhiel is now active in filesystem foo as rank 0 2024-09-13T14:22:01.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:22:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:01.921 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:01 smithi019 ceph-mon[32895]: mds.? [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] up:active 2024-09-13T14:22:01.921 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:01 smithi019 ceph-mon[32895]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 2024-09-13T14:22:01.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:01 smithi003 ceph-mon[41592]: mds.? [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] up:active 2024-09-13T14:22:01.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:01 smithi003 ceph-mon[41592]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 2024-09-13T14:22:01.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:01 smithi003 ceph-mon[29646]: mds.? [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] up:active 2024-09-13T14:22:01.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:01 smithi003 ceph-mon[29646]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 2024-09-13T14:22:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:22:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:02 smithi019 ceph-mon[32895]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:02 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:02 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:02 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:02 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:02 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:02 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[41592]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[29646]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:02 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:03.587 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"flags":0,"active_gid":14439,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":70918773},{"type":"v1","addr":"172.21.15.3:6801","nonce":70918773}]},"active_addr":"172.21.15.3:6801/70918773","active_change":"2024-09-13T14:20:33.329993+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14442,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":53,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":1118901319}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3529413292}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2893689034}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":1863609487}]}]} 2024-09-13T14:22:03.591 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:22:03.591 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:22:03.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:22:03.720 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[41592]: mds.? [v2:172.21.15.19:6832/4291362241,v1:172.21.15.19:6833/4291362241] up:boot 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[41592]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"}]: dispatch 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[41592]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3484012375' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[29646]: mds.? [v2:172.21.15.19:6832/4291362241,v1:172.21.15.19:6833/4291362241] up:boot 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[29646]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"}]: dispatch 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[29646]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:03.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3484012375' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:22:03.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:03.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:03 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:03.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:03 smithi019 ceph-mon[32895]: mds.? [v2:172.21.15.19:6832/4291362241,v1:172.21.15.19:6833/4291362241] up:boot 2024-09-13T14:22:03.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:03 smithi019 ceph-mon[32895]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:03.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:03 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"}]: dispatch 2024-09-13T14:22:03.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:03 smithi019 ceph-mon[32895]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:03.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:03 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3484012375' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:22:03.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:03 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:03.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:03 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:04.046 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:04.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:04 smithi003 ceph-mon[41592]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:04.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:04 smithi003 ceph-mon[41592]: mds.? [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] up:active 2024-09-13T14:22:04.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:04 smithi003 ceph-mon[41592]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:04.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:04 smithi003 ceph-mon[29646]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:04.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:04 smithi003 ceph-mon[29646]: mds.? [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] up:active 2024-09-13T14:22:04.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:04 smithi003 ceph-mon[29646]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:04.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:04 smithi019 ceph-mon[32895]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:04.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:04 smithi019 ceph-mon[32895]: mds.? [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] up:active 2024-09-13T14:22:04.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:04 smithi019 ceph-mon[32895]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:22:05.876 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":65,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:15:00.204452+0000","modified":"2024-09-13T14:21:54.622486+0000","last_up_change":"2024-09-13T14:20:00.222071+0000","last_in_change":"2024-09-13T14:19:42.900010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:18:02.574675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:20:35.870236+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:21:51.432728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:21:52.047402+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"a153ec29-6e01-4063-aa90-01d1de00e53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6803","nonce":3028312307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6805","nonce":3028312307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6809","nonce":3028312307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6807","nonce":3028312307}]},"public_addr":"172.21.15.3:6803/3028312307","cluster_addr":"172.21.15.3:6805/3028312307","heartbeat_back_addr":"172.21.15.3:6809/3028312307","heartbeat_front_addr":"172.21.15.3:6807/3028312307","state":["exists","up"]},{"osd":1,"uuid":"9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6811","nonce":2008999734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6813","nonce":2008999734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6817","nonce":2008999734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6815","nonce":2008999734}]},"public_addr":"172.21.15.3:6811/2008999734","cluster_addr":"172.21.15.3:6813/2008999734","heartbeat_back_addr":"172.21.15.3:6817/2008999734","heartbeat_front_addr":"172.21.15.3:6815/2008999734","state":["exists","up"]},{"osd":2,"uuid":"676ec1e5-071c-47ba-86ae-2193f448fe8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6819","nonce":2444149284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6821","nonce":2444149284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6825","nonce":2444149284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6823","nonce":2444149284}]},"public_addr":"172.21.15.3:6819/2444149284","cluster_addr":"172.21.15.3:6821/2444149284","heartbeat_back_addr":"172.21.15.3:6825/2444149284","heartbeat_front_addr":"172.21.15.3:6823/2444149284","state":["exists","up"]},{"osd":3,"uuid":"7ec0ad16-0df3-4319-b475-72cfaaa5edae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6827","nonce":2595190501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6829","nonce":2595190501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6833","nonce":2595190501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6831","nonce":2595190501}]},"public_addr":"172.21.15.3:6827/2595190501","cluster_addr":"172.21.15.3:6829/2595190501","heartbeat_back_addr":"172.21.15.3:6833/2595190501","heartbeat_front_addr":"172.21.15.3:6831/2595190501","state":["exists","up"]},{"osd":4,"uuid":"0ef54d09-4000-4695-942b-1b1cf9659983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6801","nonce":4204660131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6803","nonce":4204660131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6807","nonce":4204660131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6805","nonce":4204660131}]},"public_addr":"172.21.15.19:6801/4204660131","cluster_addr":"172.21.15.19:6803/4204660131","heartbeat_back_addr":"172.21.15.19:6807/4204660131","heartbeat_front_addr":"172.21.15.19:6805/4204660131","state":["exists","up"]},{"osd":5,"uuid":"6dfd8fce-4d3d-4a8b-90ac-556447045378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6809","nonce":3591244104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6811","nonce":3591244104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6815","nonce":3591244104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6813","nonce":3591244104}]},"public_addr":"172.21.15.19:6809/3591244104","cluster_addr":"172.21.15.19:6811/3591244104","heartbeat_back_addr":"172.21.15.19:6815/3591244104","heartbeat_front_addr":"172.21.15.19:6813/3591244104","state":["exists","up"]},{"osd":6,"uuid":"17e27318-c0ce-413d-b660-d756731ea743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6817","nonce":2162145215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6819","nonce":2162145215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6823","nonce":2162145215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6821","nonce":2162145215}]},"public_addr":"172.21.15.19:6817/2162145215","cluster_addr":"172.21.15.19:6819/2162145215","heartbeat_back_addr":"172.21.15.19:6823/2162145215","heartbeat_front_addr":"172.21.15.19:6821/2162145215","state":["exists","up"]},{"osd":7,"uuid":"1edab2d1-dad5-45c1-977c-6911541c282e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6825","nonce":2814206175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6827","nonce":2814206175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6831","nonce":2814206175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6829","nonce":2814206175}]},"public_addr":"172.21.15.19:6825/2814206175","cluster_addr":"172.21.15.19:6827/2814206175","heartbeat_back_addr":"172.21.15.19:6831/2814206175","heartbeat_front_addr":"172.21.15.19:6829/2814206175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:11.323986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:34.429745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:59.523592+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:24.423713+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:46.452261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:10.483064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:34.315912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:58.203878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6800/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/1688297948":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/738145194":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2497820386":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/3505363388":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6800/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6801/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/253916983":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/1018276975":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6801/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2723658197":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2346548909":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/3631040762":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6800/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/3615653824":"2024-09-14T14:20:33.329796+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:22:06.720 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:22:06.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:22:06.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:06 smithi019 ceph-mon[32895]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-09-13T14:22:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:06 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:06 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:06 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1745687966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:22:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:06 smithi003 ceph-mon[29646]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-09-13T14:22:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:06 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:06 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:06 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1745687966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:22:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:06 smithi003 ceph-mon[41592]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-09-13T14:22:06.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:06 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:06.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:06 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:06.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:06 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1745687966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:22:07.168 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:07.431 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:07] "GET /metrics HTTP/1.1" 200 239349 "" "Prometheus/2.43.0" 2024-09-13T14:22:08.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:07 smithi019 ceph-mon[32895]: mds.? [v2:172.21.15.19:6832/4291362241,v1:172.21.15.19:6833/4291362241] up:standby 2024-09-13T14:22:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:07 smithi019 ceph-mon[32895]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:07 smithi019 ceph-mon[32895]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:08.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:07 smithi003 ceph-mon[29646]: mds.? [v2:172.21.15.19:6832/4291362241,v1:172.21.15.19:6833/4291362241] up:standby 2024-09-13T14:22:08.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:07 smithi003 ceph-mon[29646]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:08.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:07 smithi003 ceph-mon[29646]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:08.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:07 smithi003 ceph-mon[41592]: mds.? [v2:172.21.15.19:6832/4291362241,v1:172.21.15.19:6833/4291362241] up:standby 2024-09-13T14:22:08.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:07 smithi003 ceph-mon[41592]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:22:08.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:07 smithi003 ceph-mon[41592]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:09.597 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:22:09.597 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":65,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:15:00.204452+0000","modified":"2024-09-13T14:21:54.622486+0000","last_up_change":"2024-09-13T14:20:00.222071+0000","last_in_change":"2024-09-13T14:19:42.900010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:18:02.574675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:20:35.870236+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:21:51.432728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:21:52.047402+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"a153ec29-6e01-4063-aa90-01d1de00e53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6803","nonce":3028312307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6805","nonce":3028312307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6809","nonce":3028312307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3028312307},{"type":"v1","addr":"172.21.15.3:6807","nonce":3028312307}]},"public_addr":"172.21.15.3:6803/3028312307","cluster_addr":"172.21.15.3:6805/3028312307","heartbeat_back_addr":"172.21.15.3:6809/3028312307","heartbeat_front_addr":"172.21.15.3:6807/3028312307","state":["exists","up"]},{"osd":1,"uuid":"9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6811","nonce":2008999734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6813","nonce":2008999734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6817","nonce":2008999734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2008999734},{"type":"v1","addr":"172.21.15.3:6815","nonce":2008999734}]},"public_addr":"172.21.15.3:6811/2008999734","cluster_addr":"172.21.15.3:6813/2008999734","heartbeat_back_addr":"172.21.15.3:6817/2008999734","heartbeat_front_addr":"172.21.15.3:6815/2008999734","state":["exists","up"]},{"osd":2,"uuid":"676ec1e5-071c-47ba-86ae-2193f448fe8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6819","nonce":2444149284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6821","nonce":2444149284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6825","nonce":2444149284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":2444149284},{"type":"v1","addr":"172.21.15.3:6823","nonce":2444149284}]},"public_addr":"172.21.15.3:6819/2444149284","cluster_addr":"172.21.15.3:6821/2444149284","heartbeat_back_addr":"172.21.15.3:6825/2444149284","heartbeat_front_addr":"172.21.15.3:6823/2444149284","state":["exists","up"]},{"osd":3,"uuid":"7ec0ad16-0df3-4319-b475-72cfaaa5edae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6827","nonce":2595190501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6829","nonce":2595190501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6833","nonce":2595190501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2595190501},{"type":"v1","addr":"172.21.15.3:6831","nonce":2595190501}]},"public_addr":"172.21.15.3:6827/2595190501","cluster_addr":"172.21.15.3:6829/2595190501","heartbeat_back_addr":"172.21.15.3:6833/2595190501","heartbeat_front_addr":"172.21.15.3:6831/2595190501","state":["exists","up"]},{"osd":4,"uuid":"0ef54d09-4000-4695-942b-1b1cf9659983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6801","nonce":4204660131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6803","nonce":4204660131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6807","nonce":4204660131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6805","nonce":4204660131}]},"public_addr":"172.21.15.19:6801/4204660131","cluster_addr":"172.21.15.19:6803/4204660131","heartbeat_back_addr":"172.21.15.19:6807/4204660131","heartbeat_front_addr":"172.21.15.19:6805/4204660131","state":["exists","up"]},{"osd":5,"uuid":"6dfd8fce-4d3d-4a8b-90ac-556447045378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6809","nonce":3591244104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6811","nonce":3591244104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6815","nonce":3591244104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6813","nonce":3591244104}]},"public_addr":"172.21.15.19:6809/3591244104","cluster_addr":"172.21.15.19:6811/3591244104","heartbeat_back_addr":"172.21.15.19:6815/3591244104","heartbeat_front_addr":"172.21.15.19:6813/3591244104","state":["exists","up"]},{"osd":6,"uuid":"17e27318-c0ce-413d-b660-d756731ea743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6817","nonce":2162145215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6819","nonce":2162145215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6823","nonce":2162145215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6821","nonce":2162145215}]},"public_addr":"172.21.15.19:6817/2162145215","cluster_addr":"172.21.15.19:6819/2162145215","heartbeat_back_addr":"172.21.15.19:6823/2162145215","heartbeat_front_addr":"172.21.15.19:6821/2162145215","state":["exists","up"]},{"osd":7,"uuid":"1edab2d1-dad5-45c1-977c-6911541c282e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6825","nonce":2814206175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6827","nonce":2814206175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6831","nonce":2814206175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6829","nonce":2814206175}]},"public_addr":"172.21.15.19:6825/2814206175","cluster_addr":"172.21.15.19:6827/2814206175","heartbeat_back_addr":"172.21.15.19:6831/2814206175","heartbeat_front_addr":"172.21.15.19:6829/2814206175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:11.323986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:34.429745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:59.523592+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:24.423713+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:46.452261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:10.483064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:34.315912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:58.203878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6800/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/1688297948":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/738145194":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2497820386":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/3505363388":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6800/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6801/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/253916983":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/1018276975":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6801/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2723658197":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2346548909":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/3631040762":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6800/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/3615653824":"2024-09-14T14:20:33.329796+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:22:10.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:22:10.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:22:10.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:22:10.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:22:10.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:22:10.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:22:10.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:22:10.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:22:10.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:10 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:10.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:10 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:10.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:10 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:10.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:10 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:10.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:10 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:10.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:10 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:10.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:10 smithi019 ceph-mon[32895]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:10.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:10 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2308602813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:22:10.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:10.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:10.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:10.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:10.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[29646]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2308602813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[41592]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:10.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:10 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2308602813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:22:11.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:22:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:11.441 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:12.316 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:12.316 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:12.417 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:12 smithi003 ceph-mon[41592]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:12.418 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:12 smithi003 ceph-mon[29646]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:12.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:12 smithi019 ceph-mon[32895]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:13.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:13 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:13 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:13 smithi003 ceph-mon[29646]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:13.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:13 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:13.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:13 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:13.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:13 smithi003 ceph-mon[41592]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:13 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:14.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:13 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:14.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:13 smithi019 ceph-mon[32895]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:16.311 INFO:teuthology.orchestra.run.smithi003.stdout:77309411383 2024-09-13T14:22:16.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:22:16.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:16 smithi019 ceph-mon[32895]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:16.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:16 smithi003 ceph-mon[41592]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:16.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:16 smithi003 ceph-mon[29646]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:22:17.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:17 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:17] "GET /metrics HTTP/1.1" 200 239349 "" "Prometheus/2.43.0" 2024-09-13T14:22:18.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:18 smithi019 ceph-mon[32895]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:22:18.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:18 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:18.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:18 smithi003 ceph-mon[41592]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:22:18.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:18 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:18.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:18 smithi003 ceph-mon[29646]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:22:18.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:18 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:19.208 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:19.243 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:19.245 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:20.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:20 smithi019 ceph-mon[32895]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:22:20.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:20 smithi003 ceph-mon[29646]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:22:20.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:20 smithi003 ceph-mon[41592]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:22:21.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:22:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:21.255 INFO:teuthology.orchestra.run.smithi003.stdout:133143986220 2024-09-13T14:22:21.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:22:22.540 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:22 smithi003 ceph-mon[41592]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:22.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:22 smithi003 ceph-mon[29646]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:22.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:22 smithi019 ceph-mon[32895]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:23.805 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:24.392 INFO:teuthology.orchestra.run.smithi003.stdout:214748364830 2024-09-13T14:22:24.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:22:24.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:24 smithi003 ceph-mon[41592]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:24.617 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:24 smithi003 ceph-mon[29646]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:24 smithi019 ceph-mon[32895]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:25.318 INFO:teuthology.orchestra.run.smithi003.stdout:158913789993 2024-09-13T14:22:25.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:22:26.680 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:26 smithi003 ceph-mon[41592]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:26.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:26 smithi003 ceph-mon[29646]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:26.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:26 smithi019 ceph-mon[32895]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:26.703 INFO:teuthology.orchestra.run.smithi003.stdout:55834574908 2024-09-13T14:22:26.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:22:27.407 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:27] "GET /metrics HTTP/1.1" 200 239614 "" "Prometheus/2.43.0" 2024-09-13T14:22:28.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:28 smithi019 ceph-mon[32895]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:28.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:28 smithi003 ceph-mon[41592]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:28.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:28 smithi003 ceph-mon[29646]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:29.310 INFO:teuthology.orchestra.run.smithi003.stdout:77309411385 2024-09-13T14:22:30.068 INFO:teuthology.orchestra.run.smithi003.stdout:188978561062 2024-09-13T14:22:30.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:22:30.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:30.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:30.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:30.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:30.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:30.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2051615572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:22:30.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[29646]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:30.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:30.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:30.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:30.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:30.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:30.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2051615572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:22:30.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:30 smithi003 ceph-mon[41592]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:30.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:30 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:30 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:30 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:30 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:30 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:30 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2051615572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:22:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:30 smithi019 ceph-mon[32895]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:30.552 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:30.919 INFO:teuthology.orchestra.run.smithi003.stdout:111669149748 2024-09-13T14:22:30.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:22:31.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:22:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:31.368 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411383 got 77309411385 for osd.2 2024-09-13T14:22:31.369 DEBUG:teuthology.parallel:result is None 2024-09-13T14:22:31.395 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:31.818 INFO:teuthology.orchestra.run.smithi003.stdout:34359738434 2024-09-13T14:22:31.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:22:31.824 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:32.625 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:32 smithi003 ceph-mon[41592]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:32.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:32 smithi003 ceph-mon[29646]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:32.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:32 smithi019 ceph-mon[32895]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:33.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-09-13T14:22:33.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-13T14:22:33.500 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-13T14:22:33.500 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:33.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-09-13T14:22:33.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-13T14:22:33.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-13T14:22:33.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:33.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-09-13T14:22:33.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-13T14:22:33.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-13T14:22:33.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:34.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[41592]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:34.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-09-13T14:22:34.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-13T14:22:34.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-13T14:22:34.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[41592]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:22:34.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[29646]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:34.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-09-13T14:22:34.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-13T14:22:34.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-13T14:22:34.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:34 smithi003 ceph-mon[29646]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:22:34.881 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:34 smithi019 ceph-mon[32895]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:34.881 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:34 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-09-13T14:22:34.881 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:34 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-13T14:22:34.881 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:34 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-13T14:22:34.881 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:34 smithi019 ceph-mon[32895]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:22:35.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:35 smithi003 ceph-mon[29646]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:22:35.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:35 smithi003 ceph-mon[41592]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:22:35.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:35 smithi019 ceph-mon[32895]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:22:35.991 INFO:teuthology.orchestra.run.smithi003.stdout:133143986224 2024-09-13T14:22:36.201 INFO:teuthology.orchestra.run.smithi003.stdout:214748364834 2024-09-13T14:22:36.709 INFO:teuthology.orchestra.run.smithi003.stdout:158913789996 2024-09-13T14:22:36.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:36 smithi003 ceph-mon[41592]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:36.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:36 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3045104327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:22:36.710 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:36 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/4125675723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:22:36.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:36 smithi003 ceph-mon[29646]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:36.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:36 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3045104327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:22:36.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:36 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/4125675723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:22:36.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:36 smithi019 ceph-mon[32895]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:36.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:36 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3045104327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:22:36.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:36 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/4125675723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:22:37.416 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:37 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:37] "GET /metrics HTTP/1.1" 200 239592 "" "Prometheus/2.43.0" 2024-09-13T14:22:37.444 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789993 got 158913789996 for osd.5 2024-09-13T14:22:37.444 DEBUG:teuthology.parallel:result is None 2024-09-13T14:22:37.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:37 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2213309268' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:22:37.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:37 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2213309268' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:22:37.814 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986220 got 133143986224 for osd.4 2024-09-13T14:22:37.814 DEBUG:teuthology.parallel:result is None 2024-09-13T14:22:37.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:37 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2213309268' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:22:38.403 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364830 got 214748364834 for osd.7 2024-09-13T14:22:38.452 DEBUG:teuthology.parallel:result is None 2024-09-13T14:22:38.666 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:38 smithi003 ceph-mon[41592]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:38.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:38 smithi003 ceph-mon[29646]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:38.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:38 smithi019 ceph-mon[32895]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:39.168 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:39.169 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:40 smithi003 ceph-mon[41592]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:40.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:40 smithi003 ceph-mon[29646]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:40.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:40 smithi019 ceph-mon[32895]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:41.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:22:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:42.577 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:42 smithi003 ceph-mon[41592]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:42.577 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:42 smithi003 ceph-mon[29646]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:42.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:42 smithi019 ceph-mon[32895]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:42.965 INFO:teuthology.orchestra.run.smithi003.stdout:188978561065 2024-09-13T14:22:43.501 INFO:teuthology.orchestra.run.smithi003.stdout:34359738437 2024-09-13T14:22:43.591 INFO:teuthology.orchestra.run.smithi003.stdout:111669149751 2024-09-13T14:22:43.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:43 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3115963594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:22:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:43 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3115963594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:22:43.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:43 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3115963594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:22:44.236 INFO:teuthology.orchestra.run.smithi003.stdout:55834574912 2024-09-13T14:22:44.312 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738434 got 34359738437 for osd.0 2024-09-13T14:22:44.312 DEBUG:teuthology.parallel:result is None 2024-09-13T14:22:44.549 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:44 smithi003 ceph-mon[41592]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:44.549 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:44 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2524586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:22:44.549 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:44 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/4082720486' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:22:44.549 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:44 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3981287121' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:22:44.549 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:44 smithi003 ceph-mon[29646]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:44.549 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:44 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2524586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:22:44.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:44 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/4082720486' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:22:44.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:44 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3981287121' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:22:44.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:44 smithi019 ceph-mon[32895]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:44.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:44 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2524586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:22:44.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:44 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/4082720486' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:22:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:44 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3981287121' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:22:45.392 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561062 got 188978561065 for osd.6 2024-09-13T14:22:45.392 DEBUG:teuthology.parallel:result is None 2024-09-13T14:22:45.670 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574912 for osd.1 2024-09-13T14:22:45.670 DEBUG:teuthology.parallel:result is None 2024-09-13T14:22:45.994 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149748 got 111669149751 for osd.3 2024-09-13T14:22:45.995 DEBUG:teuthology.parallel:result is None 2024-09-13T14:22:45.995 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:22:45.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:22:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:46.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:46 smithi003 ceph-mon[29646]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:46.631 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:46 smithi003 ceph-mon[41592]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:46.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:46 smithi019 ceph-mon[32895]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:47.371 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:47] "GET /metrics HTTP/1.1" 200 239592 "" "Prometheus/2.43.0" 2024-09-13T14:22:48.039 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:22:48.040 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:22:48.638 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:48 smithi003 ceph-mon[41592]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:48.638 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:48.638 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:48 smithi003 ceph-mon[29646]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:48.638 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:48.663 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-13T14:22:47.367538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386040,"kb_used_data":7176,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575176,"statfs":{"total":767960285184,"available":765516980224,"internally_reserved":0,"allocated":7348224,"data_stored":4187588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002534"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432229+0000","last_change":"2024-09-13T14:21:53.701936+0000","last_active":"2024-09-13T14:22:33.432229+0000","last_peered":"2024-09-13T14:22:33.432229+0000","last_clean":"2024-09-13T14:22:33.432229+0000","last_became_active":"2024-09-13T14:21:53.701636+0000","last_became_peered":"2024-09-13T14:21:53.701636+0000","last_unstale":"2024-09-13T14:22:33.432229+0000","last_undegraded":"2024-09-13T14:22:33.432229+0000","last_fullsized":"2024-09-13T14:22:33.432229+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:40.363836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436139+0000","last_change":"2024-09-13T14:21:52.615068+0000","last_active":"2024-09-13T14:22:33.436139+0000","last_peered":"2024-09-13T14:22:33.436139+0000","last_clean":"2024-09-13T14:22:33.436139+0000","last_became_active":"2024-09-13T14:21:52.614632+0000","last_became_peered":"2024-09-13T14:21:52.614632+0000","last_unstale":"2024-09-13T14:22:33.436139+0000","last_undegraded":"2024-09-13T14:22:33.436139+0000","last_fullsized":"2024-09-13T14:22:33.436139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:04.140321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437488+0000","last_change":"2024-09-13T14:21:52.633866+0000","last_active":"2024-09-13T14:22:33.437488+0000","last_peered":"2024-09-13T14:22:33.437488+0000","last_clean":"2024-09-13T14:22:33.437488+0000","last_became_active":"2024-09-13T14:21:52.633523+0000","last_became_peered":"2024-09-13T14:21:52.633523+0000","last_unstale":"2024-09-13T14:22:33.437488+0000","last_undegraded":"2024-09-13T14:22:33.437488+0000","last_fullsized":"2024-09-13T14:22:33.437488+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:05:33.337285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431005+0000","last_change":"2024-09-13T14:21:52.623706+0000","last_active":"2024-09-13T14:22:33.431005+0000","last_peered":"2024-09-13T14:22:33.431005+0000","last_clean":"2024-09-13T14:22:33.431005+0000","last_became_active":"2024-09-13T14:21:52.623341+0000","last_became_peered":"2024-09-13T14:21:52.623341+0000","last_unstale":"2024-09-13T14:22:33.431005+0000","last_undegraded":"2024-09-13T14:22:33.431005+0000","last_fullsized":"2024-09-13T14:22:33.431005+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:08.120479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431932+0000","last_change":"2024-09-13T14:21:53.700932+0000","last_active":"2024-09-13T14:22:33.431932+0000","last_peered":"2024-09-13T14:22:33.431932+0000","last_clean":"2024-09-13T14:22:33.431932+0000","last_became_active":"2024-09-13T14:21:53.700605+0000","last_became_peered":"2024-09-13T14:21:53.700605+0000","last_unstale":"2024-09-13T14:22:33.431932+0000","last_undegraded":"2024-09-13T14:22:33.431932+0000","last_fullsized":"2024-09-13T14:22:33.431932+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:03.340866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435930+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:22:33.435930+0000","last_peered":"2024-09-13T14:22:33.435930+0000","last_clean":"2024-09-13T14:22:33.435930+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:22:33.435930+0000","last_undegraded":"2024-09-13T14:22:33.435930+0000","last_fullsized":"2024-09-13T14:22:33.435930+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T14:22:34.901457+0000","last_change":"2024-09-13T14:22:34.901457+0000","last_active":"2024-09-13T14:22:34.901457+0000","last_peered":"2024-09-13T14:22:34.901457+0000","last_clean":"2024-09-13T14:22:34.901457+0000","last_became_active":"2024-09-13T14:22:34.901121+0000","last_became_peered":"2024-09-13T14:22:34.901121+0000","last_unstale":"2024-09-13T14:22:34.901457+0000","last_undegraded":"2024-09-13T14:22:34.901457+0000","last_fullsized":"2024-09-13T14:22:34.901457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:03.009016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.434688+0000","last_change":"2024-09-13T14:21:52.609091+0000","last_active":"2024-09-13T14:22:33.434688+0000","last_peered":"2024-09-13T14:22:33.434688+0000","last_clean":"2024-09-13T14:22:33.434688+0000","last_became_active":"2024-09-13T14:21:52.608798+0000","last_became_peered":"2024-09-13T14:21:52.608798+0000","last_unstale":"2024-09-13T14:22:33.434688+0000","last_undegraded":"2024-09-13T14:22:33.434688+0000","last_fullsized":"2024-09-13T14:22:33.434688+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:42.697256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436619+0000","last_change":"2024-09-13T14:21:52.615164+0000","last_active":"2024-09-13T14:22:33.436619+0000","last_peered":"2024-09-13T14:22:33.436619+0000","last_clean":"2024-09-13T14:22:33.436619+0000","last_became_active":"2024-09-13T14:21:52.614326+0000","last_became_peered":"2024-09-13T14:21:52.614326+0000","last_unstale":"2024-09-13T14:22:33.436619+0000","last_undegraded":"2024-09-13T14:22:33.436619+0000","last_fullsized":"2024-09-13T14:22:33.436619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:48.585715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437133+0000","last_change":"2024-09-13T14:21:52.983025+0000","last_active":"2024-09-13T14:22:33.437133+0000","last_peered":"2024-09-13T14:22:33.437133+0000","last_clean":"2024-09-13T14:22:33.437133+0000","last_became_active":"2024-09-13T14:21:52.982611+0000","last_became_peered":"2024-09-13T14:21:52.982611+0000","last_unstale":"2024-09-13T14:22:33.437133+0000","last_undegraded":"2024-09-13T14:22:33.437133+0000","last_fullsized":"2024-09-13T14:22:33.437133+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:00.851531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436340+0000","last_change":"2024-09-13T14:21:52.614799+0000","last_active":"2024-09-13T14:22:33.436340+0000","last_peered":"2024-09-13T14:22:33.436340+0000","last_clean":"2024-09-13T14:22:33.436340+0000","last_became_active":"2024-09-13T14:21:52.614520+0000","last_became_peered":"2024-09-13T14:21:52.614520+0000","last_unstale":"2024-09-13T14:22:33.436340+0000","last_undegraded":"2024-09-13T14:22:33.436340+0000","last_fullsized":"2024-09-13T14:22:33.436340+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:07.897335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.426971+0000","last_change":"2024-09-13T14:21:53.702020+0000","last_active":"2024-09-13T14:22:33.426971+0000","last_peered":"2024-09-13T14:22:33.426971+0000","last_clean":"2024-09-13T14:22:33.426971+0000","last_became_active":"2024-09-13T14:21:53.700331+0000","last_became_peered":"2024-09-13T14:21:53.700331+0000","last_unstale":"2024-09-13T14:22:33.426971+0000","last_undegraded":"2024-09-13T14:22:33.426971+0000","last_fullsized":"2024-09-13T14:22:33.426971+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:46.779119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436618+0000","last_change":"2024-09-13T14:21:52.610490+0000","last_active":"2024-09-13T14:22:33.436618+0000","last_peered":"2024-09-13T14:22:33.436618+0000","last_clean":"2024-09-13T14:22:33.436618+0000","last_became_active":"2024-09-13T14:21:52.610192+0000","last_became_peered":"2024-09-13T14:21:52.610192+0000","last_unstale":"2024-09-13T14:22:33.436618+0000","last_undegraded":"2024-09-13T14:22:33.436618+0000","last_fullsized":"2024-09-13T14:22:33.436618+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:09.564393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T14:22:34.901360+0000","last_change":"2024-09-13T14:22:34.901360+0000","last_active":"2024-09-13T14:22:34.901360+0000","last_peered":"2024-09-13T14:22:34.901360+0000","last_clean":"2024-09-13T14:22:34.901360+0000","last_became_active":"2024-09-13T14:22:34.901010+0000","last_became_peered":"2024-09-13T14:22:34.901010+0000","last_unstale":"2024-09-13T14:22:34.901360+0000","last_undegraded":"2024-09-13T14:22:34.901360+0000","last_fullsized":"2024-09-13T14:22:34.901360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:55.897004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437376+0000","last_change":"2024-09-13T14:21:52.617746+0000","last_active":"2024-09-13T14:22:33.437376+0000","last_peered":"2024-09-13T14:22:33.437376+0000","last_clean":"2024-09-13T14:22:33.437376+0000","last_became_active":"2024-09-13T14:21:52.617280+0000","last_became_peered":"2024-09-13T14:21:52.617280+0000","last_unstale":"2024-09-13T14:22:33.437376+0000","last_undegraded":"2024-09-13T14:22:33.437376+0000","last_fullsized":"2024-09-13T14:22:33.437376+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:03.243247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431142+0000","last_change":"2024-09-13T14:21:53.702188+0000","last_active":"2024-09-13T14:22:33.431142+0000","last_peered":"2024-09-13T14:22:33.431142+0000","last_clean":"2024-09-13T14:22:33.431142+0000","last_became_active":"2024-09-13T14:21:53.701891+0000","last_became_peered":"2024-09-13T14:21:53.701891+0000","last_unstale":"2024-09-13T14:22:33.431142+0000","last_undegraded":"2024-09-13T14:22:33.431142+0000","last_fullsized":"2024-09-13T14:22:33.431142+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:14.928171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432148+0000","last_change":"2024-09-13T14:21:52.629401+0000","last_active":"2024-09-13T14:22:33.432148+0000","last_peered":"2024-09-13T14:22:33.432148+0000","last_clean":"2024-09-13T14:22:33.432148+0000","last_became_active":"2024-09-13T14:21:52.629064+0000","last_became_peered":"2024-09-13T14:21:52.629064+0000","last_unstale":"2024-09-13T14:22:33.432148+0000","last_undegraded":"2024-09-13T14:22:33.432148+0000","last_fullsized":"2024-09-13T14:22:33.432148+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:00.141247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427295+0000","last_change":"2024-09-13T14:21:53.632052+0000","last_active":"2024-09-13T14:22:33.427295+0000","last_peered":"2024-09-13T14:22:33.427295+0000","last_clean":"2024-09-13T14:22:33.427295+0000","last_became_active":"2024-09-13T14:21:53.631758+0000","last_became_peered":"2024-09-13T14:21:53.631758+0000","last_unstale":"2024-09-13T14:22:33.427295+0000","last_undegraded":"2024-09-13T14:22:33.427295+0000","last_fullsized":"2024-09-13T14:22:33.427295+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:03.312429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435572+0000","last_change":"2024-09-13T14:21:52.614951+0000","last_active":"2024-09-13T14:22:33.435572+0000","last_peered":"2024-09-13T14:22:33.435572+0000","last_clean":"2024-09-13T14:22:33.435572+0000","last_became_active":"2024-09-13T14:21:52.614695+0000","last_became_peered":"2024-09-13T14:21:52.614695+0000","last_unstale":"2024-09-13T14:22:33.435572+0000","last_undegraded":"2024-09-13T14:22:33.435572+0000","last_fullsized":"2024-09-13T14:22:33.435572+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:52.672387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427546+0000","last_change":"2024-09-13T14:21:53.702626+0000","last_active":"2024-09-13T14:22:33.427546+0000","last_peered":"2024-09-13T14:22:33.427546+0000","last_clean":"2024-09-13T14:22:33.427546+0000","last_became_active":"2024-09-13T14:21:53.701967+0000","last_became_peered":"2024-09-13T14:21:53.701967+0000","last_unstale":"2024-09-13T14:22:33.427546+0000","last_undegraded":"2024-09-13T14:22:33.427546+0000","last_fullsized":"2024-09-13T14:22:33.427546+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:27.005181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427078+0000","last_change":"2024-09-13T14:21:52.613309+0000","last_active":"2024-09-13T14:22:33.427078+0000","last_peered":"2024-09-13T14:22:33.427078+0000","last_clean":"2024-09-13T14:22:33.427078+0000","last_became_active":"2024-09-13T14:21:52.613034+0000","last_became_peered":"2024-09-13T14:21:52.613034+0000","last_unstale":"2024-09-13T14:22:33.427078+0000","last_undegraded":"2024-09-13T14:22:33.427078+0000","last_fullsized":"2024-09-13T14:22:33.427078+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:18.917592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427573+0000","last_change":"2024-09-13T14:21:53.702793+0000","last_active":"2024-09-13T14:22:33.427573+0000","last_peered":"2024-09-13T14:22:33.427573+0000","last_clean":"2024-09-13T14:22:33.427573+0000","last_became_active":"2024-09-13T14:21:53.700733+0000","last_became_peered":"2024-09-13T14:21:53.700733+0000","last_unstale":"2024-09-13T14:22:33.427573+0000","last_undegraded":"2024-09-13T14:22:33.427573+0000","last_fullsized":"2024-09-13T14:22:33.427573+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:12.693658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431876+0000","last_change":"2024-09-13T14:21:52.627072+0000","last_active":"2024-09-13T14:22:33.431876+0000","last_peered":"2024-09-13T14:22:33.431876+0000","last_clean":"2024-09-13T14:22:33.431876+0000","last_became_active":"2024-09-13T14:21:52.626731+0000","last_became_peered":"2024-09-13T14:21:52.626731+0000","last_unstale":"2024-09-13T14:22:33.431876+0000","last_undegraded":"2024-09-13T14:22:33.431876+0000","last_fullsized":"2024-09-13T14:22:33.431876+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:21.096584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433464+0000","last_change":"2024-09-13T14:21:53.294753+0000","last_active":"2024-09-13T14:22:33.433464+0000","last_peered":"2024-09-13T14:22:33.433464+0000","last_clean":"2024-09-13T14:22:33.433464+0000","last_became_active":"2024-09-13T14:21:53.294309+0000","last_became_peered":"2024-09-13T14:21:53.294309+0000","last_unstale":"2024-09-13T14:22:33.433464+0000","last_undegraded":"2024-09-13T14:22:33.433464+0000","last_fullsized":"2024-09-13T14:22:33.433464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:38.991136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"53'79","reported_seq":55,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435765+0000","last_change":"2024-09-13T14:20:01.261809+0000","last_active":"2024-09-13T14:22:33.435765+0000","last_peered":"2024-09-13T14:22:33.435765+0000","last_clean":"2024-09-13T14:22:33.435765+0000","last_became_active":"2024-09-13T14:20:01.256039+0000","last_became_peered":"2024-09-13T14:20:01.256039+0000","last_unstale":"2024-09-13T14:22:33.435765+0000","last_undegraded":"2024-09-13T14:22:33.435765+0000","last_fullsized":"2024-09-13T14:22:33.435765+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:01.420572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":26,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437277+0000","last_change":"2024-09-13T14:20:41.392754+0000","last_active":"2024-09-13T14:22:33.437277+0000","last_peered":"2024-09-13T14:22:33.437277+0000","last_clean":"2024-09-13T14:22:33.437277+0000","last_became_active":"2024-09-13T14:20:37.356130+0000","last_became_peered":"2024-09-13T14:20:37.356130+0000","last_unstale":"2024-09-13T14:22:33.437277+0000","last_undegraded":"2024-09-13T14:22:33.437277+0000","last_fullsized":"2024-09-13T14:22:33.437277+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:50.167135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432013+0000","last_change":"2024-09-13T14:21:52.612472+0000","last_active":"2024-09-13T14:22:33.432013+0000","last_peered":"2024-09-13T14:22:33.432013+0000","last_clean":"2024-09-13T14:22:33.432013+0000","last_became_active":"2024-09-13T14:21:52.612054+0000","last_became_peered":"2024-09-13T14:21:52.612054+0000","last_unstale":"2024-09-13T14:22:33.432013+0000","last_undegraded":"2024-09-13T14:22:33.432013+0000","last_fullsized":"2024-09-13T14:22:33.432013+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:39.708462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435026+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:22:33.435026+0000","last_peered":"2024-09-13T14:22:33.435026+0000","last_clean":"2024-09-13T14:22:33.435026+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:22:33.435026+0000","last_undegraded":"2024-09-13T14:22:33.435026+0000","last_fullsized":"2024-09-13T14:22:33.435026+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435656+0000","last_change":"2024-09-13T14:20:41.646316+0000","last_active":"2024-09-13T14:22:33.435656+0000","last_peered":"2024-09-13T14:22:33.435656+0000","last_clean":"2024-09-13T14:22:33.435656+0000","last_became_active":"2024-09-13T14:20:37.358143+0000","last_became_peered":"2024-09-13T14:20:37.358143+0000","last_unstale":"2024-09-13T14:22:33.435656+0000","last_undegraded":"2024-09-13T14:22:33.435656+0000","last_fullsized":"2024-09-13T14:22:33.435656+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:37.774534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431566+0000","last_change":"2024-09-13T14:21:52.614291+0000","last_active":"2024-09-13T14:22:33.431566+0000","last_peered":"2024-09-13T14:22:33.431566+0000","last_clean":"2024-09-13T14:22:33.431566+0000","last_became_active":"2024-09-13T14:21:52.614011+0000","last_became_peered":"2024-09-13T14:21:52.614011+0000","last_unstale":"2024-09-13T14:22:33.431566+0000","last_undegraded":"2024-09-13T14:22:33.431566+0000","last_fullsized":"2024-09-13T14:22:33.431566+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:59.130886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431873+0000","last_change":"2024-09-13T14:21:53.700128+0000","last_active":"2024-09-13T14:22:33.431873+0000","last_peered":"2024-09-13T14:22:33.431873+0000","last_clean":"2024-09-13T14:22:33.431873+0000","last_became_active":"2024-09-13T14:21:53.699826+0000","last_became_peered":"2024-09-13T14:21:53.699826+0000","last_unstale":"2024-09-13T14:22:33.431873+0000","last_undegraded":"2024-09-13T14:22:33.431873+0000","last_fullsized":"2024-09-13T14:22:33.431873+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:35.426011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.429438+0000","last_change":"2024-09-13T14:20:41.645840+0000","last_active":"2024-09-13T14:22:33.429438+0000","last_peered":"2024-09-13T14:22:33.429438+0000","last_clean":"2024-09-13T14:22:33.429438+0000","last_became_active":"2024-09-13T14:20:37.388195+0000","last_became_peered":"2024-09-13T14:20:37.388195+0000","last_unstale":"2024-09-13T14:22:33.429438+0000","last_undegraded":"2024-09-13T14:22:33.429438+0000","last_fullsized":"2024-09-13T14:22:33.429438+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:39.037088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431193+0000","last_change":"2024-09-13T14:21:52.606822+0000","last_active":"2024-09-13T14:22:33.431193+0000","last_peered":"2024-09-13T14:22:33.431193+0000","last_clean":"2024-09-13T14:22:33.431193+0000","last_became_active":"2024-09-13T14:21:52.606517+0000","last_became_peered":"2024-09-13T14:21:52.606517+0000","last_unstale":"2024-09-13T14:22:33.431193+0000","last_undegraded":"2024-09-13T14:22:33.431193+0000","last_fullsized":"2024-09-13T14:22:33.431193+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:13:09.268798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433547+0000","last_change":"2024-09-13T14:21:52.764427+0000","last_active":"2024-09-13T14:22:33.433547+0000","last_peered":"2024-09-13T14:22:33.433547+0000","last_clean":"2024-09-13T14:22:33.433547+0000","last_became_active":"2024-09-13T14:21:52.764139+0000","last_became_peered":"2024-09-13T14:21:52.764139+0000","last_unstale":"2024-09-13T14:22:33.433547+0000","last_undegraded":"2024-09-13T14:22:33.433547+0000","last_fullsized":"2024-09-13T14:22:33.433547+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:46.462801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":27,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437331+0000","last_change":"2024-09-13T14:20:41.393368+0000","last_active":"2024-09-13T14:22:33.437331+0000","last_peered":"2024-09-13T14:22:33.437331+0000","last_clean":"2024-09-13T14:22:33.437331+0000","last_became_active":"2024-09-13T14:20:37.357807+0000","last_became_peered":"2024-09-13T14:20:37.357807+0000","last_unstale":"2024-09-13T14:22:33.437331+0000","last_undegraded":"2024-09-13T14:22:33.437331+0000","last_fullsized":"2024-09-13T14:22:33.437331+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:17.845640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427448+0000","last_change":"2024-09-13T14:21:52.615398+0000","last_active":"2024-09-13T14:22:33.427448+0000","last_peered":"2024-09-13T14:22:33.427448+0000","last_clean":"2024-09-13T14:22:33.427448+0000","last_became_active":"2024-09-13T14:21:52.614114+0000","last_became_peered":"2024-09-13T14:21:52.614114+0000","last_unstale":"2024-09-13T14:22:33.427448+0000","last_undegraded":"2024-09-13T14:22:33.427448+0000","last_fullsized":"2024-09-13T14:22:33.427448+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:20.209300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431662+0000","last_change":"2024-09-13T14:21:52.635641+0000","last_active":"2024-09-13T14:22:33.431662+0000","last_peered":"2024-09-13T14:22:33.431662+0000","last_clean":"2024-09-13T14:22:33.431662+0000","last_became_active":"2024-09-13T14:21:52.635342+0000","last_became_peered":"2024-09-13T14:21:52.635342+0000","last_unstale":"2024-09-13T14:22:33.431662+0000","last_undegraded":"2024-09-13T14:22:33.431662+0000","last_fullsized":"2024-09-13T14:22:33.431662+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:11.062694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437585+0000","last_change":"2024-09-13T14:21:52.617052+0000","last_active":"2024-09-13T14:22:33.437585+0000","last_peered":"2024-09-13T14:22:33.437585+0000","last_clean":"2024-09-13T14:22:33.437585+0000","last_became_active":"2024-09-13T14:21:52.615024+0000","last_became_peered":"2024-09-13T14:21:52.615024+0000","last_unstale":"2024-09-13T14:22:33.437585+0000","last_undegraded":"2024-09-13T14:22:33.437585+0000","last_fullsized":"2024-09-13T14:22:33.437585+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:26.971812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431800+0000","last_change":"2024-09-13T14:20:41.389300+0000","last_active":"2024-09-13T14:22:33.431800+0000","last_peered":"2024-09-13T14:22:33.431800+0000","last_clean":"2024-09-13T14:22:33.431800+0000","last_became_active":"2024-09-13T14:20:37.356625+0000","last_became_peered":"2024-09-13T14:20:37.356625+0000","last_unstale":"2024-09-13T14:22:33.431800+0000","last_undegraded":"2024-09-13T14:22:33.431800+0000","last_fullsized":"2024-09-13T14:22:33.431800+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:28.847614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433500+0000","last_change":"2024-09-13T14:21:53.294786+0000","last_active":"2024-09-13T14:22:33.433500+0000","last_peered":"2024-09-13T14:22:33.433500+0000","last_clean":"2024-09-13T14:22:33.433500+0000","last_became_active":"2024-09-13T14:21:53.294437+0000","last_became_peered":"2024-09-13T14:21:53.294437+0000","last_unstale":"2024-09-13T14:22:33.433500+0000","last_undegraded":"2024-09-13T14:22:33.433500+0000","last_fullsized":"2024-09-13T14:22:33.433500+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:12.928998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431733+0000","last_change":"2024-09-13T14:21:52.611934+0000","last_active":"2024-09-13T14:22:33.431733+0000","last_peered":"2024-09-13T14:22:33.431733+0000","last_clean":"2024-09-13T14:22:33.431733+0000","last_became_active":"2024-09-13T14:21:52.611529+0000","last_became_peered":"2024-09-13T14:21:52.611529+0000","last_unstale":"2024-09-13T14:22:33.431733+0000","last_undegraded":"2024-09-13T14:22:33.431733+0000","last_fullsized":"2024-09-13T14:22:33.431733+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:18.891936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436638+0000","last_change":"2024-09-13T14:20:41.646302+0000","last_active":"2024-09-13T14:22:33.436638+0000","last_peered":"2024-09-13T14:22:33.436638+0000","last_clean":"2024-09-13T14:22:33.436638+0000","last_became_active":"2024-09-13T14:20:37.356803+0000","last_became_peered":"2024-09-13T14:20:37.356803+0000","last_unstale":"2024-09-13T14:22:33.436638+0000","last_undegraded":"2024-09-13T14:22:33.436638+0000","last_fullsized":"2024-09-13T14:22:33.436638+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:41.431722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431209+0000","last_change":"2024-09-13T14:21:53.699505+0000","last_active":"2024-09-13T14:22:33.431209+0000","last_peered":"2024-09-13T14:22:33.431209+0000","last_clean":"2024-09-13T14:22:33.431209+0000","last_became_active":"2024-09-13T14:21:53.699096+0000","last_became_peered":"2024-09-13T14:21:53.699096+0000","last_unstale":"2024-09-13T14:22:33.431209+0000","last_undegraded":"2024-09-13T14:22:33.431209+0000","last_fullsized":"2024-09-13T14:22:33.431209+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:56.561317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433090+0000","last_change":"2024-09-13T14:21:52.627331+0000","last_active":"2024-09-13T14:22:33.433090+0000","last_peered":"2024-09-13T14:22:33.433090+0000","last_clean":"2024-09-13T14:22:33.433090+0000","last_became_active":"2024-09-13T14:21:52.627043+0000","last_became_peered":"2024-09-13T14:21:52.627043+0000","last_unstale":"2024-09-13T14:22:33.433090+0000","last_undegraded":"2024-09-13T14:22:33.433090+0000","last_fullsized":"2024-09-13T14:22:33.433090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:23.898082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432482+0000","last_change":"2024-09-13T14:20:41.389271+0000","last_active":"2024-09-13T14:22:33.432482+0000","last_peered":"2024-09-13T14:22:33.432482+0000","last_clean":"2024-09-13T14:22:33.432482+0000","last_became_active":"2024-09-13T14:20:37.355416+0000","last_became_peered":"2024-09-13T14:20:37.355416+0000","last_unstale":"2024-09-13T14:22:33.432482+0000","last_undegraded":"2024-09-13T14:22:33.432482+0000","last_fullsized":"2024-09-13T14:22:33.432482+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:47.155583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026697500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431715+0000","last_change":"2024-09-13T14:21:53.698556+0000","last_active":"2024-09-13T14:22:33.431715+0000","last_peered":"2024-09-13T14:22:33.431715+0000","last_clean":"2024-09-13T14:22:33.431715+0000","last_became_active":"2024-09-13T14:21:53.698245+0000","last_became_peered":"2024-09-13T14:21:53.698245+0000","last_unstale":"2024-09-13T14:22:33.431715+0000","last_undegraded":"2024-09-13T14:22:33.431715+0000","last_fullsized":"2024-09-13T14:22:33.431715+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:05.325683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432010+0000","last_change":"2024-09-13T14:21:52.623973+0000","last_active":"2024-09-13T14:22:33.432010+0000","last_peered":"2024-09-13T14:22:33.432010+0000","last_clean":"2024-09-13T14:22:33.432010+0000","last_became_active":"2024-09-13T14:21:52.623650+0000","last_became_peered":"2024-09-13T14:21:52.623650+0000","last_unstale":"2024-09-13T14:22:33.432010+0000","last_undegraded":"2024-09-13T14:22:33.432010+0000","last_fullsized":"2024-09-13T14:22:33.432010+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:46.904029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435117+0000","last_change":"2024-09-13T14:20:41.394322+0000","last_active":"2024-09-13T14:22:33.435117+0000","last_peered":"2024-09-13T14:22:33.435117+0000","last_clean":"2024-09-13T14:22:33.435117+0000","last_became_active":"2024-09-13T14:20:37.356874+0000","last_became_peered":"2024-09-13T14:20:37.356874+0000","last_unstale":"2024-09-13T14:22:33.435117+0000","last_undegraded":"2024-09-13T14:22:33.435117+0000","last_fullsized":"2024-09-13T14:22:33.435117+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:05.446185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.426923+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:22:33.426923+0000","last_peered":"2024-09-13T14:22:33.426923+0000","last_clean":"2024-09-13T14:22:33.426923+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:22:33.426923+0000","last_undegraded":"2024-09-13T14:22:33.426923+0000","last_fullsized":"2024-09-13T14:22:33.426923+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427307+0000","last_change":"2024-09-13T14:21:53.702034+0000","last_active":"2024-09-13T14:22:33.427307+0000","last_peered":"2024-09-13T14:22:33.427307+0000","last_clean":"2024-09-13T14:22:33.427307+0000","last_became_active":"2024-09-13T14:21:53.701326+0000","last_became_peered":"2024-09-13T14:21:53.701326+0000","last_unstale":"2024-09-13T14:22:33.427307+0000","last_undegraded":"2024-09-13T14:22:33.427307+0000","last_fullsized":"2024-09-13T14:22:33.427307+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:26.139551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"65'6","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427398+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:22:33.427398+0000","last_peered":"2024-09-13T14:22:33.427398+0000","last_clean":"2024-09-13T14:22:33.427398+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:22:33.427398+0000","last_undegraded":"2024-09-13T14:22:33.427398+0000","last_fullsized":"2024-09-13T14:22:33.427398+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433481+0000","last_change":"2024-09-13T14:21:52.983010+0000","last_active":"2024-09-13T14:22:33.433481+0000","last_peered":"2024-09-13T14:22:33.433481+0000","last_clean":"2024-09-13T14:22:33.433481+0000","last_became_active":"2024-09-13T14:21:52.982656+0000","last_became_peered":"2024-09-13T14:21:52.982656+0000","last_unstale":"2024-09-13T14:22:33.433481+0000","last_undegraded":"2024-09-13T14:22:33.433481+0000","last_fullsized":"2024-09-13T14:22:33.433481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:20.179211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436505+0000","last_change":"2024-09-13T14:21:52.631712+0000","last_active":"2024-09-13T14:22:33.436505+0000","last_peered":"2024-09-13T14:22:33.436505+0000","last_clean":"2024-09-13T14:22:33.436505+0000","last_became_active":"2024-09-13T14:21:52.631439+0000","last_became_peered":"2024-09-13T14:21:52.631439+0000","last_unstale":"2024-09-13T14:22:33.436505+0000","last_undegraded":"2024-09-13T14:22:33.436505+0000","last_fullsized":"2024-09-13T14:22:33.436505+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:15.406379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437475+0000","last_change":"2024-09-13T14:21:52.613380+0000","last_active":"2024-09-13T14:22:33.437475+0000","last_peered":"2024-09-13T14:22:33.437475+0000","last_clean":"2024-09-13T14:22:33.437475+0000","last_became_active":"2024-09-13T14:21:52.612586+0000","last_became_peered":"2024-09-13T14:21:52.612586+0000","last_unstale":"2024-09-13T14:22:33.437475+0000","last_undegraded":"2024-09-13T14:22:33.437475+0000","last_fullsized":"2024-09-13T14:22:33.437475+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:37.416155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431125+0000","last_change":"2024-09-13T14:21:53.699866+0000","last_active":"2024-09-13T14:22:33.431125+0000","last_peered":"2024-09-13T14:22:33.431125+0000","last_clean":"2024-09-13T14:22:33.431125+0000","last_became_active":"2024-09-13T14:21:53.699536+0000","last_became_peered":"2024-09-13T14:21:53.699536+0000","last_unstale":"2024-09-13T14:22:33.431125+0000","last_undegraded":"2024-09-13T14:22:33.431125+0000","last_fullsized":"2024-09-13T14:22:33.431125+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:09.594385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432448+0000","last_change":"2024-09-13T14:21:52.624373+0000","last_active":"2024-09-13T14:22:33.432448+0000","last_peered":"2024-09-13T14:22:33.432448+0000","last_clean":"2024-09-13T14:22:33.432448+0000","last_became_active":"2024-09-13T14:21:52.624139+0000","last_became_peered":"2024-09-13T14:21:52.624139+0000","last_unstale":"2024-09-13T14:22:33.432448+0000","last_undegraded":"2024-09-13T14:22:33.432448+0000","last_fullsized":"2024-09-13T14:22:33.432448+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:53:39.592214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431638+0000","last_change":"2024-09-13T14:21:53.699201+0000","last_active":"2024-09-13T14:22:33.431638+0000","last_peered":"2024-09-13T14:22:33.431638+0000","last_clean":"2024-09-13T14:22:33.431638+0000","last_became_active":"2024-09-13T14:21:53.698927+0000","last_became_peered":"2024-09-13T14:21:53.698927+0000","last_unstale":"2024-09-13T14:22:33.431638+0000","last_undegraded":"2024-09-13T14:22:33.431638+0000","last_fullsized":"2024-09-13T14:22:33.431638+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:10.138274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.434957+0000","last_change":"2024-09-13T14:21:53.701444+0000","last_active":"2024-09-13T14:22:33.434957+0000","last_peered":"2024-09-13T14:22:33.434957+0000","last_clean":"2024-09-13T14:22:33.434957+0000","last_became_active":"2024-09-13T14:21:53.701175+0000","last_became_peered":"2024-09-13T14:21:53.701175+0000","last_unstale":"2024-09-13T14:22:33.434957+0000","last_undegraded":"2024-09-13T14:22:33.434957+0000","last_fullsized":"2024-09-13T14:22:33.434957+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:39.842585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431801+0000","last_change":"2024-09-13T14:21:52.624953+0000","last_active":"2024-09-13T14:22:33.431801+0000","last_peered":"2024-09-13T14:22:33.431801+0000","last_clean":"2024-09-13T14:22:33.431801+0000","last_became_active":"2024-09-13T14:21:52.624360+0000","last_became_peered":"2024-09-13T14:21:52.624360+0000","last_unstale":"2024-09-13T14:22:33.431801+0000","last_undegraded":"2024-09-13T14:22:33.431801+0000","last_fullsized":"2024-09-13T14:22:33.431801+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:18.568935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431804+0000","last_change":"2024-09-13T14:21:53.700395+0000","last_active":"2024-09-13T14:22:33.431804+0000","last_peered":"2024-09-13T14:22:33.431804+0000","last_clean":"2024-09-13T14:22:33.431804+0000","last_became_active":"2024-09-13T14:21:53.700057+0000","last_became_peered":"2024-09-13T14:21:53.700057+0000","last_unstale":"2024-09-13T14:22:33.431804+0000","last_undegraded":"2024-09-13T14:22:33.431804+0000","last_fullsized":"2024-09-13T14:22:33.431804+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:45.627991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431421+0000","last_change":"2024-09-13T14:21:52.625583+0000","last_active":"2024-09-13T14:22:33.431421+0000","last_peered":"2024-09-13T14:22:33.431421+0000","last_clean":"2024-09-13T14:22:33.431421+0000","last_became_active":"2024-09-13T14:21:52.622107+0000","last_became_peered":"2024-09-13T14:21:52.622107+0000","last_unstale":"2024-09-13T14:22:33.431421+0000","last_undegraded":"2024-09-13T14:22:33.431421+0000","last_fullsized":"2024-09-13T14:22:33.431421+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:12.415569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432148+0000","last_change":"2024-09-13T14:21:53.701315+0000","last_active":"2024-09-13T14:22:33.432148+0000","last_peered":"2024-09-13T14:22:33.432148+0000","last_clean":"2024-09-13T14:22:33.432148+0000","last_became_active":"2024-09-13T14:21:53.701014+0000","last_became_peered":"2024-09-13T14:21:53.701014+0000","last_unstale":"2024-09-13T14:22:33.432148+0000","last_undegraded":"2024-09-13T14:22:33.432148+0000","last_fullsized":"2024-09-13T14:22:33.432148+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:48.507923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427067+0000","last_change":"2024-09-13T14:21:52.623311+0000","last_active":"2024-09-13T14:22:33.427067+0000","last_peered":"2024-09-13T14:22:33.427067+0000","last_clean":"2024-09-13T14:22:33.427067+0000","last_became_active":"2024-09-13T14:21:52.623015+0000","last_became_peered":"2024-09-13T14:21:52.623015+0000","last_unstale":"2024-09-13T14:22:33.427067+0000","last_undegraded":"2024-09-13T14:22:33.427067+0000","last_fullsized":"2024-09-13T14:22:33.427067+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:51.869610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.429501+0000","last_change":"2024-09-13T14:21:53.700992+0000","last_active":"2024-09-13T14:22:33.429501+0000","last_peered":"2024-09-13T14:22:33.429501+0000","last_clean":"2024-09-13T14:22:33.429501+0000","last_became_active":"2024-09-13T14:21:53.700681+0000","last_became_peered":"2024-09-13T14:21:53.700681+0000","last_unstale":"2024-09-13T14:22:33.429501+0000","last_undegraded":"2024-09-13T14:22:33.429501+0000","last_fullsized":"2024-09-13T14:22:33.429501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:42.030268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431331+0000","last_change":"2024-09-13T14:21:52.615630+0000","last_active":"2024-09-13T14:22:33.431331+0000","last_peered":"2024-09-13T14:22:33.431331+0000","last_clean":"2024-09-13T14:22:33.431331+0000","last_became_active":"2024-09-13T14:21:52.614872+0000","last_became_peered":"2024-09-13T14:21:52.614872+0000","last_unstale":"2024-09-13T14:22:33.431331+0000","last_undegraded":"2024-09-13T14:22:33.431331+0000","last_fullsized":"2024-09-13T14:22:33.431331+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:34.605661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T14:22:34.432604+0000","last_change":"2024-09-13T14:22:34.432604+0000","last_active":"2024-09-13T14:22:34.432604+0000","last_peered":"2024-09-13T14:22:34.432604+0000","last_clean":"2024-09-13T14:22:34.432604+0000","last_became_active":"2024-09-13T14:22:34.432418+0000","last_became_peered":"2024-09-13T14:22:34.432418+0000","last_unstale":"2024-09-13T14:22:34.432604+0000","last_undegraded":"2024-09-13T14:22:34.432604+0000","last_fullsized":"2024-09-13T14:22:34.432604+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:18.113553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437279+0000","last_change":"2024-09-13T14:21:52.617742+0000","last_active":"2024-09-13T14:22:33.437279+0000","last_peered":"2024-09-13T14:22:33.437279+0000","last_clean":"2024-09-13T14:22:33.437279+0000","last_became_active":"2024-09-13T14:21:52.616904+0000","last_became_peered":"2024-09-13T14:21:52.616904+0000","last_unstale":"2024-09-13T14:22:33.437279+0000","last_undegraded":"2024-09-13T14:22:33.437279+0000","last_fullsized":"2024-09-13T14:22:33.437279+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:53.145715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.426962+0000","last_change":"2024-09-13T14:21:53.632083+0000","last_active":"2024-09-13T14:22:33.426962+0000","last_peered":"2024-09-13T14:22:33.426962+0000","last_clean":"2024-09-13T14:22:33.426962+0000","last_became_active":"2024-09-13T14:21:53.631784+0000","last_became_peered":"2024-09-13T14:21:53.631784+0000","last_unstale":"2024-09-13T14:22:33.426962+0000","last_undegraded":"2024-09-13T14:22:33.426962+0000","last_fullsized":"2024-09-13T14:22:33.426962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:18.161083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437550+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:22:33.437550+0000","last_peered":"2024-09-13T14:22:33.437550+0000","last_clean":"2024-09-13T14:22:33.437550+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:22:33.437550+0000","last_undegraded":"2024-09-13T14:22:33.437550+0000","last_fullsized":"2024-09-13T14:22:33.437550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431365+0000","last_change":"2024-09-13T14:21:53.632016+0000","last_active":"2024-09-13T14:22:33.431365+0000","last_peered":"2024-09-13T14:22:33.431365+0000","last_clean":"2024-09-13T14:22:33.431365+0000","last_became_active":"2024-09-13T14:21:53.631724+0000","last_became_peered":"2024-09-13T14:21:53.631724+0000","last_unstale":"2024-09-13T14:22:33.431365+0000","last_undegraded":"2024-09-13T14:22:33.431365+0000","last_fullsized":"2024-09-13T14:22:33.431365+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:56.931830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433045+0000","last_change":"2024-09-13T14:21:52.620927+0000","last_active":"2024-09-13T14:22:33.433045+0000","last_peered":"2024-09-13T14:22:33.433045+0000","last_clean":"2024-09-13T14:22:33.433045+0000","last_became_active":"2024-09-13T14:21:52.620570+0000","last_became_peered":"2024-09-13T14:21:52.620570+0000","last_unstale":"2024-09-13T14:22:33.433045+0000","last_undegraded":"2024-09-13T14:22:33.433045+0000","last_fullsized":"2024-09-13T14:22:33.433045+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:28.683972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.426792+0000","last_change":"2024-09-13T14:21:53.699582+0000","last_active":"2024-09-13T14:22:33.426792+0000","last_peered":"2024-09-13T14:22:33.426792+0000","last_clean":"2024-09-13T14:22:33.426792+0000","last_became_active":"2024-09-13T14:21:53.698531+0000","last_became_peered":"2024-09-13T14:21:53.698531+0000","last_unstale":"2024-09-13T14:22:33.426792+0000","last_undegraded":"2024-09-13T14:22:33.426792+0000","last_fullsized":"2024-09-13T14:22:33.426792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:56.552336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432367+0000","last_change":"2024-09-13T14:21:52.619193+0000","last_active":"2024-09-13T14:22:33.432367+0000","last_peered":"2024-09-13T14:22:33.432367+0000","last_clean":"2024-09-13T14:22:33.432367+0000","last_became_active":"2024-09-13T14:21:52.618685+0000","last_became_peered":"2024-09-13T14:21:52.618685+0000","last_unstale":"2024-09-13T14:22:33.432367+0000","last_undegraded":"2024-09-13T14:22:33.432367+0000","last_fullsized":"2024-09-13T14:22:33.432367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:04.725931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364836,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298628,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446524,"statfs":{"total":95995035648,"available":95689240576,"internally_reserved":0,"allocated":1302528,"data_stored":893190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68200000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.65400000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.42399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72599999999999998,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.498},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79600000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76000000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55800000000000005}]},{"osd":3,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56499999999999995,"15min":0.56399999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76500000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61899999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.497},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70899999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68899999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46800000000000003}]},{"osd":5,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.44600000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58999999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.54400000000000004}]}]},{"osd":6,"up_from":44,"seq":188978561066,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298620,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446532,"statfs":{"total":95995035648,"available":95689248768,"internally_reserved":0,"allocated":1294336,"data_stored":893013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.36099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66500000000000004}]},{"osd":1,"last update":"Fri Sep 13 14:21:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.76300000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.68999999999999995}]},{"osd":2,"last update":"Fri Sep 13 14:21:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.42599999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72699999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.60099999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:21:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57599999999999996,"15min":0.57199999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.78000000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:21:43 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.48499999999999999,"15min":0.47499999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.61899999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:21:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.46700000000000003,"15min":0.45300000000000001},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.41399999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51000000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58499999999999996}]}]},{"osd":5,"up_from":37,"seq":158913789998,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298028,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447124,"statfs":{"total":95995035648,"available":95689854976,"internally_reserved":0,"allocated":688128,"data_stored":301419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.68400000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.372,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.71599999999999997,"5min":0.752,"15min":0.752},"last":0.50600000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68600000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.505,"15min":0.50800000000000001},"min":{"1min":0.40899999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66800000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.52300000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.376,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65500000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.371,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70699999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.51200000000000001,"15min":0.502},"min":{"1min":0.378,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.54900000000000004},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68300000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.45500000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.42299999999999999,"15min":0.40200000000000002},"min":{"1min":0.28299999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.42799999999999999,"15min":0.40000000000000002},"min":{"1min":0.311,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.374}]},{"osd":6,"last update":"Fri Sep 13 14:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.47499999999999998,"15min":0.46000000000000002},"min":{"1min":0.32000000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.495},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.46999999999999997,"15min":0.45400000000000001},"min":{"1min":0.32700000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.439}]},{"osd":7,"last update":"Fri Sep 13 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.314,"15min":0.314},"max":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.34599999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72899999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.38800000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986226,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298048,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447104,"statfs":{"total":95995035648,"available":95689834496,"internally_reserved":0,"allocated":708608,"data_stored":303017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71799999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.375},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.52600000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.52000000000000002},"min":{"1min":0.36399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73999999999999999,"5min":0.747,"15min":0.747},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.40899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.62},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55200000000000005,"15min":0.55000000000000004},"min":{"1min":0.33300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76200000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60499999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54700000000000004,"15min":0.54600000000000004},"min":{"1min":0.38800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.39600000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73099999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.41799999999999998,"15min":0.39600000000000002},"min":{"1min":0.32600000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70699999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.42899999999999999,"15min":0.39900000000000002},"min":{"1min":0.29899999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.73299999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.68400000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45200000000000001,"15min":0.436},"min":{"1min":0.28299999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73299999999999998,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51300000000000001,"15min":0.49099999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67700000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59499999999999997}]},{"osd":7,"last update":"Fri Sep 13 14:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81200000000000006,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.74099999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298032,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447120,"statfs":{"total":95995035648,"available":95689850880,"internally_reserved":0,"allocated":688128,"data_stored":301409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.35699999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.54900000000000004,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.40200000000000002},"min":{"1min":0.247,"5min":0.223,"15min":0.223},"max":{"1min":0.55500000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.374}]},{"osd":1,"last update":"Fri Sep 13 14:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40699999999999997,"15min":0.41999999999999998},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.26500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.54100000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.38400000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.432,"15min":0.436},"min":{"1min":0.33900000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56499999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40699999999999997,"15min":0.41199999999999998},"min":{"1min":0.29499999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.52100000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.41499999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:21:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59299999999999997,"15min":0.61299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.629},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.68899999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63100000000000001,"15min":0.64900000000000002},"min":{"1min":0.42799999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.75600000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.65900000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81100000000000005,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70899999999999996}]},{"osd":6,"last update":"Fri Sep 13 14:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.46999999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60299999999999998,"15min":0.61899999999999999},"min":{"1min":0.42299999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.84599999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72899999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:22:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58599999999999997,"15min":0.58899999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.442,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82499999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.74099999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298036,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447116,"statfs":{"total":95995035648,"available":95689846784,"internally_reserved":0,"allocated":692224,"data_stored":301923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.34499999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.55800000000000005,"5min":0.623,"15min":0.623},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36799999999999999,"15min":0.35099999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53600000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.22500000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.58599999999999997,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40500000000000003,"15min":0.39200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59699999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38500000000000001}]},{"osd":3,"last update":"Fri Sep 13 14:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28999999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.254}]},{"osd":4,"last update":"Fri Sep 13 14:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.42499999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.51000000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56299999999999994,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.434}]},{"osd":6,"last update":"Fri Sep 13 14:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.443,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79700000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.82199999999999995,"5min":1.016,"15min":1.016},"last":0.45900000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.60399999999999998,"15min":0.58899999999999997},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.497},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.54400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298032,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447120,"statfs":{"total":95995035648,"available":95689850880,"internally_reserved":0,"allocated":688128,"data_stored":301338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.373,"15min":0.34499999999999997},"min":{"1min":0.28100000000000003,"5min":0.216,"15min":0.216},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.38,"15min":0.33800000000000002},"min":{"1min":0.30399999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.58599999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.314}]},{"osd":2,"last update":"Fri Sep 13 14:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40600000000000003,"15min":0.38600000000000001},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.626,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.38800000000000001,"15min":0.374},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67400000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.35099999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.55600000000000005,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.433},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.29699999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56599999999999995,"5min":1.095,"15min":1.095},"last":0.371}]},{"osd":4,"last update":"Fri Sep 13 14:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61399999999999999,"15min":0.624},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79100000000000004,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.69099999999999995}]},{"osd":6,"last update":"Fri Sep 13 14:21:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.52700000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.624},"min":{"1min":0.47999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.84999999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Sep 13 14:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.77500000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88400000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58199999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298616,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446536,"statfs":{"total":95995035648,"available":95689252864,"internally_reserved":0,"allocated":1286144,"data_stored":892279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35299999999999998,"15min":0.33100000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.56999999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.36299999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.55900000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.52600000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40699999999999997,"15min":0.42099999999999999},"min":{"1min":0.25,"5min":0.246,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.41999999999999998,"15min":0.435},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.49299999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.51100000000000001}]},{"osd":3,"last update":"Fri Sep 13 14:22:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.45600000000000002,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63100000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.42899999999999999,"15min":0.45700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56699999999999995,"5min":0.629,"15min":0.629},"last":0.437}]},{"osd":4,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.74399999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56899999999999995,"15min":0.55200000000000005},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.70999999999999996,"5min":0.751,"15min":0.751},"last":0.63300000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.435,"5min":0.31,"15min":0.31},"max":{"1min":0.79000000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.75800000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61899999999999999,"15min":0.621},"min":{"1min":0.44500000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75900000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.37,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:22:48.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:22:48.923 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:48.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:48 smithi019 ceph-mon[32895]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:48.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:49 smithi003 ceph-mon[41592]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:22:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:49 smithi003 ceph-mon[29646]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:22:49.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:49 smithi019 ceph-mon[32895]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:22:50.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:50 smithi003 ceph-mon[29646]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:50.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:50 smithi003 ceph-mon[41592]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:50.801 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:22:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:22:50.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:50 smithi019 ceph-mon[32895]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:51.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:22:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:51.638 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-13T14:22:49.367985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386040,"kb_used_data":7176,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575176,"statfs":{"total":767960285184,"available":765516980224,"internally_reserved":0,"allocated":7348224,"data_stored":4187588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002657"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432229+0000","last_change":"2024-09-13T14:21:53.701936+0000","last_active":"2024-09-13T14:22:33.432229+0000","last_peered":"2024-09-13T14:22:33.432229+0000","last_clean":"2024-09-13T14:22:33.432229+0000","last_became_active":"2024-09-13T14:21:53.701636+0000","last_became_peered":"2024-09-13T14:21:53.701636+0000","last_unstale":"2024-09-13T14:22:33.432229+0000","last_undegraded":"2024-09-13T14:22:33.432229+0000","last_fullsized":"2024-09-13T14:22:33.432229+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:40.363836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436139+0000","last_change":"2024-09-13T14:21:52.615068+0000","last_active":"2024-09-13T14:22:33.436139+0000","last_peered":"2024-09-13T14:22:33.436139+0000","last_clean":"2024-09-13T14:22:33.436139+0000","last_became_active":"2024-09-13T14:21:52.614632+0000","last_became_peered":"2024-09-13T14:21:52.614632+0000","last_unstale":"2024-09-13T14:22:33.436139+0000","last_undegraded":"2024-09-13T14:22:33.436139+0000","last_fullsized":"2024-09-13T14:22:33.436139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:04.140321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437488+0000","last_change":"2024-09-13T14:21:52.633866+0000","last_active":"2024-09-13T14:22:33.437488+0000","last_peered":"2024-09-13T14:22:33.437488+0000","last_clean":"2024-09-13T14:22:33.437488+0000","last_became_active":"2024-09-13T14:21:52.633523+0000","last_became_peered":"2024-09-13T14:21:52.633523+0000","last_unstale":"2024-09-13T14:22:33.437488+0000","last_undegraded":"2024-09-13T14:22:33.437488+0000","last_fullsized":"2024-09-13T14:22:33.437488+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:05:33.337285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431005+0000","last_change":"2024-09-13T14:21:52.623706+0000","last_active":"2024-09-13T14:22:33.431005+0000","last_peered":"2024-09-13T14:22:33.431005+0000","last_clean":"2024-09-13T14:22:33.431005+0000","last_became_active":"2024-09-13T14:21:52.623341+0000","last_became_peered":"2024-09-13T14:21:52.623341+0000","last_unstale":"2024-09-13T14:22:33.431005+0000","last_undegraded":"2024-09-13T14:22:33.431005+0000","last_fullsized":"2024-09-13T14:22:33.431005+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:08.120479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431932+0000","last_change":"2024-09-13T14:21:53.700932+0000","last_active":"2024-09-13T14:22:33.431932+0000","last_peered":"2024-09-13T14:22:33.431932+0000","last_clean":"2024-09-13T14:22:33.431932+0000","last_became_active":"2024-09-13T14:21:53.700605+0000","last_became_peered":"2024-09-13T14:21:53.700605+0000","last_unstale":"2024-09-13T14:22:33.431932+0000","last_undegraded":"2024-09-13T14:22:33.431932+0000","last_fullsized":"2024-09-13T14:22:33.431932+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:03.340866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435930+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:22:33.435930+0000","last_peered":"2024-09-13T14:22:33.435930+0000","last_clean":"2024-09-13T14:22:33.435930+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:22:33.435930+0000","last_undegraded":"2024-09-13T14:22:33.435930+0000","last_fullsized":"2024-09-13T14:22:33.435930+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T14:22:34.901457+0000","last_change":"2024-09-13T14:22:34.901457+0000","last_active":"2024-09-13T14:22:34.901457+0000","last_peered":"2024-09-13T14:22:34.901457+0000","last_clean":"2024-09-13T14:22:34.901457+0000","last_became_active":"2024-09-13T14:22:34.901121+0000","last_became_peered":"2024-09-13T14:22:34.901121+0000","last_unstale":"2024-09-13T14:22:34.901457+0000","last_undegraded":"2024-09-13T14:22:34.901457+0000","last_fullsized":"2024-09-13T14:22:34.901457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:03.009016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.434688+0000","last_change":"2024-09-13T14:21:52.609091+0000","last_active":"2024-09-13T14:22:33.434688+0000","last_peered":"2024-09-13T14:22:33.434688+0000","last_clean":"2024-09-13T14:22:33.434688+0000","last_became_active":"2024-09-13T14:21:52.608798+0000","last_became_peered":"2024-09-13T14:21:52.608798+0000","last_unstale":"2024-09-13T14:22:33.434688+0000","last_undegraded":"2024-09-13T14:22:33.434688+0000","last_fullsized":"2024-09-13T14:22:33.434688+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:42.697256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436619+0000","last_change":"2024-09-13T14:21:52.615164+0000","last_active":"2024-09-13T14:22:33.436619+0000","last_peered":"2024-09-13T14:22:33.436619+0000","last_clean":"2024-09-13T14:22:33.436619+0000","last_became_active":"2024-09-13T14:21:52.614326+0000","last_became_peered":"2024-09-13T14:21:52.614326+0000","last_unstale":"2024-09-13T14:22:33.436619+0000","last_undegraded":"2024-09-13T14:22:33.436619+0000","last_fullsized":"2024-09-13T14:22:33.436619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:48.585715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437133+0000","last_change":"2024-09-13T14:21:52.983025+0000","last_active":"2024-09-13T14:22:33.437133+0000","last_peered":"2024-09-13T14:22:33.437133+0000","last_clean":"2024-09-13T14:22:33.437133+0000","last_became_active":"2024-09-13T14:21:52.982611+0000","last_became_peered":"2024-09-13T14:21:52.982611+0000","last_unstale":"2024-09-13T14:22:33.437133+0000","last_undegraded":"2024-09-13T14:22:33.437133+0000","last_fullsized":"2024-09-13T14:22:33.437133+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:00.851531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436340+0000","last_change":"2024-09-13T14:21:52.614799+0000","last_active":"2024-09-13T14:22:33.436340+0000","last_peered":"2024-09-13T14:22:33.436340+0000","last_clean":"2024-09-13T14:22:33.436340+0000","last_became_active":"2024-09-13T14:21:52.614520+0000","last_became_peered":"2024-09-13T14:21:52.614520+0000","last_unstale":"2024-09-13T14:22:33.436340+0000","last_undegraded":"2024-09-13T14:22:33.436340+0000","last_fullsized":"2024-09-13T14:22:33.436340+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:07.897335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.426971+0000","last_change":"2024-09-13T14:21:53.702020+0000","last_active":"2024-09-13T14:22:33.426971+0000","last_peered":"2024-09-13T14:22:33.426971+0000","last_clean":"2024-09-13T14:22:33.426971+0000","last_became_active":"2024-09-13T14:21:53.700331+0000","last_became_peered":"2024-09-13T14:21:53.700331+0000","last_unstale":"2024-09-13T14:22:33.426971+0000","last_undegraded":"2024-09-13T14:22:33.426971+0000","last_fullsized":"2024-09-13T14:22:33.426971+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:46.779119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436618+0000","last_change":"2024-09-13T14:21:52.610490+0000","last_active":"2024-09-13T14:22:33.436618+0000","last_peered":"2024-09-13T14:22:33.436618+0000","last_clean":"2024-09-13T14:22:33.436618+0000","last_became_active":"2024-09-13T14:21:52.610192+0000","last_became_peered":"2024-09-13T14:21:52.610192+0000","last_unstale":"2024-09-13T14:22:33.436618+0000","last_undegraded":"2024-09-13T14:22:33.436618+0000","last_fullsized":"2024-09-13T14:22:33.436618+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:09.564393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T14:22:34.901360+0000","last_change":"2024-09-13T14:22:34.901360+0000","last_active":"2024-09-13T14:22:34.901360+0000","last_peered":"2024-09-13T14:22:34.901360+0000","last_clean":"2024-09-13T14:22:34.901360+0000","last_became_active":"2024-09-13T14:22:34.901010+0000","last_became_peered":"2024-09-13T14:22:34.901010+0000","last_unstale":"2024-09-13T14:22:34.901360+0000","last_undegraded":"2024-09-13T14:22:34.901360+0000","last_fullsized":"2024-09-13T14:22:34.901360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:55.897004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437376+0000","last_change":"2024-09-13T14:21:52.617746+0000","last_active":"2024-09-13T14:22:33.437376+0000","last_peered":"2024-09-13T14:22:33.437376+0000","last_clean":"2024-09-13T14:22:33.437376+0000","last_became_active":"2024-09-13T14:21:52.617280+0000","last_became_peered":"2024-09-13T14:21:52.617280+0000","last_unstale":"2024-09-13T14:22:33.437376+0000","last_undegraded":"2024-09-13T14:22:33.437376+0000","last_fullsized":"2024-09-13T14:22:33.437376+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:03.243247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431142+0000","last_change":"2024-09-13T14:21:53.702188+0000","last_active":"2024-09-13T14:22:33.431142+0000","last_peered":"2024-09-13T14:22:33.431142+0000","last_clean":"2024-09-13T14:22:33.431142+0000","last_became_active":"2024-09-13T14:21:53.701891+0000","last_became_peered":"2024-09-13T14:21:53.701891+0000","last_unstale":"2024-09-13T14:22:33.431142+0000","last_undegraded":"2024-09-13T14:22:33.431142+0000","last_fullsized":"2024-09-13T14:22:33.431142+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:14.928171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432148+0000","last_change":"2024-09-13T14:21:52.629401+0000","last_active":"2024-09-13T14:22:33.432148+0000","last_peered":"2024-09-13T14:22:33.432148+0000","last_clean":"2024-09-13T14:22:33.432148+0000","last_became_active":"2024-09-13T14:21:52.629064+0000","last_became_peered":"2024-09-13T14:21:52.629064+0000","last_unstale":"2024-09-13T14:22:33.432148+0000","last_undegraded":"2024-09-13T14:22:33.432148+0000","last_fullsized":"2024-09-13T14:22:33.432148+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:00.141247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427295+0000","last_change":"2024-09-13T14:21:53.632052+0000","last_active":"2024-09-13T14:22:33.427295+0000","last_peered":"2024-09-13T14:22:33.427295+0000","last_clean":"2024-09-13T14:22:33.427295+0000","last_became_active":"2024-09-13T14:21:53.631758+0000","last_became_peered":"2024-09-13T14:21:53.631758+0000","last_unstale":"2024-09-13T14:22:33.427295+0000","last_undegraded":"2024-09-13T14:22:33.427295+0000","last_fullsized":"2024-09-13T14:22:33.427295+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:03.312429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435572+0000","last_change":"2024-09-13T14:21:52.614951+0000","last_active":"2024-09-13T14:22:33.435572+0000","last_peered":"2024-09-13T14:22:33.435572+0000","last_clean":"2024-09-13T14:22:33.435572+0000","last_became_active":"2024-09-13T14:21:52.614695+0000","last_became_peered":"2024-09-13T14:21:52.614695+0000","last_unstale":"2024-09-13T14:22:33.435572+0000","last_undegraded":"2024-09-13T14:22:33.435572+0000","last_fullsized":"2024-09-13T14:22:33.435572+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:52.672387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427546+0000","last_change":"2024-09-13T14:21:53.702626+0000","last_active":"2024-09-13T14:22:33.427546+0000","last_peered":"2024-09-13T14:22:33.427546+0000","last_clean":"2024-09-13T14:22:33.427546+0000","last_became_active":"2024-09-13T14:21:53.701967+0000","last_became_peered":"2024-09-13T14:21:53.701967+0000","last_unstale":"2024-09-13T14:22:33.427546+0000","last_undegraded":"2024-09-13T14:22:33.427546+0000","last_fullsized":"2024-09-13T14:22:33.427546+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:27.005181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427078+0000","last_change":"2024-09-13T14:21:52.613309+0000","last_active":"2024-09-13T14:22:33.427078+0000","last_peered":"2024-09-13T14:22:33.427078+0000","last_clean":"2024-09-13T14:22:33.427078+0000","last_became_active":"2024-09-13T14:21:52.613034+0000","last_became_peered":"2024-09-13T14:21:52.613034+0000","last_unstale":"2024-09-13T14:22:33.427078+0000","last_undegraded":"2024-09-13T14:22:33.427078+0000","last_fullsized":"2024-09-13T14:22:33.427078+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:18.917592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427573+0000","last_change":"2024-09-13T14:21:53.702793+0000","last_active":"2024-09-13T14:22:33.427573+0000","last_peered":"2024-09-13T14:22:33.427573+0000","last_clean":"2024-09-13T14:22:33.427573+0000","last_became_active":"2024-09-13T14:21:53.700733+0000","last_became_peered":"2024-09-13T14:21:53.700733+0000","last_unstale":"2024-09-13T14:22:33.427573+0000","last_undegraded":"2024-09-13T14:22:33.427573+0000","last_fullsized":"2024-09-13T14:22:33.427573+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:12.693658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431876+0000","last_change":"2024-09-13T14:21:52.627072+0000","last_active":"2024-09-13T14:22:33.431876+0000","last_peered":"2024-09-13T14:22:33.431876+0000","last_clean":"2024-09-13T14:22:33.431876+0000","last_became_active":"2024-09-13T14:21:52.626731+0000","last_became_peered":"2024-09-13T14:21:52.626731+0000","last_unstale":"2024-09-13T14:22:33.431876+0000","last_undegraded":"2024-09-13T14:22:33.431876+0000","last_fullsized":"2024-09-13T14:22:33.431876+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:21.096584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433464+0000","last_change":"2024-09-13T14:21:53.294753+0000","last_active":"2024-09-13T14:22:33.433464+0000","last_peered":"2024-09-13T14:22:33.433464+0000","last_clean":"2024-09-13T14:22:33.433464+0000","last_became_active":"2024-09-13T14:21:53.294309+0000","last_became_peered":"2024-09-13T14:21:53.294309+0000","last_unstale":"2024-09-13T14:22:33.433464+0000","last_undegraded":"2024-09-13T14:22:33.433464+0000","last_fullsized":"2024-09-13T14:22:33.433464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:38.991136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"53'79","reported_seq":55,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435765+0000","last_change":"2024-09-13T14:20:01.261809+0000","last_active":"2024-09-13T14:22:33.435765+0000","last_peered":"2024-09-13T14:22:33.435765+0000","last_clean":"2024-09-13T14:22:33.435765+0000","last_became_active":"2024-09-13T14:20:01.256039+0000","last_became_peered":"2024-09-13T14:20:01.256039+0000","last_unstale":"2024-09-13T14:22:33.435765+0000","last_undegraded":"2024-09-13T14:22:33.435765+0000","last_fullsized":"2024-09-13T14:22:33.435765+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:01.420572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":26,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437277+0000","last_change":"2024-09-13T14:20:41.392754+0000","last_active":"2024-09-13T14:22:33.437277+0000","last_peered":"2024-09-13T14:22:33.437277+0000","last_clean":"2024-09-13T14:22:33.437277+0000","last_became_active":"2024-09-13T14:20:37.356130+0000","last_became_peered":"2024-09-13T14:20:37.356130+0000","last_unstale":"2024-09-13T14:22:33.437277+0000","last_undegraded":"2024-09-13T14:22:33.437277+0000","last_fullsized":"2024-09-13T14:22:33.437277+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:50.167135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432013+0000","last_change":"2024-09-13T14:21:52.612472+0000","last_active":"2024-09-13T14:22:33.432013+0000","last_peered":"2024-09-13T14:22:33.432013+0000","last_clean":"2024-09-13T14:22:33.432013+0000","last_became_active":"2024-09-13T14:21:52.612054+0000","last_became_peered":"2024-09-13T14:21:52.612054+0000","last_unstale":"2024-09-13T14:22:33.432013+0000","last_undegraded":"2024-09-13T14:22:33.432013+0000","last_fullsized":"2024-09-13T14:22:33.432013+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:39.708462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435026+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:22:33.435026+0000","last_peered":"2024-09-13T14:22:33.435026+0000","last_clean":"2024-09-13T14:22:33.435026+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:22:33.435026+0000","last_undegraded":"2024-09-13T14:22:33.435026+0000","last_fullsized":"2024-09-13T14:22:33.435026+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435656+0000","last_change":"2024-09-13T14:20:41.646316+0000","last_active":"2024-09-13T14:22:33.435656+0000","last_peered":"2024-09-13T14:22:33.435656+0000","last_clean":"2024-09-13T14:22:33.435656+0000","last_became_active":"2024-09-13T14:20:37.358143+0000","last_became_peered":"2024-09-13T14:20:37.358143+0000","last_unstale":"2024-09-13T14:22:33.435656+0000","last_undegraded":"2024-09-13T14:22:33.435656+0000","last_fullsized":"2024-09-13T14:22:33.435656+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:37.774534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431566+0000","last_change":"2024-09-13T14:21:52.614291+0000","last_active":"2024-09-13T14:22:33.431566+0000","last_peered":"2024-09-13T14:22:33.431566+0000","last_clean":"2024-09-13T14:22:33.431566+0000","last_became_active":"2024-09-13T14:21:52.614011+0000","last_became_peered":"2024-09-13T14:21:52.614011+0000","last_unstale":"2024-09-13T14:22:33.431566+0000","last_undegraded":"2024-09-13T14:22:33.431566+0000","last_fullsized":"2024-09-13T14:22:33.431566+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:59.130886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431873+0000","last_change":"2024-09-13T14:21:53.700128+0000","last_active":"2024-09-13T14:22:33.431873+0000","last_peered":"2024-09-13T14:22:33.431873+0000","last_clean":"2024-09-13T14:22:33.431873+0000","last_became_active":"2024-09-13T14:21:53.699826+0000","last_became_peered":"2024-09-13T14:21:53.699826+0000","last_unstale":"2024-09-13T14:22:33.431873+0000","last_undegraded":"2024-09-13T14:22:33.431873+0000","last_fullsized":"2024-09-13T14:22:33.431873+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:35.426011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.429438+0000","last_change":"2024-09-13T14:20:41.645840+0000","last_active":"2024-09-13T14:22:33.429438+0000","last_peered":"2024-09-13T14:22:33.429438+0000","last_clean":"2024-09-13T14:22:33.429438+0000","last_became_active":"2024-09-13T14:20:37.388195+0000","last_became_peered":"2024-09-13T14:20:37.388195+0000","last_unstale":"2024-09-13T14:22:33.429438+0000","last_undegraded":"2024-09-13T14:22:33.429438+0000","last_fullsized":"2024-09-13T14:22:33.429438+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:39.037088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431193+0000","last_change":"2024-09-13T14:21:52.606822+0000","last_active":"2024-09-13T14:22:33.431193+0000","last_peered":"2024-09-13T14:22:33.431193+0000","last_clean":"2024-09-13T14:22:33.431193+0000","last_became_active":"2024-09-13T14:21:52.606517+0000","last_became_peered":"2024-09-13T14:21:52.606517+0000","last_unstale":"2024-09-13T14:22:33.431193+0000","last_undegraded":"2024-09-13T14:22:33.431193+0000","last_fullsized":"2024-09-13T14:22:33.431193+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:13:09.268798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433547+0000","last_change":"2024-09-13T14:21:52.764427+0000","last_active":"2024-09-13T14:22:33.433547+0000","last_peered":"2024-09-13T14:22:33.433547+0000","last_clean":"2024-09-13T14:22:33.433547+0000","last_became_active":"2024-09-13T14:21:52.764139+0000","last_became_peered":"2024-09-13T14:21:52.764139+0000","last_unstale":"2024-09-13T14:22:33.433547+0000","last_undegraded":"2024-09-13T14:22:33.433547+0000","last_fullsized":"2024-09-13T14:22:33.433547+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:46.462801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":27,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437331+0000","last_change":"2024-09-13T14:20:41.393368+0000","last_active":"2024-09-13T14:22:33.437331+0000","last_peered":"2024-09-13T14:22:33.437331+0000","last_clean":"2024-09-13T14:22:33.437331+0000","last_became_active":"2024-09-13T14:20:37.357807+0000","last_became_peered":"2024-09-13T14:20:37.357807+0000","last_unstale":"2024-09-13T14:22:33.437331+0000","last_undegraded":"2024-09-13T14:22:33.437331+0000","last_fullsized":"2024-09-13T14:22:33.437331+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:17.845640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427448+0000","last_change":"2024-09-13T14:21:52.615398+0000","last_active":"2024-09-13T14:22:33.427448+0000","last_peered":"2024-09-13T14:22:33.427448+0000","last_clean":"2024-09-13T14:22:33.427448+0000","last_became_active":"2024-09-13T14:21:52.614114+0000","last_became_peered":"2024-09-13T14:21:52.614114+0000","last_unstale":"2024-09-13T14:22:33.427448+0000","last_undegraded":"2024-09-13T14:22:33.427448+0000","last_fullsized":"2024-09-13T14:22:33.427448+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:20.209300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431662+0000","last_change":"2024-09-13T14:21:52.635641+0000","last_active":"2024-09-13T14:22:33.431662+0000","last_peered":"2024-09-13T14:22:33.431662+0000","last_clean":"2024-09-13T14:22:33.431662+0000","last_became_active":"2024-09-13T14:21:52.635342+0000","last_became_peered":"2024-09-13T14:21:52.635342+0000","last_unstale":"2024-09-13T14:22:33.431662+0000","last_undegraded":"2024-09-13T14:22:33.431662+0000","last_fullsized":"2024-09-13T14:22:33.431662+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:11.062694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437585+0000","last_change":"2024-09-13T14:21:52.617052+0000","last_active":"2024-09-13T14:22:33.437585+0000","last_peered":"2024-09-13T14:22:33.437585+0000","last_clean":"2024-09-13T14:22:33.437585+0000","last_became_active":"2024-09-13T14:21:52.615024+0000","last_became_peered":"2024-09-13T14:21:52.615024+0000","last_unstale":"2024-09-13T14:22:33.437585+0000","last_undegraded":"2024-09-13T14:22:33.437585+0000","last_fullsized":"2024-09-13T14:22:33.437585+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:26.971812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431800+0000","last_change":"2024-09-13T14:20:41.389300+0000","last_active":"2024-09-13T14:22:33.431800+0000","last_peered":"2024-09-13T14:22:33.431800+0000","last_clean":"2024-09-13T14:22:33.431800+0000","last_became_active":"2024-09-13T14:20:37.356625+0000","last_became_peered":"2024-09-13T14:20:37.356625+0000","last_unstale":"2024-09-13T14:22:33.431800+0000","last_undegraded":"2024-09-13T14:22:33.431800+0000","last_fullsized":"2024-09-13T14:22:33.431800+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:28.847614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433500+0000","last_change":"2024-09-13T14:21:53.294786+0000","last_active":"2024-09-13T14:22:33.433500+0000","last_peered":"2024-09-13T14:22:33.433500+0000","last_clean":"2024-09-13T14:22:33.433500+0000","last_became_active":"2024-09-13T14:21:53.294437+0000","last_became_peered":"2024-09-13T14:21:53.294437+0000","last_unstale":"2024-09-13T14:22:33.433500+0000","last_undegraded":"2024-09-13T14:22:33.433500+0000","last_fullsized":"2024-09-13T14:22:33.433500+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:12.928998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431733+0000","last_change":"2024-09-13T14:21:52.611934+0000","last_active":"2024-09-13T14:22:33.431733+0000","last_peered":"2024-09-13T14:22:33.431733+0000","last_clean":"2024-09-13T14:22:33.431733+0000","last_became_active":"2024-09-13T14:21:52.611529+0000","last_became_peered":"2024-09-13T14:21:52.611529+0000","last_unstale":"2024-09-13T14:22:33.431733+0000","last_undegraded":"2024-09-13T14:22:33.431733+0000","last_fullsized":"2024-09-13T14:22:33.431733+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:18.891936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436638+0000","last_change":"2024-09-13T14:20:41.646302+0000","last_active":"2024-09-13T14:22:33.436638+0000","last_peered":"2024-09-13T14:22:33.436638+0000","last_clean":"2024-09-13T14:22:33.436638+0000","last_became_active":"2024-09-13T14:20:37.356803+0000","last_became_peered":"2024-09-13T14:20:37.356803+0000","last_unstale":"2024-09-13T14:22:33.436638+0000","last_undegraded":"2024-09-13T14:22:33.436638+0000","last_fullsized":"2024-09-13T14:22:33.436638+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:41.431722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431209+0000","last_change":"2024-09-13T14:21:53.699505+0000","last_active":"2024-09-13T14:22:33.431209+0000","last_peered":"2024-09-13T14:22:33.431209+0000","last_clean":"2024-09-13T14:22:33.431209+0000","last_became_active":"2024-09-13T14:21:53.699096+0000","last_became_peered":"2024-09-13T14:21:53.699096+0000","last_unstale":"2024-09-13T14:22:33.431209+0000","last_undegraded":"2024-09-13T14:22:33.431209+0000","last_fullsized":"2024-09-13T14:22:33.431209+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:56.561317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433090+0000","last_change":"2024-09-13T14:21:52.627331+0000","last_active":"2024-09-13T14:22:33.433090+0000","last_peered":"2024-09-13T14:22:33.433090+0000","last_clean":"2024-09-13T14:22:33.433090+0000","last_became_active":"2024-09-13T14:21:52.627043+0000","last_became_peered":"2024-09-13T14:21:52.627043+0000","last_unstale":"2024-09-13T14:22:33.433090+0000","last_undegraded":"2024-09-13T14:22:33.433090+0000","last_fullsized":"2024-09-13T14:22:33.433090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:23.898082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432482+0000","last_change":"2024-09-13T14:20:41.389271+0000","last_active":"2024-09-13T14:22:33.432482+0000","last_peered":"2024-09-13T14:22:33.432482+0000","last_clean":"2024-09-13T14:22:33.432482+0000","last_became_active":"2024-09-13T14:20:37.355416+0000","last_became_peered":"2024-09-13T14:20:37.355416+0000","last_unstale":"2024-09-13T14:22:33.432482+0000","last_undegraded":"2024-09-13T14:22:33.432482+0000","last_fullsized":"2024-09-13T14:22:33.432482+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:47.155583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026697500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431715+0000","last_change":"2024-09-13T14:21:53.698556+0000","last_active":"2024-09-13T14:22:33.431715+0000","last_peered":"2024-09-13T14:22:33.431715+0000","last_clean":"2024-09-13T14:22:33.431715+0000","last_became_active":"2024-09-13T14:21:53.698245+0000","last_became_peered":"2024-09-13T14:21:53.698245+0000","last_unstale":"2024-09-13T14:22:33.431715+0000","last_undegraded":"2024-09-13T14:22:33.431715+0000","last_fullsized":"2024-09-13T14:22:33.431715+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:05.325683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432010+0000","last_change":"2024-09-13T14:21:52.623973+0000","last_active":"2024-09-13T14:22:33.432010+0000","last_peered":"2024-09-13T14:22:33.432010+0000","last_clean":"2024-09-13T14:22:33.432010+0000","last_became_active":"2024-09-13T14:21:52.623650+0000","last_became_peered":"2024-09-13T14:21:52.623650+0000","last_unstale":"2024-09-13T14:22:33.432010+0000","last_undegraded":"2024-09-13T14:22:33.432010+0000","last_fullsized":"2024-09-13T14:22:33.432010+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:46.904029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.435117+0000","last_change":"2024-09-13T14:20:41.394322+0000","last_active":"2024-09-13T14:22:33.435117+0000","last_peered":"2024-09-13T14:22:33.435117+0000","last_clean":"2024-09-13T14:22:33.435117+0000","last_became_active":"2024-09-13T14:20:37.356874+0000","last_became_peered":"2024-09-13T14:20:37.356874+0000","last_unstale":"2024-09-13T14:22:33.435117+0000","last_undegraded":"2024-09-13T14:22:33.435117+0000","last_fullsized":"2024-09-13T14:22:33.435117+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:05.446185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.426923+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:22:33.426923+0000","last_peered":"2024-09-13T14:22:33.426923+0000","last_clean":"2024-09-13T14:22:33.426923+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:22:33.426923+0000","last_undegraded":"2024-09-13T14:22:33.426923+0000","last_fullsized":"2024-09-13T14:22:33.426923+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427307+0000","last_change":"2024-09-13T14:21:53.702034+0000","last_active":"2024-09-13T14:22:33.427307+0000","last_peered":"2024-09-13T14:22:33.427307+0000","last_clean":"2024-09-13T14:22:33.427307+0000","last_became_active":"2024-09-13T14:21:53.701326+0000","last_became_peered":"2024-09-13T14:21:53.701326+0000","last_unstale":"2024-09-13T14:22:33.427307+0000","last_undegraded":"2024-09-13T14:22:33.427307+0000","last_fullsized":"2024-09-13T14:22:33.427307+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:26.139551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"65'6","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427398+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:22:33.427398+0000","last_peered":"2024-09-13T14:22:33.427398+0000","last_clean":"2024-09-13T14:22:33.427398+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:22:33.427398+0000","last_undegraded":"2024-09-13T14:22:33.427398+0000","last_fullsized":"2024-09-13T14:22:33.427398+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433481+0000","last_change":"2024-09-13T14:21:52.983010+0000","last_active":"2024-09-13T14:22:33.433481+0000","last_peered":"2024-09-13T14:22:33.433481+0000","last_clean":"2024-09-13T14:22:33.433481+0000","last_became_active":"2024-09-13T14:21:52.982656+0000","last_became_peered":"2024-09-13T14:21:52.982656+0000","last_unstale":"2024-09-13T14:22:33.433481+0000","last_undegraded":"2024-09-13T14:22:33.433481+0000","last_fullsized":"2024-09-13T14:22:33.433481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:20.179211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.436505+0000","last_change":"2024-09-13T14:21:52.631712+0000","last_active":"2024-09-13T14:22:33.436505+0000","last_peered":"2024-09-13T14:22:33.436505+0000","last_clean":"2024-09-13T14:22:33.436505+0000","last_became_active":"2024-09-13T14:21:52.631439+0000","last_became_peered":"2024-09-13T14:21:52.631439+0000","last_unstale":"2024-09-13T14:22:33.436505+0000","last_undegraded":"2024-09-13T14:22:33.436505+0000","last_fullsized":"2024-09-13T14:22:33.436505+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:15.406379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437475+0000","last_change":"2024-09-13T14:21:52.613380+0000","last_active":"2024-09-13T14:22:33.437475+0000","last_peered":"2024-09-13T14:22:33.437475+0000","last_clean":"2024-09-13T14:22:33.437475+0000","last_became_active":"2024-09-13T14:21:52.612586+0000","last_became_peered":"2024-09-13T14:21:52.612586+0000","last_unstale":"2024-09-13T14:22:33.437475+0000","last_undegraded":"2024-09-13T14:22:33.437475+0000","last_fullsized":"2024-09-13T14:22:33.437475+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:37.416155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431125+0000","last_change":"2024-09-13T14:21:53.699866+0000","last_active":"2024-09-13T14:22:33.431125+0000","last_peered":"2024-09-13T14:22:33.431125+0000","last_clean":"2024-09-13T14:22:33.431125+0000","last_became_active":"2024-09-13T14:21:53.699536+0000","last_became_peered":"2024-09-13T14:21:53.699536+0000","last_unstale":"2024-09-13T14:22:33.431125+0000","last_undegraded":"2024-09-13T14:22:33.431125+0000","last_fullsized":"2024-09-13T14:22:33.431125+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:09.594385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432448+0000","last_change":"2024-09-13T14:21:52.624373+0000","last_active":"2024-09-13T14:22:33.432448+0000","last_peered":"2024-09-13T14:22:33.432448+0000","last_clean":"2024-09-13T14:22:33.432448+0000","last_became_active":"2024-09-13T14:21:52.624139+0000","last_became_peered":"2024-09-13T14:21:52.624139+0000","last_unstale":"2024-09-13T14:22:33.432448+0000","last_undegraded":"2024-09-13T14:22:33.432448+0000","last_fullsized":"2024-09-13T14:22:33.432448+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:53:39.592214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431638+0000","last_change":"2024-09-13T14:21:53.699201+0000","last_active":"2024-09-13T14:22:33.431638+0000","last_peered":"2024-09-13T14:22:33.431638+0000","last_clean":"2024-09-13T14:22:33.431638+0000","last_became_active":"2024-09-13T14:21:53.698927+0000","last_became_peered":"2024-09-13T14:21:53.698927+0000","last_unstale":"2024-09-13T14:22:33.431638+0000","last_undegraded":"2024-09-13T14:22:33.431638+0000","last_fullsized":"2024-09-13T14:22:33.431638+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:10.138274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.434957+0000","last_change":"2024-09-13T14:21:53.701444+0000","last_active":"2024-09-13T14:22:33.434957+0000","last_peered":"2024-09-13T14:22:33.434957+0000","last_clean":"2024-09-13T14:22:33.434957+0000","last_became_active":"2024-09-13T14:21:53.701175+0000","last_became_peered":"2024-09-13T14:21:53.701175+0000","last_unstale":"2024-09-13T14:22:33.434957+0000","last_undegraded":"2024-09-13T14:22:33.434957+0000","last_fullsized":"2024-09-13T14:22:33.434957+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:39.842585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431801+0000","last_change":"2024-09-13T14:21:52.624953+0000","last_active":"2024-09-13T14:22:33.431801+0000","last_peered":"2024-09-13T14:22:33.431801+0000","last_clean":"2024-09-13T14:22:33.431801+0000","last_became_active":"2024-09-13T14:21:52.624360+0000","last_became_peered":"2024-09-13T14:21:52.624360+0000","last_unstale":"2024-09-13T14:22:33.431801+0000","last_undegraded":"2024-09-13T14:22:33.431801+0000","last_fullsized":"2024-09-13T14:22:33.431801+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:18.568935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431804+0000","last_change":"2024-09-13T14:21:53.700395+0000","last_active":"2024-09-13T14:22:33.431804+0000","last_peered":"2024-09-13T14:22:33.431804+0000","last_clean":"2024-09-13T14:22:33.431804+0000","last_became_active":"2024-09-13T14:21:53.700057+0000","last_became_peered":"2024-09-13T14:21:53.700057+0000","last_unstale":"2024-09-13T14:22:33.431804+0000","last_undegraded":"2024-09-13T14:22:33.431804+0000","last_fullsized":"2024-09-13T14:22:33.431804+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:45.627991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431421+0000","last_change":"2024-09-13T14:21:52.625583+0000","last_active":"2024-09-13T14:22:33.431421+0000","last_peered":"2024-09-13T14:22:33.431421+0000","last_clean":"2024-09-13T14:22:33.431421+0000","last_became_active":"2024-09-13T14:21:52.622107+0000","last_became_peered":"2024-09-13T14:21:52.622107+0000","last_unstale":"2024-09-13T14:22:33.431421+0000","last_undegraded":"2024-09-13T14:22:33.431421+0000","last_fullsized":"2024-09-13T14:22:33.431421+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:12.415569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432148+0000","last_change":"2024-09-13T14:21:53.701315+0000","last_active":"2024-09-13T14:22:33.432148+0000","last_peered":"2024-09-13T14:22:33.432148+0000","last_clean":"2024-09-13T14:22:33.432148+0000","last_became_active":"2024-09-13T14:21:53.701014+0000","last_became_peered":"2024-09-13T14:21:53.701014+0000","last_unstale":"2024-09-13T14:22:33.432148+0000","last_undegraded":"2024-09-13T14:22:33.432148+0000","last_fullsized":"2024-09-13T14:22:33.432148+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:48.507923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.427067+0000","last_change":"2024-09-13T14:21:52.623311+0000","last_active":"2024-09-13T14:22:33.427067+0000","last_peered":"2024-09-13T14:22:33.427067+0000","last_clean":"2024-09-13T14:22:33.427067+0000","last_became_active":"2024-09-13T14:21:52.623015+0000","last_became_peered":"2024-09-13T14:21:52.623015+0000","last_unstale":"2024-09-13T14:22:33.427067+0000","last_undegraded":"2024-09-13T14:22:33.427067+0000","last_fullsized":"2024-09-13T14:22:33.427067+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:51.869610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.429501+0000","last_change":"2024-09-13T14:21:53.700992+0000","last_active":"2024-09-13T14:22:33.429501+0000","last_peered":"2024-09-13T14:22:33.429501+0000","last_clean":"2024-09-13T14:22:33.429501+0000","last_became_active":"2024-09-13T14:21:53.700681+0000","last_became_peered":"2024-09-13T14:21:53.700681+0000","last_unstale":"2024-09-13T14:22:33.429501+0000","last_undegraded":"2024-09-13T14:22:33.429501+0000","last_fullsized":"2024-09-13T14:22:33.429501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:42.030268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431331+0000","last_change":"2024-09-13T14:21:52.615630+0000","last_active":"2024-09-13T14:22:33.431331+0000","last_peered":"2024-09-13T14:22:33.431331+0000","last_clean":"2024-09-13T14:22:33.431331+0000","last_became_active":"2024-09-13T14:21:52.614872+0000","last_became_peered":"2024-09-13T14:21:52.614872+0000","last_unstale":"2024-09-13T14:22:33.431331+0000","last_undegraded":"2024-09-13T14:22:33.431331+0000","last_fullsized":"2024-09-13T14:22:33.431331+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:34.605661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T14:22:34.432604+0000","last_change":"2024-09-13T14:22:34.432604+0000","last_active":"2024-09-13T14:22:34.432604+0000","last_peered":"2024-09-13T14:22:34.432604+0000","last_clean":"2024-09-13T14:22:34.432604+0000","last_became_active":"2024-09-13T14:22:34.432418+0000","last_became_peered":"2024-09-13T14:22:34.432418+0000","last_unstale":"2024-09-13T14:22:34.432604+0000","last_undegraded":"2024-09-13T14:22:34.432604+0000","last_fullsized":"2024-09-13T14:22:34.432604+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:18.113553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437279+0000","last_change":"2024-09-13T14:21:52.617742+0000","last_active":"2024-09-13T14:22:33.437279+0000","last_peered":"2024-09-13T14:22:33.437279+0000","last_clean":"2024-09-13T14:22:33.437279+0000","last_became_active":"2024-09-13T14:21:52.616904+0000","last_became_peered":"2024-09-13T14:21:52.616904+0000","last_unstale":"2024-09-13T14:22:33.437279+0000","last_undegraded":"2024-09-13T14:22:33.437279+0000","last_fullsized":"2024-09-13T14:22:33.437279+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:53.145715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.426962+0000","last_change":"2024-09-13T14:21:53.632083+0000","last_active":"2024-09-13T14:22:33.426962+0000","last_peered":"2024-09-13T14:22:33.426962+0000","last_clean":"2024-09-13T14:22:33.426962+0000","last_became_active":"2024-09-13T14:21:53.631784+0000","last_became_peered":"2024-09-13T14:21:53.631784+0000","last_unstale":"2024-09-13T14:22:33.426962+0000","last_undegraded":"2024-09-13T14:22:33.426962+0000","last_fullsized":"2024-09-13T14:22:33.426962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:18.161083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.437550+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:22:33.437550+0000","last_peered":"2024-09-13T14:22:33.437550+0000","last_clean":"2024-09-13T14:22:33.437550+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:22:33.437550+0000","last_undegraded":"2024-09-13T14:22:33.437550+0000","last_fullsized":"2024-09-13T14:22:33.437550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.431365+0000","last_change":"2024-09-13T14:21:53.632016+0000","last_active":"2024-09-13T14:22:33.431365+0000","last_peered":"2024-09-13T14:22:33.431365+0000","last_clean":"2024-09-13T14:22:33.431365+0000","last_became_active":"2024-09-13T14:21:53.631724+0000","last_became_peered":"2024-09-13T14:21:53.631724+0000","last_unstale":"2024-09-13T14:22:33.431365+0000","last_undegraded":"2024-09-13T14:22:33.431365+0000","last_fullsized":"2024-09-13T14:22:33.431365+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:56.931830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.433045+0000","last_change":"2024-09-13T14:21:52.620927+0000","last_active":"2024-09-13T14:22:33.433045+0000","last_peered":"2024-09-13T14:22:33.433045+0000","last_clean":"2024-09-13T14:22:33.433045+0000","last_became_active":"2024-09-13T14:21:52.620570+0000","last_became_peered":"2024-09-13T14:21:52.620570+0000","last_unstale":"2024-09-13T14:22:33.433045+0000","last_undegraded":"2024-09-13T14:22:33.433045+0000","last_fullsized":"2024-09-13T14:22:33.433045+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:28.683972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.426792+0000","last_change":"2024-09-13T14:21:53.699582+0000","last_active":"2024-09-13T14:22:33.426792+0000","last_peered":"2024-09-13T14:22:33.426792+0000","last_clean":"2024-09-13T14:22:33.426792+0000","last_became_active":"2024-09-13T14:21:53.698531+0000","last_became_peered":"2024-09-13T14:21:53.698531+0000","last_unstale":"2024-09-13T14:22:33.426792+0000","last_undegraded":"2024-09-13T14:22:33.426792+0000","last_fullsized":"2024-09-13T14:22:33.426792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:56.552336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:22:33.432367+0000","last_change":"2024-09-13T14:21:52.619193+0000","last_active":"2024-09-13T14:22:33.432367+0000","last_peered":"2024-09-13T14:22:33.432367+0000","last_clean":"2024-09-13T14:22:33.432367+0000","last_became_active":"2024-09-13T14:21:52.618685+0000","last_became_peered":"2024-09-13T14:21:52.618685+0000","last_unstale":"2024-09-13T14:22:33.432367+0000","last_undegraded":"2024-09-13T14:22:33.432367+0000","last_fullsized":"2024-09-13T14:22:33.432367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:04.725931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364837,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298628,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446524,"statfs":{"total":95995035648,"available":95689240576,"internally_reserved":0,"allocated":1302528,"data_stored":893190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68200000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.436},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.41099999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.42399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72599999999999998,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79600000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.46100000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76000000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56799999999999995}]},{"osd":3,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56499999999999995,"15min":0.56399999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76500000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52000000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.497},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70899999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68899999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.623}]},{"osd":5,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.50700000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58999999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65200000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561067,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298620,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446532,"statfs":{"total":95995035648,"available":95689248768,"internally_reserved":0,"allocated":1294336,"data_stored":893013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.56599999999999995},"min":{"1min":0.441,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.42599999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.49199999999999999}]},{"osd":1,"last update":"Fri Sep 13 14:22:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57799999999999996,"15min":0.58099999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Sep 13 14:22:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.58799999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76000000000000001,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.375,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.379}]},{"osd":3,"last update":"Fri Sep 13 14:22:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.38800000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77000000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.42999999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82199999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.64000000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:22:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47399999999999998,"15min":0.47199999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72999999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48599999999999999,"15min":0.47599999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69999999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.45400000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:22:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.45600000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74199999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Fri Sep 13 14:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.505},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66300000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789999,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298028,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447124,"statfs":{"total":95995035648,"available":95689854976,"internally_reserved":0,"allocated":688128,"data_stored":301419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.68400000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.372,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.71599999999999997,"5min":0.752,"15min":0.752},"last":0.60699999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68600000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.625},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.505,"15min":0.50800000000000001},"min":{"1min":0.40899999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66800000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.376,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65500000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.371,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70699999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.51200000000000001,"15min":0.502},"min":{"1min":0.378,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.54900000000000004},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68300000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.42299999999999999,"15min":0.40200000000000002},"min":{"1min":0.28299999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.436},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.42799999999999999,"15min":0.40000000000000002},"min":{"1min":0.311,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Fri Sep 13 14:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.47499999999999998,"15min":0.46000000000000002},"min":{"1min":0.32000000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.46999999999999997,"15min":0.45400000000000001},"min":{"1min":0.32700000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39200000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.314,"15min":0.314},"max":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.34599999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72899999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51100000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986227,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298048,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447104,"statfs":{"total":95995035648,"available":95689834496,"internally_reserved":0,"allocated":708608,"data_stored":303017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71799999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59899999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.52000000000000002},"min":{"1min":0.36399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73999999999999999,"5min":0.747,"15min":0.747},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.40899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.74199999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55200000000000005,"15min":0.55000000000000004},"min":{"1min":0.33300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76200000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.75700000000000001}]},{"osd":3,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54700000000000004,"15min":0.54600000000000004},"min":{"1min":0.38800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.39600000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73099999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.71899999999999997}]},{"osd":5,"last update":"Fri Sep 13 14:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.41799999999999998,"15min":0.39600000000000002},"min":{"1min":0.32600000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70699999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.42899999999999999,"15min":0.39900000000000002},"min":{"1min":0.29899999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.73299999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56599999999999995}]},{"osd":6,"last update":"Fri Sep 13 14:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45200000000000001,"15min":0.436},"min":{"1min":0.28299999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73299999999999998,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51300000000000001,"15min":0.49099999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67700000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81200000000000006,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.624}]}]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298032,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447120,"statfs":{"total":95995035648,"available":95689850880,"internally_reserved":0,"allocated":688128,"data_stored":301409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.35699999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.54900000000000004,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.40200000000000002},"min":{"1min":0.247,"5min":0.223,"15min":0.223},"max":{"1min":0.55500000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32200000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40699999999999997,"15min":0.41999999999999998},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.26500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.54100000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.432,"15min":0.436},"min":{"1min":0.33900000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56499999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40699999999999997,"15min":0.41199999999999998},"min":{"1min":0.29499999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.52100000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.34100000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:21:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59299999999999997,"15min":0.61299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.64700000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63100000000000001,"15min":0.64900000000000002},"min":{"1min":0.42799999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.75600000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.65900000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81100000000000005,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.54500000000000004}]},{"osd":6,"last update":"Fri Sep 13 14:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.46999999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.503},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60299999999999998,"15min":0.61899999999999999},"min":{"1min":0.42299999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.84599999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.56799999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:22:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58599999999999997,"15min":0.58899999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.442,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82499999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71399999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298036,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447116,"statfs":{"total":95995035648,"available":95689846784,"internally_reserved":0,"allocated":692224,"data_stored":301923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.34499999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.55800000000000005,"5min":0.623,"15min":0.623},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36799999999999999,"15min":0.35099999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53600000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.252}]},{"osd":1,"last update":"Fri Sep 13 14:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.58599999999999997,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40500000000000003,"15min":0.39200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59699999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38500000000000001}]},{"osd":3,"last update":"Fri Sep 13 14:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28999999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.34699999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.42499999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.505}]},{"osd":5,"last update":"Fri Sep 13 14:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56299999999999994,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.46600000000000003}]},{"osd":6,"last update":"Fri Sep 13 14:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.443,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79700000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.82199999999999995,"5min":1.016,"15min":1.016},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.60399999999999998,"15min":0.58899999999999997},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298032,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447120,"statfs":{"total":95995035648,"available":95689850880,"internally_reserved":0,"allocated":688128,"data_stored":301338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.38100000000000001,"15min":0.34799999999999998},"min":{"1min":0.24199999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.54300000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.38400000000000001,"15min":0.33900000000000002},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46700000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:22:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40600000000000003,"15min":0.38600000000000001},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.626,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.38800000000000001,"15min":0.374},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67400000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.245}]},{"osd":3,"last update":"Fri Sep 13 14:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.55600000000000005,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.31},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.29699999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56599999999999995,"5min":1.095,"15min":1.095},"last":0.26200000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61399999999999999,"15min":0.624},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.88}]},{"osd":5,"last update":"Fri Sep 13 14:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79100000000000004,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.64700000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.626,"15min":0.61699999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.877,"5min":1.046,"15min":1.046},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.621,"15min":0.622},"min":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81000000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.84999999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.77500000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88400000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298616,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446536,"statfs":{"total":95995035648,"available":95689252864,"internally_reserved":0,"allocated":1286144,"data_stored":892279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.35599999999999998,"15min":0.33200000000000002},"min":{"1min":0.253,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.48099999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.39700000000000002,"15min":0.36499999999999999},"min":{"1min":0.24399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.54300000000000004,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.317}]},{"osd":2,"last update":"Fri Sep 13 14:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40699999999999997,"15min":0.42099999999999999},"min":{"1min":0.25,"5min":0.246,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.41999999999999998,"15min":0.435},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.49299999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.29899999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:22:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.45600000000000002,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63100000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.42899999999999999,"15min":0.45700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56699999999999995,"5min":0.629,"15min":0.629},"last":0.498}]},{"osd":4,"last update":"Fri Sep 13 14:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44400000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56899999999999995,"15min":0.55200000000000005},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.70999999999999996,"5min":0.751,"15min":0.751},"last":0.56899999999999995}]},{"osd":6,"last update":"Fri Sep 13 14:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.57799999999999996},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":0.754,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.45800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78600000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57999999999999996}]},{"osd":7,"last update":"Fri Sep 13 14:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61899999999999999,"15min":0.621},"min":{"1min":0.44500000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75900000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.37,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:22:51.643 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:22:51.643 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:22:51.643 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:22:51.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:22:51.907 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:52.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:52 smithi003 ceph-mon[41592]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:22:52.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:52 smithi003 ceph-mon[41592]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:52.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:52 smithi003 ceph-mon[29646]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:22:52.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:52 smithi003 ceph-mon[29646]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:52.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:52 smithi019 ceph-mon[32895]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:22:52.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:52 smithi019 ceph-mon[32895]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:53.809 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:22:53.989 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:22:54.492 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:22:54.492 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:22:54.504 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-09-13T14:22:54.504 INFO:teuthology.run_tasks:Running task parallel... 2024-09-13T14:22:54.565 INFO:teuthology.task.parallel:starting parallel... 2024-09-13T14:22:54.565 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:22:54.566 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-13T14:22:54.566 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:22:54.568 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:22:54.568 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-13T14:22:54.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-13T14:22:54.569 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:22:54.569 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-13T14:22:54.570 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-09-13T14:22:54.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-09-13T14:22:54.843 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:54.844 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:54.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:54 smithi019 ceph-mon[32895]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:54.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:54 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2257969645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:22:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:54 smithi003 ceph-mon[41592]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:54 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2257969645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:22:54.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:54 smithi003 ceph-mon[29646]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:54.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:54 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2257969645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:22:56.786 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:56 smithi003 ceph-mon[41592]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:56.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:56 smithi003 ceph-mon[29646]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:56.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:56 smithi019 ceph-mon[32895]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:57.400 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:22:57] "GET /metrics HTTP/1.1" 200 239642 "" "Prometheus/2.43.0" 2024-09-13T14:22:58.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:22:58] ENGINE Bus STOPPING 2024-09-13T14:22:58.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:22:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:22:58.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:22:58] ENGINE Bus STOPPED 2024-09-13T14:22:58.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:22:58] ENGINE Bus STARTING 2024-09-13T14:22:58.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:22:58] ENGINE Serving on http://:::9283 2024-09-13T14:22:58.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:22:58] ENGINE Bus STARTED 2024-09-13T14:22:58.496 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_0' created 2024-09-13T14:22:58.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[41592]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:58.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1497946467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:22:58.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[41592]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:22:58.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[41592]: from='client.? ' entity='client.admin' 2024-09-13T14:22:58.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:58.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[29646]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:58.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1497946467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:22:58.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[29646]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:22:58.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[29646]: from='client.? ' entity='client.admin' 2024-09-13T14:22:58.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:58 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:58.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-13T14:22:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:58 smithi019 ceph-mon[32895]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:58.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:58 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1497946467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:22:58.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:58 smithi019 ceph-mon[32895]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:22:58.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:58 smithi019 ceph-mon[32895]: from='client.? ' entity='client.admin' 2024-09-13T14:22:58.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:58 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:59.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-13T14:22:59.572 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:59.796 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[41592]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:22:59.796 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[41592]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:22:59.796 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:59.796 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:59.796 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:59.797 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[29646]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:22:59.797 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[29646]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:22:59.797 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:59.797 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:59.797 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:22:59 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:22:59.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:59 smithi019 ceph-mon[32895]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:22:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:59 smithi019 ceph-mon[32895]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:22:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:59 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:59 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:22:59 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:00.250 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:00.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:00 smithi003 ceph-mon[41592]: pgmap v91: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:00.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:00 smithi003 ceph-mon[41592]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:23:00.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:00 smithi003 ceph-mon[41592]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:23:00.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:00 smithi003 ceph-mon[29646]: pgmap v91: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:00.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:00 smithi003 ceph-mon[29646]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:23:00.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:00 smithi003 ceph-mon[29646]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:23:00.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:00 smithi019 ceph-mon[32895]: pgmap v91: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:00.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:00 smithi019 ceph-mon[32895]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:23:00.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:00 smithi019 ceph-mon[32895]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:23:01.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:23:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:23:01.840 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:01 smithi003 ceph-mon[41592]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:23:01.841 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:01 smithi003 ceph-mon[29646]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:23:01.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:01 smithi019 ceph-mon[32895]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:23:02.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:02 smithi003 ceph-mon[29646]: pgmap v94: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:02.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:02 smithi003 ceph-mon[41592]: pgmap v94: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:02 smithi019 ceph-mon[32895]: pgmap v94: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:03.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-13T14:23:03.736 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:04.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:04 smithi003 ceph-mon[29646]: pgmap v95: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:04.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:04 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:04.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:04 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:04.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:04 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1918288432' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:23:04.490 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:04 smithi003 ceph-mon[41592]: pgmap v95: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:04.490 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:04 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:04.490 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:04 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:04.490 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:04 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1918288432' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:23:04.525 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-13T14:23:04.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:04 smithi019 ceph-mon[32895]: pgmap v95: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:04.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:04 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:04.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:04 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:04.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:04 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1918288432' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:23:05.809 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:05 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1918288432' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:23:05.809 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:05 smithi003 ceph-mon[41592]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:23:05.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:05 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1918288432' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:23:05.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:05 smithi003 ceph-mon[29646]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:23:05.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:05 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1918288432' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:23:05.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:05 smithi019 ceph-mon[32895]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:23:06.210 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-13T14:23:06.213 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:23:06.213 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:23:06.226 INFO:teuthology.orchestra.run.smithi003.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:23:06.271 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T14:23:06.271 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:23:06.272 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_86266 2024-09-13T14:23:06.272 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:23:06.272 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:23:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:06 smithi019 ceph-mon[32895]: pgmap v97: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:06 smithi019 ceph-mon[32895]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:23:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:06 smithi019 ceph-mon[32895]: Cluster is now healthy 2024-09-13T14:23:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:06 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:06 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:06.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:06 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:06.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:06 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:06.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:06 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:06.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[29646]: pgmap v97: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:06.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[29646]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:23:06.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[29646]: Cluster is now healthy 2024-09-13T14:23:06.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:06.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:06.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:06.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:06.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:06.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[41592]: pgmap v97: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:06.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[41592]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:23:06.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[41592]: Cluster is now healthy 2024-09-13T14:23:06.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:06.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:06.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:06.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:06.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:06 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:07.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:23:07.273 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3948 3932 245.736 245.75 0.00275754 0.00405614 2024-09-13T14:23:07.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:23:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:07] "GET /metrics HTTP/1.1" 200 242003 "" "Prometheus/2.43.0" 2024-09-13T14:23:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:07.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:07 smithi003 ceph-mon[29646]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:07.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:07 smithi003 ceph-mon[29646]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:23:07.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:07 smithi003 ceph-mon[29646]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:23:07.841 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:07 smithi003 ceph-mon[41592]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:07.841 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:07 smithi003 ceph-mon[41592]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:23:07.841 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:07 smithi003 ceph-mon[41592]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:23:07.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:07 smithi019 ceph-mon[32895]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:07 smithi019 ceph-mon[32895]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:23:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:07 smithi019 ceph-mon[32895]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:23:08.272 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 7460 7444 232.611 219.5 0.00294664 0.00428917 2024-09-13T14:23:08.801 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:08 smithi003 ceph-mon[29646]: pgmap v98: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:08.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:08 smithi003 ceph-mon[41592]: pgmap v98: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:08.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:08 smithi019 ceph-mon[32895]: pgmap v98: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:09.272 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 10734 10718 223.276 204.625 0.00303455 0.00447098 2024-09-13T14:23:09.920 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:23:09.920 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (85s) 40s ago 2m 14.5M - 0.25.0 c8568f914cd2 9994b1ee34d8 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi019 *:3000 running (2m) 57s ago 2m 82.7M - 9.4.7 954c08fa6188 dc631f7cf889 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.zmhiel smithi003 running (71s) 40s ago 70s 15.8M - 17.2.7-1627-gbbdadfa3 a994b00deb3a db8314ccc6fd 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi019.ydtwbg smithi019 running (68s) 57s ago 67s 14.3M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c0fef505f410 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi019 *:8443 running (6m) 57s ago 6m 399M - 17.2.7-1627-gbbdadfa3 a994b00deb3a f4595aba3576 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:9283,8765 running (8m) 40s ago 8m 444M - 17.2.7-1627-gbbdadfa3 a994b00deb3a fe65bae0d85f 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (8m) 40s ago 8m 50.3M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a 9c73445eb77a 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi019 running (6m) 57s ago 6m 36.9M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a b71aff50957a 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (6m) 40s ago 6m 39.5M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a a6373bc053ae 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (111s) 40s ago 108s 16.3M - 1.5.0 0da6a335fe13 a786edb7f0b5 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi019 *:9100 running (105s) 57s ago 105s 7872k - 1.5.0 0da6a335fe13 e22916e85bdc 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (6m) 40s ago 6m 44.7M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a b0e72d25bc0f 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (5m) 40s ago 5m 42.8M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a 612dd4f9bdda 2024-09-13T14:23:09.921 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (5m) 40s ago 5m 40.1M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a d32f24c0ca7e 2024-09-13T14:23:09.922 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (4m) 40s ago 4m 40.4M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a 20984c58cba9 2024-09-13T14:23:09.922 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi019 running (4m) 57s ago 4m 40.0M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a e50c61f762b0 2024-09-13T14:23:09.922 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi019 running (4m) 57s ago 4m 40.5M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a 1cd389248627 2024-09-13T14:23:09.922 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi019 running (3m) 57s ago 3m 43.5M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a aa95f3146e47 2024-09-13T14:23:09.922 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi019 running (3m) 57s ago 3m 41.1M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a 118a8cf6eadc 2024-09-13T14:23:09.922 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi019 *:9095 running (80s) 57s ago 2m 33.8M - 2.43.0 a07b618ecd1d f9fc99d6a298 2024-09-13T14:23:10.272 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 13744 13728 214.483 188.125 0.00301341 0.0046562 2024-09-13T14:23:10.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:10 smithi003 ceph-mon[29646]: pgmap v99: 89 pgs: 89 active+clean; 688 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T14:23:10.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:10 smithi003 ceph-mon[41592]: pgmap v99: 89 pgs: 89 active+clean; 688 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T14:23:10.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:10 smithi019 ceph-mon[32895]: pgmap v99: 89 pgs: 89 active+clean; 688 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T14:23:10.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for minority of mons to upgrade' 2024-09-13T14:23:11.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:23:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:23:11.273 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 16697 16681 208.475 184.562 0.00375507 0.0047899 2024-09-13T14:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:11 smithi019 ceph-mon[32895]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:11.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:11 smithi003 ceph-mon[29646]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:11.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:11 smithi003 ceph-mon[41592]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:12.273 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 19481 19465 202.727 174 0.0104788 0.00492715 2024-09-13T14:23:12.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:12 smithi019 ceph-mon[32895]: pgmap v100: 89 pgs: 89 active+clean; 688 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:23:12.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:12 smithi003 ceph-mon[41592]: pgmap v100: 89 pgs: 89 active+clean; 688 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:23:12.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:12 smithi003 ceph-mon[29646]: pgmap v100: 89 pgs: 89 active+clean; 688 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:23:13.274 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 22268 22252 198.61 174.188 0.00491594 0.00502787 2024-09-13T14:23:13.836 INFO:teuthology.orchestra.run.smithi003.stdout:wait for minority of mons to upgrade 2024-09-13T14:23:14.275 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 24844 24828 193.906 161 0.00967863 0.00515036 2024-09-13T14:23:14.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-13T14:23:14.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:14 smithi019 ceph-mon[32895]: pgmap v101: 89 pgs: 89 active+clean; 688 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:23:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:14 smithi003 ceph-mon[29646]: pgmap v101: 89 pgs: 89 active+clean; 688 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:23:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:14 smithi003 ceph-mon[41592]: pgmap v101: 89 pgs: 89 active+clean; 688 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:23:15.153 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:15.275 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 27160 27144 188.443 144.75 0.00580582 0.00530091 2024-09-13T14:23:16.275 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 29205 29189 182.367 127.812 0.00443537 0.00547709 2024-09-13T14:23:16.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:16 smithi019 ceph-mon[32895]: pgmap v102: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 144 MiB/s wr, 2.31k op/s 2024-09-13T14:23:16.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:16 smithi003 ceph-mon[41592]: pgmap v102: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 144 MiB/s wr, 2.31k op/s 2024-09-13T14:23:16.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:16 smithi003 ceph-mon[29646]: pgmap v102: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 144 MiB/s wr, 2.31k op/s 2024-09-13T14:23:17.276 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 31311 31295 177.751 131.625 0.00749156 0.00562021 2024-09-13T14:23:17.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:23:17 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:17] "GET /metrics HTTP/1.1" 200 242003 "" "Prometheus/2.43.0" 2024-09-13T14:23:18.276 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 33247 33231 173.021 121 0.0042046 0.00577459 2024-09-13T14:23:18.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:18 smithi019 ceph-mon[32895]: pgmap v103: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 130 MiB/s wr, 2.09k op/s 2024-09-13T14:23:18.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:18 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:18.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:18 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:18.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:18 smithi003 ceph-mon[29646]: pgmap v103: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 130 MiB/s wr, 2.09k op/s 2024-09-13T14:23:18.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:18 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:18.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:18 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:18.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:18 smithi003 ceph-mon[41592]: pgmap v103: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 130 MiB/s wr, 2.09k op/s 2024-09-13T14:23:18.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:18 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:18 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:19.276 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 35055 35039 168.404 113 0.0016686 0.00590908 2024-09-13T14:23:20.276 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 36691 36675 163.679 102.25 0.00814663 0.00610383 2024-09-13T14:23:20.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:20 smithi019 ceph-mon[32895]: pgmap v104: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 183 MiB/s wr, 2.92k op/s 2024-09-13T14:23:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:20 smithi003 ceph-mon[41592]: pgmap v104: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 183 MiB/s wr, 2.92k op/s 2024-09-13T14:23:20.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:20 smithi003 ceph-mon[29646]: pgmap v104: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 183 MiB/s wr, 2.92k op/s 2024-09-13T14:23:21.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:23:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:23:21.276 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 38198 38182 159.046 94.1875 0.0103849 0.00628324 2024-09-13T14:23:22.278 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 40012 39996 156.185 113.375 0.00852828 0.00639762 2024-09-13T14:23:22.610 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:22 smithi003 ceph-mon[41592]: pgmap v105: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-13T14:23:22.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:22 smithi003 ceph-mon[29646]: pgmap v105: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-13T14:23:22.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:22 smithi019 ceph-mon[32895]: pgmap v105: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-13T14:23:23.277 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 41992 41976 154.275 123.75 0.00483482 0.00647734 2024-09-13T14:23:23.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:23 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1265921027' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:23.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:23 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1265921027' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:23.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:23 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1265921027' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:24.106 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:24.277 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 43766 43750 151.863 110.875 0.00612214 0.0065798 2024-09-13T14:23:24.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:24 smithi019 ceph-mon[32895]: pgmap v106: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-13T14:23:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:24 smithi003 ceph-mon[41592]: pgmap v106: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-13T14:23:24.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:24 smithi003 ceph-mon[29646]: pgmap v106: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-13T14:23:25.278 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 45741 45725 150.367 123.438 0.00326545 0.00664403 2024-09-13T14:23:25.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:25 smithi019 ceph-mon[32895]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:25 smithi019 ceph-mon[32895]: from='client.24743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:25.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:25 smithi003 ceph-mon[41592]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:25.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:25 smithi003 ceph-mon[41592]: from='client.24743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:25.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:25 smithi003 ceph-mon[29646]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:25.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:25 smithi003 ceph-mon[29646]: from='client.24743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:26.278 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:23:26.278026+0000 min lat: 0.00125696 max lat: 0.141946 avg lat: 0.00670467 2024-09-13T14:23:26.278 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:23:26.278 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 47726 47710 149.051 124.062 0.00628706 0.00670467 2024-09-13T14:23:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:26 smithi019 ceph-mon[32895]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-09-13T14:23:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:26 smithi003 ceph-mon[41592]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-09-13T14:23:26.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:26 smithi003 ceph-mon[29646]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-09-13T14:23:27.278 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 15 49540 49525 147.352 113.438 0.00781835 0.00678053 2024-09-13T14:23:27.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:23:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:27] "GET /metrics HTTP/1.1" 200 242933 "" "Prometheus/2.43.0" 2024-09-13T14:23:27.797 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:27.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:27 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/448081311' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:27.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:27 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/448081311' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:27.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:27 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/448081311' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:28.279 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 51122 51106 145.143 98.8125 0.00857017 0.00688417 2024-09-13T14:23:28.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:28 smithi019 ceph-mon[32895]: pgmap v108: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:23:28.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:28 smithi019 ceph-mon[32895]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:28 smithi003 ceph-mon[41592]: pgmap v108: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:23:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:28 smithi003 ceph-mon[41592]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:28.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:28 smithi003 ceph-mon[29646]: pgmap v108: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:23:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:28 smithi003 ceph-mon[29646]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:29.279 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 15 52941 52926 143.777 113.75 0.00343022 0.00695029 2024-09-13T14:23:29.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:29 smithi019 ceph-mon[32895]: from='client.24755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:29.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:29 smithi003 ceph-mon[41592]: from='client.24755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:29.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:29 smithi003 ceph-mon[29646]: from='client.24755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:30.279 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 54633 54617 142.189 105.688 0.00899529 0.00702806 2024-09-13T14:23:30.906 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:30 smithi019 ceph-mon[32895]: pgmap v109: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-13T14:23:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:30 smithi003 ceph-mon[29646]: pgmap v109: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-13T14:23:30.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:30 smithi003 ceph-mon[41592]: pgmap v109: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-13T14:23:31.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:23:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:23:31.280 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 56135 56119 140.254 93.875 0.00502768 0.0071254 2024-09-13T14:23:31.495 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:31.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:31 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/4160981593' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:31 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/4160981593' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:31 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/4160981593' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:32.280 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 57597 57581 138.375 91.375 0.00745521 0.00722192 2024-09-13T14:23:32.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:32 smithi019 ceph-mon[32895]: pgmap v110: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:23:32.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:32 smithi019 ceph-mon[32895]: from='client.24764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:32 smithi003 ceph-mon[41592]: pgmap v110: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:23:32.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:32 smithi003 ceph-mon[41592]: from='client.24764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:32 smithi003 ceph-mon[29646]: pgmap v110: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:23:32.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:32 smithi003 ceph-mon[29646]: from='client.24764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.281 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 59159 59143 136.862 97.625 0.00539852 0.00730135 2024-09-13T14:23:33.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:33 smithi019 ceph-mon[32895]: from='client.24770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:33.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:33 smithi003 ceph-mon[41592]: from='client.24770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:33.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:33 smithi003 ceph-mon[29646]: from='client.24770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:34.283 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 60541 60525 135.049 86.375 0.0192457 0.00739905 2024-09-13T14:23:34.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:34 smithi019 ceph-mon[32895]: pgmap v111: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:23:34.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:34 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3438647788' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:34.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:34 smithi003 ceph-mon[29646]: pgmap v111: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:23:34.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:34 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3438647788' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:34.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:34 smithi003 ceph-mon[41592]: pgmap v111: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:23:34.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:34 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3438647788' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:35.223 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:35.285 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 61658 61642 132.792 69.8125 0.00570073 0.00752459 2024-09-13T14:23:36.285 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 62974 62958 131.107 82.25 0.00980096 0.00762285 2024-09-13T14:23:37.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:36 smithi003 ceph-mon[41592]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:37.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:36 smithi003 ceph-mon[41592]: pgmap v112: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-13T14:23:37.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:36 smithi003 ceph-mon[41592]: from='client.24785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:37.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:36 smithi003 ceph-mon[29646]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:37.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:36 smithi003 ceph-mon[29646]: pgmap v112: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-13T14:23:37.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:36 smithi003 ceph-mon[29646]: from='client.24785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:37.149 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:23:37 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:37] "GET /metrics HTTP/1.1" 200 243112 "" "Prometheus/2.43.0" 2024-09-13T14:23:37.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:36 smithi019 ceph-mon[32895]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:37.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:36 smithi019 ceph-mon[32895]: pgmap v112: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-13T14:23:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:36 smithi019 ceph-mon[32895]: from='client.24785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:37.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 64477 64461 129.908 93.9375 0.00793167 0.00769302 2024-09-13T14:23:38.285 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 65841 65825 128.512 85.25 0.0155754 0.00777592 2024-09-13T14:23:38.913 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:39.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:38 smithi019 ceph-mon[32895]: pgmap v113: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:23:39.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:38 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3623486783' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:38 smithi003 ceph-mon[29646]: pgmap v113: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:23:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:38 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3623486783' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:39.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:38 smithi003 ceph-mon[41592]: pgmap v113: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:23:39.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:38 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3623486783' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:39.285 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 67109 67093 127.02 79.25 0.0300959 0.00786738 2024-09-13T14:23:40.285 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 68440 68424 125.73 83.1875 0.00876585 0.00794866 2024-09-13T14:23:41.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:23:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:23:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:40 smithi019 ceph-mon[32895]: from='client.24797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:40 smithi019 ceph-mon[32895]: pgmap v114: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-13T14:23:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:40 smithi019 ceph-mon[32895]: from='client.24694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:40 smithi003 ceph-mon[29646]: from='client.24797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:40 smithi003 ceph-mon[29646]: pgmap v114: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-13T14:23:41.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:40 smithi003 ceph-mon[29646]: from='client.24694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:40 smithi003 ceph-mon[41592]: from='client.24797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:40 smithi003 ceph-mon[41592]: pgmap v114: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-13T14:23:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:40 smithi003 ceph-mon[41592]: from='client.24694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.285 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 69876 69860 124.703 89.75 0.00451876 0.00801407 2024-09-13T14:23:42.285 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 71373 71357 123.838 93.5625 0.0224907 0.00807063 2024-09-13T14:23:42.531 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:42.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:42 smithi003 ceph-mon[41592]: pgmap v115: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T14:23:42.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:42 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2571271955' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:42 smithi003 ceph-mon[29646]: pgmap v115: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T14:23:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:42 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2571271955' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:43.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:42 smithi019 ceph-mon[32895]: pgmap v115: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T14:23:43.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:42 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2571271955' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:43.286 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 15 72818 72803 122.931 90.375 0.00528304 0.00812951 2024-09-13T14:23:44.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:43 smithi003 ceph-mon[41592]: from='client.24806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:43 smithi003 ceph-mon[29646]: from='client.24806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:43 smithi019 ceph-mon[32895]: from='client.24806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.286 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 74343 74327 122.203 95.25 0.0143809 0.0081781 2024-09-13T14:23:44.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[41592]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[41592]: pgmap v116: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:23:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:44.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[29646]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[29646]: pgmap v116: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:23:44.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:44.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:44.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:44 smithi019 ceph-mon[32895]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:44 smithi019 ceph-mon[32895]: pgmap v116: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:23:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:23:45.286 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 75742 75726 121.311 87.4375 0.00932723 0.00823825 2024-09-13T14:23:46.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:45 smithi003 ceph-mon[41592]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:23:46.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:45 smithi003 ceph-mon[41592]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:23:46.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:45 smithi003 ceph-mon[41592]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:23:46.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:45 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2706140659' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:46.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:45 smithi003 ceph-mon[29646]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:23:46.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:45 smithi003 ceph-mon[29646]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:23:46.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:45 smithi003 ceph-mon[29646]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:23:46.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:45 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2706140659' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:46.082 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:45 smithi019 ceph-mon[32895]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:23:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:45 smithi019 ceph-mon[32895]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:23:46.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:45 smithi019 ceph-mon[32895]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:23:46.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:45 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2706140659' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:46.286 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:23:46.286727+0000 min lat: 0.00125696 max lat: 0.141946 avg lat: 0.00830561 2024-09-13T14:23:46.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:23:46.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 77047 77031 120.318 81.5625 0.00580398 0.00830561 2024-09-13T14:23:47.147 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:46 smithi003 ceph-mon[29646]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi019 2024-09-13T14:23:47.147 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:46 smithi003 ceph-mon[29646]: pgmap v117: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T14:23:47.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:46 smithi003 ceph-mon[41592]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi019 2024-09-13T14:23:47.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:46 smithi003 ceph-mon[41592]: pgmap v117: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T14:23:47.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:46 smithi019 ceph-mon[32895]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi019 2024-09-13T14:23:47.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:46 smithi019 ceph-mon[32895]: pgmap v117: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T14:23:47.286 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 78293 78277 119.282 77.875 0.00443281 0.0083776 2024-09-13T14:23:47.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:23:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:47] "GET /metrics HTTP/1.1" 200 243112 "" "Prometheus/2.43.0" 2024-09-13T14:23:48.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:47 smithi019 ceph-mon[32895]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:48.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:47 smithi019 ceph-mon[32895]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:48.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:47 smithi003 ceph-mon[29646]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:48.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:47 smithi003 ceph-mon[29646]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:47 smithi003 ceph-mon[41592]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:47 smithi003 ceph-mon[41592]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:48.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 79530 79514 118.283 77.3125 0.0136922 0.00844955 2024-09-13T14:23:49.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:48 smithi019 ceph-mon[32895]: pgmap v118: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:23:49.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:48 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:49.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:48 smithi003 ceph-mon[41592]: pgmap v118: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:23:49.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:48 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:49.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:48 smithi003 ceph-mon[29646]: pgmap v118: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:23:49.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:48 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:49.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 15 80761 80746 117.323 77 0.0280846 0.00851797 2024-09-13T14:23:49.554 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:49.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:49 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2731468538' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:49 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2731468538' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:50.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:49 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2731468538' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:50.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 81981 81965 116.388 76.1875 0.0204997 0.00858695 2024-09-13T14:23:51.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:23:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:23:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:50 smithi019 ceph-mon[32895]: pgmap v119: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:23:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:50 smithi019 ceph-mon[32895]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:50 smithi003 ceph-mon[41592]: pgmap v119: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:23:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:50 smithi003 ceph-mon[41592]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:51.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:50 smithi003 ceph-mon[29646]: pgmap v119: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:23:51.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:50 smithi003 ceph-mon[29646]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:51.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 83192 83176 115.484 75.6875 0.0084033 0.00865403 2024-09-13T14:23:52.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:51 smithi019 ceph-mon[32895]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:52.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:51 smithi003 ceph-mon[41592]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:52.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:51 smithi003 ceph-mon[29646]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:52.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 84388 84372 114.599 74.75 0.0112001 0.00872105 2024-09-13T14:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:53.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:52 smithi019 ceph-mon[32895]: pgmap v120: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:23:53.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:52 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/551632666' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:53.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:52 smithi003 ceph-mon[41592]: pgmap v120: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:23:53.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:52 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/551632666' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:53.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:52 smithi003 ceph-mon[29646]: pgmap v120: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:23:53.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:52 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/551632666' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:53.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 85550 85534 113.706 72.625 0.0104785 0.0087896 2024-09-13T14:23:54.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 86737 86721 112.882 74.1875 0.0127766 0.00885406 2024-09-13T14:23:55.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:54 smithi003 ceph-mon[41592]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:55.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:54 smithi003 ceph-mon[41592]: pgmap v121: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-13T14:23:55.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:54 smithi003 ceph-mon[41592]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:55.186 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:54 smithi019 ceph-mon[32895]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:55.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:54 smithi019 ceph-mon[32895]: pgmap v121: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-13T14:23:55.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:54 smithi019 ceph-mon[32895]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:55.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:54 smithi003 ceph-mon[29646]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:55.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:54 smithi003 ceph-mon[29646]: pgmap v121: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-13T14:23:55.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:54 smithi003 ceph-mon[29646]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:55.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 15 87906 87891 112.071 73.125 0.0125482 0.00891726 2024-09-13T14:23:56.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 88972 88956 111.161 66.5625 0.0259441 0.00898974 2024-09-13T14:23:56.381 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:23:57.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:56 smithi003 ceph-mon[41592]: pgmap v122: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-13T14:23:57.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:56 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1172708917' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:57.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:56 smithi003 ceph-mon[29646]: pgmap v122: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-13T14:23:57.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:56 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1172708917' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:57.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:56 smithi019 ceph-mon[32895]: pgmap v122: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-13T14:23:57.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:56 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1172708917' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:57.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 90086 90070 110.347 69.625 0.0112096 0.00905718 2024-09-13T14:23:57.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:23:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:23:57] "GET /metrics HTTP/1.1" 200 243172 "" "Prometheus/2.43.0" 2024-09-13T14:23:58.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:57 smithi019 ceph-mon[32895]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:57 smithi019 ceph-mon[32895]: from='client.24875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:58.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:57 smithi003 ceph-mon[41592]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:58.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:57 smithi003 ceph-mon[41592]: from='client.24875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:58.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:57 smithi003 ceph-mon[29646]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:57 smithi003 ceph-mon[29646]: from='client.24875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:58.287 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 91201 91185 109.565 69.6875 0.010642 0.00912172 2024-09-13T14:23:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:58 smithi019 ceph-mon[32895]: pgmap v123: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T14:23:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:58 smithi003 ceph-mon[41592]: pgmap v123: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T14:23:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:58 smithi003 ceph-mon[29646]: pgmap v123: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T14:23:59.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 92323 92307 108.821 70.125 0.0112518 0.00918452 2024-09-13T14:23:59.805 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:00.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:23:59 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2978546272' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:00.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:23:59 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2978546272' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:00.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:23:59 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2978546272' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:00.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 93425 93409 108.081 68.875 0.00994669 0.00924741 2024-09-13T14:24:01.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:24:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:00 smithi019 ceph-mon[32895]: pgmap v124: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:24:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:00 smithi019 ceph-mon[32895]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:00 smithi019 ceph-mon[32895]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:00 smithi003 ceph-mon[41592]: pgmap v124: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:24:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:00 smithi003 ceph-mon[41592]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:00 smithi003 ceph-mon[41592]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:00 smithi003 ceph-mon[29646]: pgmap v124: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:24:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:00 smithi003 ceph-mon[29646]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:00 smithi003 ceph-mon[29646]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:01.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 94541 94525 107.384 69.75 0.00747788 0.00930685 2024-09-13T14:24:02.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:01 smithi019 ceph-mon[32895]: pgmap v125: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:24:02.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:01 smithi003 ceph-mon[29646]: pgmap v125: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:24:02.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:01 smithi003 ceph-mon[41592]: pgmap v125: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:24:02.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 95600 95584 106.648 66.1875 0.0123964 0.00937115 2024-09-13T14:24:03.083 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:02 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3822241032' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:03.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:02 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3822241032' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:02 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3822241032' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:03.223 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:03.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 96611 96595 105.886 63.1875 0.0107896 0.00943902 2024-09-13T14:24:04.083 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:03 smithi019 ceph-mon[32895]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.083 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:03 smithi019 ceph-mon[32895]: pgmap v126: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-13T14:24:04.083 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:03 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:04.083 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:03 smithi019 ceph-mon[32895]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:03 smithi003 ceph-mon[29646]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:03 smithi003 ceph-mon[29646]: pgmap v126: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-13T14:24:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:03 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:03 smithi003 ceph-mon[29646]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:03 smithi003 ceph-mon[41592]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:03 smithi003 ceph-mon[41592]: pgmap v126: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-13T14:24:04.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:03 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:04.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:03 smithi003 ceph-mon[41592]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 97681 97665 105.213 66.875 0.0179639 0.00949968 2024-09-13T14:24:05.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 98729 98713 104.54 65.5 0.00971693 0.00956068 2024-09-13T14:24:06.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:24:06.288470+0000 min lat: 0.00125696 max lat: 0.141946 avg lat: 0.00962201 2024-09-13T14:24:06.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:24:06.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 99761 99745 103.873 64.5 0.0104388 0.00962201 2024-09-13T14:24:06.695 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:06 smithi019 ceph-mon[32895]: pgmap v127: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:24:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:06 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3877887177' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:06 smithi003 ceph-mon[29646]: pgmap v127: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:24:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:06 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3877887177' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:06 smithi003 ceph-mon[41592]: pgmap v127: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:24:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:06 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3877887177' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:07.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 100782 100766 103.216 63.8125 0.0158603 0.00968251 2024-09-13T14:24:07.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:07] "GET /metrics HTTP/1.1" 200 243246 "" "Prometheus/2.43.0" 2024-09-13T14:24:07.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:07 smithi019 ceph-mon[32895]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:07.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:07 smithi003 ceph-mon[41592]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:07.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:07 smithi003 ceph-mon[29646]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:08.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 101847 101831 102.625 66.5625 0.0084093 0.00973944 2024-09-13T14:24:08.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:08 smithi019 ceph-mon[32895]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:08 smithi019 ceph-mon[32895]: pgmap v128: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s 2024-09-13T14:24:08.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:08 smithi003 ceph-mon[41592]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:08.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:08 smithi003 ceph-mon[41592]: pgmap v128: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s 2024-09-13T14:24:08.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:08 smithi003 ceph-mon[29646]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:08.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:08 smithi003 ceph-mon[29646]: pgmap v128: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s 2024-09-13T14:24:09.288 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 102840 102824 101.981 62.0625 0.0148731 0.00980056 2024-09-13T14:24:10.154 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:10.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 103832 103816 101.357 62 0.0160545 0.00986123 2024-09-13T14:24:10.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:10 smithi019 ceph-mon[32895]: pgmap v129: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-13T14:24:10.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:10 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2671687874' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:10.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:10 smithi003 ceph-mon[29646]: pgmap v129: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-13T14:24:10.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:10 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2671687874' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:10 smithi003 ceph-mon[41592]: pgmap v129: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-13T14:24:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:10 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2671687874' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:11.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:24:11.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 104787 104771 100.716 59.6875 0.011144 0.00992402 2024-09-13T14:24:11.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:11 smithi003 ceph-mon[41592]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:11.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:11 smithi003 ceph-mon[41592]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:11.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:11 smithi003 ceph-mon[29646]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:11.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:11 smithi003 ceph-mon[29646]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:11.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:11 smithi019 ceph-mon[32895]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:11.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:11 smithi019 ceph-mon[32895]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:12.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 105705 105689 100.059 57.375 0.00646458 0.0099892 2024-09-13T14:24:12.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:12 smithi019 ceph-mon[32895]: pgmap v130: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:24:12.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:12 smithi003 ceph-mon[41592]: pgmap v130: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:24:12.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:12 smithi003 ceph-mon[29646]: pgmap v130: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:24:13.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 106665 106649 99.461 60 0.0102849 0.0100488 2024-09-13T14:24:13.555 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:13.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:13 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1238334847' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:13.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:13 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1238334847' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:13.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:13 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1238334847' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:14.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 107588 107572 98.8467 57.6875 0.0364471 0.0101103 2024-09-13T14:24:14.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:14 smithi019 ceph-mon[32895]: pgmap v131: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s 2024-09-13T14:24:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:14 smithi019 ceph-mon[32895]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:14 smithi003 ceph-mon[41592]: pgmap v131: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s 2024-09-13T14:24:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:14 smithi003 ceph-mon[41592]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:14 smithi003 ceph-mon[29646]: pgmap v131: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s 2024-09-13T14:24:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:14 smithi003 ceph-mon[29646]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:15.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 108518 108502 98.2566 58.125 0.0106533 0.0101726 2024-09-13T14:24:15.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:15 smithi019 ceph-mon[32895]: from='client.24965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:15.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:15 smithi003 ceph-mon[29646]: from='client.24965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:15.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:15 smithi003 ceph-mon[41592]: from='client.24965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:16.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 109457 109441 97.6913 58.6875 0.0182811 0.0102307 2024-09-13T14:24:16.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:16 smithi019 ceph-mon[32895]: pgmap v132: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-13T14:24:16.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:16 smithi003 ceph-mon[41592]: pgmap v132: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-13T14:24:16.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:16 smithi003 ceph-mon[29646]: pgmap v132: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-13T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:17.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 110363 110347 97.1129 56.625 0.0181894 0.0102918 2024-09-13T14:24:17.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:17 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:17] "GET /metrics HTTP/1.1" 200 243246 "" "Prometheus/2.43.0" 2024-09-13T14:24:17.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:17 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2910195454' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:17.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:17 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2910195454' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:17.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:17 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2910195454' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:18.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 111269 111253 96.5506 56.625 0.00893621 0.0103521 2024-09-13T14:24:18.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:18 smithi019 ceph-mon[32895]: from='client.24817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:18.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:18 smithi019 ceph-mon[32895]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:18.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:18 smithi019 ceph-mon[32895]: pgmap v133: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s 2024-09-13T14:24:18.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:18 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:18.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:18 smithi003 ceph-mon[41592]: from='client.24817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:18.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:18 smithi003 ceph-mon[41592]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:18.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:18 smithi003 ceph-mon[41592]: pgmap v133: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s 2024-09-13T14:24:18.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:18 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:18.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:18 smithi003 ceph-mon[29646]: from='client.24817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:18.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:18 smithi003 ceph-mon[29646]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:18.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:18 smithi003 ceph-mon[29646]: pgmap v133: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s 2024-09-13T14:24:18.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:18 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:19.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 112173 112157 96.002 56.5 0.00928292 0.010411 2024-09-13T14:24:20.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 113082 113066 95.4724 56.8125 0.0126912 0.0104688 2024-09-13T14:24:20.345 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:20.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:20 smithi003 ceph-mon[29646]: pgmap v134: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-09-13T14:24:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:20 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/623107524' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:20.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:20 smithi003 ceph-mon[41592]: pgmap v134: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-09-13T14:24:20.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:20 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/623107524' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:20.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:20 smithi019 ceph-mon[32895]: pgmap v134: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-09-13T14:24:20.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:20 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/623107524' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:21.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[34961]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:24:21.289 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 113860 113844 94.8477 48.625 0.00345621 0.0105209 2024-09-13T14:24:21.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:21 smithi019 ceph-mon[32895]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:21.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:21 smithi019 ceph-mon[32895]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:21.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:21 smithi003 ceph-mon[41592]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:21.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:21 smithi003 ceph-mon[41592]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:21.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:21 smithi003 ceph-mon[29646]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:21.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:21 smithi003 ceph-mon[29646]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:22.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 114430 114414 94.0686 35.625 0.00518128 0.0106255 2024-09-13T14:24:22.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:22 smithi019 ceph-mon[32895]: pgmap v135: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:24:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:22 smithi003 ceph-mon[29646]: pgmap v135: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:24:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:22 smithi003 ceph-mon[41592]: pgmap v135: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:24:23.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 115191 115175 93.4646 47.5625 0.008963 0.010694 2024-09-13T14:24:23.794 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:23.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:23 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2527961300' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:23.794 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:23 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2527961300' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:23.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:23 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2527961300' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:24.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 116161 116145 93.0436 60.625 0.0156187 0.0107428 2024-09-13T14:24:24.843 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:24 smithi019 ceph-mon[32895]: pgmap v136: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-13T14:24:24.844 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:24 smithi019 ceph-mon[32895]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:24 smithi003 ceph-mon[41592]: pgmap v136: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-13T14:24:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:24 smithi003 ceph-mon[41592]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:24.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:24 smithi003 ceph-mon[29646]: pgmap v136: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-13T14:24:24.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:24 smithi003 ceph-mon[29646]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:25.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 117193 117177 92.6822 64.5 0.00866781 0.0107848 2024-09-13T14:24:25.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:25 smithi019 ceph-mon[32895]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:25.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:25 smithi003 ceph-mon[41592]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:25.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:25 smithi003 ceph-mon[29646]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:26.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:24:26.290353+0000 min lat: 0.00125696 max lat: 0.64275 avg lat: 0.0108241 2024-09-13T14:24:26.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:24:26.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 118244 118228 92.3448 65.6875 0.00897382 0.0108241 2024-09-13T14:24:26.880 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:26 smithi019 ceph-mon[32895]: pgmap v137: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:24:26.881 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:26 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:26.881 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:26 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:24:26.881 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:26 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:24:26.881 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:26 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[41592]: pgmap v137: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:24:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:24:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:24:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[29646]: pgmap v137: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:24:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:26.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:24:26.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:24:26.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:26 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:27.256 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:27.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 119219 119203 91.957 60.9375 0.00846657 0.0108697 2024-09-13T14:24:27.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:27] "GET /metrics HTTP/1.1" 200 243251 "" "Prometheus/2.43.0" 2024-09-13T14:24:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:27 smithi019 ceph-mon[32895]: Upgrade: Updating mgr.x 2024-09-13T14:24:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:27 smithi019 ceph-mon[32895]: Deploying daemon mgr.x on smithi019 2024-09-13T14:24:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:27 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/4237483918' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:27.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:27 smithi003 ceph-mon[41592]: Upgrade: Updating mgr.x 2024-09-13T14:24:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:27 smithi003 ceph-mon[41592]: Deploying daemon mgr.x on smithi019 2024-09-13T14:24:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:27 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/4237483918' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:27.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:27 smithi003 ceph-mon[29646]: Upgrade: Updating mgr.x 2024-09-13T14:24:27.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:27 smithi003 ceph-mon[29646]: Deploying daemon mgr.x on smithi019 2024-09-13T14:24:27.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:27 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/4237483918' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:28.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 120193 120177 91.5779 60.875 0.0331807 0.0109139 2024-09-13T14:24:28.600 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:28 smithi019 systemd[1]: Stopping Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:24:28.931 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:28 smithi019 podman[68156]: 2024-09-13 14:24:28.827447426 +0000 UTC m=+0.186687074 container died f4595aba35763e5274893a4b034ef1a9b36166d8d3d21dcbb122485db463a4fc (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, ceph=True) 2024-09-13T14:24:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:28 smithi019 ceph-mon[32895]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:28 smithi019 ceph-mon[32895]: pgmap v138: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 45 MiB/s wr, 716 op/s 2024-09-13T14:24:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:28 smithi019 ceph-mon[32895]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:28 smithi003 ceph-mon[29646]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:28 smithi003 ceph-mon[29646]: pgmap v138: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 45 MiB/s wr, 716 op/s 2024-09-13T14:24:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:28 smithi003 ceph-mon[29646]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:28.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:28 smithi003 ceph-mon[41592]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:28.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:28 smithi003 ceph-mon[41592]: pgmap v138: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 45 MiB/s wr, 716 op/s 2024-09-13T14:24:28.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:28 smithi003 ceph-mon[41592]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:29.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 121203 121187 91.2351 63.125 0.00664453 0.0109557 2024-09-13T14:24:29.681 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:29 smithi019 podman[68156]: 2024-09-13 14:24:29.431401105 +0000 UTC m=+0.790640745 container cleanup f4595aba35763e5274893a4b034ef1a9b36166d8d3d21dcbb122485db463a4fc (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:24:29.682 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:29 smithi019 bash[68156]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x 2024-09-13T14:24:29.993 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:29 smithi019 podman[68172]: 2024-09-13 14:24:29.741879103 +0000 UTC m=+0.911533477 container remove f4595aba35763e5274893a4b034ef1a9b36166d8d3d21dcbb122485db463a4fc (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:24:29.993 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:29 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:24:30.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 122142 122126 90.8476 58.6875 0.0183341 0.0110022 2024-09-13T14:24:30.316 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:29 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-13T14:24:30.316 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:29 smithi019 systemd[1]: Stopped Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:24:30.316 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:29 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service: Consumed 10.924s CPU time. 2024-09-13T14:24:30.605 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:30 smithi019 systemd[1]: Starting Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:24:30.771 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:30.859 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:30 smithi019 podman[68291]: 2024-09-13 14:24:30.582394418 +0000 UTC m=+0.042165625 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:30.859 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:30 smithi019 podman[68291]: 2024-09-13 14:24:30.718840815 +0000 UTC m=+0.178612013 container create 733f346f63cf610f56f232be68a293d2e54b9efd6ec2df96fa6b3e291c60658d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T14:24:30.860 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:30 smithi019 ceph-mon[32895]: pgmap v139: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 46 MiB/s wr, 743 op/s 2024-09-13T14:24:30.860 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:30 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1058345462' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:30.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:30 smithi003 ceph-mon[29646]: pgmap v139: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 46 MiB/s wr, 743 op/s 2024-09-13T14:24:30.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:30 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1058345462' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:30.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:30 smithi003 ceph-mon[41592]: pgmap v139: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 46 MiB/s wr, 743 op/s 2024-09-13T14:24:30.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:30 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1058345462' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:31.148 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:31 smithi019 podman[68291]: 2024-09-13 14:24:31.030368828 +0000 UTC m=+0.490140026 container init 733f346f63cf610f56f232be68a293d2e54b9efd6ec2df96fa6b3e291c60658d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:24:31.148 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:31 smithi019 podman[68291]: 2024-09-13 14:24:31.039377532 +0000 UTC m=+0.499148733 container start 733f346f63cf610f56f232be68a293d2e54b9efd6ec2df96fa6b3e291c60658d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-13T14:24:31.148 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:31 smithi019 ceph-mgr[68329]: -- 172.21.15.19:0/1857675519 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55b5e7acb520 con 0x55b5e7a9f000 2024-09-13T14:24:31.148 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:31 smithi019 ceph-mgr[68329]: -- 172.21.15.19:0/1857675519 <== mon.2 v2:172.21.15.3:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b5e7ac10e0 con 0x55b5e7a9f000 2024-09-13T14:24:31.290 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 123063 123047 90.456 57.5625 0.0140692 0.0110496 2024-09-13T14:24:31.432 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:31 smithi019 bash[68291]: 733f346f63cf610f56f232be68a293d2e54b9efd6ec2df96fa6b3e291c60658d 2024-09-13T14:24:31.432 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:31 smithi019 systemd[1]: Started Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:24:31.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:31 smithi019 ceph-mon[32895]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:31 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:31 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:31 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:31.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:31 smithi003 ceph-mon[29646]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:31.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:31 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:31.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:31 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:31 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:31 smithi003 ceph-mon[41592]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:31 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:31 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:31 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:32.291 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 123888 123872 90.0038 51.5625 0.0234893 0.0111053 2024-09-13T14:24:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:32 smithi019 ceph-mon[32895]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:32 smithi019 ceph-mon[32895]: pgmap v140: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:24:32.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:32 smithi003 ceph-mon[41592]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:32.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:32 smithi003 ceph-mon[41592]: pgmap v140: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:24:32.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:32 smithi003 ceph-mon[29646]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:32.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:32 smithi003 ceph-mon[29646]: pgmap v140: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:24:33.291 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 124705 124689 89.5562 51.0625 0.0268416 0.0111608 2024-09-13T14:24:33.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:33 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:33.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:33 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:33.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:33 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:34.247 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:34.291 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 125484 125468 89.0917 48.6875 0.0216324 0.0112191 2024-09-13T14:24:34.835 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:34 smithi019 ceph-mon[32895]: pgmap v141: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 754 op/s 2024-09-13T14:24:34.835 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:34 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2633809244' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:34.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:34 smithi003 ceph-mon[29646]: pgmap v141: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 754 op/s 2024-09-13T14:24:34.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:34 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2633809244' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:34.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:34 smithi003 ceph-mon[41592]: pgmap v141: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 754 op/s 2024-09-13T14:24:34.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:34 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2633809244' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:35.291 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 126306 126290 88.6678 51.375 0.00759821 0.011273 2024-09-13T14:24:35.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:35 smithi019 ceph-mon[32895]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:35 smithi019 ceph-mon[32895]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:35 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:35 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:35.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:35 smithi003 ceph-mon[41592]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:35 smithi003 ceph-mon[41592]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:35 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:35.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:35 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:35.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:35 smithi003 ceph-mon[29646]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:35 smithi003 ceph-mon[29646]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:35 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:35.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:35 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:36.291 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0134 2024-09-13T14:24:36.291 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 127035 2024-09-13T14:24:36.291 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T14:24:36.291 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 88.2056 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 40.7776 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 245.75 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 35.625 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1411 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 652.441 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3932 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 570 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0113327 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0108863 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 0.64275 2024-09-13T14:24:36.292 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00125696 2024-09-13T14:24:36.296 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:24:36.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:36 smithi019 ceph-mon[32895]: pgmap v142: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:24:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:36 smithi003 ceph-mon[41592]: pgmap v142: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:24:36.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:36 smithi003 ceph-mon[29646]: pgmap v142: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:24:37.435 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:37 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:37] "GET /metrics HTTP/1.1" 200 243269 "" "Prometheus/2.43.0" 2024-09-13T14:24:37.595 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:37.913 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:37 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3812092188' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:37.913 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:37.913 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:37 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:37.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:37 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3812092188' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:37.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:37.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:37 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:37.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:37 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3812092188' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:37.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:37.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:37 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:38.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:38 smithi003 ceph-mon[29646]: pgmap v143: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 774 op/s 2024-09-13T14:24:38.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:38 smithi003 ceph-mon[29646]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:38.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:38.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:38 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:38.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:38 smithi003 ceph-mon[41592]: pgmap v143: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 774 op/s 2024-09-13T14:24:38.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:38 smithi003 ceph-mon[41592]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:38.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:38.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:38 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:38.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:38 smithi019 ceph-mon[32895]: pgmap v143: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 774 op/s 2024-09-13T14:24:38.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:38 smithi019 ceph-mon[32895]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:38.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:38.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:38 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:39.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:39 smithi019 ceph-mon[32895]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:39.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:39 smithi003 ceph-mon[29646]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:39.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:39 smithi003 ceph-mon[41592]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:40.622 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:24:40] ENGINE Bus STARTING 2024-09-13T14:24:40.622 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: CherryPy Checker: 2024-09-13T14:24:40.622 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: The Application mounted at '' has an empty config. 2024-09-13T14:24:40.622 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:40.622 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:24:40] ENGINE Serving on http://:::9283 2024-09-13T14:24:40.622 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:24:40] ENGINE Bus STARTED 2024-09-13T14:24:40.908 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:24:40.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-mon[32895]: pgmap v144: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 794 op/s 2024-09-13T14:24:40.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-mon[32895]: Standby manager daemon x restarted 2024-09-13T14:24:40.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-mon[32895]: Standby manager daemon x started 2024-09-13T14:24:40.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:24:40.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:24:40.908 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:24:40.909 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:24:40.909 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:40 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2197601397' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:40.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[41592]: pgmap v144: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 794 op/s 2024-09-13T14:24:40.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[41592]: Standby manager daemon x restarted 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[41592]: Standby manager daemon x started 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2197601397' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[29646]: pgmap v144: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 794 op/s 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[29646]: Standby manager daemon x restarted 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[29646]: Standby manager daemon x started 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:24:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:24:40.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:24:40.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:24:40.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:40 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2197601397' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:40.956 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:24:41.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:41 smithi019 ceph-mon[32895]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-13T14:24:41.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:41 smithi003 ceph-mon[41592]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-13T14:24:41.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:41 smithi003 ceph-mon[29646]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-13T14:24:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:42 smithi019 ceph-mon[32895]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:42 smithi019 ceph-mon[32895]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:42 smithi019 ceph-mon[32895]: pgmap v145: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 54 MiB/s wr, 1.07k op/s 2024-09-13T14:24:42.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:42 smithi003 ceph-mon[41592]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:42 smithi003 ceph-mon[41592]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:42 smithi003 ceph-mon[41592]: pgmap v145: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 54 MiB/s wr, 1.07k op/s 2024-09-13T14:24:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:42 smithi003 ceph-mon[29646]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:42 smithi003 ceph-mon[29646]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:42 smithi003 ceph-mon[29646]: pgmap v145: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 54 MiB/s wr, 1.07k op/s 2024-09-13T14:24:44.260 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ignoring --setuser ceph since I am not root 2024-09-13T14:24:44.260 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ignoring --setgroup ceph since I am not root 2024-09-13T14:24:44.260 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:44.163+0000 7f9edd8edec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:24:44.260 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:44.258+0000 7f9edd8edec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:24:44.415 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:43 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:24:43] ENGINE Bus STOPPING 2024-09-13T14:24:44.576 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:44.365+0000 7f9edd8edec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:24:44.681 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:24:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:24:44.682 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:24:44] ENGINE Bus STOPPED 2024-09-13T14:24:44.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:44.574+0000 7f9edd8edec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:24:44.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:44.794+0000 7f9edd8edec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:24:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: pgmap v146: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 30 MiB/s wr, 677 op/s 2024-09-13T14:24:44.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:44.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:44.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:44.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:44.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:44.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:44.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:24:44.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: Activating manager daemon x 2024-09-13T14:24:44.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:24:44.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:24:44.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: mgrmap e22: x(active, starting, since 0.0216424s) 2024-09-13T14:24:44.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:44.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:44.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:44.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"}]: dispatch 2024-09-13T14:24:44.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"}]: dispatch 2024-09-13T14:24:44.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:24:44.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:24:44.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:44.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1193213402' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[41592]: Manager daemon x is now available 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: pgmap v146: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 30 MiB/s wr, 677 op/s 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: Activating manager daemon x 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:24:44.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:24:44.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: mgrmap e22: x(active, starting, since 0.0216424s) 2024-09-13T14:24:44.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:44.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:44.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:44.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"}]: dispatch 2024-09-13T14:24:44.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"}]: dispatch 2024-09-13T14:24:44.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:24:44.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1193213402' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:44.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:44 smithi003 ceph-mon[29646]: Manager daemon x is now available 2024-09-13T14:24:44.941 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-09-13T14:24:44.941 INFO:teuthology.orchestra.run.smithi003.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-09-13T14:24:45.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:44.935+0000 7f64d4cee640 -1 mgr.server reply reply (95) Operation not supported Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-09-13T14:24:45.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-09-13T14:24:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: pgmap v146: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 30 MiB/s wr, 677 op/s 2024-09-13T14:24:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: Activating manager daemon x 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.14439 172.21.15.3:0/356261083' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: mgrmap e22: x(active, starting, since 0.0216424s) 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:24:45.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:24:45.185 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1193213402' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:24:45.185 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:44 smithi019 ceph-mon[32895]: Manager daemon x is now available 2024-09-13T14:24:45.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:45 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:45.228+0000 7f9edd8edec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:24:45.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:45 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:45.332+0000 7f9edd8edec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:24:45.595 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:45 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:45.593+0000 7f9edd8edec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:24:45.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:24:46.113 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:24:46.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:45 smithi003 ceph-mon[41592]: mgrmap e23: x(active, since 1.01371s) 2024-09-13T14:24:46.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:45 smithi003 ceph-mon[41592]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:46.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:45 smithi003 ceph-mon[41592]: pgmap v3: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail 2024-09-13T14:24:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:45 smithi003 ceph-mon[29646]: mgrmap e23: x(active, since 1.01371s) 2024-09-13T14:24:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:45 smithi003 ceph-mon[29646]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:45 smithi003 ceph-mon[29646]: pgmap v3: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail 2024-09-13T14:24:46.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:45 smithi019 ceph-mon[32895]: mgrmap e23: x(active, since 1.01371s) 2024-09-13T14:24:46.455 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:45 smithi019 ceph-mon[32895]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:46.455 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:45 smithi019 ceph-mon[32895]: pgmap v3: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail 2024-09-13T14:24:46.983 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:24:46] ENGINE Bus STARTING 2024-09-13T14:24:46.983 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.981+0000 7f64ac720640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:46.983 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: pgmap v4: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail 2024-09-13T14:24:46.983 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:46.983 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:46.983 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:46.983 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:46.984 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:46.984 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:46.984 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:46.984 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:24:46.984 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:46.984 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:24:46.984 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:24:46.984 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:24:46.984 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:24:47.329 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:47.017+0000 7f9edd8edec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:24:47.329 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:47.109+0000 7f9edd8edec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:24:47.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: pgmap v4: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:24:47.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: pgmap v4: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:24:47.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:24:47.332 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:24:47.332 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:24:47.332 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:24:47.332 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:46 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:24:47.361 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.981+0000 7f64ac720640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:47.361 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.981+0000 7f64ac720640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:47.361 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.981+0000 7f64ac720640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:47.361 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.981+0000 7f64ac720640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:47.361 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.987+0000 7f64b1ca8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:47.362 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.987+0000 7f64b1ca8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:47.362 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.987+0000 7f64b1ca8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:47.362 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.987+0000 7f64b1ca8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:47.362 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:46.987+0000 7f64b1ca8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:47.362 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:24:47] ENGINE Serving on http://:::9283 2024-09-13T14:24:47.362 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:24:47] ENGINE Bus STARTED 2024-09-13T14:24:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-mon[41592]: Checking for cert/key for grafana.a 2024-09-13T14:24:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-mon[41592]: Migrating grafana.a cert to cert store 2024-09-13T14:24:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-mon[41592]: Migrating grafana.a key to cert store 2024-09-13T14:24:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-mon[41592]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:24:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-mon[29646]: Checking for cert/key for grafana.a 2024-09-13T14:24:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-mon[29646]: Migrating grafana.a cert to cert store 2024-09-13T14:24:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-mon[29646]: Migrating grafana.a key to cert store 2024-09-13T14:24:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:47 smithi003 ceph-mon[29646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:24:48.256 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:47 smithi019 ceph-mon[32895]: Checking for cert/key for grafana.a 2024-09-13T14:24:48.257 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:47 smithi019 ceph-mon[32895]: Migrating grafana.a cert to cert store 2024-09-13T14:24:48.257 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:47 smithi019 ceph-mon[32895]: Migrating grafana.a key to cert store 2024-09-13T14:24:48.257 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:47 smithi019 ceph-mon[32895]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:24:48.597 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:48.237+0000 7f9edd8edec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:24:48.598 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:48.398+0000 7f9edd8edec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:24:48.598 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:48.596+0000 7f9edd8edec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:24:48.883 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:48.688+0000 7f9edd8edec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:24:49.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-mon[29646]: Deploying cephadm binary to smithi019 2024-09-13T14:24:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-mon[29646]: Deploying cephadm binary to smithi003 2024-09-13T14:24:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-mon[29646]: pgmap v5: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail 2024-09-13T14:24:49.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-mon[41592]: Deploying cephadm binary to smithi019 2024-09-13T14:24:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-mon[41592]: Deploying cephadm binary to smithi003 2024-09-13T14:24:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-mon[41592]: pgmap v5: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail 2024-09-13T14:24:49.184 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:48.879+0000 7f9edd8edec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:24:49.184 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:48.970+0000 7f9edd8edec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:24:49.185 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:49.041+0000 7f9edd8edec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:24:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:49 smithi019 ceph-mon[32895]: Deploying cephadm binary to smithi019 2024-09-13T14:24:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:49 smithi019 ceph-mon[32895]: Deploying cephadm binary to smithi003 2024-09-13T14:24:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:49 smithi019 ceph-mon[32895]: pgmap v5: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail 2024-09-13T14:24:49.522 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:49.235+0000 7f9edd8edec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:24:49.522 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:49.344+0000 7f9edd8edec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:24:49.522 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:49.448+0000 7f9edd8edec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:24:49.570 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:24:49.570 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (3m) 14s ago 4m 28.5M - 0.25.0 c8568f914cd2 9994b1ee34d8 2024-09-13T14:24:49.571 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi019 *:3000 running (3m) 12s ago 3m 88.6M - 9.4.7 954c08fa6188 dc631f7cf889 2024-09-13T14:24:49.571 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.zmhiel smithi003 running (2m) 14s ago 2m 16.6M - 17.2.7-1627-gbbdadfa3 a994b00deb3a db8314ccc6fd 2024-09-13T14:24:49.571 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi019.ydtwbg smithi019 running (2m) 12s ago 2m 14.5M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c0fef505f410 2024-09-13T14:24:49.571 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi019 *:8443,9283 running (18s) 12s ago 7m 69.2M - 19.3.0-4697-gd58f7306 fec07016a2b0 733f346f63cf 2024-09-13T14:24:49.571 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:9283,8765 running (9m) 14s ago 9m 451M - 17.2.7-1627-gbbdadfa3 a994b00deb3a fe65bae0d85f 2024-09-13T14:24:49.571 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (9m) 14s ago 9m 51.7M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a 9c73445eb77a 2024-09-13T14:24:49.572 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi019 running (8m) 12s ago 8m 40.3M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a b71aff50957a 2024-09-13T14:24:49.572 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (8m) 14s ago 8m 41.5M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a a6373bc053ae 2024-09-13T14:24:49.572 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (3m) 14s ago 3m 16.1M - 1.5.0 0da6a335fe13 a786edb7f0b5 2024-09-13T14:24:49.572 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi019 *:9100 running (3m) 12s ago 3m 15.4M - 1.5.0 0da6a335fe13 e22916e85bdc 2024-09-13T14:24:49.572 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (7m) 14s ago 7m 335M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a b0e72d25bc0f 2024-09-13T14:24:49.572 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (7m) 14s ago 7m 222M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a 612dd4f9bdda 2024-09-13T14:24:49.573 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (6m) 14s ago 6m 275M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a d32f24c0ca7e 2024-09-13T14:24:49.573 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (6m) 14s ago 6m 151M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a 20984c58cba9 2024-09-13T14:24:49.573 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi019 running (6m) 12s ago 6m 331M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a e50c61f762b0 2024-09-13T14:24:49.573 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi019 running (5m) 12s ago 5m 363M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a 1cd389248627 2024-09-13T14:24:49.573 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi019 running (5m) 12s ago 5m 192M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a aa95f3146e47 2024-09-13T14:24:49.573 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi019 running (4m) 12s ago 4m 227M 4017M 17.2.7-1627-gbbdadfa3 a994b00deb3a 118a8cf6eadc 2024-09-13T14:24:49.574 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi019 *:9095 running (3m) 12s ago 4m 57.8M - 2.43.0 a07b618ecd1d f9fc99d6a298 2024-09-13T14:24:49.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:49.585+0000 7f9edd8edec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:24:49.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:49.670+0000 7f9edd8edec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:24:49.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:49 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:49.767+0000 7f9edd8edec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:24:50.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:50 smithi003 ceph-mon[29646]: mgrmap e24: x(active, since 5s) 2024-09-13T14:24:50.242 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:50 smithi003 ceph-mon[41592]: mgrmap e24: x(active, since 5s) 2024-09-13T14:24:50.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade pause' 2024-09-13T14:24:50.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-mon[32895]: mgrmap e24: x(active, since 5s) 2024-09-13T14:24:50.738 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:24:50.908 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:50.551+0000 7f648b19e640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:24:50] ENGINE Error in HTTPServer.serve 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: Traceback (most recent call last): 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: self._connections.run(self.expiration_interval) 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: self._run(expiration_interval) 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: s = self.context.wrap_socket( 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:24:50.909 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: return self.sslsocket_class._create( 2024-09-13T14:24:50.910 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:24:50.910 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: self.do_handshake() 2024-09-13T14:24:50.910 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:24:50.910 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: self._sslobj.do_handshake() 2024-09-13T14:24:50.910 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:24:50.910 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:24:50.910 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:24:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:50] "GET /metrics HTTP/1.1" 200 36034 "" "Prometheus/2.43.0" 2024-09-13T14:24:51.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-mon[41592]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:51.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-mon[41592]: pgmap v6: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 90 op/s 2024-09-13T14:24:51.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-mon[29646]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:51.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-mon[29646]: pgmap v6: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 90 op/s 2024-09-13T14:24:51.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:51.062+0000 7f9edd8edec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:24:51.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:24:51] ENGINE Bus STARTING 2024-09-13T14:24:51.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: CherryPy Checker: 2024-09-13T14:24:51.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: The Application mounted at '' has an empty config. 2024-09-13T14:24:51.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: 2024-09-13T14:24:51.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:24:51] ENGINE Serving on http://:::9283 2024-09-13T14:24:51.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: [13/Sep/2024:14:24:51] ENGINE Bus STARTED 2024-09-13T14:24:51.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:51 smithi019 ceph-mon[32895]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:51.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:51 smithi019 ceph-mon[32895]: pgmap v6: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 90 op/s 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: [13/Sep/2024:14:24:50] ENGINE Bus STARTING 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: [13/Sep/2024:14:24:50] ENGINE Serving on http://172.21.15.19:8765 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: [13/Sep/2024:14:24:50] ENGINE Serving on https://172.21.15.19:7150 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: [13/Sep/2024:14:24:50] ENGINE Bus STARTED 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: [13/Sep/2024:14:24:50] ENGINE Error in HTTPServer.serve 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: self.do_handshake() 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: Standby manager daemon y started 2024-09-13T14:24:52.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: mgrmap e25: x(active, since 7s), standbys: y 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: [13/Sep/2024:14:24:50] ENGINE Bus STARTING 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: [13/Sep/2024:14:24:50] ENGINE Serving on http://172.21.15.19:8765 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: [13/Sep/2024:14:24:50] ENGINE Serving on https://172.21.15.19:7150 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: [13/Sep/2024:14:24:50] ENGINE Bus STARTED 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: [13/Sep/2024:14:24:50] ENGINE Error in HTTPServer.serve 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:24:52.185 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-09-13T14:24:52.186 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:24:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:24:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:24:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: Standby manager daemon y started 2024-09-13T14:24:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:24:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:24:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:24:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:24:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: mgrmap e25: x(active, since 7s), standbys: y 2024-09-13T14:24:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:52 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:24:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: [13/Sep/2024:14:24:50] ENGINE Bus STARTING 2024-09-13T14:24:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: [13/Sep/2024:14:24:50] ENGINE Serving on http://172.21.15.19:8765 2024-09-13T14:24:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: [13/Sep/2024:14:24:50] ENGINE Serving on https://172.21.15.19:7150 2024-09-13T14:24:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: [13/Sep/2024:14:24:50] ENGINE Bus STARTED 2024-09-13T14:24:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: [13/Sep/2024:14:24:50] ENGINE Error in HTTPServer.serve 2024-09-13T14:24:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout: Traceback (most recent call last): 2024-09-13T14:24:52.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:24:52.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:24:52.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:24:52.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._run(expiration_interval) 2024-09-13T14:24:52.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:24:52.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:24:52.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:24:52.433 INFO:journalctl@ceph.mon.b.smithi019.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:24:52.434 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:24:52.434 INFO:journalctl@ceph.mon.b.smithi019.stdout: s = self.context.wrap_socket( 2024-09-13T14:24:52.434 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:24:52.434 INFO:journalctl@ceph.mon.b.smithi019.stdout: return self.sslsocket_class._create( 2024-09-13T14:24:52.434 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:24:52.434 INFO:journalctl@ceph.mon.b.smithi019.stdout: self.do_handshake() 2024-09-13T14:24:52.434 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:24:52.434 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._sslobj.do_handshake() 2024-09-13T14:24:52.435 INFO:journalctl@ceph.mon.b.smithi019.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:24:52.435 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: Standby manager daemon y started 2024-09-13T14:24:52.435 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:24:52.435 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:24:52.435 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:24:52.435 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.3:0/4011874505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:24:52.436 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: mgrmap e25: x(active, since 7s), standbys: y 2024-09-13T14:24:52.436 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:52 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:24:53.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:53 smithi003 ceph-mon[41592]: pgmap v7: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 25 KiB/s rd, 439 B/s wr, 552 op/s 2024-09-13T14:24:53.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:53 smithi003 ceph-mon[29646]: pgmap v7: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 25 KiB/s rd, 439 B/s wr, 552 op/s 2024-09-13T14:24:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:53 smithi019 ceph-mon[32895]: pgmap v7: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 25 KiB/s rd, 439 B/s wr, 552 op/s 2024-09-13T14:24:53.518 INFO:teuthology.orchestra.run.smithi003.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:54.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:24:54.529 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:54 smithi003 ceph-mon[41592]: from='client.15225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:54.529 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:54 smithi003 ceph-mon[41592]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:54.529 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:54 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:54.529 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:54 smithi003 ceph-mon[41592]: pgmap v8: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 429 op/s 2024-09-13T14:24:54.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:54 smithi003 ceph-mon[29646]: from='client.15225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:54.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:54 smithi003 ceph-mon[29646]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:54.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:54 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:54.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:54 smithi003 ceph-mon[29646]: pgmap v8: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 429 op/s 2024-09-13T14:24:54.719 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:24:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:54 smithi019 ceph-mon[32895]: from='client.15225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:54 smithi019 ceph-mon[32895]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:54 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:54 smithi019 ceph-mon[32895]: pgmap v8: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 429 op/s 2024-09-13T14:24:57.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:57 smithi003 ceph-mon[41592]: pgmap v9: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 16 KiB/s rd, 279 B/s wr, 749 op/s 2024-09-13T14:24:57.436 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:24:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:24:57.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:57 smithi003 ceph-mon[29646]: pgmap v9: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 16 KiB/s rd, 279 B/s wr, 749 op/s 2024-09-13T14:24:57.436 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:57 smithi019 ceph-mon[32895]: pgmap v9: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 16 KiB/s rd, 279 B/s wr, 749 op/s 2024-09-13T14:24:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:59 smithi019 ceph-mon[32895]: pgmap v10: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 685 op/s 2024-09-13T14:24:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:59 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:24:59 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:59.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:59 smithi003 ceph-mon[41592]: pgmap v10: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 685 op/s 2024-09-13T14:24:59.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:59 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:59.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:24:59 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:59 smithi003 ceph-mon[29646]: pgmap v10: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 685 op/s 2024-09-13T14:24:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:59 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:24:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:24:59 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:25:00] "GET /metrics HTTP/1.1" 200 36034 "" "Prometheus/2.43.0" 2024-09-13T14:25:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: pgmap v11: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 729 op/s 2024-09-13T14:25:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi003 to 2481M 2024-09-13T14:25:01.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:00 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: pgmap v11: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 729 op/s 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi003 to 2481M 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: pgmap v11: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 729 op/s 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:01.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi003 to 2481M 2024-09-13T14:25:01.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:01.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:00 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:02.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:02 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:02.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:02 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:02.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:02 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:02.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:02 smithi019 ceph-mon[32895]: pgmap v12: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 1.02k op/s 2024-09-13T14:25:02.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:02 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:02.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:02 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:02.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:02 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:02.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:02 smithi003 ceph-mon[41592]: pgmap v12: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 1.02k op/s 2024-09-13T14:25:02.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:02 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:02.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:02 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:02.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:02 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:02.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:02 smithi003 ceph-mon[29646]: pgmap v12: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 1.02k op/s 2024-09-13T14:25:05.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:05 smithi019 ceph-mon[32895]: pgmap v13: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T14:25:05.385 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:05 smithi003 ceph-mon[41592]: pgmap v13: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T14:25:05.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:05 smithi003 ceph-mon[29646]: pgmap v13: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T14:25:07.113 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:07 smithi003 ceph-mon[41592]: pgmap v14: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-13T14:25:07.113 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:07 smithi003 ceph-mon[29646]: pgmap v14: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-13T14:25:07.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:07 smithi019 ceph-mon[32895]: pgmap v14: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-13T14:25:07.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:25:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:25:08.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:08.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:08.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: Adjusting osd_memory_target on smithi019 to 2481M 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:25:08.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: Adjusting osd_memory_target on smithi019 to 2481M 2024-09-13T14:25:08.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:08.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:08.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:08.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:25:08.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:08 smithi003 ceph-mon[29646]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:25:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: Adjusting osd_memory_target on smithi019 to 2481M 2024-09-13T14:25:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:25:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:08 smithi019 ceph-mon[32895]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:25:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:25:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:25:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:25:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: pgmap v15: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T14:25:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:25:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:09 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: pgmap v15: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: pgmap v15: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:09.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:09 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:10 smithi019 ceph-mon[32895]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:25:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:10 smithi019 ceph-mon[32895]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:25:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:10 smithi019 ceph-mon[32895]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:25:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:10 smithi019 ceph-mon[32895]: Deploying daemon alertmanager.a on smithi003 2024-09-13T14:25:10.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:10 smithi003 ceph-mon[29646]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:25:10.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:10 smithi003 ceph-mon[29646]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:25:10.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:10 smithi003 ceph-mon[29646]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:25:10.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:10 smithi003 ceph-mon[29646]: Deploying daemon alertmanager.a on smithi003 2024-09-13T14:25:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:10 smithi003 ceph-mon[41592]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:25:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:10 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:25:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:10 smithi003 ceph-mon[41592]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:25:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:10 smithi003 ceph-mon[41592]: Deploying daemon alertmanager.a on smithi003 2024-09-13T14:25:11.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:25:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:25:10] "GET /metrics HTTP/1.1" 200 36244 "" "Prometheus/2.43.0" 2024-09-13T14:25:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:11 smithi019 ceph-mon[32895]: pgmap v16: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 781 op/s 2024-09-13T14:25:11.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:11 smithi003 ceph-mon[29646]: pgmap v16: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 781 op/s 2024-09-13T14:25:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:11 smithi003 ceph-mon[41592]: pgmap v16: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 781 op/s 2024-09-13T14:25:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:13 smithi019 ceph-mon[32895]: pgmap v17: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:25:13.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:13 smithi003 ceph-mon[29646]: pgmap v17: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:25:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:13 smithi003 ceph-mon[41592]: pgmap v17: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:25:15.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:15 smithi019 ceph-mon[32895]: pgmap v18: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-13T14:25:15.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:15 smithi003 ceph-mon[29646]: pgmap v18: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-13T14:25:15.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:15 smithi003 ceph-mon[41592]: pgmap v18: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-13T14:25:17.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:17 smithi019 ceph-mon[32895]: pgmap v19: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:25:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:17 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:17.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:17 smithi003 ceph-mon[29646]: pgmap v19: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:25:17.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:17 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:17.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:25:17 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:25:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:17 smithi003 ceph-mon[41592]: pgmap v19: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:25:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:17 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:18.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:18 smithi003 ceph-mon[29646]: pgmap v20: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-13T14:25:18.316 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:18 smithi003 ceph-mon[41592]: pgmap v20: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-13T14:25:18.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:18 smithi019 ceph-mon[32895]: pgmap v20: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-13T14:25:19.932 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:19 smithi003 systemd[1]: Stopping Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:20.182 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:19 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[77793]: ts=2024-09-13T14:25:19.930Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:25:20.182 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:19 smithi003 podman[92014]: 2024-09-13 14:25:19.985795016 +0000 UTC m=+0.200421576 container died 9994b1ee34d8e3ba211c04279e16b33b7e0e98bfae609e48654a2000f73b6791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:20.639 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:20 smithi003 podman[92014]: 2024-09-13 14:25:20.386486093 +0000 UTC m=+0.601112649 container cleanup 9994b1ee34d8e3ba211c04279e16b33b7e0e98bfae609e48654a2000f73b6791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:20.639 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:20 smithi003 bash[92014]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T14:25:20.932 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:20 smithi003 podman[92031]: 2024-09-13 14:25:20.638205826 +0000 UTC m=+0.652438368 container remove 9994b1ee34d8e3ba211c04279e16b33b7e0e98bfae609e48654a2000f73b6791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:20.933 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:20 smithi003 podman[92031]: 2024-09-13 14:25:20.739454244 +0000 UTC m=+0.753686794 volume remove d04f150da32cf5bc66dbad61cddf02c4da282417af68e97f2813a5c1ce8dcad3 2024-09-13T14:25:21.181 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:25:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:25:20] "GET /metrics HTTP/1.1" 200 36241 "" "Prometheus/2.43.0" 2024-09-13T14:25:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:21 smithi019 ceph-mon[32895]: pgmap v21: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 750 op/s 2024-09-13T14:25:21.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:21 smithi003 ceph-mon[41592]: pgmap v21: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 750 op/s 2024-09-13T14:25:21.433 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:21 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T14:25:21.433 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:21 smithi003 systemd[1]: Stopped Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:21.433 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:21 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.485s CPU time. 2024-09-13T14:25:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:21 smithi003 ceph-mon[29646]: pgmap v21: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 750 op/s 2024-09-13T14:25:21.805 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:21 smithi003 systemd[1]: Starting Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:22.182 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:21 smithi003 podman[92153]: 2024-09-13 14:25:21.710667017 +0000 UTC m=+0.033629133 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-13T14:25:22.827 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:22 smithi003 podman[92153]: 2024-09-13 14:25:22.533273439 +0000 UTC m=+0.856235536 volume create 55a07eca77a17f394407a9f194e29610997633f60b685fd897382190c8136a63 2024-09-13T14:25:22.827 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:22 smithi003 podman[92153]: 2024-09-13 14:25:22.62676753 +0000 UTC m=+0.949729627 container create b74b601869dcd87580033730e1508b6ca64775fa6e77d47d379bf70a1208a369 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:23 smithi003 ceph-mon[29646]: pgmap v22: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T14:25:23.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:22 smithi003 podman[92153]: 2024-09-13 14:25:22.945991585 +0000 UTC m=+1.268953697 container init b74b601869dcd87580033730e1508b6ca64775fa6e77d47d379bf70a1208a369 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:23.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:22 smithi003 podman[92153]: 2024-09-13 14:25:22.952672769 +0000 UTC m=+1.275634864 container start b74b601869dcd87580033730e1508b6ca64775fa6e77d47d379bf70a1208a369 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:23.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:22 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:22.971Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-13T14:25:23.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:22 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:22.971Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-13T14:25:23.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:22 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:22.974Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.3 port=9094 2024-09-13T14:25:23.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:22 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:22.977Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:25:23.184 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:23 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:23.017Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:25:23.184 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:23 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:23.018Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:25:23.184 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:23 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:23.022Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:25:23.184 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:23 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:23.022Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:25:23.184 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:23 smithi003 bash[92153]: b74b601869dcd87580033730e1508b6ca64775fa6e77d47d379bf70a1208a369 2024-09-13T14:25:23.184 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:23 smithi003 systemd[1]: Started Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:23.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:23 smithi003 ceph-mon[41592]: pgmap v22: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T14:25:23.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:23 smithi019 ceph-mon[32895]: pgmap v22: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T14:25:24.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:24 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:24.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:24 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:24.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:24 smithi019 ceph-mon[32895]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-13T14:25:24.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:24 smithi019 ceph-mon[32895]: Deploying daemon node-exporter.a on smithi003 2024-09-13T14:25:24.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:24 smithi019 ceph-mon[32895]: pgmap v23: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-09-13T14:25:24.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:24.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:24.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[29646]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-13T14:25:24.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[29646]: Deploying daemon node-exporter.a on smithi003 2024-09-13T14:25:24.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[29646]: pgmap v23: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-09-13T14:25:24.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:24.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:24.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[41592]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-13T14:25:24.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[41592]: Deploying daemon node-exporter.a on smithi003 2024-09-13T14:25:24.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-mon[41592]: pgmap v23: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-09-13T14:25:24.434 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:23 smithi003 systemd[1]: Stopping Ceph node-exporter.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:24.730 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:24 smithi003 podman[92340]: 2024-09-13 14:25:24.435149597 +0000 UTC m=+0.170612526 container died a786edb7f0b5897fa412ad4fa9aa841e3e40b2df9900f65dacdf5e6b32239700 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:25.122 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:24 smithi003 podman[92340]: 2024-09-13 14:25:24.811761175 +0000 UTC m=+0.547224099 container cleanup a786edb7f0b5897fa412ad4fa9aa841e3e40b2df9900f65dacdf5e6b32239700 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:25.122 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:24 smithi003 bash[92340]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T14:25:25.123 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:24.977Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000727763s 2024-09-13T14:25:25.375 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:25 smithi003 podman[92357]: 2024-09-13 14:25:25.122011267 +0000 UTC m=+0.686930940 container remove a786edb7f0b5897fa412ad4fa9aa841e3e40b2df9900f65dacdf5e6b32239700 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:25.376 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:25 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:25:25.376 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:25 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T14:25:25.376 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:25 smithi003 systemd[1]: Stopped Ceph node-exporter.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:25.376 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:25 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 2.731s CPU time. 2024-09-13T14:25:25.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:25 smithi003 systemd[1]: Starting Ceph node-exporter.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:26.390 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:26 smithi003 bash[92480]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-13T14:25:26.390 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:26 smithi003 bash[92480]: Getting image source signatures 2024-09-13T14:25:26.682 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:26 smithi003 bash[92480]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T14:25:26.682 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:26 smithi003 bash[92480]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T14:25:26.682 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:26 smithi003 bash[92480]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T14:25:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:27 smithi019 ceph-mon[32895]: pgmap v24: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-13T14:25:27.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:27 smithi003 ceph-mon[41592]: pgmap v24: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-13T14:25:27.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:25:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:25:27.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:27 smithi003 ceph-mon[29646]: pgmap v24: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-13T14:25:28.432 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:28 smithi003 bash[92480]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T14:25:28.932 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:28 smithi003 bash[92480]: Writing manifest to image destination 2024-09-13T14:25:29.291 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-mon[41592]: pgmap v25: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-13T14:25:29.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-mon[29646]: pgmap v25: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-13T14:25:29.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:29 smithi019 ceph-mon[32895]: pgmap v25: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-13T14:25:29.598 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 podman[92480]: 2024-09-13 14:25:29.214143313 +0000 UTC m=+3.362909607 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T14:25:29.598 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 podman[92480]: 2024-09-13 14:25:29.328722144 +0000 UTC m=+3.477488437 container create 30d257fdd98b6b331315138d554f432b8af83eca585f5afb1ff8520faf859de3 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:29.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 podman[92480]: 2024-09-13 14:25:29.5982623 +0000 UTC m=+3.747028587 container init 30d257fdd98b6b331315138d554f432b8af83eca585f5afb1ff8520faf859de3 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:29.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 podman[92480]: 2024-09-13 14:25:29.606662047 +0000 UTC m=+3.755428339 container start 30d257fdd98b6b331315138d554f432b8af83eca585f5afb1ff8520faf859de3 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:29.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.610Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T14:25:29.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.610Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T14:25:29.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.610Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:25:29.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.610Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.611Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.611Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:25:29.934 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:25:29.935 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:25:29.935 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:25:29.935 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:25:29.935 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:25:29.935 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:25:29.936 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:25:29.937 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.612Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.613Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a[92548]: ts=2024-09-13T14:25:29.613Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 bash[92480]: 30d257fdd98b6b331315138d554f432b8af83eca585f5afb1ff8520faf859de3 2024-09-13T14:25:29.938 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 14:25:29 smithi003 systemd[1]: Started Ceph node-exporter.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:30 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:30 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:30 smithi019 ceph-mon[32895]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:25:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:30 smithi019 ceph-mon[32895]: pgmap v26: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-13T14:25:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:30 smithi019 ceph-mon[32895]: Deploying daemon prometheus.a on smithi019 2024-09-13T14:25:31.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:25:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:25:30] "GET /metrics HTTP/1.1" 200 36241 "" "Prometheus/2.43.0" 2024-09-13T14:25:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[41592]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:25:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[41592]: pgmap v26: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-13T14:25:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[41592]: Deploying daemon prometheus.a on smithi019 2024-09-13T14:25:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:31.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[29646]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:25:31.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[29646]: pgmap v26: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-13T14:25:31.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:30 smithi003 ceph-mon[29646]: Deploying daemon prometheus.a on smithi019 2024-09-13T14:25:32.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:31 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:32.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:31 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:32.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:31 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:33.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:32 smithi019 ceph-mon[32895]: pgmap v27: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-13T14:25:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:32 smithi003 ceph-mon[41592]: pgmap v27: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-13T14:25:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:32 smithi003 ceph-mon[29646]: pgmap v27: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-13T14:25:33.183 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 14:25:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T14:25:32.980Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003530776s 2024-09-13T14:25:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:35 smithi019 ceph-mon[32895]: pgmap v28: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-13T14:25:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:35 smithi003 ceph-mon[41592]: pgmap v28: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-13T14:25:35.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:35 smithi003 ceph-mon[29646]: pgmap v28: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-13T14:25:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:37 smithi019 ceph-mon[32895]: pgmap v29: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:25:37.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:37 smithi003 ceph-mon[41592]: pgmap v29: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:25:37.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:37 smithi003 ceph-mon[29646]: pgmap v29: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:25:37.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:25:37 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[29897]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:25:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:39 smithi019 ceph-mon[32895]: pgmap v30: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-13T14:25:39.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:39 smithi003 ceph-mon[41592]: pgmap v30: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-13T14:25:39.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:39 smithi003 ceph-mon[29646]: pgmap v30: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-13T14:25:41.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:41 smithi019 ceph-mon[32895]: pgmap v31: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-13T14:25:41.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:25:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:25:40] "GET /metrics HTTP/1.1" 200 36237 "" "Prometheus/2.43.0" 2024-09-13T14:25:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:41 smithi003 ceph-mon[41592]: pgmap v31: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-13T14:25:41.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:41 smithi003 ceph-mon[29646]: pgmap v31: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-13T14:25:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:43 smithi019 ceph-mon[32895]: pgmap v32: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:25:43.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:43 smithi003 ceph-mon[41592]: pgmap v32: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:25:43.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:43 smithi003 ceph-mon[29646]: pgmap v32: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:25:45.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:44 smithi019 systemd[1]: Stopping Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:45.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-mon[32895]: pgmap v33: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 669 op/s 2024-09-13T14:25:45.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:45 smithi003 ceph-mon[41592]: pgmap v33: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 669 op/s 2024-09-13T14:25:45.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:45 smithi003 ceph-mon[29646]: pgmap v33: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 669 op/s 2024-09-13T14:25:45.624 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.307Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:25:45.624 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.308Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:25:45.624 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.308Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:25:45.624 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.308Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:25:45.624 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.308Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:25:45.625 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.308Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:25:45.625 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.308Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:25:45.625 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.308Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:25:45.625 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.308Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:25:45.625 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.310Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:25:45.625 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.310Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:25:45.625 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[65333]: ts=2024-09-13T14:25:45.310Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:25:45.625 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 podman[76961]: 2024-09-13 14:25:45.319677523 +0000 UTC m=+0.154729860 container died f9fc99d6a298947e7da34345e801bcb2a80a86b830aa9505f10b496c458f3cc5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:45.931 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 podman[76961]: 2024-09-13 14:25:45.721253469 +0000 UTC m=+0.556305810 container cleanup f9fc99d6a298947e7da34345e801bcb2a80a86b830aa9505f10b496c458f3cc5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:45.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:45 smithi019 bash[76961]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:25:46.277 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:46 smithi019 podman[76977]: 2024-09-13 14:25:46.014713782 +0000 UTC m=+0.687895288 container remove f9fc99d6a298947e7da34345e801bcb2a80a86b830aa9505f10b496c458f3cc5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:46.639 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:46 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:25:46.639 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:46 smithi019 systemd[1]: Stopped Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:46.639 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:46 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 2.234s CPU time. 2024-09-13T14:25:46.931 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:46 smithi019 systemd[1]: Starting Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:47.246 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-mon[32895]: pgmap v34: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:25:47.246 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:47.247 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:46 smithi019 podman[77094]: 2024-09-13 14:25:46.892706853 +0000 UTC m=+0.039421413 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-13T14:25:47.247 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 podman[77094]: 2024-09-13 14:25:47.099415418 +0000 UTC m=+0.246129974 container create fb69e3c0bb9bab0a22cfb09a2af4f197aa698828e7c4380c79185bc91e52accf (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:47 smithi003 ceph-mon[41592]: pgmap v34: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:25:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:47 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:47 smithi003 ceph-mon[29646]: pgmap v34: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:25:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:47 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:25:47.501 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 podman[77094]: 2024-09-13 14:25:47.394396638 +0000 UTC m=+0.541111204 container init fb69e3c0bb9bab0a22cfb09a2af4f197aa698828e7c4380c79185bc91e52accf (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:47.501 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 podman[77094]: 2024-09-13 14:25:47.405950082 +0000 UTC m=+0.552664641 container start fb69e3c0bb9bab0a22cfb09a2af4f197aa698828e7c4380c79185bc91e52accf (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:25:47.501 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.444Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-13T14:25:47.501 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.444Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-13T14:25:47.501 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.444Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi019 (none))" 2024-09-13T14:25:47.501 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.444Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:25:47.501 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.444Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:25:47.501 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.447Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:25:47.501 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.448Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.451Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.451Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.456Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.456Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.198µs 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.456Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.457Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.494Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.494Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.494Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=49.534µs wal_replay_duration=37.275531ms wbl_replay_duration=192ns total_replay_duration=37.573224ms 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.499Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.499Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-13T14:25:47.502 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.499Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:25:47.931 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 bash[77094]: fb69e3c0bb9bab0a22cfb09a2af4f197aa698828e7c4380c79185bc91e52accf 2024-09-13T14:25:47.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 systemd[1]: Started Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:47.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.533Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=33.63886ms db_storage=1.561µs remote_storage=1.897µs web_handler=467ns query_engine=890ns scrape=603.543µs scrape_sd=221.862µs notify=24.842µs notify_sd=17.574µs rules=32.049459ms tracing=6.129µs 2024-09-13T14:25:47.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.533Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-13T14:25:47.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:47.533Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:25:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:48 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:48 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:48 smithi019 ceph-mon[32895]: Reconfiguring grafana.a (dependencies changed)... 2024-09-13T14:25:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:48 smithi019 ceph-mon[32895]: Reconfiguring daemon grafana.a on smithi019 2024-09-13T14:25:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:48 smithi019 ceph-mon[32895]: pgmap v35: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-13T14:25:48.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:48.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:48.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[41592]: Reconfiguring grafana.a (dependencies changed)... 2024-09-13T14:25:48.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[41592]: Reconfiguring daemon grafana.a on smithi019 2024-09-13T14:25:48.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[41592]: pgmap v35: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-13T14:25:48.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:48.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:48.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[29646]: Reconfiguring grafana.a (dependencies changed)... 2024-09-13T14:25:48.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[29646]: Reconfiguring daemon grafana.a on smithi019 2024-09-13T14:25:48.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:48 smithi003 ceph-mon[29646]: pgmap v35: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-13T14:25:51.307 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:51 smithi019 ceph-mon[32895]: pgmap v36: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-13T14:25:51.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:51 smithi003 ceph-mon[41592]: pgmap v36: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-13T14:25:51.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:51 smithi003 ceph-mon[29646]: pgmap v36: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-13T14:25:51.681 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:51 smithi019 systemd[1]: Stopping Ceph grafana.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:52.182 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:51 smithi019 podman[77365]: 2024-09-13 14:25:51.871428395 +0000 UTC m=+0.179119418 container died dc631f7cf889e8943d8dde5e015a29858b440bbed0b09215a3f0a632fecd8c60 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, name=ubi8, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, architecture=x86_64, vendor=Red Hat, Inc., io.buildah.version=1.32.0, vcs-type=git, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-13T14:25:53.023 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:52 smithi019 podman[77365]: 2024-09-13 14:25:52.803931938 +0000 UTC m=+1.111622962 container cleanup dc631f7cf889e8943d8dde5e015a29858b440bbed0b09215a3f0a632fecd8c60 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, distribution-scope=public, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, version=8.5, maintainer=Paul Cuzner ) 2024-09-13T14:25:53.023 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:52 smithi019 bash[77365]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a 2024-09-13T14:25:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:53 smithi019 ceph-mon[32895]: pgmap v37: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:25:53.432 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:53 smithi019 podman[77381]: 2024-09-13 14:25:53.055782174 +0000 UTC m=+1.180278256 container remove dc631f7cf889e8943d8dde5e015a29858b440bbed0b09215a3f0a632fecd8c60 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, com.redhat.component=ubi8-container, architecture=x86_64, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , io.buildah.version=1.32.0, name=ubi8, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5) 2024-09-13T14:25:53.432 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:53 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:25:53.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:53 smithi003 ceph-mon[29646]: pgmap v37: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:25:53.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:53 smithi003 ceph-mon[41592]: pgmap v37: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:25:53.854 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:53 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-13T14:25:53.854 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:53 smithi019 systemd[1]: Stopped Ceph grafana.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:53.854 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:53 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@grafana.a.service: Consumed 6.502s CPU time. 2024-09-13T14:25:53.854 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:53 smithi019 systemd[1]: Starting Ceph grafana.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:54.182 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:53 smithi019 podman[77464]: 2024-09-13 14:25:53.756561834 +0000 UTC m=+0.035729292 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-13T14:25:54.182 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:53 smithi019 podman[77464]: 2024-09-13 14:25:53.900086083 +0000 UTC m=+0.179253533 container create 1d1343f3e5db9bde828ff653b38ea57003ddf791409dab273613d1e4141f00a4 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, name=ubi8, architecture=x86_64, vcs-type=git, io.openshift.expose-services=, distribution-scope=public, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.buildah.version=1.32.0) 2024-09-13T14:25:54.573 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 podman[77464]: 2024-09-13 14:25:54.196594188 +0000 UTC m=+0.475761649 container init 1d1343f3e5db9bde828ff653b38ea57003ddf791409dab273613d1e4141f00a4 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, maintainer=Paul Cuzner , release=239.1651231664, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, description=Ceph Grafana Container, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5) 2024-09-13T14:25:54.573 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 podman[77464]: 2024-09-13 14:25:54.211346729 +0000 UTC m=+0.490514181 container start 1d1343f3e5db9bde828ff653b38ea57003ddf791409dab273613d1e4141f00a4 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, io.openshift.tags=base rhel8, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, vendor=Red Hat, Inc., io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, version=8.5, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, com.redhat.component=ubi8-container, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, maintainer=Paul Cuzner ) 2024-09-13T14:25:54.573 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 bash[77464]: 1d1343f3e5db9bde828ff653b38ea57003ddf791409dab273613d1e4141f00a4 2024-09-13T14:25:54.573 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 systemd[1]: Started Ceph grafana.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:54.934 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.572492908Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-13T14:25:54.934 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.573717597Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-13T14:25:54.934 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.573954676Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.574076072Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.574190415Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.574306389Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.574418907Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.574528056Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.574636939Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.574745099Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.574851817Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.574966958Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.575075122Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=settings t=2024-09-13T14:25:54.575189603Z level=info msg="App mode production" 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=sqlstore t=2024-09-13T14:25:54.575387501Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=sqlstore t=2024-09-13T14:25:54.575548517Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T14:25:54.935 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=migrator t=2024-09-13T14:25:54.601153653Z level=info msg="Starting DB migrations" 2024-09-13T14:25:54.936 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=migrator t=2024-09-13T14:25:54.610057622Z level=info msg="migrations completed" performed=0 skipped=481 duration=717.593µs 2024-09-13T14:25:54.936 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=sqlstore t=2024-09-13T14:25:54.611077662Z level=info msg="Created default organization" 2024-09-13T14:25:54.936 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=plugin.loader t=2024-09-13T14:25:54.719393617Z level=info msg="Plugin registered" pluginID=input 2024-09-13T14:25:54.936 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=plugin.loader t=2024-09-13T14:25:54.779270406Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-13T14:25:54.936 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=plugin.loader t=2024-09-13T14:25:54.779401302Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-13T14:25:54.936 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=secrets t=2024-09-13T14:25:54.779528853Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T14:25:54.936 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=query_data t=2024-09-13T14:25:54.782199633Z level=info msg="Query Service initialization" 2024-09-13T14:25:54.936 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=live.push_http t=2024-09-13T14:25:54.786106195Z level=info msg="Live Push Gateway initialization" 2024-09-13T14:25:55.245 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:55 smithi019 ceph-mon[32895]: pgmap v38: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-09-13T14:25:55.245 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:55 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:55.245 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:55 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:55 smithi003 ceph-mon[29646]: pgmap v38: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-09-13T14:25:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:55 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:55 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:55 smithi003 ceph-mon[41592]: pgmap v38: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-09-13T14:25:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:55 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:55 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:25:55.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:55 smithi019 systemd[1]: Stopping Ceph node-exporter.b for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:56.042 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:55 smithi019 podman[77642]: 2024-09-13 14:25:55.778002457 +0000 UTC m=+0.196733371 container died e22916e85bdc446b204f9cc8828482adca626c8d757b245a70a3ab7a571db547 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:25:56.296 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-mon[32895]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-13T14:25:56.296 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-mon[32895]: Deploying daemon node-exporter.b on smithi019 2024-09-13T14:25:56.296 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=infra.usagestats.collector t=2024-09-13T14:25:56.194626266Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T14:25:56.296 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=provisioning.datasources t=2024-09-13T14:25:56.284884612Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-09-13T14:25:56.296 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=provisioning.datasources t=2024-09-13T14:25:56.285170622Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-09-13T14:25:56.296 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=provisioning.datasources t=2024-09-13T14:25:56.295776197Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T14:25:56.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:56 smithi003 ceph-mon[41592]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-13T14:25:56.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:56 smithi003 ceph-mon[41592]: Deploying daemon node-exporter.b on smithi019 2024-09-13T14:25:56.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:56 smithi003 ceph-mon[29646]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-13T14:25:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:56 smithi003 ceph-mon[29646]: Deploying daemon node-exporter.b on smithi019 2024-09-13T14:25:56.622 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:56 smithi019 podman[77642]: 2024-09-13 14:25:56.411476174 +0000 UTC m=+0.830207080 container cleanup e22916e85bdc446b204f9cc8828482adca626c8d757b245a70a3ab7a571db547 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:25:56.623 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:56 smithi019 bash[77642]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T14:25:56.623 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=provisioning.alerting t=2024-09-13T14:25:56.403210905Z level=info msg="starting to provision alerting" 2024-09-13T14:25:56.623 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=provisioning.alerting t=2024-09-13T14:25:56.403322894Z level=info msg="finished to provision alerting" 2024-09-13T14:25:56.623 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=grafanaStorageLogger t=2024-09-13T14:25:56.404681593Z level=info msg="storage starting" 2024-09-13T14:25:56.624 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=ngalert.state.manager t=2024-09-13T14:25:56.4051393Z level=info msg="Warming state cache for startup" 2024-09-13T14:25:56.624 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=http.server t=2024-09-13T14:25:56.40760651Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T14:25:56.624 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=ngalert.state.manager t=2024-09-13T14:25:56.417542051Z level=info msg="State cache has been initialized" states=0 duration=12.400813ms 2024-09-13T14:25:56.624 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=ticker t=2024-09-13T14:25:56.417854668Z level=info msg=starting first_tick=2024-09-13T14:26:00Z 2024-09-13T14:25:56.624 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=ngalert.multiorg.alertmanager t=2024-09-13T14:25:56.417965385Z level=info msg="starting MultiOrg Alertmanager" 2024-09-13T14:25:56.931 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:56 smithi019 podman[77658]: 2024-09-13 14:25:56.621773469 +0000 UTC m=+0.844636568 container remove e22916e85bdc446b204f9cc8828482adca626c8d757b245a70a3ab7a571db547 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:25:56.932 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:56 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:25:57.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:25:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:25:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:57 smithi019 ceph-mon[32895]: pgmap v39: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-13T14:25:57.432 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:57 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T14:25:57.432 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:57 smithi019 systemd[1]: Stopped Ceph node-exporter.b for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:57.432 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:57 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 2.845s CPU time. 2024-09-13T14:25:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:57 smithi003 ceph-mon[29646]: pgmap v39: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-13T14:25:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:57 smithi003 ceph-mon[41592]: pgmap v39: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-13T14:25:57.879 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:57 smithi019 systemd[1]: Starting Ceph node-exporter.b for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:58.181 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:57 smithi019 bash[77775]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-13T14:25:58.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:58 smithi019 bash[77775]: Getting image source signatures 2024-09-13T14:25:58.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:58 smithi019 bash[77775]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T14:25:58.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:58 smithi019 bash[77775]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T14:25:58.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:25:58 smithi019 bash[77775]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T14:25:58.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade resume' 2024-09-13T14:25:59.387 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:25:59 smithi019 ceph-mon[32895]: pgmap v40: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-13T14:25:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:25:59 smithi003 ceph-mon[29646]: pgmap v40: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-13T14:25:59.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:25:59 smithi003 ceph-mon[41592]: pgmap v40: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-13T14:26:00.681 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:00 smithi019 bash[77775]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T14:26:00.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:00 smithi019 bash[77775]: Writing manifest to image destination 2024-09-13T14:26:01.047 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:00 smithi019 podman[77775]: 2024-09-13 14:26:00.658731328 +0000 UTC m=+2.991800855 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T14:26:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:01 smithi003 ceph-mon[41592]: pgmap v41: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-13T14:26:01.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:01 smithi003 ceph-mon[29646]: pgmap v41: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-13T14:26:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-mon[32895]: pgmap v41: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-13T14:26:01.432 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:26:01] "GET /metrics HTTP/1.1" 200 36242 "" "Prometheus/2.51.0" 2024-09-13T14:26:01.432 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 podman[77775]: 2024-09-13 14:26:01.297036424 +0000 UTC m=+3.630105937 container create 738f7b07d0d84a6f3f13ce1d070ad599b7aef1ab956ffea3cf6abfb5a4f47c31 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:26:01.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 podman[77775]: 2024-09-13 14:26:01.608160731 +0000 UTC m=+3.941230240 container init 738f7b07d0d84a6f3f13ce1d070ad599b7aef1ab956ffea3cf6abfb5a4f47c31 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:26:01.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 podman[77775]: 2024-09-13 14:26:01.613462618 +0000 UTC m=+3.946532130 container start 738f7b07d0d84a6f3f13ce1d070ad599b7aef1ab956ffea3cf6abfb5a4f47c31 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:26:01.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.616Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T14:26:01.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.616Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T14:26:01.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.616Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:26:01.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:26:01.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:26:01.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:26:01.934 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:26:01.935 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:26:01.936 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.617Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:26:01.937 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:26:01.938 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:26:01.938 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:26:01.938 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:26:01.938 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:26:01.938 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:26:01.938 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:26:01.938 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b[77844]: ts=2024-09-13T14:26:01.618Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:26:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:02 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:02.432 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:02 smithi019 bash[77775]: 738f7b07d0d84a6f3f13ce1d070ad599b7aef1ab956ffea3cf6abfb5a4f47c31 2024-09-13T14:26:02.432 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 14:26:02 smithi019 systemd[1]: Started Ceph node-exporter.b for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:26:02.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:02 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:02 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:03.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[41592]: pgmap v42: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:26:03.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:03.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:03.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:26:03.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:03.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:26:03.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[29646]: pgmap v42: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:26:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:26:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:26:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:03 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:03 smithi019 ceph-mon[32895]: pgmap v42: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:26:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:03 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:03 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:03 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:26:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:03 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:03 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:26:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:03 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:03.663 INFO:teuthology.orchestra.run.smithi003.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:04.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[41592]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:26:04.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[41592]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:04.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[41592]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:26:04.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:04.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:04.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:04.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[29646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:26:04.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[29646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:04.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[29646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:26:04.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:04.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:04.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:04 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:04.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:04 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:04 smithi019 ceph-mon[32895]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:26:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:04 smithi019 ceph-mon[32895]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:04 smithi019 ceph-mon[32895]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:26:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:04 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:04 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:04 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:05.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:05 smithi019 ceph-mon[32895]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:05.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:05 smithi019 ceph-mon[32895]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:05.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:05 smithi019 ceph-mon[32895]: pgmap v43: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 694 op/s 2024-09-13T14:26:05.402 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:05 smithi003 ceph-mon[41592]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:05.402 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:05 smithi003 ceph-mon[41592]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:05.402 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:05 smithi003 ceph-mon[41592]: pgmap v43: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 694 op/s 2024-09-13T14:26:05.402 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:05 smithi003 ceph-mon[29646]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:05.402 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:05 smithi003 ceph-mon[29646]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:05.402 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:05 smithi003 ceph-mon[29646]: pgmap v43: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 694 op/s 2024-09-13T14:26:05.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for majority of mons to upgrade' 2024-09-13T14:26:05.956 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:06.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:06.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:07.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:07 smithi003 ceph-mon[41592]: pgmap v44: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:26:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:07 smithi003 ceph-mon[29646]: pgmap v44: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:26:07.318 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:07 smithi019 ceph-mon[32895]: pgmap v44: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:26:07.319 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:06 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:06.950Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:08.536 INFO:teuthology.orchestra.run.smithi003.stdout:wait for majority of mons to upgrade 2024-09-13T14:26:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:09 smithi003 ceph-mon[41592]: pgmap v45: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 679 op/s 2024-09-13T14:26:09.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:09 smithi003 ceph-mon[29646]: pgmap v45: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 679 op/s 2024-09-13T14:26:09.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:09 smithi019 ceph-mon[32895]: pgmap v45: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 679 op/s 2024-09-13T14:26:09.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ! ceph mon versions | grep $sha1 | egrep '"'"': [23]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-13T14:26:10.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:10 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:10.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:10 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:10.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:10 smithi019 ceph-mon[32895]: pgmap v46: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 679 op/s 2024-09-13T14:26:10.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:10 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:10.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:10 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:10.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:10 smithi003 ceph-mon[29646]: pgmap v46: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 679 op/s 2024-09-13T14:26:10.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:10 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:10.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:10 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:10.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:10 smithi003 ceph-mon[41592]: pgmap v46: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 679 op/s 2024-09-13T14:26:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:11.681 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:26:11] "GET /metrics HTTP/1.1" 200 36242 "" "Prometheus/2.51.0" 2024-09-13T14:26:12.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:12.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:12.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:12.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:12.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:12.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:12.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:12.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:12.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:12.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:12.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:12.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:11 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:12.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:11 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:11 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:11 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:11 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:11 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:11 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:13.397 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:12 smithi003 ceph-mon[41592]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:26:13.397 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:12 smithi003 ceph-mon[41592]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:26:13.397 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:12 smithi003 ceph-mon[41592]: pgmap v47: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-09-13T14:26:13.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:12 smithi003 ceph-mon[29646]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:26:13.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:12 smithi003 ceph-mon[29646]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:26:13.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:12 smithi003 ceph-mon[29646]: pgmap v47: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-09-13T14:26:13.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:12 smithi019 ceph-mon[32895]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:26:13.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:12 smithi019 ceph-mon[32895]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:26:13.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:12 smithi019 ceph-mon[32895]: pgmap v47: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-09-13T14:26:14.336 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:14.337 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:13 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1480609960' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:14.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:13 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1480609960' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:13 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1480609960' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:14.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:14.151Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:15.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[29646]: pgmap v48: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 657 op/s 2024-09-13T14:26:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:26:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:15.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[41592]: pgmap v48: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 657 op/s 2024-09-13T14:26:15.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:15.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:15.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:15.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:26:15.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:14 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:15.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:14 smithi019 ceph-mon[32895]: pgmap v48: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 657 op/s 2024-09-13T14:26:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:14 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:14 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:14 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:14 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:26:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:14 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:16.005 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:16.005 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:15.742Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:16.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:16 smithi003 ceph-mon[29646]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:16.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:16 smithi003 ceph-mon[29646]: Upgrade: Updating mgr.y 2024-09-13T14:26:16.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:16 smithi003 ceph-mon[29646]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:16.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:16 smithi003 ceph-mon[29646]: Deploying daemon mgr.y on smithi003 2024-09-13T14:26:16.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:16 smithi003 ceph-mon[41592]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:16.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:16 smithi003 ceph-mon[41592]: Upgrade: Updating mgr.y 2024-09-13T14:26:16.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:16 smithi003 ceph-mon[41592]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:16.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:16 smithi003 ceph-mon[41592]: Deploying daemon mgr.y on smithi003 2024-09-13T14:26:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:16 smithi019 ceph-mon[32895]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:16 smithi019 ceph-mon[32895]: Upgrade: Updating mgr.y 2024-09-13T14:26:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:16 smithi019 ceph-mon[32895]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:16 smithi019 ceph-mon[32895]: Deploying daemon mgr.y on smithi003 2024-09-13T14:26:17.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:17 smithi019 ceph-mon[32895]: pgmap v49: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-13T14:26:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:17 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:17.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:16.950Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:17.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:17 smithi003 ceph-mon[41592]: pgmap v49: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-13T14:26:17.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:17 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:17 smithi003 ceph-mon[29646]: pgmap v49: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-13T14:26:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:17 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:17.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:17 smithi003 systemd[1]: Stopping Ceph mgr.y for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:26:17.923 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:17 smithi003 podman[94186]: 2024-09-13 14:26:17.59443056 +0000 UTC m=+0.195918198 container died fe65bae0d85f36f80352cd9a95b3f221f37cdd513ce72f92f82acc3dfb3fd7ec (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-13T14:26:18.068 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:18.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:18 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1651203161' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:18.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:18 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1651203161' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:18.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:18 smithi003 podman[94186]: 2024-09-13 14:26:18.038456969 +0000 UTC m=+0.639944601 container cleanup fe65bae0d85f36f80352cd9a95b3f221f37cdd513ce72f92f82acc3dfb3fd7ec (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-13T14:26:18.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:18 smithi003 bash[94186]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y 2024-09-13T14:26:18.410 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:18 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1651203161' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:18.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:18 smithi003 podman[94208]: 2024-09-13 14:26:18.257088185 +0000 UTC m=+0.657983534 container remove fe65bae0d85f36f80352cd9a95b3f221f37cdd513ce72f92f82acc3dfb3fd7ec (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-13T14:26:18.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:18 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:26:19.036 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:18 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-13T14:26:19.036 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:18 smithi003 systemd[1]: Stopped Ceph mgr.y for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:26:19.036 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:18 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y.service: Consumed 48.807s CPU time. 2024-09-13T14:26:19.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:19 smithi003 ceph-mon[41592]: pgmap v50: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 669 op/s 2024-09-13T14:26:19.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:19 smithi003 ceph-mon[41592]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:19.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:19 smithi003 ceph-mon[29646]: pgmap v50: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 669 op/s 2024-09-13T14:26:19.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:19 smithi003 ceph-mon[29646]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:19.370 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:19 smithi003 systemd[1]: Starting Ceph mgr.y for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:26:19.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:19 smithi019 ceph-mon[32895]: pgmap v50: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 669 op/s 2024-09-13T14:26:19.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:19 smithi019 ceph-mon[32895]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:19.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:19 smithi003 podman[94376]: 2024-09-13 14:26:19.279381653 +0000 UTC m=+0.042664553 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:19.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:19 smithi003 podman[94376]: 2024-09-13 14:26:19.408440825 +0000 UTC m=+0.171723709 container create e8e7986695ad32198064f64e4dc22a78705e99a8631ebb404845d6b5a373358d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-13T14:26:20.052 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:19 smithi003 podman[94376]: 2024-09-13 14:26:19.703470854 +0000 UTC m=+0.466753749 container init e8e7986695ad32198064f64e4dc22a78705e99a8631ebb404845d6b5a373358d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-13T14:26:20.052 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:19 smithi003 podman[94376]: 2024-09-13 14:26:19.713301669 +0000 UTC m=+0.476584561 container start e8e7986695ad32198064f64e4dc22a78705e99a8631ebb404845d6b5a373358d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-13T14:26:20.052 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:19 smithi003 ceph-mgr[94411]: -- 172.21.15.3:0/2067553811 <== mon.1 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5f1d830e0 con 0x55a5f1d61000 2024-09-13T14:26:20.052 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:19 smithi003 bash[94376]: e8e7986695ad32198064f64e4dc22a78705e99a8631ebb404845d6b5a373358d 2024-09-13T14:26:20.052 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:19 smithi003 systemd[1]: Started Ceph mgr.y for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:26:20.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:20 smithi019 ceph-mon[32895]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:20.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:20 smithi019 ceph-mon[32895]: mgrmap e26: x(active, since 95s), standbys: y 2024-09-13T14:26:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:20 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:20 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:20 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:20.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:20.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:20.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[41592]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:20.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[41592]: mgrmap e26: x(active, since 95s), standbys: y 2024-09-13T14:26:20.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:20.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:20.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:20.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[29646]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:20.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[29646]: mgrmap e26: x(active, since 95s), standbys: y 2024-09-13T14:26:20.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:20.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:20.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:20 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:21.364 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:21 smithi019 ceph-mon[32895]: pgmap v51: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 669 op/s 2024-09-13T14:26:21.364 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:21 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:21.365 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:21 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:21.365 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:21 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:26:21] "GET /metrics HTTP/1.1" 200 36241 "" "Prometheus/2.51.0" 2024-09-13T14:26:21.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:21 smithi003 ceph-mon[41592]: pgmap v51: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 669 op/s 2024-09-13T14:26:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:21 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:21 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:21 smithi003 ceph-mon[29646]: pgmap v51: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 669 op/s 2024-09-13T14:26:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:21 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:21 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:21.914 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:22.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:22 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3808681790' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:22.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:22 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3808681790' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:22.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:22 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3808681790' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:23.358 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:23 smithi003 ceph-mon[41592]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:23.358 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:23 smithi003 ceph-mon[41592]: pgmap v52: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.02k op/s 2024-09-13T14:26:23.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:23 smithi003 ceph-mon[29646]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:23.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:23 smithi003 ceph-mon[29646]: pgmap v52: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.02k op/s 2024-09-13T14:26:23.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:23 smithi019 ceph-mon[32895]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:23 smithi019 ceph-mon[32895]: pgmap v52: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.02k op/s 2024-09-13T14:26:24.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:24 smithi019 ceph-mon[32895]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:24.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:24.149Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:24.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:24 smithi003 ceph-mon[29646]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:24.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:24 smithi003 ceph-mon[41592]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:25.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:25 smithi003 ceph-mon[29646]: pgmap v53: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 703 op/s 2024-09-13T14:26:25.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:25 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:25.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:25 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:25.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:25 smithi003 ceph-mon[41592]: pgmap v53: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 703 op/s 2024-09-13T14:26:25.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:25 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:25.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:25 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:25.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:25 smithi019 ceph-mon[32895]: pgmap v53: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 703 op/s 2024-09-13T14:26:25.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:25 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:25.435 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:25 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:25.716 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:25.931 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:25.742Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:25.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:25.743Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:26.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:26 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2493836872' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:26.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:26 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:26.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:26 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:26.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:26 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2493836872' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:26.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:26 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:26.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:26 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:26 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2493836872' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:26 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:26 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:27.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:26.950Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:27 smithi019 ceph-mon[32895]: from='client.25163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:27 smithi019 ceph-mon[32895]: pgmap v54: 89 pgs: 89 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.01k op/s 2024-09-13T14:26:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:27 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:27 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:27.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:27 smithi003 ceph-mon[41592]: from='client.25163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:27.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:27 smithi003 ceph-mon[41592]: pgmap v54: 89 pgs: 89 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.01k op/s 2024-09-13T14:26:27.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:27 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:27.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:27 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:27.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:27 smithi003 ceph-mon[29646]: from='client.25163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:27.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:27 smithi003 ceph-mon[29646]: pgmap v54: 89 pgs: 89 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.01k op/s 2024-09-13T14:26:27.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:27 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:27.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:27 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:28.397 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-mon[41592]: from='client.25009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:28.397 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-mon[41592]: pgmap v55: 89 pgs: 89 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 657 op/s 2024-09-13T14:26:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-mon[29646]: from='client.25009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-mon[29646]: pgmap v55: 89 pgs: 89 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 657 op/s 2024-09-13T14:26:28.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:28 smithi019 ceph-mon[32895]: from='client.25009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:28.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:28 smithi019 ceph-mon[32895]: pgmap v55: 89 pgs: 89 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 657 op/s 2024-09-13T14:26:28.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:26:28] ENGINE Bus STARTING 2024-09-13T14:26:28.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: CherryPy Checker: 2024-09-13T14:26:28.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: The Application mounted at '' has an empty config. 2024-09-13T14:26:28.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:28.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:26:28] ENGINE Serving on http://:::9283 2024-09-13T14:26:28.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:26:28] ENGINE Bus STARTED 2024-09-13T14:26:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:29 smithi019 ceph-mon[32895]: Standby manager daemon y restarted 2024-09-13T14:26:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:29 smithi019 ceph-mon[32895]: Standby manager daemon y started 2024-09-13T14:26:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:29 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:26:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:29 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:26:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:29 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:26:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:29 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:26:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:29 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3783278211' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[29646]: Standby manager daemon y restarted 2024-09-13T14:26:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[29646]: Standby manager daemon y started 2024-09-13T14:26:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:26:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:26:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:26:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:26:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3783278211' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:29.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[41592]: Standby manager daemon y restarted 2024-09-13T14:26:29.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[41592]: Standby manager daemon y started 2024-09-13T14:26:29.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:26:29.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:26:29.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:26:29.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:26:29.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:29 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3783278211' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:29.525 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:30.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:30.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:30 smithi019 ceph-mon[32895]: mgrmap e27: x(active, since 105s), standbys: y 2024-09-13T14:26:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:30 smithi019 ceph-mon[32895]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:30 smithi019 ceph-mon[32895]: pgmap v56: 89 pgs: 89 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 657 op/s 2024-09-13T14:26:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:30 smithi019 ceph-mon[32895]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:30.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:30 smithi003 ceph-mon[41592]: mgrmap e27: x(active, since 105s), standbys: y 2024-09-13T14:26:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:30 smithi003 ceph-mon[41592]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:30 smithi003 ceph-mon[41592]: pgmap v56: 89 pgs: 89 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 657 op/s 2024-09-13T14:26:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:30 smithi003 ceph-mon[41592]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:30 smithi003 ceph-mon[29646]: mgrmap e27: x(active, since 105s), standbys: y 2024-09-13T14:26:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:30 smithi003 ceph-mon[29646]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:30 smithi003 ceph-mon[29646]: pgmap v56: 89 pgs: 89 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 657 op/s 2024-09-13T14:26:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:30 smithi003 ceph-mon[29646]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:31.661 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:26:31] "GET /metrics HTTP/1.1" 200 36241 "" "Prometheus/2.51.0" 2024-09-13T14:26:31.931 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ignoring --setuser ceph since I am not root 2024-09-13T14:26:31.932 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ignoring --setgroup ceph since I am not root 2024-09-13T14:26:31.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:26:31] ENGINE Bus STOPPING 2024-09-13T14:26:31.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:26:31] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:26:31.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:26:31] ENGINE Bus STOPPED 2024-09-13T14:26:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: Activating manager daemon y 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24877 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: mgrmap e28: y(active, starting, since 0.0166436s) 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"}]: dispatch 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"}]: dispatch 2024-09-13T14:26:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:26:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:32 smithi019 ceph-mon[32895]: Manager daemon y is now available 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: Activating manager daemon y 2024-09-13T14:26:32.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24877 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: mgrmap e28: y(active, starting, since 0.0166436s) 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:26:32.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[41592]: Manager daemon y is now available 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:32.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24877 172.21.15.19:0/3342794375' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: Activating manager daemon y 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24877 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: mgrmap e28: y(active, starting, since 0.0166436s) 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"}]: dispatch 2024-09-13T14:26:32.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:26:32.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-mon[29646]: Manager daemon y is now available 2024-09-13T14:26:33.077 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:26:32] ENGINE Bus STARTING 2024-09-13T14:26:33.251 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:26:33] ENGINE Serving on http://:::9283 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:26:33] ENGINE Bus STARTED 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.162+0000 7f14baf81640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.162+0000 7f14baf81640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.162+0000 7f14baf81640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.162+0000 7f14baf81640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.162+0000 7f14baf81640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.164+0000 7f14c180e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.164+0000 7f14c180e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.164+0000 7f14c180e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.434 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.164+0000 7f14c180e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.434 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:33.164+0000 7f14c180e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:26:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: mgrmap e29: y(active, since 1.02102s) 2024-09-13T14:26:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: pgmap v3: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3989575142' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:26:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:26:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:33.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:33.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:33.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:33 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: mgrmap e29: y(active, since 1.02102s) 2024-09-13T14:26:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: pgmap v3: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3989575142' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: mgrmap e29: y(active, since 1.02102s) 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: pgmap v3: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3989575142' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:33.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:26:33.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:26:33.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:33.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:33.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:33.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:33.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:33 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:34.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:34.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:34.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:34 smithi003 ceph-mon[41592]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:34.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:34 smithi003 ceph-mon[41592]: pgmap v4: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:34.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:34 smithi003 ceph-mon[29646]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:34.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:34 smithi003 ceph-mon[29646]: pgmap v4: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:34.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:34 smithi019 ceph-mon[32895]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:34.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:34 smithi019 ceph-mon[32895]: pgmap v4: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:35.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-mon[29646]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:35.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-mon[41592]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:35.920 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:35.437+0000 7f14a4294640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:26:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: Traceback (most recent call last): 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: self._connections.run(self.expiration_interval) 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: self._run(expiration_interval) 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: s = self.context.wrap_socket( 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:26:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: return self.sslsocket_class._create( 2024-09-13T14:26:35.922 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:26:35.922 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: self.do_handshake() 2024-09-13T14:26:35.922 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:26:35.922 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: self._sslobj.do_handshake() 2024-09-13T14:26:35.922 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:26:35.922 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:26:35 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:26:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:35 smithi019 ceph-mon[32895]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:35.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:35.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[41592]: [13/Sep/2024:14:26:35] ENGINE Bus STARTING 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[41592]: [13/Sep/2024:14:26:35] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[41592]: [13/Sep/2024:14:26:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:26:36.933 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: self.do_handshake() 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[41592]: [13/Sep/2024:14:26:35] ENGINE Serving on http://172.21.15.3:8765 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[41592]: [13/Sep/2024:14:26:35] ENGINE Bus STARTED 2024-09-13T14:26:36.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[41592]: pgmap v5: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:36.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[41592]: mgrmap e30: y(active, since 4s) 2024-09-13T14:26:36.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/839613803' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:36.935 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:36 smithi019 ceph-mon[32895]: [13/Sep/2024:14:26:35] ENGINE Bus STARTING 2024-09-13T14:26:36.935 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:36 smithi019 ceph-mon[32895]: [13/Sep/2024:14:26:35] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:26:36.935 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:36 smithi019 ceph-mon[32895]: [13/Sep/2024:14:26:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:26:36.935 INFO:journalctl@ceph.mon.b.smithi019.stdout: Traceback (most recent call last): 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._run(expiration_interval) 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: s = self.context.wrap_socket( 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: return self.sslsocket_class._create( 2024-09-13T14:26:36.936 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout: self.do_handshake() 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._sslobj.do_handshake() 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:36 smithi019 ceph-mon[32895]: [13/Sep/2024:14:26:35] ENGINE Serving on http://172.21.15.3:8765 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:36 smithi019 ceph-mon[32895]: [13/Sep/2024:14:26:35] ENGINE Bus STARTED 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:36 smithi019 ceph-mon[32895]: pgmap v5: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:36 smithi019 ceph-mon[32895]: mgrmap e30: y(active, since 4s) 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:36 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/839613803' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[29646]: [13/Sep/2024:14:26:35] ENGINE Bus STARTING 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[29646]: [13/Sep/2024:14:26:35] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:26:36.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[29646]: [13/Sep/2024:14:26:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:26:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:26:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-09-13T14:26:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:26:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:26:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:26:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[29646]: [13/Sep/2024:14:26:35] ENGINE Serving on http://172.21.15.3:8765 2024-09-13T14:26:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[29646]: [13/Sep/2024:14:26:35] ENGINE Bus STARTED 2024-09-13T14:26:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[29646]: pgmap v5: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[29646]: mgrmap e30: y(active, since 4s) 2024-09-13T14:26:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:36 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/839613803' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:36.994 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:37.235 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:38.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:38 smithi019 ceph-mon[32895]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:38 smithi019 ceph-mon[32895]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:38 smithi019 ceph-mon[32895]: pgmap v6: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:38 smithi019 ceph-mon[32895]: mgrmap e31: y(active, since 6s) 2024-09-13T14:26:38.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:38 smithi003 ceph-mon[41592]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:38.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:38 smithi003 ceph-mon[41592]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:38.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:38 smithi003 ceph-mon[41592]: pgmap v6: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:38.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:38 smithi003 ceph-mon[41592]: mgrmap e31: y(active, since 6s) 2024-09-13T14:26:38.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:38 smithi003 ceph-mon[29646]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:38.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:38 smithi003 ceph-mon[29646]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:38.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:38 smithi003 ceph-mon[29646]: pgmap v6: 89 pgs: 89 active+clean; 1.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:26:38.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:38 smithi003 ceph-mon[29646]: mgrmap e31: y(active, since 6s) 2024-09-13T14:26:40.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:39 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:40.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:39 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:40.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:39 smithi019 ceph-mon[32895]: pgmap v7: 89 pgs: 89 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 25 KiB/s rd, 440 B/s wr, 480 op/s 2024-09-13T14:26:40.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:39 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:40.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:39 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:40.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:39 smithi003 ceph-mon[41592]: pgmap v7: 89 pgs: 89 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 25 KiB/s rd, 440 B/s wr, 480 op/s 2024-09-13T14:26:40.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:39 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:40.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:39 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:40.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:39 smithi003 ceph-mon[29646]: pgmap v7: 89 pgs: 89 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 25 KiB/s rd, 440 B/s wr, 480 op/s 2024-09-13T14:26:40.474 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:40.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:40.474 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:26:40] ENGINE Bus STARTING 2024-09-13T14:26:40.474 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: CherryPy Checker: 2024-09-13T14:26:40.474 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: The Application mounted at '' has an empty config. 2024-09-13T14:26:40.474 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: 2024-09-13T14:26:40.474 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:26:40] ENGINE Serving on http://:::9283 2024-09-13T14:26:40.475 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: [13/Sep/2024:14:26:40] ENGINE Bus STARTED 2024-09-13T14:26:40.607 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:41.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/4173373568' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: Standby manager daemon x started 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:26:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:40 smithi019 ceph-mon[32895]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/4173373568' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: Standby manager daemon x started 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:26:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[29646]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/4173373568' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: Standby manager daemon x started 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/2774913134' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:26:41.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:40 smithi003 ceph-mon[41592]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:41.681 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:41 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:26:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.51.0" 2024-09-13T14:26:42.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:42.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:42.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:26:42.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[41592]: mgrmap e32: y(active, since 9s), standbys: x 2024-09-13T14:26:42.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[41592]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:42.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[41592]: pgmap v8: 89 pgs: 89 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 342 B/s wr, 373 op/s 2024-09-13T14:26:42.196 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:42.196 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:42.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:26:42.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[29646]: mgrmap e32: y(active, since 9s), standbys: x 2024-09-13T14:26:42.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[29646]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:42.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:41 smithi003 ceph-mon[29646]: pgmap v8: 89 pgs: 89 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 342 B/s wr, 373 op/s 2024-09-13T14:26:42.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:41 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:41 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:41 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:26:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:41 smithi019 ceph-mon[32895]: mgrmap e32: y(active, since 9s), standbys: x 2024-09-13T14:26:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:41 smithi019 ceph-mon[32895]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:41 smithi019 ceph-mon[32895]: pgmap v8: 89 pgs: 89 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 342 B/s wr, 373 op/s 2024-09-13T14:26:44.316 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:44.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:44 smithi019 ceph-mon[32895]: pgmap v9: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 280 B/s wr, 654 op/s 2024-09-13T14:26:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:44 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3640833676' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:44.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:44 smithi003 ceph-mon[29646]: pgmap v9: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 280 B/s wr, 654 op/s 2024-09-13T14:26:44.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:44 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3640833676' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:44.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:44 smithi003 ceph-mon[41592]: pgmap v9: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 280 B/s wr, 654 op/s 2024-09-13T14:26:44.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:44 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3640833676' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:45.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[41592]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:45.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:45.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:45.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:45.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:45.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:45 smithi019 ceph-mon[32895]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:45.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:45 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:45.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:45 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:45.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:45 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:45.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:45 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:45.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[29646]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:45.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:45.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:45.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:45.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:45 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:46.950 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: from='client.25280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:46.950 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:26:46.950 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:26:46.950 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:26:46.950 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:26:46.950 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: pgmap v10: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 597 op/s 2024-09-13T14:26:46.950 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:26:46.950 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:46.951 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:46.951 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:46.951 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:46.951 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: from='client.25280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:26:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:26:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:26:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:26:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: pgmap v10: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 597 op/s 2024-09-13T14:26:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:26:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: from='client.25280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:47.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:26:47.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:26:47.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:26:47.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:26:47.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: pgmap v10: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 597 op/s 2024-09-13T14:26:47.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:26:47.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:46 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:47.434 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:47.803 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-mon[32895]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:26:47.803 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-mon[32895]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:26:47.803 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-mon[32895]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:26:47.803 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-mon[32895]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:26:47.803 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/441493961' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:47.804 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:47.506Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.19:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.19:8765: connect: connection refused" 2024-09-13T14:26:47.804 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:47.511Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.19:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.19:8765: connect: connection refused" 2024-09-13T14:26:47.804 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:47.511Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.19:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.19:8765: connect: connection refused" 2024-09-13T14:26:47.804 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:47.513Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.19:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.19:8765: connect: connection refused" 2024-09-13T14:26:47.804 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:47.514Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.19:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.19:8765: connect: connection refused" 2024-09-13T14:26:47.804 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:47 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:47.514Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.19:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.19:8765: connect: connection refused" 2024-09-13T14:26:47.991 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:47.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:26:47.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[41592]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:26:47.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:26:47.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[41592]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:26:47.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/441493961' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:47.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[29646]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:26:47.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[29646]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:26:47.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[29646]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:26:47.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[29646]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:26:47.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:47 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/441493961' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:48 smithi019 ceph-mon[32895]: Reconfiguring daemon prometheus.a on smithi019 2024-09-13T14:26:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:48 smithi019 ceph-mon[32895]: pgmap v11: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 597 op/s 2024-09-13T14:26:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:48 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:49.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:48 smithi003 ceph-mon[41592]: Reconfiguring daemon prometheus.a on smithi019 2024-09-13T14:26:49.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:48 smithi003 ceph-mon[41592]: pgmap v11: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 597 op/s 2024-09-13T14:26:49.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:48 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:48 smithi003 ceph-mon[29646]: Reconfiguring daemon prometheus.a on smithi019 2024-09-13T14:26:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:48 smithi003 ceph-mon[29646]: pgmap v11: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 597 op/s 2024-09-13T14:26:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:48 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:49.716 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:49 smithi019 ceph-mon[32895]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:49.716 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:49 smithi019 ceph-mon[32895]: from='client.25292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:50.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:49 smithi003 ceph-mon[41592]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:50.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:49 smithi003 ceph-mon[41592]: from='client.25292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:50.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:49 smithi003 ceph-mon[29646]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:50.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:49 smithi003 ceph-mon[29646]: from='client.25292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:50.257 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:50.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:51.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:50 smithi019 ceph-mon[32895]: pgmap v12: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 942 op/s 2024-09-13T14:26:51.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:50 smithi003 ceph-mon[29646]: pgmap v12: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 942 op/s 2024-09-13T14:26:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:50 smithi003 ceph-mon[41592]: pgmap v12: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 942 op/s 2024-09-13T14:26:51.559 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:51 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[68304]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:26:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.51.0" 2024-09-13T14:26:51.657 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:51.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:51 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/640410160' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:51.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:51 smithi019 systemd[1]: Stopping Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:26:51.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:51 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/640410160' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:51.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:51 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/640410160' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:52.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.150Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:26:52.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.150Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:26:52.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.150Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:26:52.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.150Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:26:52.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.150Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:26:52.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.150Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:26:52.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.150Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-13T14:26:52.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.150Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-13T14:26:52.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.151Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-13T14:26:52.433 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.153Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:26:52.433 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.153Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-13T14:26:52.433 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[77107]: ts=2024-09-13T14:26:52.153Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-13T14:26:52.433 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 podman[81056]: 2024-09-13 14:26:52.16302316 +0000 UTC m=+0.163697589 container died fb69e3c0bb9bab0a22cfb09a2af4f197aa698828e7c4380c79185bc91e52accf (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:26:53.175 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 podman[81056]: 2024-09-13 14:26:52.881580122 +0000 UTC m=+0.882254550 container cleanup fb69e3c0bb9bab0a22cfb09a2af4f197aa698828e7c4380c79185bc91e52accf (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:26:53.175 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:52 smithi019 bash[81056]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:26:53.175 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-mon[32895]: pgmap v13: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-09-13T14:26:53.176 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:52 smithi019 ceph-mon[32895]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:53.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:52 smithi003 ceph-mon[41592]: pgmap v13: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-09-13T14:26:53.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:52 smithi003 ceph-mon[41592]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:53.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:52 smithi003 ceph-mon[29646]: pgmap v13: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-09-13T14:26:53.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:52 smithi003 ceph-mon[29646]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:53.428 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:53 smithi019 podman[81076]: 2024-09-13 14:26:53.174746158 +0000 UTC m=+1.000167437 container remove fb69e3c0bb9bab0a22cfb09a2af4f197aa698828e7c4380c79185bc91e52accf (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:26:53.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:53 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:26:53.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:53 smithi019 systemd[1]: Stopped Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:26:53.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:53 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.712s CPU time. 2024-09-13T14:26:53.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:53 smithi019 systemd[1]: Starting Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:26:54.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:53 smithi019 podman[81157]: 2024-09-13 14:26:53.708239246 +0000 UTC m=+0.068279122 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-13T14:26:54.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:53 smithi019 podman[81157]: 2024-09-13 14:26:53.850976382 +0000 UTC m=+0.211016247 container create 0b10348ed2daad9e85f8fd1b82edaf6e6be981522e1b701656de477c9d594541 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:26:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:53 smithi019 ceph-mon[32895]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:54.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:53 smithi003 ceph-mon[41592]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:54.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:53 smithi003 ceph-mon[29646]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:55.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:54 smithi003 ceph-mon[41592]: pgmap v14: 89 pgs: 89 active+clean; 861 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T14:26:55.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:54 smithi003 ceph-mon[29646]: pgmap v14: 89 pgs: 89 active+clean; 861 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T14:26:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 podman[81157]: 2024-09-13 14:26:54.687655871 +0000 UTC m=+1.047695734 container init 0b10348ed2daad9e85f8fd1b82edaf6e6be981522e1b701656de477c9d594541 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:26:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 podman[81157]: 2024-09-13 14:26:54.695597372 +0000 UTC m=+1.055637244 container start 0b10348ed2daad9e85f8fd1b82edaf6e6be981522e1b701656de477c9d594541 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:26:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.728Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-13T14:26:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.728Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-13T14:26:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.728Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi019 (none))" 2024-09-13T14:26:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.728Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:26:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.728Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.735Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.736Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.739Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.739Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.744Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.744Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.111µs 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.744Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.744Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 bash[81157]: 0b10348ed2daad9e85f8fd1b82edaf6e6be981522e1b701656de477c9d594541 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 systemd[1]: Started Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.791Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.802Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-13T14:26:55.184 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.802Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-13T14:26:55.185 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.802Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=48.553µs wal_replay_duration=57.823526ms wbl_replay_duration=140ns total_replay_duration=57.905695ms 2024-09-13T14:26:55.185 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.808Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:26:55.185 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.808Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-13T14:26:55.185 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.808Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:26:55.185 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.851Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=42.171349ms db_storage=1.988µs remote_storage=1.433µs web_handler=412ns query_engine=868ns scrape=667.281µs scrape_sd=164.1µs notify=23µs notify_sd=14.842µs rules=40.586759ms tracing=8.919µs 2024-09-13T14:26:55.185 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.851Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-13T14:26:55.185 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:54.851Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:26:55.185 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:54 smithi019 ceph-mon[32895]: pgmap v14: 89 pgs: 89 active+clean; 861 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T14:26:55.264 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:55 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/459659337' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:55 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:55 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:55 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:55 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:56.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/459659337' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:56.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:56.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:56.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:56.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/459659337' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:26:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:55 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:56 smithi019 ceph-mon[32895]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:56 smithi019 ceph-mon[32895]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:56 smithi019 ceph-mon[32895]: pgmap v15: 89 pgs: 89 active+clean; 861 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-09-13T14:26:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:56 smithi019 ceph-mon[32895]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:57.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:26:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:26:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:57.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:56 smithi003 ceph-mon[41592]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:57.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:56 smithi003 ceph-mon[41592]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:57.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:56 smithi003 ceph-mon[41592]: pgmap v15: 89 pgs: 89 active+clean; 861 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-09-13T14:26:57.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:56 smithi003 ceph-mon[41592]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:56 smithi003 ceph-mon[29646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:26:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:56 smithi003 ceph-mon[29646]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:57.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:56 smithi003 ceph-mon[29646]: pgmap v15: 89 pgs: 89 active+clean; 861 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-09-13T14:26:57.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:56 smithi003 ceph-mon[29646]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:57 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:57 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:57 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:26:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:57 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:57 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:57 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:57 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:26:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:57 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:57 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:57 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:26:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:57 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:26:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:57 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:58.894 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:26:58.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:58 smithi019 ceph-mon[32895]: Upgrade: Updating mgr.x 2024-09-13T14:26:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:58 smithi019 ceph-mon[32895]: Deploying daemon mgr.x on smithi019 2024-09-13T14:26:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:58 smithi019 ceph-mon[32895]: pgmap v16: 89 pgs: 89 active+clean; 861 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-09-13T14:26:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:26:58 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3958340051' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:58 smithi003 ceph-mon[29646]: Upgrade: Updating mgr.x 2024-09-13T14:26:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:58 smithi003 ceph-mon[29646]: Deploying daemon mgr.x on smithi019 2024-09-13T14:26:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:58 smithi003 ceph-mon[29646]: pgmap v16: 89 pgs: 89 active+clean; 861 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-09-13T14:26:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:26:58 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3958340051' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:26:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:58 smithi003 ceph-mon[41592]: Upgrade: Updating mgr.x 2024-09-13T14:26:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:58 smithi003 ceph-mon[41592]: Deploying daemon mgr.x on smithi019 2024-09-13T14:26:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:58 smithi003 ceph-mon[41592]: pgmap v16: 89 pgs: 89 active+clean; 861 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-09-13T14:26:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:26:58 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3958340051' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:00.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:26:59 smithi019 systemd[1]: Stopping Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:27:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:00.682 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:00 smithi019 podman[81531]: 2024-09-13 14:27:00.407877444 +0000 UTC m=+0.213539145 container died 733f346f63cf610f56f232be68a293d2e54b9efd6ec2df96fa6b3e291c60658d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T14:27:01.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:00 smithi019 ceph-mon[32895]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:00 smithi019 ceph-mon[32895]: from='client.25123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:00 smithi019 ceph-mon[32895]: pgmap v17: 89 pgs: 89 active+clean; 600 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:27:01.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:00 smithi019 podman[81531]: 2024-09-13 14:27:00.91307714 +0000 UTC m=+0.718738830 container cleanup 733f346f63cf610f56f232be68a293d2e54b9efd6ec2df96fa6b3e291c60658d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-13T14:27:01.182 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:00 smithi019 bash[81531]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x 2024-09-13T14:27:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:00 smithi003 ceph-mon[41592]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:00 smithi003 ceph-mon[41592]: from='client.25123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:00 smithi003 ceph-mon[41592]: pgmap v17: 89 pgs: 89 active+clean; 600 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:27:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:00 smithi003 ceph-mon[29646]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:00 smithi003 ceph-mon[29646]: from='client.25123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:00 smithi003 ceph-mon[29646]: pgmap v17: 89 pgs: 89 active+clean; 600 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:27:01.470 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:01 smithi019 podman[81546]: 2024-09-13 14:27:01.212676599 +0000 UTC m=+0.796801819 container remove 733f346f63cf610f56f232be68a293d2e54b9efd6ec2df96fa6b3e291c60658d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=) 2024-09-13T14:27:01.470 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:01 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:27:01.800 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:01 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-13T14:27:01.800 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:01 smithi019 systemd[1]: Stopped Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:27:01.800 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:01 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service: Consumed 33.237s CPU time. 2024-09-13T14:27:02.136 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:01 smithi019 systemd[1]: Starting Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:27:02.392 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:02 smithi019 podman[81664]: 2024-09-13 14:27:02.039884598 +0000 UTC m=+0.038767453 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:27:02.493 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:02.682 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:02 smithi019 podman[81664]: 2024-09-13 14:27:02.391525858 +0000 UTC m=+0.390408704 container create f0b9ab84e7f361356fa5fe2e63d65f9e2d9d29176535f9aa00e50077e3e47fe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-13T14:27:03.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:02 smithi003 ceph-mon[41592]: pgmap v18: 89 pgs: 89 active+clean; 600 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-09-13T14:27:03.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:02 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1341021819' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:03.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:02 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:03.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:02 smithi003 ceph-mon[29646]: pgmap v18: 89 pgs: 89 active+clean; 600 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-09-13T14:27:03.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:02 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1341021819' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:03.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:02 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:03.183 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:02 smithi019 podman[81664]: 2024-09-13 14:27:02.699214044 +0000 UTC m=+0.698096901 container init f0b9ab84e7f361356fa5fe2e63d65f9e2d9d29176535f9aa00e50077e3e47fe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-13T14:27:03.184 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:02 smithi019 podman[81664]: 2024-09-13 14:27:02.709340172 +0000 UTC m=+0.708223027 container start f0b9ab84e7f361356fa5fe2e63d65f9e2d9d29176535f9aa00e50077e3e47fe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-13T14:27:03.184 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:02 smithi019 bash[81664]: f0b9ab84e7f361356fa5fe2e63d65f9e2d9d29176535f9aa00e50077e3e47fe8 2024-09-13T14:27:03.184 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:02 smithi019 systemd[1]: Started Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:27:03.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:02 smithi019 ceph-mon[32895]: pgmap v18: 89 pgs: 89 active+clean; 600 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-09-13T14:27:03.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:02 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1341021819' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:03.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:02 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:04.078 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:03 smithi019 ceph-mon[32895]: from='client.25129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:04.079 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:03 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:04.079 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:03 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:04.079 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:03 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:27:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:03 smithi003 ceph-mon[41592]: from='client.25129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:03 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:03 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:03 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:27:04.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:03 smithi003 ceph-mon[29646]: from='client.25129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:04.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:03 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:03 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:03 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:27:04.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:04 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:27:04] "GET /metrics HTTP/1.1" 200 36229 "" "Prometheus/2.51.0" 2024-09-13T14:27:05.161 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:04 smithi019 ceph-mon[32895]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:05.161 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:04 smithi019 ceph-mon[32895]: pgmap v19: 89 pgs: 89 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:27:05.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:04 smithi003 ceph-mon[29646]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:05.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:04 smithi003 ceph-mon[29646]: pgmap v19: 89 pgs: 89 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:27:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:04 smithi003 ceph-mon[41592]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:04 smithi003 ceph-mon[41592]: pgmap v19: 89 pgs: 89 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:27:05.922 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:05 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1689241665' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:05.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:05 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/1689241665' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:06.116 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:06.179 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:06.179 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:06.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:05 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1689241665' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:06.808 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:06 smithi019 ceph-mon[32895]: pgmap v20: 89 pgs: 89 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 693 op/s 2024-09-13T14:27:06.808 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:06 smithi019 ceph-mon[32895]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:06.808 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:06 smithi019 ceph-mon[32895]: from='client.25370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:07.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:06 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:06.953Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:07.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:06 smithi003 ceph-mon[41592]: pgmap v20: 89 pgs: 89 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 693 op/s 2024-09-13T14:27:07.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:06 smithi003 ceph-mon[41592]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:07.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:06 smithi003 ceph-mon[41592]: from='client.25370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:06 smithi003 ceph-mon[29646]: pgmap v20: 89 pgs: 89 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 693 op/s 2024-09-13T14:27:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:06 smithi003 ceph-mon[29646]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:06 smithi003 ceph-mon[29646]: from='client.25370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:07 smithi019 ceph-mon[32895]: pgmap v21: 89 pgs: 89 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 693 op/s 2024-09-13T14:27:08.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:07 smithi003 ceph-mon[41592]: pgmap v21: 89 pgs: 89 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 693 op/s 2024-09-13T14:27:08.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:07 smithi003 ceph-mon[29646]: pgmap v21: 89 pgs: 89 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 693 op/s 2024-09-13T14:27:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:09 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3171742236' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:09 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3171742236' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:09.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:09 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3171742236' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:09.549 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:10.473 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 127035 objects 2024-09-13T14:27:10.474 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :154.177 2024-09-13T14:27:10.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:10 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:10.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:10 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:10.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:10 smithi003 ceph-mon[29646]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:10.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:10 smithi003 ceph-mon[29646]: pgmap v22: 89 pgs: 89 active+clean; 123 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 983 op/s 2024-09-13T14:27:10.686 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:10.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:10 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:10.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:10 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:10.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:10 smithi003 ceph-mon[41592]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:10.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:10 smithi003 ceph-mon[41592]: pgmap v22: 89 pgs: 89 active+clean; 123 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 983 op/s 2024-09-13T14:27:10.687 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:10 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:10.687 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:10 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:10.687 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:10 smithi019 ceph-mon[32895]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:10.687 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:10 smithi019 ceph-mon[32895]: pgmap v22: 89 pgs: 89 active+clean; 123 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 983 op/s 2024-09-13T14:27:11.505 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: [13/Sep/2024:14:27:11] ENGINE Bus STARTING 2024-09-13T14:27:11.505 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: CherryPy Checker: 2024-09-13T14:27:11.505 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: The Application mounted at '' has an empty config. 2024-09-13T14:27:11.505 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: 2024-09-13T14:27:11.506 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: [13/Sep/2024:14:27:11] ENGINE Serving on http://:::9283 2024-09-13T14:27:11.506 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: [13/Sep/2024:14:27:11] ENGINE Bus STARTED 2024-09-13T14:27:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-mon[32895]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-mon[32895]: Standby manager daemon x restarted 2024-09-13T14:27:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-mon[32895]: Standby manager daemon x started 2024-09-13T14:27:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:27:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:27:11.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:27:11.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:11 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:27:11.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[29646]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:11.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:11.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:11.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[29646]: Standby manager daemon x restarted 2024-09-13T14:27:11.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[29646]: Standby manager daemon x started 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[29646]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[41592]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[41592]: Standby manager daemon x restarted 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[41592]: Standby manager daemon x started 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:27:11.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:27:11.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:11 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/1103175803' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:27:12.657 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:12.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:12 smithi003 ceph-mon[29646]: mgrmap e33: y(active, since 39s), standbys: x 2024-09-13T14:27:12.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:12 smithi003 ceph-mon[29646]: pgmap v23: 89 pgs: 89 active+clean; 123 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 636 op/s 2024-09-13T14:27:12.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:12 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/3500862768' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:12.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:12 smithi019 ceph-mon[32895]: mgrmap e33: y(active, since 39s), standbys: x 2024-09-13T14:27:12.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:12 smithi019 ceph-mon[32895]: pgmap v23: 89 pgs: 89 active+clean; 123 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 636 op/s 2024-09-13T14:27:12.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:12 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3500862768' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:12 smithi003 ceph-mon[41592]: mgrmap e33: y(active, since 39s), standbys: x 2024-09-13T14:27:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:12 smithi003 ceph-mon[41592]: pgmap v23: 89 pgs: 89 active+clean; 123 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 636 op/s 2024-09-13T14:27:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:12 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3500862768' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:13.671 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:13 smithi019 ceph-mon[32895]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:13.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:13 smithi003 ceph-mon[41592]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:13 smithi003 ceph-mon[29646]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:14.376 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:27:14] "GET /metrics HTTP/1.1" 200 36229 "" "Prometheus/2.51.0" 2024-09-13T14:27:14.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: pgmap v24: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:27:15.192 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:27:15.193 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:27:15.193 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:27:15.193 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:27:15.193 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:27:15.193 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:27:15.193 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:14 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:27:15.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: pgmap v24: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:27:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: pgmap v24: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:27:15.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:14 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:27:15.607 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:15.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[29646]: Upgrade: Setting container_image for all mgr 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[29646]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[29646]: from='client.? 172.21.15.3:0/2564412416' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[29646]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[29646]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[41592]: Upgrade: Setting container_image for all mgr 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[41592]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2564412416' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:15.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:27:15.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:27:15.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:15 smithi003 ceph-mon[41592]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:27:16.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:16.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:15 smithi019 ceph-mon[32895]: Upgrade: Setting container_image for all mgr 2024-09-13T14:27:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:15 smithi019 ceph-mon[32895]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:27:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:15 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2564412416' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:15 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='mgr.y' 2024-09-13T14:27:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:15 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:27:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:15 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:27:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:15 smithi019 ceph-mon[32895]: from='mgr.24982 172.21.15.3:0/2916500961' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:27:16.375 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-13T14:27:16.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-13T14:27:16.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:16 smithi003 ceph-mon[29646]: Upgrade: Updating mon.a 2024-09-13T14:27:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:16 smithi003 ceph-mon[29646]: Deploying daemon mon.a on smithi003 2024-09-13T14:27:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:16 smithi003 ceph-mon[29646]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:16 smithi003 ceph-mon[29646]: pgmap v25: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 453 op/s 2024-09-13T14:27:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:16 smithi003 ceph-mon[41592]: Upgrade: Updating mon.a 2024-09-13T14:27:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:16 smithi003 ceph-mon[41592]: Deploying daemon mon.a on smithi003 2024-09-13T14:27:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:16 smithi003 ceph-mon[41592]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:16 smithi003 ceph-mon[41592]: pgmap v25: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 453 op/s 2024-09-13T14:27:17.171 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:17.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:16 smithi019 ceph-mon[32895]: Upgrade: Updating mon.a 2024-09-13T14:27:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:16 smithi019 ceph-mon[32895]: Deploying daemon mon.a on smithi003 2024-09-13T14:27:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:16 smithi019 ceph-mon[32895]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:16 smithi019 ceph-mon[32895]: pgmap v25: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 453 op/s 2024-09-13T14:27:17.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:17 smithi003 systemd[1]: Stopping Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:27:17.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:17 smithi003 ceph-mon[41592]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:17.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:17 smithi003 ceph-mon[29646]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:18.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:17 smithi019 ceph-mon[32895]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:18.568 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:18.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:18 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[29623]: 2024-09-13T14:27:18.578+0000 7efddc16a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:27:18.667 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:18 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[29623]: 2024-09-13T14:27:18.578+0000 7efddc16a640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-13T14:27:18.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:18 smithi003 podman[103738]: 2024-09-13 14:27:18.75828601 +0000 UTC m=+0.317959145 container died 9c73445eb77ab5a00da5b9208f7059a9479328857246d1402b1f55f88a7bdef3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-09-13T14:27:19.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:19 smithi003 podman[103738]: 2024-09-13 14:27:19.128684507 +0000 UTC m=+0.688357682 container cleanup 9c73445eb77ab5a00da5b9208f7059a9479328857246d1402b1f55f88a7bdef3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T14:27:19.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:19 smithi003 bash[103738]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a 2024-09-13T14:27:19.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:19 smithi003 podman[103768]: 2024-09-13 14:27:19.522564289 +0000 UTC m=+0.890276784 container remove 9c73445eb77ab5a00da5b9208f7059a9479328857246d1402b1f55f88a7bdef3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T14:27:19.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:19 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T14:27:19.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:19 smithi003 systemd[1]: Stopped Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:27:19.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:19 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a.service: Consumed 15.576s CPU time. 2024-09-13T14:27:19.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:19 smithi003 systemd[1]: Starting Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:27:20.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 podman[103969]: 2024-09-13 14:27:19.97509384 +0000 UTC m=+0.016398247 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:27:20.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 podman[103969]: 2024-09-13 14:27:20.090446065 +0000 UTC m=+0.131750456 container create 265013e6136c03d49868cb6aaa0cd49f1ea1b7e16800a371517e0d910d676bca (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-13T14:27:20.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 podman[103969]: 2024-09-13 14:27:20.301205001 +0000 UTC m=+0.342509403 container init 265013e6136c03d49868cb6aaa0cd49f1ea1b7e16800a371517e0d910d676bca (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:27:20.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 podman[103969]: 2024-09-13 14:27:20.306723221 +0000 UTC m=+0.348027611 container start 265013e6136c03d49868cb6aaa0cd49f1ea1b7e16800a371517e0d910d676bca (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-13T14:27:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:27:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 7 2024-09-13T14:27:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: pidfile_write: ignore empty --pid-file 2024-09-13T14:27:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: load: jerasure load: lrc 2024-09-13T14:27:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:27:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Git sha 0 2024-09-13T14:27:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: DB SUMMARY 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: DB Session ID: T0YDO2FNQRVWBUS7KNIC 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1179 Bytes 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000033.sst 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000031.log size: 1410490 ; 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:27:20.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.env: 0x558b1c0e1160 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.info_log: 0x558b1e633880 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.statistics: (nil) 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.use_fsync: 0 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:27:20.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.db_log_dir: 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.wal_dir: 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:27:20.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.write_buffer_manager: 0x558b1e636e60 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.unordered_write: 0 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:27:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.row_cache: None 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.wal_filter: None 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.wal_compression: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:27:20.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_open_files: -1 2024-09-13T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:27:20.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:27:20.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:27:20.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:27:20.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Compression algorithms supported: 2024-09-13T14:27:20.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: kZSTD supported: 0 2024-09-13T14:27:20.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:27:20.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:27:20.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:27:20.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:27:20.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:27:20.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:27:20.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:27:20.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:27:20.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:27:20.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-13T14:27:20.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:27:20.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:27:20.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.merge_operator: 2024-09-13T14:27:20.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_filter: None 2024-09-13T14:27:20.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:27:20.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:27:20.695 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:27:20.695 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:27:20.695 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558b1e6334e0) 2024-09-13T14:27:20.695 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:27:20.695 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:27:20.695 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:27:20.696 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:27:20.696 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_type: 0 2024-09-13T14:27:20.696 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_index_type: 0 2024-09-13T14:27:20.696 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_shortening: 1 2024-09-13T14:27:20.696 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:27:20.696 INFO:journalctl@ceph.mon.a.smithi003.stdout: checksum: 4 2024-09-13T14:27:20.697 INFO:journalctl@ceph.mon.a.smithi003.stdout: no_block_cache: 0 2024-09-13T14:27:20.697 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache: 0x558b1e626850 2024-09-13T14:27:20.697 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:27:20.697 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_options: 2024-09-13T14:27:20.697 INFO:journalctl@ceph.mon.a.smithi003.stdout: capacity : 536870912 2024-09-13T14:27:20.697 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_shard_bits : 4 2024-09-13T14:27:20.698 INFO:journalctl@ceph.mon.a.smithi003.stdout: strict_capacity_limit : 0 2024-09-13T14:27:20.698 INFO:journalctl@ceph.mon.a.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:27:20.698 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_compressed: (nil) 2024-09-13T14:27:20.698 INFO:journalctl@ceph.mon.a.smithi003.stdout: persistent_cache: (nil) 2024-09-13T14:27:20.698 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size: 4096 2024-09-13T14:27:20.699 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size_deviation: 10 2024-09-13T14:27:20.699 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_restart_interval: 16 2024-09-13T14:27:20.699 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_block_restart_interval: 1 2024-09-13T14:27:20.699 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_block_size: 4096 2024-09-13T14:27:20.699 INFO:journalctl@ceph.mon.a.smithi003.stdout: partition_filters: 0 2024-09-13T14:27:20.699 INFO:journalctl@ceph.mon.a.smithi003.stdout: use_delta_encoding: 1 2024-09-13T14:27:20.700 INFO:journalctl@ceph.mon.a.smithi003.stdout: filter_policy: bloomfilter 2024-09-13T14:27:20.700 INFO:journalctl@ceph.mon.a.smithi003.stdout: whole_key_filtering: 1 2024-09-13T14:27:20.700 INFO:journalctl@ceph.mon.a.smithi003.stdout: verify_compression: 0 2024-09-13T14:27:20.700 INFO:journalctl@ceph.mon.a.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:27:20.700 INFO:journalctl@ceph.mon.a.smithi003.stdout: format_version: 5 2024-09-13T14:27:20.701 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_index_compression: 1 2024-09-13T14:27:20.701 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_align: 0 2024-09-13T14:27:20.701 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_auto_readahead_size: 262144 2024-09-13T14:27:20.701 INFO:journalctl@ceph.mon.a.smithi003.stdout: prepopulate_block_cache: 0 2024-09-13T14:27:20.701 INFO:journalctl@ceph.mon.a.smithi003.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:27:20.701 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:27:20.702 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:27:20.702 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:27:20.702 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression: NoCompression 2024-09-13T14:27:20.702 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:27:20.702 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:27:20.702 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:27:20.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.num_levels: 7 2024-09-13T14:27:20.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:27:20.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:27:20.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:27:20.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:27:20.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:27:20.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:27:20.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:27:20.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:27:20.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:27:20.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:27:20.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:27:20.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:27:20.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:27:20.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:27:20.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:27:20.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:27:20.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:27:20.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:27:20.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:27:20.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:27:20.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:27:20.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:27:20.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:27:20.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:27:20.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:27:20.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:27:20.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:27:20.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:27:20.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:27:20.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:27:20.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:27:20.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:27:20.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:27:20.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:27:20.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:27:20.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:27:20.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:27:20.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:27:20.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:27:20.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:27:20.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:27:20.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:27:20.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:27:20.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:27:20.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:27:20.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:27:20.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:27:20.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:27:20.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:27:20.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.ttl: 2592000 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:27:20.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:27:20.713 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:27:20.713 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:27:20.713 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:27:20.714 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:27:20.714 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:27:20.714 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:27:20.714 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:27:20.714 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-09-13T14:27:20.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:27:20.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 12142, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:27:20.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-09-13T14:27:20.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e32e2c4f-7b67-41e7-a71f-fc3bac25accc 2024-09-13T14:27:20.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237640334238, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-09-13T14:27:20.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-09-13T14:27:20.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237640339265, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 1280160, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12136, "largest_seqno": 12867, "table_properties": {"data_size": 1275973, "index_size": 2251, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 901, "raw_key_size": 8041, "raw_average_key_size": 23, "raw_value_size": 1268530, "raw_average_value_size": 3687, "num_data_blocks": 103, "num_entries": 344, "num_filter_entries": 344, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237640, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e32e2c4f-7b67-41e7-a71f-fc3bac25accc", "db_session_id": "T0YDO2FNQRVWBUS7KNIC", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:27:20.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237640339355, "job": 1, "event": "recovery_finished"} 2024-09-13T14:27:20.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-09-13T14:27:20.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:27:20.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:27:20.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558b1e656e00 2024-09-13T14:27:20.717 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: DB pointer 0x558b1e756000 2024-09-13T14:27:20.717 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:27:20.717 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:27:20.717 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** DB Stats ** 2024-09-13T14:27:20.717 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:27:20.717 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:27:20.717 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:27:20.717 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:27:20.717 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:27:20.718 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:27:20.718 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:27:20.718 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 1/0 1.22 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 435.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: L6 1/0 7.05 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 8.27 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 435.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 435.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 435.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:27:20.719 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 124.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 124.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:27:20.720 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache BinnedLRUCache@0x558b1e626850#7 capacity: 512.00 MB usage: 3.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.03 KB,0.000196695%) IndexBlock(1,2.33 KB,0.000444055%) Misc(1,0.00 KB,0%) 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: starting mon.a rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: mon.a@-1(???) e3 preinit fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: mon.a@-1(???).mds e9 new map 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: mon.a@-1(???).mds e9 print_map 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout: e9 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout: legacy client fscid: 1 2024-09-13T14:27:20.721 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: Filesystem 'foo' (1) 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: fs_name foo 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: epoch 8 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: created 2024-09-13T14:21:52.613871+0000 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: modified 2024-09-13T14:22:03.666371+0000 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: tableserver 0 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: root 0 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: session_timeout 60 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: session_autoclose 300 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_file_size 1099511627776 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_xattr_size 65536 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: required_client_features {} 2024-09-13T14:27:20.722 INFO:journalctl@ceph.mon.a.smithi003.stdout: last_failure 0 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: last_failure_osd_epoch 0 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_mds 1 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: in 0 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: up {0=14646} 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: failed 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: damaged 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: stopped 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_pools [4] 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_pool 3 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: inline_data disabled 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: balancer 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: bal_rank_mask -1 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: standby_count_wanted 1 2024-09-13T14:27:20.723 INFO:journalctl@ceph.mon.a.smithi003.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:27:20.724 INFO:journalctl@ceph.mon.a.smithi003.stdout: [mds.foo.smithi003.zmhiel{0:14646} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:27:20.724 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:27:20.724 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:27:20.724 INFO:journalctl@ceph.mon.a.smithi003.stdout: Standby daemons: 2024-09-13T14:27:20.724 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-09-13T14:27:20.724 INFO:journalctl@ceph.mon.a.smithi003.stdout: [mds.foo.smithi019.ydtwbg{-1:24520} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.19:6832/4291362241,v1:172.21.15.19:6833/4291362241] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:27:20.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: mon.a@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:27:20.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: mon.a@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:27:20.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: mon.a@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:27:20.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: mon.a@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:27:20.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 ceph-mon[104007]: mon.a@-1(???).paxosservice(auth 1..19) refresh upgraded, format 0 -> 3 2024-09-13T14:27:20.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 bash[103969]: 265013e6136c03d49868cb6aaa0cd49f1ea1b7e16800a371517e0d910d676bca 2024-09-13T14:27:20.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:20 smithi003 systemd[1]: Started Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:27:21.526 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_0' removed 2024-09-13T14:27:22.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ignoring --setuser ceph since I am not root 2024-09-13T14:27:22.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ignoring --setgroup ceph since I am not root 2024-09-13T14:27:22.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mgr[94411]: -- 172.21.15.3:0/1881582512 <== mon.1 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5617400070e0 con 0x56173ffe5000 2024-09-13T14:27:22.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:22.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:22.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3630830534' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T14:27:22.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: pgmap v27: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 453 op/s 2024-09-13T14:27:22.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: mon.a calling monitor election 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2163418116' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: from='client.? ' entity='' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: monmap epoch 3 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: last_changed 2024-09-13T14:16:35.060163+0000 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: created 2024-09-13T14:14:57.332230+0000 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: min_mon_release 17 (quincy) 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: election_strategy: 1 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: 1: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.b 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:27:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: mgrmap e33: y(active, since 50s), standbys: x 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: overall HEALTH_OK 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: from='mgr.24982 ' entity='' 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[41592]: mgrmap e34: y(active, since 50s), standbys: x 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3630830534' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: pgmap v27: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 453 op/s 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: mon.a calling monitor election 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2163418116' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:27:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: from='client.? ' entity='' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: monmap epoch 3 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: last_changed 2024-09-13T14:16:35.060163+0000 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: created 2024-09-13T14:14:57.332230+0000 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: min_mon_release 17 (quincy) 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: election_strategy: 1 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: 1: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.b 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: mgrmap e33: y(active, since 50s), standbys: x 2024-09-13T14:27:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: overall HEALTH_OK 2024-09-13T14:27:22.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: from='mgr.24982 ' entity='' 2024-09-13T14:27:22.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:21 smithi003 ceph-mon[104007]: mgrmap e34: y(active, since 50s), standbys: x 2024-09-13T14:27:22.186 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:22.186 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:22.186 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3630830534' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T14:27:22.186 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: pgmap v27: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 453 op/s 2024-09-13T14:27:22.186 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: mon.a calling monitor election 2024-09-13T14:27:22.186 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2163418116' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:22.186 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: from='client.? ' entity='' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: monmap epoch 3 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: last_changed 2024-09-13T14:16:35.060163+0000 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: created 2024-09-13T14:14:57.332230+0000 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: min_mon_release 17 (quincy) 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: election_strategy: 1 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: 1: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.b 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:27:22.187 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: mgrmap e33: y(active, since 50s), standbys: x 2024-09-13T14:27:22.188 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: overall HEALTH_OK 2024-09-13T14:27:22.188 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: from='mgr.24982 ' entity='' 2024-09-13T14:27:22.188 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mon[32895]: mgrmap e34: y(active, since 50s), standbys: x 2024-09-13T14:27:22.188 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: ignoring --setuser ceph since I am not root 2024-09-13T14:27:22.188 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: ignoring --setgroup ceph since I am not root 2024-09-13T14:27:22.188 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:21 smithi019 ceph-mgr[81698]: -- 172.21.15.19:0/112395562 <== mon.1 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb1cc0b0e0 con 0x55fb1cbe8800 2024-09-13T14:27:22.650 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:27:22.651 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:27:22.651 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-13T14:27:22.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-13T14:27:22.926 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:22 smithi003 ceph-mon[41592]: from='client.? ' entity='' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:27:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:22 smithi003 ceph-mon[41592]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:27:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:22 smithi003 ceph-mon[104007]: from='client.? ' entity='' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:27:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:22 smithi003 ceph-mon[104007]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:27:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:22 smithi019 ceph-mon[32895]: from='client.? ' entity='' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:27:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:22 smithi019 ceph-mon[32895]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:27:24.837 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_1' created 2024-09-13T14:27:25.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:24 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1991261818' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:27:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:24 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1991261818' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:27:25.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:24 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1991261818' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:27:25.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-13T14:27:25.812 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:25 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1991261818' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:27:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:25 smithi003 ceph-mon[104007]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:27:25.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:25 smithi003 ceph-mon[104007]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:27:26.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:25 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1991261818' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:27:26.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:25 smithi003 ceph-mon[41592]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:27:26.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:25 smithi003 ceph-mon[41592]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:27:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:25 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1991261818' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:27:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:25 smithi019 ceph-mon[32895]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:27:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:25 smithi019 ceph-mon[32895]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:27:27.211 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:27.211 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: [13/Sep/2024:14:27:27] ENGINE Bus STARTING 2024-09-13T14:27:27.211 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: CherryPy Checker: 2024-09-13T14:27:27.211 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: The Application mounted at '' has an empty config. 2024-09-13T14:27:27.212 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: 2024-09-13T14:27:27.212 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: [13/Sep/2024:14:27:27] ENGINE Serving on http://:::9283 2024-09-13T14:27:27.212 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x[81676]: [13/Sep/2024:14:27:27] ENGINE Bus STARTED 2024-09-13T14:27:28.062 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.800+0000 7f0f4f771640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.062 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.800+0000 7f0f4c76b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.062 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.800+0000 7f0f4c76b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.062 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.800+0000 7f0f4c76b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.063 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.800+0000 7f0f4c76b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.063 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.800+0000 7f0f4c76b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.063 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.801+0000 7f0f4f771640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.063 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.801+0000 7f0f4f771640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.063 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.801+0000 7f0f4f771640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.063 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:27.801+0000 7f0f4f771640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:27:28.063 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:27:28.063 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: Standby manager daemon x restarted 2024-09-13T14:27:28.063 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: Standby manager daemon x started 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: Active manager daemon y restarted 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: Activating manager daemon y 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: mgrmap e35: y(active, starting, since 0.00728438s), standbys: x 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:27:28.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:27:28.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"} : dispatch 2024-09-13T14:27:28.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"} : dispatch 2024-09-13T14:27:28.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T14:27:28.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T14:27:28.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:27:28.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:27:28.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:27:28.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:27:28.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: Manager daemon y is now available 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: Standby manager daemon x restarted 2024-09-13T14:27:28.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: Standby manager daemon x started 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: Active manager daemon y restarted 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: Activating manager daemon y 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: mgrmap e35: y(active, starting, since 0.00728438s), standbys: x 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"} : dispatch 2024-09-13T14:27:28.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T14:27:28.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: Manager daemon y is now available 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/874217196' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/874217196' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:27:28.069 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:28.070 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:28.070 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T14:27:28.070 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:27:28.070 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:27 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:27:28.179 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:27:28.179 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: Standby manager daemon x restarted 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: Standby manager daemon x started 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.? 172.21.15.19:0/1862995252' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: Active manager daemon y restarted 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: Activating manager daemon y 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: mgrmap e35: y(active, starting, since 0.00728438s), standbys: x 2024-09-13T14:27:28.180 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi003.zmhiel"} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi019.ydtwbg"} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T14:27:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: Manager daemon y is now available 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/874217196' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:27:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:27 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:27:28.267 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-13T14:27:28.375 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:28.266+0000 7f0f7ea9d640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-13T14:27:28.375 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:27:28] ENGINE Bus STARTING 2024-09-13T14:27:28.375 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: CherryPy Checker: 2024-09-13T14:27:28.375 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: The Application mounted at '' has an empty config. 2024-09-13T14:27:28.375 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:28.375 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:27:28] ENGINE Serving on http://:::9283 2024-09-13T14:27:28.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:14:27:28] ENGINE Bus STARTED 2024-09-13T14:27:29.588 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[41592]: mgrmap e36: y(active, since 1.01244s), standbys: x 2024-09-13T14:27:29.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[41592]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:27:29.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[41592]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:29.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[41592]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:27:29.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[41592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:27:29.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[41592]: pgmap v4: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:29.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[104007]: mgrmap e36: y(active, since 1.01244s), standbys: x 2024-09-13T14:27:29.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[104007]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[104007]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-mon[104007]: pgmap v4: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:29.392+0000 7f0f3257d640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:27:29] ENGINE Error in HTTPServer.serve 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: Traceback (most recent call last): 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: self._connections.run(self.expiration_interval) 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: self._run(expiration_interval) 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:27:29.590 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: s = self.context.wrap_socket( 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: return self.sslsocket_class._create( 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: self.do_handshake() 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: self._sslobj.do_handshake() 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:27:29.591 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:27:29.619 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:29 smithi019 ceph-mon[32895]: mgrmap e36: y(active, since 1.01244s), standbys: x 2024-09-13T14:27:29.619 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:29 smithi019 ceph-mon[32895]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:27:29.619 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:29 smithi019 ceph-mon[32895]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:29.619 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:29 smithi019 ceph-mon[32895]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:27:29.620 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:29 smithi019 ceph-mon[32895]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:27:29.620 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:29 smithi019 ceph-mon[32895]: pgmap v4: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:30.019 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-13T14:27:30.022 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:27:30.022 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:27:30.069 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T14:27:30.069 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:27:30.069 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_105041 2024-09-13T14:27:30.070 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:27:30.070 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:27:30.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:30 smithi019 ceph-mon[32895]: [13/Sep/2024:14:27:29] ENGINE Bus STARTING 2024-09-13T14:27:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:30 smithi019 ceph-mon[32895]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-13T14:27:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:30 smithi019 ceph-mon[32895]: [13/Sep/2024:14:27:29] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:27:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:30 smithi019 ceph-mon[32895]: [13/Sep/2024:14:27:29] ENGINE Error in HTTPServer.serve 2024-09-13T14:27:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout: Traceback (most recent call last): 2024-09-13T14:27:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:27:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:27:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:27:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._run(expiration_interval) 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: s = self.context.wrap_socket( 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: return self.sslsocket_class._create( 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: self.do_handshake() 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: self._sslobj.do_handshake() 2024-09-13T14:27:30.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:27:30.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:30 smithi019 ceph-mon[32895]: [13/Sep/2024:14:27:29] ENGINE Serving on http://172.21.15.3:8765 2024-09-13T14:27:30.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:30 smithi019 ceph-mon[32895]: [13/Sep/2024:14:27:29] ENGINE Bus STARTED 2024-09-13T14:27:30.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[104007]: [13/Sep/2024:14:27:29] ENGINE Bus STARTING 2024-09-13T14:27:30.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[104007]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-13T14:27:30.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[104007]: [13/Sep/2024:14:27:29] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:27:30.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[104007]: [13/Sep/2024:14:27:29] ENGINE Error in HTTPServer.serve 2024-09-13T14:27:30.684 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:27:30.684 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:27:30.685 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[104007]: [13/Sep/2024:14:27:29] ENGINE Serving on http://172.21.15.3:8765 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[104007]: [13/Sep/2024:14:27:29] ENGINE Bus STARTED 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[41592]: [13/Sep/2024:14:27:29] ENGINE Bus STARTING 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[41592]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[41592]: [13/Sep/2024:14:27:29] ENGINE Serving on https://172.21.15.3:7150 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[41592]: [13/Sep/2024:14:27:29] ENGINE Error in HTTPServer.serve 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.c.smithi003.stdout: Traceback (most recent call last): 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:27:30.686 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._run(expiration_interval) 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: s = self.context.wrap_socket( 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: return self.sslsocket_class._create( 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: self.do_handshake() 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:27:30.687 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._sslobj.do_handshake() 2024-09-13T14:27:30.688 INFO:journalctl@ceph.mon.c.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:27:30.688 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[41592]: [13/Sep/2024:14:27:29] ENGINE Serving on http://172.21.15.3:8765 2024-09-13T14:27:30.688 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:30 smithi003 ceph-mon[41592]: [13/Sep/2024:14:27:29] ENGINE Bus STARTED 2024-09-13T14:27:31.070 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3445 3429 214.301 214.312 0.00379339 0.00465025 2024-09-13T14:27:31.543 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:31 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2892588193' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:31.544 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2892588193' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:31.666 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:31.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:31 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2892588193' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:32.070 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 6694 6678 208.671 203.062 0.00263825 0.00478179 2024-09-13T14:27:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:32 smithi019 ceph-mon[32895]: pgmap v6: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:32 smithi019 ceph-mon[32895]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-13T14:27:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:32 smithi019 ceph-mon[32895]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:32 smithi003 ceph-mon[41592]: pgmap v6: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:32 smithi003 ceph-mon[41592]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-13T14:27:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:32 smithi003 ceph-mon[41592]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:32 smithi003 ceph-mon[104007]: pgmap v6: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:32 smithi003 ceph-mon[104007]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-13T14:27:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:32 smithi003 ceph-mon[104007]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:33.070 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 9860 9844 205.066 197.875 0.00654895 0.00486788 2024-09-13T14:27:33.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:33 smithi019 ceph-mon[32895]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:33.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:33 smithi003 ceph-mon[41592]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:33.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:33 smithi003 ceph-mon[104007]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:34.071 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 13009 12993 202.924 196.812 0.00556366 0.00492257 2024-09-13T14:27:34.350 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:27:34] "GET /metrics HTTP/1.1" 200 33488 "" "Prometheus/2.51.0" 2024-09-13T14:27:34.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[104007]: pgmap v7: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:34.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:34.149Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:34 smithi019 ceph-mon[32895]: pgmap v7: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:34 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:34 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:34 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:34 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[41592]: pgmap v7: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:34 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 15527 15511 193.814 157.375 0.00603841 0.0051518 2024-09-13T14:27:35.591 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:35.665 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:35 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.665 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:35 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.665 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:35 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/3187540019' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:35.665 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:35 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.665 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:35 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/3187540019' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3187540019' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:35 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:35.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:35.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:36.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 18135 18119 188.677 163 0.00368435 0.00529303 2024-09-13T14:27:36.622 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:36 smithi019 ceph-mon[32895]: pgmap v8: 89 pgs: 89 active+clean; 656 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 25 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-13T14:27:36.622 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:36 smithi019 ceph-mon[32895]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:36.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:36 smithi003 ceph-mon[41592]: pgmap v8: 89 pgs: 89 active+clean; 656 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 25 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-13T14:27:36.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:36 smithi003 ceph-mon[41592]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:36.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:36 smithi003 ceph-mon[104007]: pgmap v8: 89 pgs: 89 active+clean; 656 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 25 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-13T14:27:36.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:36 smithi003 ceph-mon[104007]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:37.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 20837 20821 185.846 168.875 0.00561948 0.00537486 2024-09-13T14:27:37.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:37.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:37 smithi003 ceph-mon[41592]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:37.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:37 smithi003 ceph-mon[104007]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:37.700 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:37 smithi019 ceph-mon[32895]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:38.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 23304 23288 181.888 154.188 0.00336295 0.00549151 2024-09-13T14:27:38.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:38 smithi003 ceph-mon[41592]: pgmap v9: 89 pgs: 89 active+clean; 656 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-13T14:27:38.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:38 smithi003 ceph-mon[104007]: pgmap v9: 89 pgs: 89 active+clean; 656 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-13T14:27:38.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:38 smithi019 ceph-mon[32895]: pgmap v9: 89 pgs: 89 active+clean; 656 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-13T14:27:39.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 25410 25394 176.302 131.625 0.00391714 0.00566559 2024-09-13T14:27:39.398 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:39.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:39 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/2353327759' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:39.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2353327759' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:39.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:39 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/2353327759' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:40.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 27506 27490 171.771 131 0.00449113 0.00581584 2024-09-13T14:27:40.635 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:40 smithi019 ceph-mon[32895]: pgmap v10: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-13T14:27:40.635 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:40 smithi019 ceph-mon[32895]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:40.636 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:40 smithi003 ceph-mon[104007]: pgmap v10: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-13T14:27:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:40 smithi003 ceph-mon[104007]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:40 smithi003 ceph-mon[41592]: pgmap v10: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-13T14:27:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:40 smithi003 ceph-mon[41592]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:41.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 29750 29734 168.905 140.25 0.00726261 0.00591494 2024-09-13T14:27:42.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 31809 31793 165.553 128.688 0.00500187 0.00603457 2024-09-13T14:27:42.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:41 smithi019 ceph-mon[32895]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:42.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:41 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:42.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:41 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:42.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:41 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:42.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:41 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:42.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:41 smithi019 ceph-mon[32895]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:27:42.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:41 smithi019 ceph-mon[32895]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:27:42.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[41592]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:42.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:42.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:42.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:42.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:42.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[41592]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:27:42.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:27:42.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[104007]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:42.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:42.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:42.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:42.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:42.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[104007]: Updating smithi003:/etc/ceph/ceph.conf 2024-09-13T14:27:42.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:41 smithi003 ceph-mon[104007]: Updating smithi019:/etc/ceph/ceph.conf 2024-09-13T14:27:43.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 33600 33584 161.429 111.938 0.00458686 0.00618907 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: pgmap v11: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 123 MiB/s wr, 1.98k op/s 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:42 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/221211244' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: pgmap v11: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 123 MiB/s wr, 1.98k op/s 2024-09-13T14:27:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/221211244' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:43.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:43.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:43.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: pgmap v11: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 123 MiB/s wr, 1.98k op/s 2024-09-13T14:27:43.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:43.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:43.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:43.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:42 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/221211244' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:43.225 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:43.808 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[41592]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:43.808 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[41592]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:43.809 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:43.809 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:43.809 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:27:43.809 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:27:43.809 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-13T14:27:43.809 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[41592]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:27:43.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[104007]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:43.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[104007]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:43.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:43.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:43.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:27:43.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:27:43.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-13T14:27:43.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:43 smithi003 ceph-mon[104007]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:27:44.072 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 15 35535 35520 158.54 121 0.00398631 0.00630288 2024-09-13T14:27:44.149 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:43 smithi019 ceph-mon[32895]: Updating smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:44.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:43 smithi019 ceph-mon[32895]: Updating smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:44.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:43 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:44.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:43 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:44.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:43 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:27:44.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:43 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:27:44.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:43 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-13T14:27:44.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:43 smithi019 ceph-mon[32895]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:27:44.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:44 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:44.149Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:27:44] "GET /metrics HTTP/1.1" 200 36225 "" "Prometheus/2.51.0" 2024-09-13T14:27:45.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 37666 37650 156.845 133.125 0.00598945 0.00637044 2024-09-13T14:27:45.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:44 smithi019 ceph-mon[32895]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:44 smithi019 ceph-mon[32895]: pgmap v12: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 123 MiB/s wr, 1.98k op/s 2024-09-13T14:27:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:44 smithi019 ceph-mon[32895]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:45.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:44 smithi003 ceph-mon[41592]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:45.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:44 smithi003 ceph-mon[41592]: pgmap v12: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 123 MiB/s wr, 1.98k op/s 2024-09-13T14:27:45.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:44 smithi003 ceph-mon[41592]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:45.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:44 smithi003 ceph-mon[104007]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:45.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:44 smithi003 ceph-mon[104007]: pgmap v12: 89 pgs: 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 123 MiB/s wr, 1.98k op/s 2024-09-13T14:27:45.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:44 smithi003 ceph-mon[104007]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:46.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 39622 39606 154.683 122.25 0.0145786 0.00645892 2024-09-13T14:27:46.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:46.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:45 smithi019 ceph-mon[32895]: Upgrade: Updating mon.c 2024-09-13T14:27:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:45 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:45 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:45 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:46.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:45 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:46.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:45 smithi019 ceph-mon[32895]: Deploying daemon mon.c on smithi003 2024-09-13T14:27:46.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[41592]: Upgrade: Updating mon.c 2024-09-13T14:27:46.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:46.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:46.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:46.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[41592]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:46.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[41592]: Deploying daemon mon.c on smithi003 2024-09-13T14:27:46.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[104007]: Upgrade: Updating mon.c 2024-09-13T14:27:46.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:27:46.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:46.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:46.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:46.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:45 smithi003 ceph-mon[104007]: Deploying daemon mon.c on smithi003 2024-09-13T14:27:47.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 41348 41332 151.929 107.875 0.0074055 0.0065759 2024-09-13T14:27:47.075 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:47.075 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:46 smithi019 ceph-mon[32895]: pgmap v13: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 15 KiB/s rd, 177 MiB/s wr, 2.83k op/s 2024-09-13T14:27:47.075 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:46 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1725581655' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:47.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:46 smithi003 ceph-mon[104007]: pgmap v13: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 15 KiB/s rd, 177 MiB/s wr, 2.83k op/s 2024-09-13T14:27:47.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:46 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1725581655' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:47.083 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:46 smithi003 ceph-mon[41592]: pgmap v13: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 15 KiB/s rd, 177 MiB/s wr, 2.83k op/s 2024-09-13T14:27:47.083 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:46 smithi003 ceph-mon[41592]: from='client.? 172.21.15.3:0/1725581655' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:27:47.084 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:47 smithi003 systemd[1]: Stopping Ceph mon.c for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:27:47.844 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c[41569]: 2024-09-13T14:27:47.577+0000 7f506b235640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:27:47.845 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c[41569]: 2024-09-13T14:27:47.577+0000 7f506b235640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-13T14:27:47.845 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:47 smithi003 podman[107084]: 2024-09-13 14:27:47.813985018 +0000 UTC m=+0.385958550 container stop a6373bc053ae585cf9f7c2cebbf38690d5872405a03302298ba5346edceb9188 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa) 2024-09-13T14:27:47.845 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:47 smithi003 podman[107084]: 2024-09-13 14:27:47.843972997 +0000 UTC m=+0.415946525 container died a6373bc053ae585cf9f7c2cebbf38690d5872405a03302298ba5346edceb9188 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-13T14:27:48.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 43107 43091 149.595 109.938 0.00377382 0.00667934 2024-09-13T14:27:48.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:48 smithi003 podman[107084]: 2024-09-13 14:27:48.366176695 +0000 UTC m=+0.938150228 container cleanup a6373bc053ae585cf9f7c2cebbf38690d5872405a03302298ba5346edceb9188 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa) 2024-09-13T14:27:48.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:48 smithi003 bash[107084]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c 2024-09-13T14:27:48.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:48 smithi003 podman[107104]: 2024-09-13 14:27:48.602008485 +0000 UTC m=+0.778393925 container remove a6373bc053ae585cf9f7c2cebbf38690d5872405a03302298ba5346edceb9188 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:27:49.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 44939 44923 147.748 114.5 0.00882182 0.00676223 2024-09-13T14:27:49.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:49 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-13T14:27:49.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:49 smithi003 systemd[1]: Stopped Ceph mon.c for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:27:49.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:49 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.c.service: Consumed 9.205s CPU time. 2024-09-13T14:27:49.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:49 smithi003 systemd[1]: Starting Ceph mon.c for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:27:49.982 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:49 smithi003 podman[107224]: 2024-09-13 14:27:49.599490966 +0000 UTC m=+0.038245910 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:27:49.982 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:49 smithi003 podman[107224]: 2024-09-13 14:27:49.711945871 +0000 UTC m=+0.150700799 container create 8539c77bb8fb82cfe0bf98a107e5865f14820f407d5498e59675b8c33498d798 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-13T14:27:50.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:27:50.073373+0000 min lat: 0.00112889 max lat: 0.0976582 avg lat: 0.00686329 2024-09-13T14:27:50.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:27:50.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 46609 46593 145.579 104.375 0.0128232 0.00686329 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:49 smithi003 podman[107224]: 2024-09-13 14:27:49.981804434 +0000 UTC m=+0.420559374 container init 8539c77bb8fb82cfe0bf98a107e5865f14820f407d5498e59675b8c33498d798 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:49 smithi003 podman[107224]: 2024-09-13 14:27:49.99370105 +0000 UTC m=+0.432455978 container start 8539c77bb8fb82cfe0bf98a107e5865f14820f407d5498e59675b8c33498d798 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 6 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: pidfile_write: ignore empty --pid-file 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: load: jerasure load: lrc 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Git sha 0 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: DB SUMMARY 2024-09-13T14:27:50.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: DB Session ID: 11OUX9Y0Y2MR6TFLHEE1 2024-09-13T14:27:50.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:27:50.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:27:50.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1104 Bytes 2024-09-13T14:27:50.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-13T14:27:50.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 5092846 ; 2024-09-13T14:27:50.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:27:50.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:27:50.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.env: 0x5639d2cf2160 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.info_log: 0x5639d3c37880 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.statistics: (nil) 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.use_fsync: 0 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:27:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:27:50.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:27:50.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:27:50.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:27:50.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:27:50.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:27:50.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:27:50.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:27:50.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:27:50.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.db_log_dir: 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.wal_dir: 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.write_buffer_manager: 0x5639d3c3ae60 2024-09-13T14:27:50.443 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:27:50.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:27:50.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:27:50.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:27:50.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:27:50.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:27:50.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:27:50.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:27:50.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.unordered_write: 0 2024-09-13T14:27:50.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:27:50.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:27:50.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:27:50.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:27:50.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.row_cache: None 2024-09-13T14:27:50.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.wal_filter: None 2024-09-13T14:27:50.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:27:50.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:27:50.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:27:50.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:27:50.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.wal_compression: 0 2024-09-13T14:27:50.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:27:50.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:27:50.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:27:50.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:27:50.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:27:50.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:27:50.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:27:50.446 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:27:50.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:27:50.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:27:50.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:27:50.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:27:50.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:27:50.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:27:50.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:27:50.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:27:50.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:27:50.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:27:50.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:27:50.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:27:50.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:27:50.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:27:50.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_open_files: -1 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Compression algorithms supported: 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: kZSTD supported: 0 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:27:50.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.merge_operator: 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_filter: None 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5639d3c374e0) 2024-09-13T14:27:50.450 INFO:journalctl@ceph.mon.c.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_type: 0 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_block_index_type: 0 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_shortening: 1 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: checksum: 4 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: no_block_cache: 0 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache: 0x5639d3c2a850 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:27:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_options: 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: capacity : 536870912 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: num_shard_bits : 4 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: strict_capacity_limit : 0 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_compressed: (nil) 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: persistent_cache: (nil) 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_size: 4096 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_size_deviation: 10 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_restart_interval: 16 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_block_restart_interval: 1 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: metadata_block_size: 4096 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: partition_filters: 0 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: use_delta_encoding: 1 2024-09-13T14:27:50.452 INFO:journalctl@ceph.mon.c.smithi003.stdout: filter_policy: bloomfilter 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: whole_key_filtering: 1 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: verify_compression: 0 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: format_version: 5 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: enable_index_compression: 1 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_align: 0 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: max_auto_readahead_size: 262144 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: prepopulate_block_cache: 0 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:27:50.453 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression: NoCompression 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.num_levels: 7 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:27:50.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:27:50.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:27:50.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:27:50.457 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:27:50.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.ttl: 2592000 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:27:50.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:27:50.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12744, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1daf4819-06f4-4b08-948c-244f6581c9ad 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237670055180, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237670074307, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3161922, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12741, "largest_seqno": 14182, "table_properties": {"data_size": 3155179, "index_size": 4098, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1605, "raw_key_size": 14495, "raw_average_key_size": 23, "raw_value_size": 3141915, "raw_average_value_size": 5051, "num_data_blocks": 186, "num_entries": 622, "num_filter_entries": 622, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237670, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1daf4819-06f4-4b08-948c-244f6581c9ad", "db_session_id": "11OUX9Y0Y2MR6TFLHEE1", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237670074934, "job": 1, "event": "recovery_finished"} 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5639d3c5ae00 2024-09-13T14:27:50.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: DB pointer 0x5639d3d5a000 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** DB Stats ** 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:27:50.462 INFO:journalctl@ceph.mon.c.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: L0 1/0 3.02 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 423.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: L6 1/0 7.05 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: Sum 2/0 10.07 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 423.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 423.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 423.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:27:50.463 INFO:journalctl@ceph.mon.c.smithi003.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative compaction: 0.00 GB write, 89.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval compaction: 0.00 GB write, 89.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: Block cache BinnedLRUCache@0x5639d3c2a850#6 capacity: 512.00 MB usage: 61.78 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.6e-05 secs_since: 0 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: Block cache entry stats(count,size,portion): DataBlock(4,30.31 KB,0.00578165%) FilterBlock(2,10.81 KB,0.00206232%) IndexBlock(2,20.66 KB,0.00393987%) Misc(1,0.00 KB,0%) 2024-09-13T14:27:50.464 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:27:50.465 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:27:50.465 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: starting mon.c rank 2 at public addrs [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] at bind addrs [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:50.465 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: mon.c@-1(???) e3 preinit fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:50.465 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: mon.c@-1(???).mds e9 new map 2024-09-13T14:27:50.465 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: mon.c@-1(???).mds e9 print_map 2024-09-13T14:27:50.465 INFO:journalctl@ceph.mon.c.smithi003.stdout: e9 2024-09-13T14:27:50.465 INFO:journalctl@ceph.mon.c.smithi003.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:27:50.465 INFO:journalctl@ceph.mon.c.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:27:50.466 INFO:journalctl@ceph.mon.c.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:27:50.466 INFO:journalctl@ceph.mon.c.smithi003.stdout: legacy client fscid: 1 2024-09-13T14:27:50.466 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:27:50.466 INFO:journalctl@ceph.mon.c.smithi003.stdout: Filesystem 'foo' (1) 2024-09-13T14:27:50.466 INFO:journalctl@ceph.mon.c.smithi003.stdout: fs_name foo 2024-09-13T14:27:50.466 INFO:journalctl@ceph.mon.c.smithi003.stdout: epoch 8 2024-09-13T14:27:50.467 INFO:journalctl@ceph.mon.c.smithi003.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: created 2024-09-13T14:21:52.613871+0000 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: modified 2024-09-13T14:22:03.666371+0000 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: tableserver 0 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: root 0 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: session_timeout 60 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: session_autoclose 300 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: max_file_size 1099511627776 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: max_xattr_size 65536 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: required_client_features {} 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: last_failure 0 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: last_failure_osd_epoch 0 2024-09-13T14:27:50.468 INFO:journalctl@ceph.mon.c.smithi003.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:27:50.469 INFO:journalctl@ceph.mon.c.smithi003.stdout: max_mds 1 2024-09-13T14:27:50.469 INFO:journalctl@ceph.mon.c.smithi003.stdout: in 0 2024-09-13T14:27:50.469 INFO:journalctl@ceph.mon.c.smithi003.stdout: up {0=14646} 2024-09-13T14:27:50.469 INFO:journalctl@ceph.mon.c.smithi003.stdout: failed 2024-09-13T14:27:50.469 INFO:journalctl@ceph.mon.c.smithi003.stdout: damaged 2024-09-13T14:27:50.469 INFO:journalctl@ceph.mon.c.smithi003.stdout: stopped 2024-09-13T14:27:50.469 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_pools [4] 2024-09-13T14:27:50.469 INFO:journalctl@ceph.mon.c.smithi003.stdout: metadata_pool 3 2024-09-13T14:27:50.469 INFO:journalctl@ceph.mon.c.smithi003.stdout: inline_data disabled 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: balancer 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: bal_rank_mask -1 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: standby_count_wanted 1 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: [mds.foo.smithi003.zmhiel{0:14646} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: Standby daemons: 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout: [mds.foo.smithi019.ydtwbg{-1:24520} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.19:6832/4291362241,v1:172.21.15.19:6833/4291362241] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: mon.c@-1(???).osd e79 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:27:50.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: mon.c@-1(???).osd e79 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:27:50.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: mon.c@-1(???).osd e79 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:27:50.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: mon.c@-1(???).osd e79 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:27:50.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 ceph-mon[107278]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-09-13T14:27:50.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 bash[107224]: 8539c77bb8fb82cfe0bf98a107e5865f14820f407d5498e59675b8c33498d798 2024-09-13T14:27:50.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:50 smithi003 systemd[1]: Started Ceph mon.c for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:27:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:51.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 48430 48414 144.066 113.812 0.00997738 0.00693597 2024-09-13T14:27:52.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 50349 50333 142.969 119.938 0.0139305 0.00698918 2024-09-13T14:27:53.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 52223 52207 141.845 117.125 0.00511948 0.00704315 2024-09-13T14:27:54.073 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 54048 54032 140.687 114.062 0.00801898 0.00710273 2024-09-13T14:27:54.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:54.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:27:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:27:54] "GET /metrics HTTP/1.1" 200 36225 "" "Prometheus/2.51.0" 2024-09-13T14:27:55.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 55767 55751 139.356 107.438 0.0101246 0.00717049 2024-09-13T14:27:56.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 57338 57322 137.773 98.1875 0.00912248 0.00725388 2024-09-13T14:27:56.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:55 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:56.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:55 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:56.647 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: mon.c calling monitor election 2024-09-13T14:27:56.647 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: mon.b calling monitor election 2024-09-13T14:27:56.647 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: mon.a calling monitor election 2024-09-13T14:27:56.647 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:27:56.647 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: monmap epoch 3 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: last_changed 2024-09-13T14:16:35.060163+0000 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: created 2024-09-13T14:14:57.332230+0000 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: min_mon_release 17 (quincy) 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: election_strategy: 1 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: 1: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.b 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: mgrmap e38: y(active, since 29s), standbys: x 2024-09-13T14:27:56.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[107278]: overall HEALTH_OK 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: mon.c calling monitor election 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: mon.b calling monitor election 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: mon.a calling monitor election 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: monmap epoch 3 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: last_changed 2024-09-13T14:16:35.060163+0000 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: created 2024-09-13T14:14:57.332230+0000 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: min_mon_release 17 (quincy) 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: election_strategy: 1 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: 1: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.b 2024-09-13T14:27:56.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-09-13T14:27:56.650 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:27:56.650 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:27:56.650 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: mgrmap e38: y(active, since 29s), standbys: x 2024-09-13T14:27:56.650 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:56 smithi003 ceph-mon[104007]: overall HEALTH_OK 2024-09-13T14:27:56.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: mon.c calling monitor election 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: mon.b calling monitor election 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: mon.a calling monitor election 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: monmap epoch 3 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: last_changed 2024-09-13T14:16:35.060163+0000 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: created 2024-09-13T14:14:57.332230+0000 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: min_mon_release 17 (quincy) 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: election_strategy: 1 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: 1: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.b 2024-09-13T14:27:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-09-13T14:27:56.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:27:56.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:27:56.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: mgrmap e38: y(active, since 29s), standbys: x 2024-09-13T14:27:56.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-mon[32895]: overall HEALTH_OK 2024-09-13T14:27:56.964 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:27:57.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 58851 58835 136.172 94.5625 0.0130341 0.00733791 2024-09-13T14:27:57.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:27:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:27:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:58.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 60280 60264 134.498 89.3125 0.0122032 0.00743028 2024-09-13T14:27:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:58 smithi019 ceph-mon[32895]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:58 smithi019 ceph-mon[32895]: pgmap v19: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:27:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:58 smithi019 ceph-mon[32895]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:27:58 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:58.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:58 smithi003 ceph-mon[107278]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:58.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:58 smithi003 ceph-mon[107278]: pgmap v19: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:27:58.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:58 smithi003 ceph-mon[107278]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:58.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:27:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:58.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:58 smithi003 ceph-mon[104007]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:58.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:58 smithi003 ceph-mon[104007]: pgmap v19: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:27:58.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:58 smithi003 ceph-mon[104007]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:58.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:27:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:59.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 61782 61766 133.097 93.875 0.00719377 0.00750832 2024-09-13T14:28:00.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 63252 63236 131.723 91.875 0.0115454 0.00758592 2024-09-13T14:28:00.173 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:28:00.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:00 smithi003 ceph-mon[104007]: pgmap v20: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-09-13T14:28:00.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:00 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1947519727' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:28:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:00 smithi003 ceph-mon[107278]: pgmap v20: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-09-13T14:28:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:00 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1947519727' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:28:00.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:28:00 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:28:00.099+0000 7f0f7ea9d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T14:28:00.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:00 smithi019 ceph-mon[32895]: pgmap v20: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-09-13T14:28:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:00 smithi019 ceph-mon[32895]: from='client.? 172.21.15.3:0/1947519727' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:28:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:01.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 64483 64467 129.956 76.9375 0.00764415 0.00768945 2024-09-13T14:28:02.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 65805 65789 128.476 82.625 0.00430423 0.0077783 2024-09-13T14:28:02.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:02 smithi003 ceph-mon[104007]: pgmap v21: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:28:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:02 smithi003 ceph-mon[107278]: pgmap v21: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:28:02.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:02 smithi019 ceph-mon[32895]: pgmap v21: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:28:03.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 67154 67138 127.138 84.3125 0.00868108 0.00785983 2024-09-13T14:28:04.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 68564 68548 125.99 88.125 0.003908 0.00793116 2024-09-13T14:28:04.360 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:28:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:28:04] "GET /metrics HTTP/1.1" 200 36230 "" "Prometheus/2.51.0" 2024-09-13T14:28:04.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:04 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:04 smithi019 ceph-mon[32895]: pgmap v22: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:28:04.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:28:04.680 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:04 smithi003 ceph-mon[107278]: pgmap v22: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:28:04.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:04 smithi003 ceph-mon[104007]: pgmap v22: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:28:05.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 69953 69937 124.861 86.8125 0.00920444 0.00800223 2024-09-13T14:28:05.152 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:06.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 71170 71154 123.506 76.0625 0.00370585 0.00809148 2024-09-13T14:28:06.167 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:06.167 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:06.167 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:06 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:06.168 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:06 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:06.168 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:06 smithi019 ceph-mon[32895]: pgmap v23: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-13T14:28:06.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:06.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:06.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:06 smithi003 ceph-mon[107278]: pgmap v23: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-13T14:28:06.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:06.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:06.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:06 smithi003 ceph-mon[104007]: pgmap v23: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-13T14:28:07.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 72455 72439 122.338 80.3125 0.0243252 0.00816838 2024-09-13T14:28:07.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:06 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:07.835 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:07 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:07.835 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:07 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:07.835 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:07 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:07.836 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:07 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:07.859 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:07 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:07.859 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:07 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:08.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 15 73674 73659 121.126 76.25 0.00919628 0.00825045 2024-09-13T14:28:08.671 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:28:08.671 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (2m) 3s ago 7m 19.9M - 0.27.0 11f11916f8cd b74b601869dc 2024-09-13T14:28:08.672 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi019 *:3000 running (2m) 35s ago 7m 75.1M - 9.4.7 954c08fa6188 1d1343f3e5db 2024-09-13T14:28:08.672 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.zmhiel smithi003 running (6m) 3s ago 6m 19.2M - 17.2.7-1627-gbbdadfa3 a994b00deb3a db8314ccc6fd 2024-09-13T14:28:08.672 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi019.ydtwbg smithi019 running (6m) 35s ago 6m 15.5M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c0fef505f410 2024-09-13T14:28:08.672 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi019 *:8443,9283,8765 running (66s) 35s ago 11m 450M - 19.3.0-4697-gd58f7306 fec07016a2b0 f0b9ab84e7f3 2024-09-13T14:28:08.672 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:8443,9283,8765 running (109s) 3s ago 13m 504M - 19.3.0-4697-gd58f7306 fec07016a2b0 e8e7986695ad 2024-09-13T14:28:08.672 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (48s) 3s ago 13m 42.2M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 265013e6136c 2024-09-13T14:28:08.673 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi019 running (11m) 35s ago 11m 52.2M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a b71aff50957a 2024-09-13T14:28:08.673 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (19s) 3s ago 11m 23.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 8539c77bb8fb 2024-09-13T14:28:08.673 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (2m) 3s ago 6m 9697k - 1.7.0 72c9c2088986 30d257fdd98b 2024-09-13T14:28:08.673 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi019 *:9100 running (2m) 35s ago 6m 9865k - 1.7.0 72c9c2088986 738f7b07d0d8 2024-09-13T14:28:08.673 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (11m) 3s ago 11m 318M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a b0e72d25bc0f 2024-09-13T14:28:08.673 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (10m) 3s ago 10m 227M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 612dd4f9bdda 2024-09-13T14:28:08.673 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (10m) 3s ago 10m 305M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a d32f24c0ca7e 2024-09-13T14:28:08.674 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (9m) 3s ago 9m 264M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 20984c58cba9 2024-09-13T14:28:08.674 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi019 running (9m) 35s ago 9m 281M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a e50c61f762b0 2024-09-13T14:28:08.674 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi019 running (9m) 35s ago 9m 306M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 1cd389248627 2024-09-13T14:28:08.674 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi019 running (8m) 35s ago 8m 170M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a aa95f3146e47 2024-09-13T14:28:08.674 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi019 running (8m) 35s ago 8m 202M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 118a8cf6eadc 2024-09-13T14:28:08.674 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi019 *:9095 running (74s) 35s ago 7m 43.7M - 2.51.0 1d3b7f56885b 0b10348ed2da 2024-09-13T14:28:08.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:08 smithi003 ceph-mon[104007]: pgmap v24: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-13T14:28:08.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:08 smithi003 ceph-mon[107278]: pgmap v24: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-13T14:28:09.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 74762 74746 119.762 67.9375 0.0143024 0.0083444 2024-09-13T14:28:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:08 smithi019 ceph-mon[32895]: pgmap v24: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-13T14:28:10.078 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:28:10.077786+0000 min lat: 0.00112889 max lat: 0.150445 avg lat: 0.0084024 2024-09-13T14:28:10.078 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:28:10.078 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 76150 76134 118.936 86.75 0.00455829 0.0084024 2024-09-13T14:28:10.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:09 smithi019 ceph-mon[32895]: from='client.25357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:10.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:09 smithi003 ceph-mon[107278]: from='client.25357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:10.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:09 smithi003 ceph-mon[104007]: from='client.25357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:10.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade pause' 2024-09-13T14:28:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:11.078 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 77401 77385 117.941 78.1875 0.0193554 0.00847345 2024-09-13T14:28:11.093 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:10 smithi003 ceph-mon[107278]: pgmap v25: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-09-13T14:28:11.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:10 smithi003 ceph-mon[104007]: pgmap v25: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-09-13T14:28:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:10 smithi019 ceph-mon[32895]: pgmap v25: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-09-13T14:28:12.078 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 78652 78636 116.995 78.1875 0.00589441 0.00854179 2024-09-13T14:28:12.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:12 smithi003 ceph-mon[104007]: pgmap v26: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:28:12.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:12.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:12 smithi003 ceph-mon[107278]: pgmap v26: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:28:12.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:13.078 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 79888 79872 116.07 77.25 0.011171 0.00861027 2024-09-13T14:28:13.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:12 smithi019 ceph-mon[32895]: pgmap v26: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:28:13.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:12 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:13.775 INFO:teuthology.orchestra.run.smithi003.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:14.078 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 81073 81057 115.116 74.0625 0.00626901 0.00868172 2024-09-13T14:28:14.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:14.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:28:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:28:14] "GET /metrics HTTP/1.1" 200 36223 "" "Prometheus/2.51.0" 2024-09-13T14:28:14.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:28:15.078 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 82362 82346 114.348 80.5625 0.0260819 0.00873933 2024-09-13T14:28:15.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:14 smithi003 ceph-mon[104007]: pgmap v27: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:28:15.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:14 smithi003 ceph-mon[104007]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:15.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:14 smithi003 ceph-mon[104007]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:15.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:14 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:15.085 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:14 smithi003 ceph-mon[107278]: pgmap v27: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:28:15.085 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:14 smithi003 ceph-mon[107278]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:15.085 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:14 smithi003 ceph-mon[107278]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:15.085 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:14 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:14 smithi019 ceph-mon[32895]: pgmap v27: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:28:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:14 smithi019 ceph-mon[32895]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:14 smithi019 ceph-mon[32895]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:14 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:15.559 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:16.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 83499 83483 113.406 71.0625 0.0216371 0.00881219 2024-09-13T14:28:16.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:16.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:15.742Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:16.885 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: pgmap v28: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:28:16.885 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.885 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.885 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:16.885 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: pgmap v28: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:16.886 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:16.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:16.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:16.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: pgmap v28: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:28:17.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 84751 84735 112.658 78.25 0.0130649 0.00887099 2024-09-13T14:28:17.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:18.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 86028 86012 111.974 79.8125 0.0160145 0.00892553 2024-09-13T14:28:18.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:18 smithi019 ceph-mon[32895]: pgmap v29: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T14:28:18.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:18 smithi003 ceph-mon[104007]: pgmap v29: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T14:28:18.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:18 smithi003 ceph-mon[107278]: pgmap v29: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T14:28:19.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 87298 87282 111.309 79.375 0.0127093 0.00897895 2024-09-13T14:28:20.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 88550 88534 110.647 78.25 0.0216399 0.009032 2024-09-13T14:28:20.491 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:20.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:20 smithi019 ceph-mon[32895]: pgmap v30: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:28:20.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:20 smithi003 ceph-mon[104007]: pgmap v30: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:28:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:20 smithi003 ceph-mon[107278]: pgmap v30: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:28:21.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 89820 89804 110.034 79.375 0.00654987 0.00908268 2024-09-13T14:28:22.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 91059 91043 109.407 77.4375 0.0229661 0.00913468 2024-09-13T14:28:22.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:22 smithi019 ceph-mon[32895]: pgmap v31: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:28:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:22 smithi003 ceph-mon[107278]: pgmap v31: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:28:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:22 smithi003 ceph-mon[104007]: pgmap v31: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:28:23.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 92308 92292 108.816 78.0625 0.00946717 0.00918423 2024-09-13T14:28:24.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 93533 93517 108.218 76.5625 0.00729016 0.00923501 2024-09-13T14:28:24.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:24.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:28:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:28:24] "GET /metrics HTTP/1.1" 200 36223 "" "Prometheus/2.51.0" 2024-09-13T14:28:24.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:24 smithi019 ceph-mon[32895]: pgmap v32: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:28:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:24 smithi003 ceph-mon[107278]: pgmap v32: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:28:24.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:24 smithi003 ceph-mon[104007]: pgmap v32: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:28:25.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 94731 94715 107.612 74.875 0.00956726 0.00928742 2024-09-13T14:28:26.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 95932 95916 107.031 75.0625 0.00705637 0.00933709 2024-09-13T14:28:26.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:26.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:25.742Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:26 smithi019 ceph-mon[32895]: pgmap v33: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:28:26.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:26 smithi003 ceph-mon[104007]: pgmap v33: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:28:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:26 smithi003 ceph-mon[107278]: pgmap v33: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:28:27.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 97116 97100 106.451 74 0.00552152 0.00938842 2024-09-13T14:28:27.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:28.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 98275 98259 105.865 72.4375 0.0166855 0.00944069 2024-09-13T14:28:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:28 smithi019 ceph-mon[32895]: pgmap v34: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:28:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:28 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:28.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:28 smithi003 ceph-mon[104007]: pgmap v34: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:28:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:28.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:28 smithi003 ceph-mon[107278]: pgmap v34: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:28:28.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:29.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 99440 99424 105.304 72.8125 0.00870532 0.00949068 2024-09-13T14:28:30.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:28:30.080040+0000 min lat: 0.00112889 max lat: 0.150445 avg lat: 0.00954133 2024-09-13T14:28:30.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:28:30.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 100595 100579 104.752 72.1875 0.0104126 0.00954133 2024-09-13T14:28:30.517 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:30.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:30 smithi019 ceph-mon[32895]: pgmap v35: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T14:28:30.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:30 smithi003 ceph-mon[107278]: pgmap v35: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T14:28:30.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:30 smithi003 ceph-mon[104007]: pgmap v35: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T14:28:31.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 101739 101723 104.207 71.5 0.00736822 0.0095911 2024-09-13T14:28:32.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 102884 102868 103.681 71.5625 0.0166794 0.00964001 2024-09-13T14:28:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:32 smithi019 ceph-mon[32895]: pgmap v36: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:28:32.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:32 smithi003 ceph-mon[107278]: pgmap v36: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:28:32.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:32 smithi003 ceph-mon[104007]: pgmap v36: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:28:33.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 104013 103997 103.155 70.5625 0.0219329 0.009689 2024-09-13T14:28:34.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 105130 105114 102.634 69.8125 0.0201302 0.00973804 2024-09-13T14:28:34.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:34.157Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:28:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:28:34] "GET /metrics HTTP/1.1" 200 36225 "" "Prometheus/2.51.0" 2024-09-13T14:28:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:34 smithi019 ceph-mon[32895]: pgmap v37: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:28:34.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:34 smithi003 ceph-mon[104007]: pgmap v37: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:28:34.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:34 smithi003 ceph-mon[107278]: pgmap v37: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:28:35.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 106257 106241 102.138 70.4375 0.0107488 0.00978554 2024-09-13T14:28:36.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 107348 107332 101.624 68.1875 0.0134108 0.00983507 2024-09-13T14:28:36.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:36.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:36.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:36 smithi019 ceph-mon[32895]: pgmap v38: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:28:36.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:36 smithi003 ceph-mon[107278]: pgmap v38: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:28:36.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:36 smithi003 ceph-mon[104007]: pgmap v38: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:28:37.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 108361 108345 101.052 63.3125 0.0160386 0.00989049 2024-09-13T14:28:37.328 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:38.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 109296 109280 100.425 58.4375 0.00903753 0.00995167 2024-09-13T14:28:38.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:38 smithi003 ceph-mon[104007]: pgmap v39: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-13T14:28:38.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:38 smithi003 ceph-mon[107278]: pgmap v39: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-13T14:28:38.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:38 smithi019 ceph-mon[32895]: pgmap v39: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-13T14:28:39.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 110283 110267 99.8639 61.6875 0.00937659 0.0100085 2024-09-13T14:28:40.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 111348 111332 99.3881 66.5625 0.0109219 0.0100562 2024-09-13T14:28:40.547 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:40.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:40 smithi019 ceph-mon[32895]: pgmap v40: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:28:40.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:40 smithi003 ceph-mon[107278]: pgmap v40: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:28:40.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:40 smithi003 ceph-mon[104007]: pgmap v40: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:28:41.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 112398 112382 98.9125 65.625 0.010065 0.0101042 2024-09-13T14:28:42.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 113455 113439 98.4562 66.0625 0.0128421 0.0101513 2024-09-13T14:28:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:42 smithi019 ceph-mon[32895]: pgmap v41: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s 2024-09-13T14:28:42.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:42 smithi003 ceph-mon[107278]: pgmap v41: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s 2024-09-13T14:28:42.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:42 smithi003 ceph-mon[104007]: pgmap v41: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s 2024-09-13T14:28:43.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 114455 114439 97.9636 62.5 0.0181784 0.0102027 2024-09-13T14:28:43.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:43 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:43.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:43.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:44.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 115497 115481 97.5198 65.125 0.0121279 0.010249 2024-09-13T14:28:44.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:44 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:44.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:28:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:28:44] "GET /metrics HTTP/1.1" 200 36223 "" "Prometheus/2.51.0" 2024-09-13T14:28:44.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:44 smithi019 ceph-mon[32895]: pgmap v42: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s 2024-09-13T14:28:44.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:44 smithi003 ceph-mon[104007]: pgmap v42: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s 2024-09-13T14:28:44.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:44 smithi003 ceph-mon[107278]: pgmap v42: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s 2024-09-13T14:28:45.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 116546 116530 97.0936 65.5625 0.0165884 0.0102942 2024-09-13T14:28:46.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 117582 117566 96.668 64.75 0.024474 0.0103393 2024-09-13T14:28:46.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:46.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:46 smithi019 ceph-mon[32895]: pgmap v43: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-13T14:28:46.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:46 smithi003 ceph-mon[104007]: pgmap v43: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-13T14:28:46.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:46 smithi003 ceph-mon[107278]: pgmap v43: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-13T14:28:47.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 118631 118615 96.264 65.5625 0.00650688 0.0103825 2024-09-13T14:28:47.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:48.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 119631 119615 95.831 62.5 0.0127302 0.0104301 2024-09-13T14:28:48.886 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:48 smithi019 ceph-mon[32895]: pgmap v44: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 54 MiB/s wr, 868 op/s 2024-09-13T14:28:48.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:48 smithi003 ceph-mon[104007]: pgmap v44: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 54 MiB/s wr, 868 op/s 2024-09-13T14:28:48.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:48 smithi003 ceph-mon[107278]: pgmap v44: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 54 MiB/s wr, 868 op/s 2024-09-13T14:28:49.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 120621 120605 95.4012 61.875 0.00985997 0.0104759 2024-09-13T14:28:50.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:28:50.081862+0000 min lat: 0.00112889 max lat: 0.150445 avg lat: 0.010522 2024-09-13T14:28:50.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:28:50.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 121619 121603 94.9883 62.375 0.0113486 0.010522 2024-09-13T14:28:50.587 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:50.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:50 smithi019 ceph-mon[32895]: pgmap v45: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:28:50.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:50 smithi003 ceph-mon[104007]: pgmap v45: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:28:50.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:50 smithi003 ceph-mon[107278]: pgmap v45: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:28:51.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 122602 122586 94.574 61.4375 0.0192988 0.0105681 2024-09-13T14:28:52.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 123575 123559 94.1623 60.8125 0.0183124 0.0106144 2024-09-13T14:28:52.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:52 smithi019 ceph-mon[32895]: pgmap v46: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-13T14:28:52.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:52 smithi003 ceph-mon[104007]: pgmap v46: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-13T14:28:52.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:52 smithi003 ceph-mon[107278]: pgmap v46: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-13T14:28:53.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 15 124568 124553 93.7762 62.125 0.015579 0.010658 2024-09-13T14:28:54.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 125537 125521 93.38 60.5 0.0122785 0.0107039 2024-09-13T14:28:54.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:28:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:28:54] "GET /metrics HTTP/1.1" 200 36223 "" "Prometheus/2.51.0" 2024-09-13T14:28:54.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:54 smithi019 ceph-mon[32895]: pgmap v47: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-13T14:28:54.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:54 smithi003 ceph-mon[104007]: pgmap v47: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-13T14:28:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:54 smithi003 ceph-mon[107278]: pgmap v47: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-13T14:28:55.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 126502 126486 92.991 60.3125 0.00849269 0.0107485 2024-09-13T14:28:56.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 127456 127440 92.6029 59.625 0.00601731 0.0107936 2024-09-13T14:28:56.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:55 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:56.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:55 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:56.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:56 smithi019 ceph-mon[32895]: pgmap v48: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-13T14:28:56.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:56 smithi003 ceph-mon[104007]: pgmap v48: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-13T14:28:56.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:56 smithi003 ceph-mon[107278]: pgmap v48: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-13T14:28:57.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 128414 128398 92.2267 59.875 0.0181286 0.0108374 2024-09-13T14:28:57.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:28:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:28:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:58.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 129356 129340 91.8477 58.875 0.0113384 0.0108825 2024-09-13T14:28:58.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:58 smithi019 ceph-mon[32895]: pgmap v49: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 53 MiB/s wr, 842 op/s 2024-09-13T14:28:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:28:58 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:58.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:58 smithi003 ceph-mon[104007]: pgmap v49: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 53 MiB/s wr, 842 op/s 2024-09-13T14:28:58.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:28:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:58 smithi003 ceph-mon[107278]: pgmap v49: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 53 MiB/s wr, 842 op/s 2024-09-13T14:28:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:28:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:59.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 130320 130304 91.4926 60.25 0.0142373 0.0109246 2024-09-13T14:29:00.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0079 2024-09-13T14:29:00.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 131266 2024-09-13T14:29:00.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T14:29:00.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T14:29:00.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 91.149 2024-09-13T14:29:00.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 35.8081 2024-09-13T14:29:00.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 214.312 2024-09-13T14:29:00.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 58.4375 2024-09-13T14:29:00.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1458 2024-09-13T14:29:00.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 572.93 2024-09-13T14:29:00.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3429 2024-09-13T14:29:00.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 935 2024-09-13T14:29:00.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0109666 2024-09-13T14:29:00.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.00687473 2024-09-13T14:29:00.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 0.150445 2024-09-13T14:29:00.085 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00112889 2024-09-13T14:29:00.088 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:29:00.615 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:00.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:00 smithi019 ceph-mon[32895]: pgmap v50: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T14:29:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:00 smithi019 ceph-mon[32895]: mgrmap e39: y(active, since 92s), standbys: x 2024-09-13T14:29:00.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:00 smithi003 ceph-mon[104007]: pgmap v50: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T14:29:00.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:00 smithi003 ceph-mon[104007]: mgrmap e39: y(active, since 92s), standbys: x 2024-09-13T14:29:00.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:00 smithi003 ceph-mon[107278]: pgmap v50: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T14:29:00.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:00 smithi003 ceph-mon[107278]: mgrmap e39: y(active, since 92s), standbys: x 2024-09-13T14:29:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:02 smithi019 ceph-mon[32895]: pgmap v51: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 810 op/s 2024-09-13T14:29:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:02 smithi003 ceph-mon[107278]: pgmap v51: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 810 op/s 2024-09-13T14:29:02.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:02 smithi003 ceph-mon[104007]: pgmap v51: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 810 op/s 2024-09-13T14:29:04.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:04 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:04.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:29:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:29:04] "GET /metrics HTTP/1.1" 200 36225 "" "Prometheus/2.51.0" 2024-09-13T14:29:04.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:04 smithi003 ceph-mon[104007]: pgmap v52: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 810 op/s 2024-09-13T14:29:04.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:04 smithi019 ceph-mon[32895]: pgmap v52: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 810 op/s 2024-09-13T14:29:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:04 smithi003 ceph-mon[107278]: pgmap v52: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 810 op/s 2024-09-13T14:29:06.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:06.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:06 smithi019 ceph-mon[32895]: pgmap v53: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 59 MiB/s wr, 1.21k op/s 2024-09-13T14:29:06.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:06 smithi003 ceph-mon[104007]: pgmap v53: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 59 MiB/s wr, 1.21k op/s 2024-09-13T14:29:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:06 smithi003 ceph-mon[107278]: pgmap v53: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 59 MiB/s wr, 1.21k op/s 2024-09-13T14:29:07.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:06 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:08.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:08 smithi019 ceph-mon[32895]: pgmap v54: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 802 op/s 2024-09-13T14:29:08.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:08 smithi003 ceph-mon[104007]: pgmap v54: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 802 op/s 2024-09-13T14:29:08.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:08 smithi003 ceph-mon[107278]: pgmap v54: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 802 op/s 2024-09-13T14:29:10.631 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:10.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:10 smithi019 ceph-mon[32895]: pgmap v55: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 1.21k op/s 2024-09-13T14:29:10.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:10 smithi003 ceph-mon[104007]: pgmap v55: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 1.21k op/s 2024-09-13T14:29:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:10 smithi003 ceph-mon[107278]: pgmap v55: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 1.21k op/s 2024-09-13T14:29:12.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:12 smithi019 ceph-mon[32895]: pgmap v56: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 8.6 MiB/s wr, 809 op/s 2024-09-13T14:29:12.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:12 smithi003 ceph-mon[104007]: pgmap v56: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 8.6 MiB/s wr, 809 op/s 2024-09-13T14:29:12.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:12 smithi003 ceph-mon[107278]: pgmap v56: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 8.6 MiB/s wr, 809 op/s 2024-09-13T14:29:13.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:13 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:13.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:13.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:14.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:14.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:29:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:29:14] "GET /metrics HTTP/1.1" 200 36224 "" "Prometheus/2.51.0" 2024-09-13T14:29:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:14 smithi019 ceph-mon[32895]: pgmap v57: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 8.6 MiB/s wr, 809 op/s 2024-09-13T14:29:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:14 smithi003 ceph-mon[104007]: pgmap v57: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 8.6 MiB/s wr, 809 op/s 2024-09-13T14:29:14.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:14 smithi003 ceph-mon[107278]: pgmap v57: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 8.6 MiB/s wr, 809 op/s 2024-09-13T14:29:16.181 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:16.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:16 smithi019 ceph-mon[32895]: pgmap v58: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 8.6 MiB/s wr, 1.21k op/s 2024-09-13T14:29:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:16 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:16 smithi003 ceph-mon[104007]: pgmap v58: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 8.6 MiB/s wr, 1.21k op/s 2024-09-13T14:29:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:16 smithi003 ceph-mon[107278]: pgmap v58: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 8.6 MiB/s wr, 1.21k op/s 2024-09-13T14:29:16.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:17.379 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:18.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade resume' 2024-09-13T14:29:18.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:18 smithi019 ceph-mon[32895]: pgmap v59: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 808 op/s 2024-09-13T14:29:18.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:18 smithi003 ceph-mon[104007]: pgmap v59: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 808 op/s 2024-09-13T14:29:18.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:18 smithi003 ceph-mon[107278]: pgmap v59: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 808 op/s 2024-09-13T14:29:19.742 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:20.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:21.058 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:20 smithi019 ceph-mon[32895]: pgmap v60: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-13T14:29:21.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:20 smithi003 ceph-mon[107278]: pgmap v60: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-13T14:29:21.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:20 smithi003 ceph-mon[104007]: pgmap v60: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-13T14:29:22.560 INFO:teuthology.orchestra.run.smithi003.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:22 smithi003 ceph-mon[104007]: pgmap v61: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 775 op/s 2024-09-13T14:29:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:22 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:22 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:22 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:22 smithi003 ceph-mon[107278]: pgmap v61: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 775 op/s 2024-09-13T14:29:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:22 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:22 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:22 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:22 smithi019 ceph-mon[32895]: pgmap v61: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 775 op/s 2024-09-13T14:29:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:22 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:22 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:22 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:23.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for all mons to upgrade' 2024-09-13T14:29:23.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:23 smithi003 ceph-mon[104007]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:23.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:23 smithi003 ceph-mon[104007]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:23.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:23 smithi003 ceph-mon[107278]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:23.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:23 smithi003 ceph-mon[107278]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:24.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:23 smithi019 ceph-mon[32895]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:24.150 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:23 smithi019 ceph-mon[32895]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:24.282 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:24.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:24.149Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:29:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:29:24] "GET /metrics HTTP/1.1" 200 36224 "" "Prometheus/2.51.0" 2024-09-13T14:29:25.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:24 smithi019 ceph-mon[32895]: pgmap v62: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 776 op/s 2024-09-13T14:29:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:24 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:24 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:24 smithi003 ceph-mon[107278]: pgmap v62: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 776 op/s 2024-09-13T14:29:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:24 smithi003 ceph-mon[104007]: pgmap v62: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 776 op/s 2024-09-13T14:29:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:25.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:25.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:25.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:25.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:29:25.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-13T14:29:26.118 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:26.119 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:26.119 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:25 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:26.119 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:25 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:26.119 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:25 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:26.119 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:25 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:26.119 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:25 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:26.119 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:25 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:29:26.119 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:25 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-13T14:29:26.229 INFO:teuthology.orchestra.run.smithi003.stdout:wait for all mons to upgrade 2024-09-13T14:29:26.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:26 smithi003 ceph-mon[104007]: pgmap v63: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-13T14:29:26.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:26 smithi003 ceph-mon[104007]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:29:26.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:26 smithi003 ceph-mon[107278]: pgmap v63: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-13T14:29:26.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:26 smithi003 ceph-mon[107278]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:29:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:26 smithi019 ceph-mon[32895]: pgmap v63: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-13T14:29:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:26 smithi019 ceph-mon[32895]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:29:26.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ! ceph mon versions | grep $sha1 | grep '"'"': 3'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-13T14:29:27.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:27.452 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:28.770 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[107278]: pgmap v64: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 726 op/s 2024-09-13T14:29:28.771 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[107278]: Upgrade: Updating mon.b 2024-09-13T14:29:28.771 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:28.771 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:29:28.771 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:29:28.771 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:28.771 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[107278]: Deploying daemon mon.b on smithi019 2024-09-13T14:29:28.771 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:28.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[104007]: pgmap v64: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 726 op/s 2024-09-13T14:29:28.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[104007]: Upgrade: Updating mon.b 2024-09-13T14:29:28.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:28.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:29:28.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:29:28.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:28.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[104007]: Deploying daemon mon.b on smithi019 2024-09-13T14:29:28.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:28 smithi019 ceph-mon[32895]: pgmap v64: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 726 op/s 2024-09-13T14:29:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:28 smithi019 ceph-mon[32895]: Upgrade: Updating mon.b 2024-09-13T14:29:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:28 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:28 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:29:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:28 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:29:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:28 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:28 smithi019 ceph-mon[32895]: Deploying daemon mon.b on smithi019 2024-09-13T14:29:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:28 smithi019 ceph-mon[32895]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:29 smithi019 systemd[1]: Stopping Ceph mon.b for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:29:30.552 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:29:30.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b[32864]: 2024-09-13T14:29:30.207+0000 7f69dfd6e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:29:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b[32864]: 2024-09-13T14:29:30.208+0000 7f69dfd6e640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-13T14:29:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:30 smithi019 podman[91117]: 2024-09-13 14:29:30.383175235 +0000 UTC m=+0.308419348 container died b71aff50957ad91f07d8864d6d8dca25d614cdf3bc18eb80b00733cc68f35535 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-13T14:29:31.064 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:30 smithi019 podman[91117]: 2024-09-13 14:29:30.795219492 +0000 UTC m=+0.720463603 container cleanup b71aff50957ad91f07d8864d6d8dca25d614cdf3bc18eb80b00733cc68f35535 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:29:31.064 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:30 smithi019 bash[91117]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b 2024-09-13T14:29:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:31 smithi019 podman[91136]: 2024-09-13 14:29:31.063522378 +0000 UTC m=+0.677931630 container remove b71aff50957ad91f07d8864d6d8dca25d614cdf3bc18eb80b00733cc68f35535 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:29:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:31 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-13T14:29:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:31 smithi019 systemd[1]: Stopped Ceph mon.b for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:29:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:31 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.b.service: Consumed 11.193s CPU time. 2024-09-13T14:29:31.880 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:31 smithi019 systemd[1]: Starting Ceph mon.b for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:29:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:31 smithi019 podman[91259]: 2024-09-13 14:29:31.785771318 +0000 UTC m=+0.029196125 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:31 smithi019 podman[91259]: 2024-09-13 14:29:31.898663018 +0000 UTC m=+0.142087821 container create 301a92c4f814e9b5af3e5b323e79f4dcfda970d2191a32a6e53ad69098662246 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731) 2024-09-13T14:29:32.477 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 podman[91259]: 2024-09-13 14:29:32.424267305 +0000 UTC m=+0.667692109 container init 301a92c4f814e9b5af3e5b323e79f4dcfda970d2191a32a6e53ad69098662246 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 podman[91259]: 2024-09-13 14:29:32.434714062 +0000 UTC m=+0.678138862 container start 301a92c4f814e9b5af3e5b323e79f4dcfda970d2191a32a6e53ad69098662246 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 6 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: pidfile_write: ignore empty --pid-file 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: load: jerasure load: lrc 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Git sha 0 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: DB SUMMARY 2024-09-13T14:29:32.478 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: DB Session ID: 3WYIVUT8H9G795FOUJFK 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1104 Bytes 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 7080898 ; 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.env: 0x5565ca8a5160 2024-09-13T14:29:32.479 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:29:32.480 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.info_log: 0x5565ccd3b880 2024-09-13T14:29:32.480 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:29:32.480 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.statistics: (nil) 2024-09-13T14:29:32.480 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.use_fsync: 0 2024-09-13T14:29:32.480 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:29:32.480 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.db_log_dir: 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.wal_dir: 2024-09-13T14:29:32.481 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.write_buffer_manager: 0x5565ccd3ee60 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:29:32.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.unordered_write: 0 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.row_cache: None 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.wal_filter: None 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:29:32.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.wal_compression: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:29:32.484 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:29:32.485 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_open_files: -1 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Compression algorithms supported: 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: kZSTD supported: 0 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:29:32.486 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-09-13T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.merge_operator: 2024-09-13T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_filter: None 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5565ccd3b4e0) 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout: index_type: 0 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout: data_block_index_type: 0 2024-09-13T14:29:32.488 INFO:journalctl@ceph.mon.b.smithi019.stdout: index_shortening: 1 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: checksum: 4 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: no_block_cache: 0 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: block_cache: 0x5565ccd2e850 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: block_cache_options: 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: capacity : 536870912 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: num_shard_bits : 4 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: strict_capacity_limit : 0 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: block_cache_compressed: (nil) 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: persistent_cache: (nil) 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: block_size: 4096 2024-09-13T14:29:32.489 INFO:journalctl@ceph.mon.b.smithi019.stdout: block_size_deviation: 10 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: block_restart_interval: 16 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: index_block_restart_interval: 1 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: metadata_block_size: 4096 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: partition_filters: 0 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: use_delta_encoding: 1 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: filter_policy: bloomfilter 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: whole_key_filtering: 1 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: verify_compression: 0 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: format_version: 5 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: enable_index_compression: 1 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: block_align: 0 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: max_auto_readahead_size: 262144 2024-09-13T14:29:32.490 INFO:journalctl@ceph.mon.b.smithi019.stdout: prepopulate_block_cache: 0 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression: NoCompression 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.num_levels: 7 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:29:32.491 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:29:32.492 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:29:32.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:29:32.494 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:29:32.495 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.ttl: 2592000 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:29:32.496 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12826, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d0141353-001a-4827-87e7-5a24f4c838f4 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237772476123, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-13T14:29:32.497 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-13T14:29:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237772501184, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4382292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12823, "largest_seqno": 15605, "table_properties": {"data_size": 4371007, "index_size": 7102, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3141, "raw_key_size": 26464, "raw_average_key_size": 21, "raw_value_size": 4346565, "raw_average_value_size": 3516, "num_data_blocks": 321, "num_entries": 1236, "num_filter_entries": 1236, "num_deletions": 253, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237772, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d0141353-001a-4827-87e7-5a24f4c838f4", "db_session_id": "3WYIVUT8H9G795FOUJFK", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:29:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237772501362, "job": 1, "event": "recovery_finished"} 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5565ccd5ee00 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: DB pointer 0x5565cce5e000 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: ** DB Stats ** 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:29:32.683 INFO:journalctl@ceph.mon.b.smithi019.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: ** Compaction Stats [default] ** 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: L0 1/0 4.18 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 388.1 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: L6 1/0 7.05 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: Sum 2/0 11.23 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 388.1 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-09-13T14:29:32.684 INFO:journalctl@ceph.mon.b.smithi019.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 388.1 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: ** Compaction Stats [default] ** 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 388.1 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:29:32.685 INFO:journalctl@ceph.mon.b.smithi019.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout: Cumulative compaction: 0.00 GB write, 129.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout: Interval compaction: 0.00 GB write, 129.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout: Block cache BinnedLRUCache@0x5565ccd2e850#6 capacity: 512.00 MB usage: 60.70 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout: Block cache entry stats(count,size,portion): DataBlock(3,24.23 KB,0.00462234%) FilterBlock(2,12.31 KB,0.00234842%) IndexBlock(2,24.16 KB,0.00460744%) Misc(1,0.00 KB,0%) 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout: 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: starting mon.b rank 1 at public addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] at bind addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: mon.b@-1(???) e3 preinit fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: mon.b@-1(???).mds e9 new map 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: mon.b@-1(???).mds e9 print_map 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout: e9 2024-09-13T14:29:32.686 INFO:journalctl@ceph.mon.b.smithi019.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: legacy client fscid: 1 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: Filesystem 'foo' (1) 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: fs_name foo 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: epoch 8 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: created 2024-09-13T14:21:52.613871+0000 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: modified 2024-09-13T14:22:03.666371+0000 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: tableserver 0 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: root 0 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: session_timeout 60 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: session_autoclose 300 2024-09-13T14:29:32.687 INFO:journalctl@ceph.mon.b.smithi019.stdout: max_file_size 1099511627776 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: max_xattr_size 65536 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: required_client_features {} 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: last_failure 0 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: last_failure_osd_epoch 0 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: max_mds 1 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: in 0 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: up {0=14646} 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: failed 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: damaged 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: stopped 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: data_pools [4] 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: metadata_pool 3 2024-09-13T14:29:32.688 INFO:journalctl@ceph.mon.b.smithi019.stdout: inline_data disabled 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: balancer 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: bal_rank_mask -1 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: standby_count_wanted 1 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: [mds.foo.smithi003.zmhiel{0:14646} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.3:6834/1473925488,v1:172.21.15.3:6835/1473925488] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: Standby daemons: 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout: [mds.foo.smithi019.ydtwbg{-1:24520} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.19:6832/4291362241,v1:172.21.15.19:6833/4291362241] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: mon.b@-1(???).osd e79 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: mon.b@-1(???).osd e79 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: mon.b@-1(???).osd e79 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:29:32.689 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: mon.b@-1(???).osd e79 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:29:32.690 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 ceph-mon[91296]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-09-13T14:29:32.690 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 bash[91259]: 301a92c4f814e9b5af3e5b323e79f4dcfda970d2191a32a6e53ad69098662246 2024-09-13T14:29:32.690 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:32 smithi019 systemd[1]: Started Ceph mon.b for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:29:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:29:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:33 smithi019 ceph-mon[91296]: pgmap v65: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:29:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:33 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1239096733' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:29:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:33 smithi003 ceph-mon[107278]: pgmap v65: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:29:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:33 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1239096733' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:29:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:33 smithi003 ceph-mon[104007]: pgmap v65: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:29:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:33 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1239096733' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:29:34.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:34.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:29:34] "GET /metrics HTTP/1.1" 200 36224 "" "Prometheus/2.51.0" 2024-09-13T14:29:34.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:29:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: pgmap v67: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: mon.b calling monitor election 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: mon.a calling monitor election 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: mon.c calling monitor election 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: monmap epoch 4 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: last_changed 2024-09-13T14:29:33.613721+0000 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: created 2024-09-13T14:14:57.332230+0000 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: min_mon_release 19 (squid) 2024-09-13T14:29:34.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: election_strategy: 1 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: 1: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.b 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: mgrmap e39: y(active, since 2m), standbys: x 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: overall HEALTH_OK 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2535789076' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:34.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: pgmap v67: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: mon.b calling monitor election 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: mon.a calling monitor election 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: mon.c calling monitor election 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: monmap epoch 4 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:29:34.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: last_changed 2024-09-13T14:29:33.613721+0000 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: created 2024-09-13T14:14:57.332230+0000 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: min_mon_release 19 (squid) 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: election_strategy: 1 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: 1: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.b 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: mgrmap e39: y(active, since 2m), standbys: x 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: overall HEALTH_OK 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:34.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2535789076' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: pgmap v67: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: mon.b calling monitor election 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: mon.a calling monitor election 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: mon.c calling monitor election 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:29:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: monmap epoch 4 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: last_changed 2024-09-13T14:29:33.613721+0000 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: created 2024-09-13T14:14:57.332230+0000 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: min_mon_release 19 (squid) 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: election_strategy: 1 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: 1: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.b 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: fsmap foo:1 {0=foo.smithi003.zmhiel=up:active} 1 up:standby 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: mgrmap e39: y(active, since 2m), standbys: x 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: overall HEALTH_OK 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:34.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2535789076' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:29:34.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:34.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:35.931 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:35.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:36.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:36 smithi003 ceph-mon[107278]: pgmap v68: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T14:29:36.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:36 smithi003 ceph-mon[104007]: pgmap v68: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T14:29:36.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:36 smithi019 ceph-mon[91296]: pgmap v68: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T14:29:37.328 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:37.829 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:29:37 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:29:37.528+0000 7f0f7ea9d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (4m) 14s ago 8m 19.9M - 0.27.0 11f11916f8cd b74b601869dc 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi019 *:3000 running (3m) 15s ago 8m 81.9M - 9.4.7 954c08fa6188 1d1343f3e5db 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.zmhiel smithi003 running (7m) 14s ago 7m 19.6M - 17.2.7-1627-gbbdadfa3 a994b00deb3a db8314ccc6fd 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi019.ydtwbg smithi019 running (7m) 15s ago 7m 15.5M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c0fef505f410 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi019 *:8443,9283,8765 running (2m) 15s ago 12m 453M - 19.3.0-4697-gd58f7306 fec07016a2b0 f0b9ab84e7f3 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:8443,9283,8765 running (3m) 14s ago 14m 508M - 19.3.0-4697-gd58f7306 fec07016a2b0 e8e7986695ad 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (2m) 14s ago 14m 44.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 265013e6136c 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi019 starting - - - 2048M 2024-09-13T14:29:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (108s) 14s ago 13m 35.3M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 8539c77bb8fb 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (4m) 14s ago 8m 11.7M - 1.7.0 72c9c2088986 30d257fdd98b 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi019 *:9100 running (3m) 15s ago 8m 11.4M - 1.7.0 72c9c2088986 738f7b07d0d8 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (12m) 14s ago 12m 374M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a b0e72d25bc0f 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (12m) 14s ago 12m 326M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 612dd4f9bdda 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (11m) 14s ago 11m 455M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a d32f24c0ca7e 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (11m) 14s ago 11m 476M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 20984c58cba9 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi019 running (10m) 15s ago 10m 398M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a e50c61f762b0 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi019 running (10m) 15s ago 10m 431M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 1cd389248627 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi019 running (10m) 15s ago 10m 302M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a aa95f3146e47 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi019 running (9m) 15s ago 9m 379M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 118a8cf6eadc 2024-09-13T14:29:38.391 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi019 *:9095 running (2m) 15s ago 9m 46.4M - 2.51.0 1d3b7f56885b 0b10348ed2da 2024-09-13T14:29:38.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:38 smithi003 ceph-mon[107278]: pgmap v69: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-13T14:29:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:38 smithi003 ceph-mon[104007]: pgmap v69: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-13T14:29:38.802 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:38 smithi019 ceph-mon[91296]: pgmap v69: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-13T14:29:39.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade pause' 2024-09-13T14:29:39.629 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:39 smithi019 ceph-mon[91296]: from='client.54103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:39.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:39 smithi003 ceph-mon[107278]: from='client.54103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:39.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:39 smithi003 ceph-mon[104007]: from='client.54103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:40.656 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:40 smithi019 ceph-mon[91296]: pgmap v70: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:29:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:40 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:40 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:40.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:40 smithi003 ceph-mon[104007]: pgmap v70: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:29:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:40 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:40 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:40 smithi003 ceph-mon[107278]: pgmap v70: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:29:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:40 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:40 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:42.420 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:42.420 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:42.420 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:42 smithi019 ceph-mon[91296]: pgmap v71: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-13T14:29:42.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:42.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:42.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:42 smithi003 ceph-mon[104007]: pgmap v71: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-13T14:29:42.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:42.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:42.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:42 smithi003 ceph-mon[107278]: pgmap v71: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-13T14:29:43.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:43.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:43.748 INFO:teuthology.orchestra.run.smithi003.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:44.399 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:44 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:44.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:29:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:29:44] "GET /metrics HTTP/1.1" 200 36235 "" "Prometheus/2.51.0" 2024-09-13T14:29:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:44 smithi019 ceph-mon[91296]: pgmap v72: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-13T14:29:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:44 smithi019 ceph-mon[91296]: from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:44.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:44 smithi019 ceph-mon[91296]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:44.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:44 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:44 smithi003 ceph-mon[107278]: pgmap v72: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-13T14:29:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:44 smithi003 ceph-mon[107278]: from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:44 smithi003 ceph-mon[107278]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:44 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:44.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:44 smithi003 ceph-mon[104007]: pgmap v72: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-13T14:29:44.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:44 smithi003 ceph-mon[104007]: from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:44.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:44 smithi003 ceph-mon[104007]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:44.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:44 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:45.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:29:45.547 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:45.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:45.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:45 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[104007]: pgmap v73: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:29:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:29:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:29:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:46.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[107278]: pgmap v73: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:29:46.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:46.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:46.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:46.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:46.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:46.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:29:46.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:29:46.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:46 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:47.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-mon[91296]: pgmap v73: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:29:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:47.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:29:47.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:29:47.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:46 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:47.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:47 smithi003 ceph-mon[107278]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:29:47.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:47 smithi003 ceph-mon[107278]: Reconfiguring daemon mon.a on smithi003 2024-09-13T14:29:47.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:47 smithi003 ceph-mon[104007]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:29:47.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:47 smithi003 ceph-mon[104007]: Reconfiguring daemon mon.a on smithi003 2024-09-13T14:29:48.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:47 smithi019 ceph-mon[91296]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:29:48.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:47 smithi019 ceph-mon[91296]: Reconfiguring daemon mon.a on smithi003 2024-09-13T14:29:48.902 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:48 smithi003 ceph-mon[107278]: pgmap v74: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 682 op/s 2024-09-13T14:29:48.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:48 smithi003 ceph-mon[104007]: pgmap v74: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 682 op/s 2024-09-13T14:29:49.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:48 smithi019 ceph-mon[91296]: pgmap v74: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 682 op/s 2024-09-13T14:29:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:50 smithi019 ceph-mon[91296]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:29:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:29:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:29:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:50 smithi019 ceph-mon[91296]: Reconfiguring daemon mgr.y on smithi003 2024-09-13T14:29:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:50 smithi019 ceph-mon[91296]: pgmap v75: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:29:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[107278]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:29:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:29:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:29:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[107278]: Reconfiguring daemon mgr.y on smithi003 2024-09-13T14:29:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[107278]: pgmap v75: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:29:50.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:50.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:50.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[104007]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:29:50.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:29:50.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:29:50.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:50.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[104007]: Reconfiguring daemon mgr.y on smithi003 2024-09-13T14:29:50.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:50 smithi003 ceph-mon[104007]: pgmap v75: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:29:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:52 smithi019 ceph-mon[91296]: pgmap v76: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 689 op/s 2024-09-13T14:29:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:52.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:52 smithi019 ceph-mon[91296]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:29:52.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:29:52.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:29:52.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:52.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:52 smithi019 ceph-mon[91296]: Reconfiguring daemon mon.c on smithi003 2024-09-13T14:29:52.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[107278]: pgmap v76: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 689 op/s 2024-09-13T14:29:52.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:52.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:52.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[107278]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:29:52.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:29:52.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:29:52.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:52.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[107278]: Reconfiguring daemon mon.c on smithi003 2024-09-13T14:29:52.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[104007]: pgmap v76: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 689 op/s 2024-09-13T14:29:52.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:52.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:52.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[104007]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:29:52.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:29:52.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:29:52.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:52.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:52 smithi003 ceph-mon[104007]: Reconfiguring daemon mon.c on smithi003 2024-09-13T14:29:54.352 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:29:54] "GET /metrics HTTP/1.1" 200 36235 "" "Prometheus/2.51.0" 2024-09-13T14:29:54.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:54.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:54 smithi019 ceph-mon[91296]: pgmap v77: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 689 op/s 2024-09-13T14:29:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:54 smithi019 ceph-mon[91296]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:29:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:29:54.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:54.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:54 smithi019 ceph-mon[91296]: Reconfiguring daemon osd.0 on smithi003 2024-09-13T14:29:54.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[107278]: pgmap v77: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 689 op/s 2024-09-13T14:29:54.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[107278]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[107278]: Reconfiguring daemon osd.0 on smithi003 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[104007]: pgmap v77: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 689 op/s 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[104007]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:29:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:29:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:54 smithi003 ceph-mon[104007]: Reconfiguring daemon osd.0 on smithi003 2024-09-13T14:29:56.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:55 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:56.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:55 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:56 smithi019 ceph-mon[91296]: pgmap v78: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:29:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:56 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:56 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:56 smithi019 ceph-mon[91296]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:29:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:56 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:29:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:56 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:56 smithi019 ceph-mon[91296]: Reconfiguring daemon osd.1 on smithi003 2024-09-13T14:29:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[107278]: pgmap v78: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:29:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[107278]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:29:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:29:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[107278]: Reconfiguring daemon osd.1 on smithi003 2024-09-13T14:29:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[104007]: pgmap v78: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T14:29:56.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:56.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:56.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[104007]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:29:56.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:29:56.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:56.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:56 smithi003 ceph-mon[104007]: Reconfiguring daemon osd.1 on smithi003 2024-09-13T14:29:57.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:29:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:29:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:58 smithi019 ceph-mon[91296]: pgmap v79: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-09-13T14:29:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:58 smithi019 ceph-mon[91296]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:29:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:29:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:58 smithi019 ceph-mon[91296]: Reconfiguring daemon osd.2 on smithi003 2024-09-13T14:29:58.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:29:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[107278]: pgmap v79: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-09-13T14:29:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[107278]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:29:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:29:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[107278]: Reconfiguring daemon osd.2 on smithi003 2024-09-13T14:29:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[104007]: pgmap v79: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-09-13T14:29:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:29:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[104007]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:29:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:29:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[104007]: Reconfiguring daemon osd.2 on smithi003 2024-09-13T14:29:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:29:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:00.567 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[104007]: pgmap v80: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:30:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[104007]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:30:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:30:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[104007]: Reconfiguring daemon osd.3 on smithi003 2024-09-13T14:30:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[104007]: overall HEALTH_OK 2024-09-13T14:30:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[107278]: pgmap v80: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:30:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[107278]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:30:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:30:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[107278]: Reconfiguring daemon osd.3 on smithi003 2024-09-13T14:30:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:00 smithi003 ceph-mon[107278]: overall HEALTH_OK 2024-09-13T14:30:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:00 smithi019 ceph-mon[91296]: pgmap v80: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:30:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:00 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:00 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:00 smithi019 ceph-mon[91296]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:30:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:00 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:30:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:00 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:00.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:00 smithi019 ceph-mon[91296]: Reconfiguring daemon osd.3 on smithi003 2024-09-13T14:30:00.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:00 smithi019 ceph-mon[91296]: overall HEALTH_OK 2024-09-13T14:30:02.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:02 smithi019 ceph-mon[91296]: pgmap v81: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 705 op/s 2024-09-13T14:30:02.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:02 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:02.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:02 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:02.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:02 smithi019 ceph-mon[91296]: Reconfiguring mds.foo.smithi003.zmhiel (monmap changed)... 2024-09-13T14:30:02.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:02 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.zmhiel", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:30:02.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:02 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:02.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:02 smithi019 ceph-mon[91296]: Reconfiguring daemon mds.foo.smithi003.zmhiel on smithi003 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[107278]: pgmap v81: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 705 op/s 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[107278]: Reconfiguring mds.foo.smithi003.zmhiel (monmap changed)... 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.zmhiel", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[107278]: Reconfiguring daemon mds.foo.smithi003.zmhiel on smithi003 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[104007]: pgmap v81: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 705 op/s 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:02.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[104007]: Reconfiguring mds.foo.smithi003.zmhiel (monmap changed)... 2024-09-13T14:30:02.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.zmhiel", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:30:02.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:02.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:02 smithi003 ceph-mon[104007]: Reconfiguring daemon mds.foo.smithi003.zmhiel on smithi003 2024-09-13T14:30:04.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:04 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:30:04] "GET /metrics HTTP/1.1" 200 36234 "" "Prometheus/2.51.0" 2024-09-13T14:30:04.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:04 smithi019 ceph-mon[91296]: pgmap v82: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 705 op/s 2024-09-13T14:30:04.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:04.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:04.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:04 smithi019 ceph-mon[91296]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:30:04.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:30:04.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:30:04.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:04.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:04 smithi019 ceph-mon[91296]: Reconfiguring daemon mon.b on smithi019 2024-09-13T14:30:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[107278]: pgmap v82: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 705 op/s 2024-09-13T14:30:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[107278]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:30:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:30:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:30:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[107278]: Reconfiguring daemon mon.b on smithi019 2024-09-13T14:30:04.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[104007]: pgmap v82: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 705 op/s 2024-09-13T14:30:04.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:04.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:04.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[104007]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:30:04.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:30:04.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:30:04.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:04.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:04 smithi003 ceph-mon[104007]: Reconfiguring daemon mon.b on smithi019 2024-09-13T14:30:06.020 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:05.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:06.020 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:06 smithi019 ceph-mon[91296]: pgmap v83: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T14:30:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:06 smithi019 ceph-mon[91296]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:30:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:30:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:30:06.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:06.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:06 smithi019 ceph-mon[91296]: Reconfiguring daemon mgr.x on smithi019 2024-09-13T14:30:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[107278]: pgmap v83: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T14:30:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[107278]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:30:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:30:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:30:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[107278]: Reconfiguring daemon mgr.x on smithi019 2024-09-13T14:30:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[104007]: pgmap v83: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T14:30:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[104007]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:30:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:30:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:30:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:06.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:06 smithi003 ceph-mon[104007]: Reconfiguring daemon mgr.x on smithi019 2024-09-13T14:30:07.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:06 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:08.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[107278]: pgmap v84: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 708 op/s 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[107278]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[107278]: Reconfiguring daemon osd.4 on smithi019 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[104007]: pgmap v84: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 708 op/s 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:08.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[104007]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:30:08.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:30:08.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:08.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:08 smithi003 ceph-mon[104007]: Reconfiguring daemon osd.4 on smithi019 2024-09-13T14:30:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:08 smithi019 ceph-mon[91296]: pgmap v84: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 708 op/s 2024-09-13T14:30:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:08 smithi019 ceph-mon[91296]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:30:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:30:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:08 smithi019 ceph-mon[91296]: Reconfiguring daemon osd.4 on smithi019 2024-09-13T14:30:10.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:10 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:10.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:10 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:10.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:10 smithi019 ceph-mon[91296]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:30:10.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:10 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:30:10.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:10 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:10.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:10 smithi019 ceph-mon[91296]: Reconfiguring daemon osd.5 on smithi019 2024-09-13T14:30:10.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:10 smithi019 ceph-mon[91296]: pgmap v85: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:30:10.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[107278]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:30:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:30:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[107278]: Reconfiguring daemon osd.5 on smithi019 2024-09-13T14:30:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[107278]: pgmap v85: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:30:10.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:10.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:10.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[104007]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:30:10.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:30:10.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:10.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[104007]: Reconfiguring daemon osd.5 on smithi019 2024-09-13T14:30:10.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:10 smithi003 ceph-mon[104007]: pgmap v85: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:30:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:12 smithi019 ceph-mon[91296]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:30:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:30:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:12 smithi019 ceph-mon[91296]: Reconfiguring daemon osd.6 on smithi019 2024-09-13T14:30:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:12 smithi019 ceph-mon[91296]: pgmap v86: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-13T14:30:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[107278]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:30:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:30:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[107278]: Reconfiguring daemon osd.6 on smithi019 2024-09-13T14:30:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[107278]: pgmap v86: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-13T14:30:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:12.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:12.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[104007]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:30:12.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:30:12.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:12.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[104007]: Reconfiguring daemon osd.6 on smithi019 2024-09-13T14:30:12.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:12 smithi003 ceph-mon[104007]: pgmap v86: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-13T14:30:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:30:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:30:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:14.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:14 smithi019 ceph-mon[91296]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:30:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:14 smithi019 ceph-mon[91296]: Reconfiguring daemon osd.7 on smithi019 2024-09-13T14:30:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:14 smithi019 ceph-mon[91296]: pgmap v87: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-13T14:30:14.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:14 smithi003 ceph-mon[104007]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:30:14.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:14 smithi003 ceph-mon[104007]: Reconfiguring daemon osd.7 on smithi019 2024-09-13T14:30:14.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:14 smithi003 ceph-mon[104007]: pgmap v87: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-13T14:30:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:30:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:30:14] "GET /metrics HTTP/1.1" 200 36227 "" "Prometheus/2.51.0" 2024-09-13T14:30:14.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:14 smithi003 ceph-mon[107278]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:30:14.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:14 smithi003 ceph-mon[107278]: Reconfiguring daemon osd.7 on smithi019 2024-09-13T14:30:14.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:14 smithi003 ceph-mon[107278]: pgmap v87: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-13T14:30:15.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:15.932 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:16 smithi019 ceph-mon[91296]: pgmap v88: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:30:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:16 smithi019 ceph-mon[91296]: Reconfiguring mds.foo.smithi019.ydtwbg (monmap changed)... 2024-09-13T14:30:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi019.ydtwbg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:30:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:16 smithi019 ceph-mon[91296]: Reconfiguring daemon mds.foo.smithi019.ydtwbg on smithi019 2024-09-13T14:30:17.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[104007]: pgmap v88: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:30:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[104007]: Reconfiguring mds.foo.smithi019.ydtwbg (monmap changed)... 2024-09-13T14:30:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi019.ydtwbg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:30:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[104007]: Reconfiguring daemon mds.foo.smithi019.ydtwbg on smithi019 2024-09-13T14:30:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[107278]: pgmap v88: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:30:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:17.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[107278]: Reconfiguring mds.foo.smithi019.ydtwbg (monmap changed)... 2024-09-13T14:30:17.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi019.ydtwbg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:30:17.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:17.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:16 smithi003 ceph-mon[107278]: Reconfiguring daemon mds.foo.smithi019.ydtwbg on smithi019 2024-09-13T14:30:17.199 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:18.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[104007]: pgmap v89: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 720 op/s 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[107278]: pgmap v89: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 720 op/s 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:18.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:18.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:18.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:18.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:18.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:18 smithi019 ceph-mon[91296]: pgmap v89: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 720 op/s 2024-09-13T14:30:18.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:18.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:18.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:18.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:18.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:18.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:20.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:20 smithi019 ceph-mon[91296]: pgmap v90: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:30:20.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:20 smithi003 ceph-mon[104007]: pgmap v90: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:30:20.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:20 smithi003 ceph-mon[107278]: pgmap v90: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:30:22.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:22 smithi019 ceph-mon[91296]: pgmap v91: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 724 op/s 2024-09-13T14:30:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:22 smithi003 ceph-mon[104007]: pgmap v91: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 724 op/s 2024-09-13T14:30:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:22 smithi003 ceph-mon[107278]: pgmap v91: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 724 op/s 2024-09-13T14:30:24.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:24.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:30:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:30:24] "GET /metrics HTTP/1.1" 200 36227 "" "Prometheus/2.51.0" 2024-09-13T14:30:24.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:24 smithi019 ceph-mon[91296]: pgmap v92: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 724 op/s 2024-09-13T14:30:24.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:24 smithi003 ceph-mon[104007]: pgmap v92: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 724 op/s 2024-09-13T14:30:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:24 smithi003 ceph-mon[107278]: pgmap v92: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 724 op/s 2024-09-13T14:30:26.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:26.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:26 smithi019 ceph-mon[91296]: pgmap v93: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-13T14:30:26.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:26 smithi003 ceph-mon[104007]: pgmap v93: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-13T14:30:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:26 smithi003 ceph-mon[107278]: pgmap v93: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-13T14:30:27.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:28.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:28 smithi019 ceph-mon[91296]: pgmap v94: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T14:30:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:28 smithi003 ceph-mon[107278]: pgmap v94: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T14:30:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:28.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:28 smithi003 ceph-mon[104007]: pgmap v94: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T14:30:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:30.470 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:30.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:30 smithi019 ceph-mon[91296]: pgmap v95: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-13T14:30:30.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:30 smithi003 ceph-mon[107278]: pgmap v95: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-13T14:30:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:30 smithi003 ceph-mon[104007]: pgmap v95: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-13T14:30:32.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:32 smithi019 ceph-mon[91296]: pgmap v96: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-09-13T14:30:32.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:32 smithi003 ceph-mon[104007]: pgmap v96: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-09-13T14:30:32.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:32 smithi003 ceph-mon[107278]: pgmap v96: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-09-13T14:30:34.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:34.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:30:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:30:34] "GET /metrics HTTP/1.1" 200 36236 "" "Prometheus/2.51.0" 2024-09-13T14:30:34.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:34 smithi019 ceph-mon[91296]: pgmap v97: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-09-13T14:30:34.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:34 smithi003 ceph-mon[104007]: pgmap v97: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-09-13T14:30:34.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:34 smithi003 ceph-mon[107278]: pgmap v97: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-09-13T14:30:36.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:36.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:36.734 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:36 smithi003 ceph-mon[107278]: pgmap v98: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:30:36.735 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:36 smithi003 ceph-mon[104007]: pgmap v98: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:30:36.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:36 smithi019 ceph-mon[91296]: pgmap v98: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:30:37.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi003\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi003\", job=\"node\", machine=\"x86_64\", nodename=\"smithi003\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:38.641 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:38 smithi019 ceph-mon[91296]: pgmap v99: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-13T14:30:38.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:38 smithi003 ceph-mon[107278]: pgmap v99: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-13T14:30:38.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:38 smithi003 ceph-mon[104007]: pgmap v99: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-13T14:30:40.492 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:40 smithi019 ceph-mon[91296]: pgmap v100: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:30:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:40 smithi003 ceph-mon[104007]: pgmap v100: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:30:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:40 smithi003 ceph-mon[107278]: pgmap v100: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:30:42.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:42 smithi019 ceph-mon[91296]: pgmap v101: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-13T14:30:42.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:42 smithi003 ceph-mon[104007]: pgmap v101: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-13T14:30:42.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:42 smithi003 ceph-mon[107278]: pgmap v101: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-13T14:30:43.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:43.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:43.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:30:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:30:44] "GET /metrics HTTP/1.1" 200 36226 "" "Prometheus/2.51.0" 2024-09-13T14:30:44.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:44 smithi019 ceph-mon[91296]: pgmap v102: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-13T14:30:44.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:44 smithi003 ceph-mon[104007]: pgmap v102: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-13T14:30:44.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:44 smithi003 ceph-mon[107278]: pgmap v102: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-13T14:30:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:46 smithi019 ceph-mon[91296]: pgmap v103: 89 pgs: 89 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-13T14:30:46.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:46 smithi003 ceph-mon[104007]: pgmap v103: 89 pgs: 89 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-13T14:30:46.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:46 smithi003 ceph-mon[107278]: pgmap v103: 89 pgs: 89 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-13T14:30:48.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:48 smithi019 ceph-mon[91296]: pgmap v104: 89 pgs: 89 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 742 op/s 2024-09-13T14:30:48.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:48 smithi003 ceph-mon[104007]: pgmap v104: 89 pgs: 89 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 742 op/s 2024-09-13T14:30:48.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:48 smithi003 ceph-mon[107278]: pgmap v104: 89 pgs: 89 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 742 op/s 2024-09-13T14:30:49.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade resume' 2024-09-13T14:30:50.062 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:50.527 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:30:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:30:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:50.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:50 smithi019 ceph-mon[91296]: pgmap v105: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:30:50.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:50 smithi003 ceph-mon[104007]: pgmap v105: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:30:50.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:50 smithi003 ceph-mon[107278]: pgmap v105: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:30:52.543 INFO:teuthology.orchestra.run.smithi003.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:52.930 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:52 smithi003 ceph-mon[107278]: pgmap v106: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-13T14:30:52.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:52 smithi019 ceph-mon[91296]: pgmap v106: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-13T14:30:52.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:52 smithi003 ceph-mon[104007]: pgmap v106: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-13T14:30:53.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for half of osds to upgrade' 2024-09-13T14:30:53.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:53 smithi019 ceph-mon[91296]: from='client.54112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:53 smithi019 ceph-mon[91296]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:53.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:53 smithi003 ceph-mon[107278]: from='client.54112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:53.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:53 smithi003 ceph-mon[107278]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:53.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:53.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:53.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:53 smithi003 ceph-mon[104007]: from='client.54112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:53.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:53 smithi003 ceph-mon[104007]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:53.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:30:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:30:54] "GET /metrics HTTP/1.1" 200 36226 "" "Prometheus/2.51.0" 2024-09-13T14:30:54.526 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:54 smithi019 ceph-mon[91296]: pgmap v107: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-13T14:30:54.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:54 smithi003 ceph-mon[107278]: pgmap v107: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-13T14:30:54.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:54 smithi003 ceph-mon[104007]: pgmap v107: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-13T14:30:56.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:56 smithi003 ceph-mon[107278]: pgmap v108: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:30:56.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:56 smithi003 ceph-mon[104007]: pgmap v108: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:30:56.757 INFO:teuthology.orchestra.run.smithi003.stdout:wait for half of osds to upgrade 2024-09-13T14:30:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:56 smithi019 ceph-mon[91296]: pgmap v108: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-13T14:30:58.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ! ceph osd versions | grep $sha1 | egrep '"'"': [45678]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-13T14:30:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:58 smithi019 ceph-mon[91296]: pgmap v109: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 723 op/s 2024-09-13T14:30:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:30:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:59.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:58 smithi003 ceph-mon[104007]: pgmap v109: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 723 op/s 2024-09-13T14:30:59.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:59.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:59.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:30:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:59.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:58 smithi003 ceph-mon[107278]: pgmap v109: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 723 op/s 2024-09-13T14:30:59.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:59.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:30:59.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:30:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:00.034 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:31:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:31:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:00.921 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:00 smithi003 ceph-mon[107278]: pgmap v110: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:31:00.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:00 smithi003 ceph-mon[104007]: pgmap v110: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:31:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:00 smithi019 ceph-mon[91296]: pgmap v110: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:31:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:02 smithi003 ceph-mon[107278]: pgmap v111: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 695 op/s 2024-09-13T14:31:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:02 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:02 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:02 smithi003 ceph-mon[104007]: pgmap v111: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 695 op/s 2024-09-13T14:31:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:02 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:02 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:02 smithi019 ceph-mon[91296]: pgmap v111: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 695 op/s 2024-09-13T14:31:03.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:02 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:03.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:02 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:03.764 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:04.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:04.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:04.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:04.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:04.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:04.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4071064818' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:04.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:31:04.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:31:04.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:31:04.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:31:04.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:31:04.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:04.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4071064818' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:04.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:04.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:31:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:31:04.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:31:04.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:31:04.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:31:04.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:31:04.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:31:04.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:04.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4071064818' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:04.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:04.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:31:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:31:04] "GET /metrics HTTP/1.1" 200 36226 "" "Prometheus/2.51.0" 2024-09-13T14:31:05.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[107278]: Upgrade: Setting container_image for all mon 2024-09-13T14:31:05.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[107278]: Upgrade: Setting container_image for all crash 2024-09-13T14:31:05.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:31:05.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[107278]: Upgrade: osd.0 is safe to restart 2024-09-13T14:31:05.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[107278]: pgmap v112: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 695 op/s 2024-09-13T14:31:05.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[107278]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:05.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[104007]: Upgrade: Setting container_image for all mon 2024-09-13T14:31:05.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[104007]: Upgrade: Setting container_image for all crash 2024-09-13T14:31:05.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:31:05.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[104007]: Upgrade: osd.0 is safe to restart 2024-09-13T14:31:05.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[104007]: pgmap v112: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 695 op/s 2024-09-13T14:31:05.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:04 smithi003 ceph-mon[104007]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:04 smithi019 ceph-mon[91296]: Upgrade: Setting container_image for all mon 2024-09-13T14:31:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:04 smithi019 ceph-mon[91296]: Upgrade: Setting container_image for all crash 2024-09-13T14:31:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:04 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:31:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:04 smithi019 ceph-mon[91296]: Upgrade: osd.0 is safe to restart 2024-09-13T14:31:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:04 smithi019 ceph-mon[91296]: pgmap v112: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 695 op/s 2024-09-13T14:31:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:04 smithi019 ceph-mon[91296]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:06.082 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:05 smithi019 ceph-mon[91296]: from='client.54121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:06.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:05 smithi003 ceph-mon[104007]: from='client.54121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:06.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:05 smithi003 ceph-mon[107278]: from='client.54121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:07.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:06 smithi003 ceph-mon[107278]: pgmap v113: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T14:31:07.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:07.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:31:07.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:07.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:06 smithi003 ceph-mon[104007]: pgmap v113: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T14:31:07.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:07.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:31:07.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:07.095 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:06 smithi019 ceph-mon[91296]: pgmap v113: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T14:31:07.095 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:07.095 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:31:07.095 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:07.448 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:07.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:07 smithi003 ceph-mon[107278]: Upgrade: Updating osd.0 2024-09-13T14:31:07.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:07 smithi003 ceph-mon[107278]: Deploying daemon osd.0 on smithi003 2024-09-13T14:31:07.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:07 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3511048929' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:07.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:07 smithi003 ceph-mon[104007]: Upgrade: Updating osd.0 2024-09-13T14:31:07.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:07 smithi003 ceph-mon[104007]: Deploying daemon osd.0 on smithi003 2024-09-13T14:31:07.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:07 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3511048929' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:07 smithi019 ceph-mon[91296]: Upgrade: Updating osd.0 2024-09-13T14:31:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:07 smithi019 ceph-mon[91296]: Deploying daemon osd.0 on smithi003 2024-09-13T14:31:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:07 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3511048929' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:08.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:08 smithi003 ceph-mon[104007]: pgmap v114: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 676 op/s 2024-09-13T14:31:08.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:08 smithi003 ceph-mon[104007]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:08.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:08 smithi003 ceph-mon[107278]: pgmap v114: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 676 op/s 2024-09-13T14:31:08.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:08 smithi003 ceph-mon[107278]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:08 smithi019 ceph-mon[91296]: pgmap v114: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 676 op/s 2024-09-13T14:31:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:08 smithi019 ceph-mon[91296]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:10.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:09 smithi019 ceph-mon[91296]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:10.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:09 smithi003 ceph-mon[104007]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:10.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:09 smithi003 ceph-mon[107278]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:31:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:31:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:10.946 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:10 smithi003 ceph-mon[107278]: pgmap v115: 89 pgs: 89 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 983 op/s 2024-09-13T14:31:10.946 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/723561713' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:10.947 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:10 smithi003 ceph-mon[104007]: pgmap v115: 89 pgs: 89 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 983 op/s 2024-09-13T14:31:10.947 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/723561713' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:11.156 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:10 smithi019 ceph-mon[91296]: pgmap v115: 89 pgs: 89 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 983 op/s 2024-09-13T14:31:11.259 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/723561713' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:11.682 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:11 smithi003 systemd[1]: Stopping Ceph osd.0 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:31:12.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:11 smithi019 ceph-mon[91296]: osd.0 marked itself down and dead 2024-09-13T14:31:12.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:11 smithi003 ceph-mon[104007]: osd.0 marked itself down and dead 2024-09-13T14:31:12.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:11 smithi003 ceph-mon[107278]: osd.0 marked itself down and dead 2024-09-13T14:31:12.183 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:11 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[45851]: 2024-09-13T14:31:11.764+0000 7f35f5eea640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:31:12.183 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:11 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[45851]: 2024-09-13T14:31:11.764+0000 7f35f5eea640 -1 osd.0 79 *** Got signal Terminated *** 2024-09-13T14:31:12.183 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:11 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[45851]: 2024-09-13T14:31:11.764+0000 7f35f5eea640 -1 osd.0 79 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:31:12.535 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:12 smithi003 podman[116128]: 2024-09-13 14:31:12.269568959 +0000 UTC m=+0.665581162 container died b0e72d25bc0f79cbd9118925adfac19e0949fcd2ffb0ca0bd609bcb59c14c5a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-09-13T14:31:12.809 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:12 smithi003 podman[116128]: 2024-09-13 14:31:12.733156671 +0000 UTC m=+1.129168875 container cleanup b0e72d25bc0f79cbd9118925adfac19e0949fcd2ffb0ca0bd609bcb59c14c5a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, CEPH_POINT_RELEASE=) 2024-09-13T14:31:12.809 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:12 smithi003 bash[116128]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0 2024-09-13T14:31:13.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:12 smithi019 ceph-mon[91296]: from='client.54136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:13.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:12 smithi019 ceph-mon[91296]: pgmap v116: 89 pgs: 89 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 636 op/s 2024-09-13T14:31:13.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:12 smithi019 ceph-mon[91296]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:13.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:12 smithi019 ceph-mon[91296]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:31:13.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:12 smithi019 ceph-mon[91296]: osdmap e80: 8 total, 7 up, 8 in 2024-09-13T14:31:13.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:13.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:13.183 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:13 smithi003 podman[116151]: 2024-09-13 14:31:13.035389045 +0000 UTC m=+0.761578874 container remove b0e72d25bc0f79cbd9118925adfac19e0949fcd2ffb0ca0bd609bcb59c14c5a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-13T14:31:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[107278]: from='client.54136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[107278]: pgmap v116: 89 pgs: 89 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 636 op/s 2024-09-13T14:31:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[107278]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[107278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:31:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[107278]: osdmap e80: 8 total, 7 up, 8 in 2024-09-13T14:31:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:13.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[104007]: from='client.54136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:13.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[104007]: pgmap v116: 89 pgs: 89 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 636 op/s 2024-09-13T14:31:13.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[104007]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:13.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[104007]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:31:13.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[104007]: osdmap e80: 8 total, 7 up, 8 in 2024-09-13T14:31:13.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:13.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:13.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:13 smithi003 ceph-mon[104007]: osdmap e81: 8 total, 7 up, 8 in 2024-09-13T14:31:13.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:13 smithi003 ceph-mon[107278]: osdmap e81: 8 total, 7 up, 8 in 2024-09-13T14:31:13.933 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:13 smithi003 podman[116231]: 2024-09-13 14:31:13.523833514 +0000 UTC m=+0.032892884 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:31:13.933 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:13 smithi003 podman[116231]: 2024-09-13 14:31:13.65302861 +0000 UTC m=+0.162087980 container create 684d7ec47b2780462661ec03765452ca5d72911c411f5d1711f520e65a3f974c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-13T14:31:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:13 smithi019 ceph-mon[91296]: osdmap e81: 8 total, 7 up, 8 in 2024-09-13T14:31:14.298 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:31:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:31:14] "GET /metrics HTTP/1.1" 200 36290 "" "Prometheus/2.51.0" 2024-09-13T14:31:14.298 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:13 smithi003 podman[116231]: 2024-09-13 14:31:13.981994842 +0000 UTC m=+0.491054214 container init 684d7ec47b2780462661ec03765452ca5d72911c411f5d1711f520e65a3f974c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-13T14:31:14.298 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:13 smithi003 podman[116231]: 2024-09-13 14:31:13.994418869 +0000 UTC m=+0.503478236 container start 684d7ec47b2780462661ec03765452ca5d72911c411f5d1711f520e65a3f974c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-13T14:31:14.298 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:14 smithi003 podman[116231]: 2024-09-13 14:31:14.091884711 +0000 UTC m=+0.600944099 container attach 684d7ec47b2780462661ec03765452ca5d72911c411f5d1711f520e65a3f974c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-09-13T14:31:14.663 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:14 smithi003 podman[116231]: 2024-09-13 14:31:14.298150761 +0000 UTC m=+0.807210127 container died 684d7ec47b2780462661ec03765452ca5d72911c411f5d1711f520e65a3f974c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-13T14:31:14.809 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:14 smithi003 ceph-mon[107278]: pgmap v119: 89 pgs: 13 stale+active+clean, 76 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 460 op/s 2024-09-13T14:31:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1767441088' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:14 smithi003 ceph-mon[104007]: pgmap v119: 89 pgs: 13 stale+active+clean, 76 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 460 op/s 2024-09-13T14:31:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1767441088' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:14 smithi019 ceph-mon[91296]: pgmap v119: 89 pgs: 13 stale+active+clean, 76 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 460 op/s 2024-09-13T14:31:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1767441088' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:15.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:15 smithi003 podman[116300]: 2024-09-13 14:31:15.088524005 +0000 UTC m=+0.780217980 container remove 684d7ec47b2780462661ec03765452ca5d72911c411f5d1711f520e65a3f974c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:31:15.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:15 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-13T14:31:15.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:15 smithi003 systemd[1]: Stopped Ceph osd.0 for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:31:15.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:15 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.0.service: Consumed 12min 8.534s CPU time. 2024-09-13T14:31:15.776 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:15 smithi003 systemd[1]: Starting Ceph osd.0 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:31:16.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:15 smithi003 ceph-mon[104007]: from='client.34329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:16.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:15 smithi003 ceph-mon[107278]: from='client.34329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:16.093 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:15 smithi003 podman[116439]: 2024-09-13 14:31:15.680731032 +0000 UTC m=+0.029046805 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:31:16.093 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:15 smithi003 podman[116439]: 2024-09-13 14:31:15.814397535 +0000 UTC m=+0.162713311 container create 885e074ef4130cc2cdef961c1c94c7f1a6b1176c0843294e8665d942e7d912fa (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:31:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:15 smithi019 ceph-mon[91296]: from='client.34329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:16.366 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 podman[116439]: 2024-09-13 14:31:16.092158606 +0000 UTC m=+0.440474390 container init 885e074ef4130cc2cdef961c1c94c7f1a6b1176c0843294e8665d942e7d912fa (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:31:16.366 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 podman[116439]: 2024-09-13 14:31:16.100123402 +0000 UTC m=+0.448439174 container start 885e074ef4130cc2cdef961c1c94c7f1a6b1176c0843294e8665d942e7d912fa (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T14:31:16.367 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 podman[116439]: 2024-09-13 14:31:16.225893866 +0000 UTC m=+0.574209646 container attach 885e074ef4130cc2cdef961c1c94c7f1a6b1176c0843294e8665d942e7d912fa (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-13T14:31:16.367 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:16.367 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:16.367 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:16.367 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:16.367 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:31:16.367 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:31:16.367 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:16.367 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:16.682 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:16.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:16.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:31:16.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:31:16.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:31:16.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:31:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:16 smithi019 ceph-mon[91296]: pgmap v120: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.0 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 4293/51480 objects degraded (8.339%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:16 smithi019 ceph-mon[91296]: from='client.44189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:16 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 4293/51480 objects degraded (8.339%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-mon[104007]: pgmap v120: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.0 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 4293/51480 objects degraded (8.339%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-mon[104007]: from='client.44189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 4293/51480 objects degraded (8.339%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-mon[107278]: pgmap v120: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.0 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 4293/51480 objects degraded (8.339%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-mon[107278]: from='client.44189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 4293/51480 objects degraded (8.339%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:17.183 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:31:17.183 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:31:17.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:31:17.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:31:17.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:31:17.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:31:17.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:31:17.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:31:17.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate[116453]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:31:17.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 bash[116439]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:31:17.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:16 smithi003 podman[116439]: 2024-09-13 14:31:16.843582286 +0000 UTC m=+1.191898059 container died 885e074ef4130cc2cdef961c1c94c7f1a6b1176c0843294e8665d942e7d912fa (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731) 2024-09-13T14:31:17.933 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:17 smithi003 podman[116585]: 2024-09-13 14:31:17.640235546 +0000 UTC m=+0.784308812 container remove 885e074ef4130cc2cdef961c1c94c7f1a6b1176c0843294e8665d942e7d912fa (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:31:18.328 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:17 smithi003 podman[116660]: 2024-09-13 14:31:17.903567127 +0000 UTC m=+0.041647916 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:31:18.328 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:18 smithi003 podman[116660]: 2024-09-13 14:31:18.033163894 +0000 UTC m=+0.171244665 container create 8e2479752fc34e051b9551a21a573d547f13b87644b6a02be72a3e2f04ac8d79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-13T14:31:18.329 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:18 smithi003 podman[116660]: 2024-09-13 14:31:18.328191864 +0000 UTC m=+0.466272642 container init 8e2479752fc34e051b9551a21a573d547f13b87644b6a02be72a3e2f04ac8d79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-13T14:31:18.447 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:18.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:18 smithi003 podman[116660]: 2024-09-13 14:31:18.349771888 +0000 UTC m=+0.487852662 container start 8e2479752fc34e051b9551a21a573d547f13b87644b6a02be72a3e2f04ac8d79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-13T14:31:18.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:18 smithi003 bash[116660]: 8e2479752fc34e051b9551a21a573d547f13b87644b6a02be72a3e2f04ac8d79 2024-09-13T14:31:18.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:18 smithi003 systemd[1]: Started Ceph osd.0 for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:31:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:18 smithi019 ceph-mon[91296]: pgmap v121: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.0 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 381 op/s; 4293/51480 objects degraded (8.339%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/239446073' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:19.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[107278]: pgmap v121: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.0 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 381 op/s; 4293/51480 objects degraded (8.339%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:19.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/239446073' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:19.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:19.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:19.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:19.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[104007]: pgmap v121: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.0 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 381 op/s; 4293/51480 objects degraded (8.339%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:19.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/239446073' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:19.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:19.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:19.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:20.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:19 smithi019 ceph-mon[91296]: from='client.54157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:20.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:19 smithi003 ceph-mon[104007]: from='client.54157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:20.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:19 smithi003 ceph-mon[107278]: from='client.54157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:20.183 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:19 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[116698]: 2024-09-13T14:31:19.994+0000 7f9dc948c640 -1 Falling back to public interface 2024-09-13T14:31:20.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:31:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:31:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:20 smithi019 ceph-mon[91296]: from='client.54166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:20 smithi019 ceph-mon[91296]: pgmap v122: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 779 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 966 op/s; 3161/37428 objects degraded (8.446%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:21.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:20 smithi003 ceph-mon[107278]: from='client.54166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:21.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:20 smithi003 ceph-mon[107278]: pgmap v122: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 779 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 966 op/s; 3161/37428 objects degraded (8.446%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:21.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:20 smithi003 ceph-mon[104007]: from='client.54166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:20 smithi003 ceph-mon[104007]: pgmap v122: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 779 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 966 op/s; 3161/37428 objects degraded (8.446%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:22.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/298668329' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:22.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/298668329' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:22.128 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:22.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/298668329' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:22 smithi019 ceph-mon[91296]: pgmap v123: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 779 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 3161/37428 objects degraded (8.446%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:23.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:22 smithi003 ceph-mon[104007]: pgmap v123: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 779 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 3161/37428 objects degraded (8.446%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:23.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:22 smithi003 ceph-mon[107278]: pgmap v123: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 779 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 3161/37428 objects degraded (8.446%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:23 smithi019 ceph-mon[91296]: from='client.44210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:23 smithi019 ceph-mon[91296]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:23 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:23 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:24.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:23 smithi003 ceph-mon[104007]: from='client.44210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:23 smithi003 ceph-mon[104007]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:23 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:23 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:24.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:31:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:31:24] "GET /metrics HTTP/1.1" 200 36290 "" "Prometheus/2.51.0" 2024-09-13T14:31:24.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:23 smithi003 ceph-mon[107278]: from='client.44210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:24.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:23 smithi003 ceph-mon[107278]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:24.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:23 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:24.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:23 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:25.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:24 smithi019 ceph-mon[91296]: pgmap v124: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 779 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 3161/37428 objects degraded (8.446%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:24 smithi003 ceph-mon[107278]: pgmap v124: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 779 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 3161/37428 objects degraded (8.446%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:25.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:24 smithi003 ceph-mon[104007]: pgmap v124: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 779 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 3161/37428 objects degraded (8.446%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:25.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:25.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:25.840 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:26.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:25 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1130441706' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:26.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3161/37428 objects degraded (8.446%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:26.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:25 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1130441706' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:26.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3161/37428 objects degraded (8.446%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:25 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1130441706' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3161/37428 objects degraded (8.446%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:27.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:26 smithi003 ceph-mon[107278]: pgmap v125: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 494 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2002/23781 objects degraded (8.418%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:27.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:26 smithi003 ceph-mon[107278]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:27.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:26 smithi003 ceph-mon[104007]: pgmap v125: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 494 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2002/23781 objects degraded (8.418%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:27.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:26 smithi003 ceph-mon[104007]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:27.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:26 smithi019 ceph-mon[91296]: pgmap v125: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 494 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2002/23781 objects degraded (8.418%); 0 B/s, 0 objects/s recovering 2024-09-13T14:31:27.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:26 smithi019 ceph-mon[91296]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:27.895 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[116698]: 2024-09-13T14:31:27.677+0000 7f9dc948c640 -1 osd.0 79 log_to_monitors true 2024-09-13T14:31:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:27 smithi019 ceph-mon[91296]: from='client.34365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:27 smithi019 ceph-mon[91296]: from='osd.0 [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:31:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:27 smithi019 ceph-mon[91296]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:31:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:28.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[104007]: from='client.34365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[104007]: from='osd.0 [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:31:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[104007]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:31:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[107278]: from='client.34365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[107278]: from='osd.0 [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:31:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[107278]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:31:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:28.184 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 14:31:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[116698]: 2024-09-13T14:31:27.964+0000 7f9dc1236640 -1 osd.0 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:31:29.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:28 smithi019 ceph-mon[91296]: pgmap v126: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 494 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 769 op/s; 2002/23781 objects degraded (8.418%) 2024-09-13T14:31:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:28 smithi019 ceph-mon[91296]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:31:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:28 smithi019 ceph-mon[91296]: from='osd.0 [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:31:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:28 smithi019 ceph-mon[91296]: osdmap e82: 8 total, 7 up, 8 in 2024-09-13T14:31:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:28 smithi019 ceph-mon[91296]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:31:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[104007]: pgmap v126: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 494 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 769 op/s; 2002/23781 objects degraded (8.418%) 2024-09-13T14:31:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[104007]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:31:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[104007]: from='osd.0 [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:31:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[104007]: osdmap e82: 8 total, 7 up, 8 in 2024-09-13T14:31:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[104007]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:31:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[107278]: pgmap v126: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 494 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 769 op/s; 2002/23781 objects degraded (8.418%) 2024-09-13T14:31:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[107278]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:31:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[107278]: from='osd.0 [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:31:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[107278]: osdmap e82: 8 total, 7 up, 8 in 2024-09-13T14:31:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:28 smithi003 ceph-mon[107278]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:31:29.297 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:30.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[107278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:31:30.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[107278]: osd.0 [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808] boot 2024-09-13T14:31:30.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[107278]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:31:30.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:31:30.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3443970903' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:30.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[104007]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:31:30.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[104007]: osd.0 [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808] boot 2024-09-13T14:31:30.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[104007]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:31:30.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:31:30.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:29 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3443970903' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:30.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:29 smithi019 ceph-mon[91296]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:31:30.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:29 smithi019 ceph-mon[91296]: osd.0 [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808] boot 2024-09-13T14:31:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:29 smithi019 ceph-mon[91296]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:31:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:31:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:29 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3443970903' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:30.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:31:30.320+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:31:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:31:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='client.44246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: pgmap v129: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 238 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 1018/11469 objects degraded (8.876%) 2024-09-13T14:31:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='client.44252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:31:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1018/11469 objects degraded (8.876%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='client.44246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: pgmap v129: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 238 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 1018/11469 objects degraded (8.876%) 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='client.44252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1018/11469 objects degraded (8.876%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='client.44246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: pgmap v129: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 238 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 1018/11469 objects degraded (8.876%) 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='client.44252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:31.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:31.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:31.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:31.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:31.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:31.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:31:31.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:31:31.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1018/11469 objects degraded (8.876%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:31 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:31:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:31 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:31:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:31 smithi019 ceph-mon[91296]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:31:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:31 smithi019 ceph-mon[91296]: pgmap v132: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 238 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 1018/11469 objects degraded (8.876%) 2024-09-13T14:31:32.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:31 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:31:32.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:31 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:31:32.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:31 smithi003 ceph-mon[107278]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:31:32.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:31 smithi003 ceph-mon[107278]: pgmap v132: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 238 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 1018/11469 objects degraded (8.876%) 2024-09-13T14:31:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:31 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:31:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:31 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:31:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:31 smithi003 ceph-mon[104007]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:31:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:31 smithi003 ceph-mon[104007]: pgmap v132: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 238 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 1018/11469 objects degraded (8.876%) 2024-09-13T14:31:32.715 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3503496979' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:32 smithi003 ceph-mon[107278]: from='client.54214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:33.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3503496979' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:33.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:32 smithi003 ceph-mon[104007]: from='client.54214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:33.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3503496979' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:32 smithi019 ceph-mon[91296]: from='client.54214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:34.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:33 smithi019 ceph-mon[91296]: from='client.54220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:33 smithi019 ceph-mon[91296]: pgmap v133: 89 pgs: 10 active+undersized, 9 active+undersized+degraded, 70 active+clean; 238 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 756 op/s; 1015/11469 objects degraded (8.850%) 2024-09-13T14:31:34.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:33 smithi003 ceph-mon[107278]: from='client.54220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:34.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:33 smithi003 ceph-mon[107278]: pgmap v133: 89 pgs: 10 active+undersized, 9 active+undersized+degraded, 70 active+clean; 238 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 756 op/s; 1015/11469 objects degraded (8.850%) 2024-09-13T14:31:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:33 smithi003 ceph-mon[104007]: from='client.54220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:33 smithi003 ceph-mon[104007]: pgmap v133: 89 pgs: 10 active+undersized, 9 active+undersized+degraded, 70 active+clean; 238 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 756 op/s; 1015/11469 objects degraded (8.850%) 2024-09-13T14:31:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:31:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:31:34] "GET /metrics HTTP/1.1" 200 36353 "" "Prometheus/2.51.0" 2024-09-13T14:31:35.625 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 131266 objects 2024-09-13T14:31:35.625 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :155.537 2024-09-13T14:31:36.089 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:36 smithi019 ceph-mon[91296]: pgmap v134: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 108 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 324 op/s; 2615/5220 objects degraded (50.096%); 0 B/s, 16 objects/s recovering 2024-09-13T14:31:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:36 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2615/5220 objects degraded (50.096%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:36 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4182414523' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:36 smithi003 ceph-mon[107278]: pgmap v134: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 108 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 324 op/s; 2615/5220 objects degraded (50.096%); 0 B/s, 16 objects/s recovering 2024-09-13T14:31:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:36 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2615/5220 objects degraded (50.096%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:36 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4182414523' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:36 smithi003 ceph-mon[104007]: pgmap v134: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 108 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 324 op/s; 2615/5220 objects degraded (50.096%); 0 B/s, 16 objects/s recovering 2024-09-13T14:31:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:36 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2615/5220 objects degraded (50.096%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:36 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4182414523' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:37 smithi019 ceph-mon[91296]: from='client.44273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:37 smithi019 ceph-mon[91296]: from='client.44279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:37.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:37 smithi003 ceph-mon[107278]: from='client.44273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:37.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:37 smithi003 ceph-mon[107278]: from='client.44279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:37 smithi003 ceph-mon[104007]: from='client.44273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:37 smithi003 ceph-mon[104007]: from='client.44279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:38 smithi019 ceph-mon[91296]: pgmap v135: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 108 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 260 op/s; 2615/5220 objects degraded (50.096%); 0 B/s, 13 objects/s recovering 2024-09-13T14:31:38.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:38 smithi003 ceph-mon[107278]: pgmap v135: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 108 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 260 op/s; 2615/5220 objects degraded (50.096%); 0 B/s, 13 objects/s recovering 2024-09-13T14:31:38.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:38 smithi003 ceph-mon[104007]: pgmap v135: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 108 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 260 op/s; 2615/5220 objects degraded (50.096%); 0 B/s, 13 objects/s recovering 2024-09-13T14:31:39.034 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:39.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:39 smithi019 ceph-mon[91296]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:31:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/674362124' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:39.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:39 smithi003 ceph-mon[107278]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:31:39.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/674362124' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:39.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:39 smithi003 ceph-mon[104007]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:31:39.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/674362124' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:40.181 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-13T14:31:40.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-13T14:31:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:40.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[107278]: from='client.44291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:40.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[107278]: from='client.44297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:40.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[107278]: pgmap v137: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 1754/414 objects degraded (423.671%); 0 B/s, 22 objects/s recovering 2024-09-13T14:31:40.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[107278]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:31:40.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1754/414 objects degraded (423.671%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:40.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[107278]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:31:40.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[104007]: from='client.44291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:40.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[104007]: from='client.44297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:40.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[104007]: pgmap v137: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 1754/414 objects degraded (423.671%); 0 B/s, 22 objects/s recovering 2024-09-13T14:31:40.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[104007]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:31:40.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1754/414 objects degraded (423.671%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:40.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:40 smithi003 ceph-mon[104007]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:31:40.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:31:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:31:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:40 smithi019 ceph-mon[91296]: from='client.44291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:40 smithi019 ceph-mon[91296]: from='client.44297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:40 smithi019 ceph-mon[91296]: pgmap v137: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 1754/414 objects degraded (423.671%); 0 B/s, 22 objects/s recovering 2024-09-13T14:31:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:40 smithi019 ceph-mon[91296]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:31:40.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1754/414 objects degraded (423.671%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:40.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:40 smithi019 ceph-mon[91296]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:31:42.049 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:42 smithi019 ceph-mon[91296]: pgmap v140: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 266 op/s; 1754/414 objects degraded (423.671%); 0 B/s, 13 objects/s recovering 2024-09-13T14:31:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:42 smithi019 ceph-mon[91296]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:31:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:42 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3004595017' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:42.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:42 smithi003 ceph-mon[104007]: pgmap v140: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 266 op/s; 1754/414 objects degraded (423.671%); 0 B/s, 13 objects/s recovering 2024-09-13T14:31:42.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:42 smithi003 ceph-mon[104007]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:31:42.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:42 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3004595017' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:42 smithi003 ceph-mon[107278]: pgmap v140: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 266 op/s; 1754/414 objects degraded (423.671%); 0 B/s, 13 objects/s recovering 2024-09-13T14:31:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:42 smithi003 ceph-mon[107278]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:31:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:42 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3004595017' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:43.419 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_1' removed 2024-09-13T14:31:43.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:43 smithi019 ceph-mon[91296]: from='client.44309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:43 smithi019 ceph-mon[91296]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/458113185' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:31:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:43 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:31:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:43.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[104007]: from='client.44309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:43.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[104007]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/458113185' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[107278]: from='client.44309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[107278]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/458113185' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:44.416 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:31:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:31:44] "GET /metrics HTTP/1.1" 200 36347 "" "Prometheus/2.51.0" 2024-09-13T14:31:44.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:44 smithi019 ceph-mon[91296]: pgmap v142: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 304 op/s; 1754/78 objects degraded (2248.718%); 0 B/s, 177 objects/s recovering 2024-09-13T14:31:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:44 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:31:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:44 smithi019 ceph-mon[91296]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:31:44.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:44 smithi003 ceph-mon[104007]: pgmap v142: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 304 op/s; 1754/78 objects degraded (2248.718%); 0 B/s, 177 objects/s recovering 2024-09-13T14:31:44.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:44 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:31:44.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:44 smithi003 ceph-mon[104007]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:31:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:44 smithi003 ceph-mon[107278]: pgmap v142: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 304 op/s; 1754/78 objects degraded (2248.718%); 0 B/s, 177 objects/s recovering 2024-09-13T14:31:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:44 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:31:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:44 smithi003 ceph-mon[107278]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:31:44.930 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:31:44.931 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:31:44.932 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-13T14:31:44.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-13T14:31:45.004 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:45.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:45 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3109907301' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:45.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:45 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:31:45.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1754/78 objects degraded (2248.718%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:45.570 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:45 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3109907301' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:45.570 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:45 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:31:45.570 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1754/78 objects degraded (2248.718%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:45.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:45 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3109907301' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:45 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:31:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1754/78 objects degraded (2248.718%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:46.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[107278]: from='client.44333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:46.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[107278]: from='client.44339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:46.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[107278]: Upgrade: osd.1 is safe to restart 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[107278]: pgmap v144: 73 pgs: 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1754/78 objects degraded (2248.718%), 2 pgs degraded) 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[104007]: from='client.44333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[104007]: from='client.44339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[104007]: Upgrade: osd.1 is safe to restart 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[104007]: pgmap v144: 73 pgs: 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1754/78 objects degraded (2248.718%), 2 pgs degraded) 2024-09-13T14:31:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:46 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:31:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:46 smithi019 ceph-mon[91296]: from='client.44333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:46 smithi019 ceph-mon[91296]: from='client.44339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:46 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:31:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:46 smithi019 ceph-mon[91296]: Upgrade: osd.1 is safe to restart 2024-09-13T14:31:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:46 smithi019 ceph-mon[91296]: pgmap v144: 73 pgs: 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:46 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1754/78 objects degraded (2248.718%), 2 pgs degraded) 2024-09-13T14:31:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:46 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:31:47.927 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:48.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: pgmap v145: 73 pgs: 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:48.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/596313461' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:48.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: Upgrade: Updating osd.1 2024-09-13T14:31:48.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:48.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:31:48.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:48.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: Deploying daemon osd.1 on smithi003 2024-09-13T14:31:48.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:48.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/586591223' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: pgmap v145: 73 pgs: 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/596313461' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: Upgrade: Updating osd.1 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: Deploying daemon osd.1 on smithi003 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:48.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/586591223' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:31:48.471 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:48 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:31:48.697 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_2' created 2024-09-13T14:31:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: pgmap v145: 73 pgs: 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/596313461' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: Upgrade: Updating osd.1 2024-09-13T14:31:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:31:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: Deploying daemon osd.1 on smithi003 2024-09-13T14:31:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:48.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/586591223' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:31:48.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:48 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:31:49.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:49 smithi003 ceph-mon[104007]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:49.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:49 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:31:49.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:49 smithi003 ceph-mon[104007]: osdmap e91: 8 total, 8 up, 8 in 2024-09-13T14:31:49.743 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:49 smithi003 ceph-mon[107278]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:49.744 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:49 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:31:49.744 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:49 smithi003 ceph-mon[107278]: osdmap e91: 8 total, 8 up, 8 in 2024-09-13T14:31:49.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-13T14:31:50.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:49 smithi019 ceph-mon[91296]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:50.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:49 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:31:50.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:49 smithi019 ceph-mon[91296]: osdmap e91: 8 total, 8 up, 8 in 2024-09-13T14:31:50.331 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:50.685 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:31:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:31:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:50.945 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:50 smithi003 ceph-mon[107278]: pgmap v147: 89 pgs: 5 creating+peering, 11 unknown, 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:50.945 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:50 smithi003 ceph-mon[107278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:31:50.945 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:50 smithi003 ceph-mon[107278]: osdmap e92: 8 total, 8 up, 8 in 2024-09-13T14:31:50.945 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:50 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2722181668' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:50.946 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:50 smithi003 ceph-mon[104007]: pgmap v147: 89 pgs: 5 creating+peering, 11 unknown, 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:50.946 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:50 smithi003 ceph-mon[104007]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:31:50.946 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:50 smithi003 ceph-mon[104007]: osdmap e92: 8 total, 8 up, 8 in 2024-09-13T14:31:50.946 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:50 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2722181668' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:50 smithi019 ceph-mon[91296]: pgmap v147: 89 pgs: 5 creating+peering, 11 unknown, 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:50 smithi019 ceph-mon[91296]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:31:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:50 smithi019 ceph-mon[91296]: osdmap e92: 8 total, 8 up, 8 in 2024-09-13T14:31:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:50 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2722181668' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:51.910 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:51 smithi003 ceph-mon[107278]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T14:31:51.910 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:51 smithi003 ceph-mon[107278]: from='client.34491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:51.910 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:51 smithi003 systemd[1]: Stopping Ceph osd.1 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:31:51.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:51 smithi003 ceph-mon[104007]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T14:31:51.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:51 smithi003 ceph-mon[104007]: from='client.34491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:52.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:51 smithi019 ceph-mon[91296]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T14:31:52.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:51 smithi019 ceph-mon[91296]: from='client.34491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:52.432 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[51969]: 2024-09-13T14:31:52.321+0000 7fe3dee77640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:31:52.432 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[51969]: 2024-09-13T14:31:52.321+0000 7fe3dee77640 -1 osd.1 93 *** Got signal Terminated *** 2024-09-13T14:31:52.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[51969]: 2024-09-13T14:31:52.321+0000 7fe3dee77640 -1 osd.1 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:31:52.717 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:52 smithi003 ceph-mon[104007]: from='client.44381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:52.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:52 smithi003 ceph-mon[104007]: pgmap v150: 89 pgs: 5 creating+peering, 11 unknown, 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:52.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:52 smithi003 ceph-mon[104007]: osd.1 marked itself down and dead 2024-09-13T14:31:52.718 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:52 smithi003 ceph-mon[107278]: from='client.44381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:52.718 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:52 smithi003 ceph-mon[107278]: pgmap v150: 89 pgs: 5 creating+peering, 11 unknown, 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:52.718 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:52 smithi003 ceph-mon[107278]: osd.1 marked itself down and dead 2024-09-13T14:31:53.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:52 smithi019 ceph-mon[91296]: from='client.44381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:53.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:52 smithi019 ceph-mon[91296]: pgmap v150: 89 pgs: 5 creating+peering, 11 unknown, 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:53.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:52 smithi019 ceph-mon[91296]: osd.1 marked itself down and dead 2024-09-13T14:31:53.609 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:53 smithi003 podman[122532]: 2024-09-13 14:31:53.291691112 +0000 UTC m=+1.069537940 container died 612dd4f9bddaef6b7a512d41284ec6f166d351e6ab5e0c91912c6527ac9bbc2c (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0) 2024-09-13T14:31:53.709 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-13T14:31:53.737 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:53.932 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:53 smithi003 podman[122532]: 2024-09-13 14:31:53.706833499 +0000 UTC m=+1.484680326 container cleanup 612dd4f9bddaef6b7a512d41284ec6f166d351e6ab5e0c91912c6527ac9bbc2c (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-13T14:31:53.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:53 smithi003 bash[122532]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1 2024-09-13T14:31:53.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[107278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:31:53.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[107278]: osdmap e94: 8 total, 7 up, 8 in 2024-09-13T14:31:53.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:53.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3131369548' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:31:53.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2561685286' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:53.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3131369548' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:31:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[104007]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:31:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[104007]: osdmap e94: 8 total, 7 up, 8 in 2024-09-13T14:31:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3131369548' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:31:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2561685286' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3131369548' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:31:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:53 smithi019 ceph-mon[91296]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:31:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:53 smithi019 ceph-mon[91296]: osdmap e94: 8 total, 7 up, 8 in 2024-09-13T14:31:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3131369548' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:31:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2561685286' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:54.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3131369548' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:31:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:31:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:31:54] "GET /metrics HTTP/1.1" 200 36347 "" "Prometheus/2.51.0" 2024-09-13T14:31:54.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:54 smithi003 podman[122637]: 2024-09-13 14:31:54.151611909 +0000 UTC m=+0.850159910 container remove 612dd4f9bddaef6b7a512d41284ec6f166d351e6ab5e0c91912c6527ac9bbc2c (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-13T14:31:54.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:54 smithi003 ceph-mon[107278]: pgmap v152: 89 pgs: 6 peering, 9 stale+active+clean, 5 creating+peering, 7 unknown, 62 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:54.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:54 smithi003 ceph-mon[107278]: from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:54.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:54 smithi003 ceph-mon[107278]: osdmap e95: 8 total, 7 up, 8 in 2024-09-13T14:31:54.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:54 smithi003 ceph-mon[104007]: pgmap v152: 89 pgs: 6 peering, 9 stale+active+clean, 5 creating+peering, 7 unknown, 62 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:54.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:54 smithi003 ceph-mon[104007]: from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:54.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:54 smithi003 ceph-mon[104007]: osdmap e95: 8 total, 7 up, 8 in 2024-09-13T14:31:55.170 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:54 smithi003 podman[122797]: 2024-09-13 14:31:54.79772176 +0000 UTC m=+0.016774959 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:31:55.171 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:55 smithi003 podman[122797]: 2024-09-13 14:31:55.03876289 +0000 UTC m=+0.257816076 container create 705ae139acbab7684c88e7255aeb61ce2569bbfc2ca3859aa2fe17a3dbf476c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T14:31:55.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:54 smithi019 ceph-mon[91296]: pgmap v152: 89 pgs: 6 peering, 9 stale+active+clean, 5 creating+peering, 7 unknown, 62 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:55.185 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:54 smithi019 ceph-mon[91296]: from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:55.185 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:54 smithi019 ceph-mon[91296]: osdmap e95: 8 total, 7 up, 8 in 2024-09-13T14:31:55.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:55 smithi003 podman[122797]: 2024-09-13 14:31:55.282864331 +0000 UTC m=+0.501917518 container init 705ae139acbab7684c88e7255aeb61ce2569bbfc2ca3859aa2fe17a3dbf476c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-13T14:31:55.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:55 smithi003 podman[122797]: 2024-09-13 14:31:55.28900958 +0000 UTC m=+0.508062758 container start 705ae139acbab7684c88e7255aeb61ce2569bbfc2ca3859aa2fe17a3dbf476c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-13T14:31:55.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:55 smithi003 podman[122797]: 2024-09-13 14:31:55.374345737 +0000 UTC m=+0.593398925 container attach 705ae139acbab7684c88e7255aeb61ce2569bbfc2ca3859aa2fe17a3dbf476c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-13T14:31:55.605 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-13T14:31:55.608 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:31:55.608 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:31:55.656 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T14:31:55.656 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:31:55.656 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_122851 2024-09-13T14:31:55.656 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:31:55.656 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:31:55.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:55 smithi003 ceph-mon[104007]: from='client.44405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:55.810 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:55 smithi003 ceph-mon[107278]: from='client.44405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:55.811 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:55 smithi003 podman[122797]: 2024-09-13 14:31:55.488241569 +0000 UTC m=+0.707294766 container died 705ae139acbab7684c88e7255aeb61ce2569bbfc2ca3859aa2fe17a3dbf476c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-13T14:31:56.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:55 smithi019 ceph-mon[91296]: from='client.44405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:56.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:56 smithi003 podman[122841]: 2024-09-13 14:31:56.112745165 +0000 UTC m=+0.618206037 container remove 705ae139acbab7684c88e7255aeb61ce2569bbfc2ca3859aa2fe17a3dbf476c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:31:56.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:56 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-13T14:31:56.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:56 smithi003 systemd[1]: Stopped Ceph osd.1 for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:31:56.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:56 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.1.service: Consumed 10min 23.500s CPU time. 2024-09-13T14:31:56.656 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 15 3908 3893 243.296 243.312 0.00388795 0.00409469 2024-09-13T14:31:56.837 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:56 smithi003 systemd[1]: Starting Ceph osd.1 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:31:56.837 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:56 smithi003 ceph-mon[107278]: pgmap v154: 89 pgs: 12 active+undersized, 26 peering, 51 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:56.837 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:56 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:31:56.837 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:56 smithi003 ceph-mon[107278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:31:56.837 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1606136658' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:56.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:56 smithi003 ceph-mon[104007]: pgmap v154: 89 pgs: 12 active+undersized, 26 peering, 51 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:56.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:56 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:31:56.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:56 smithi003 ceph-mon[104007]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:31:56.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1606136658' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:57.168 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:56 smithi003 podman[123000]: 2024-09-13 14:31:56.743572445 +0000 UTC m=+0.036173338 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:31:57.168 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:56 smithi003 podman[123000]: 2024-09-13 14:31:56.897797205 +0000 UTC m=+0.190398071 container create bf9f537b16fce19ad545907a8ae595e2a64aa020da2af43367469d2ef3308d59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:31:57.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:56 smithi019 ceph-mon[91296]: pgmap v154: 89 pgs: 12 active+undersized, 26 peering, 51 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:56 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:31:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:56 smithi019 ceph-mon[91296]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:31:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1606136658' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:31:57.285 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:31:57.432 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 podman[123000]: 2024-09-13 14:31:57.167768662 +0000 UTC m=+0.460369549 container init bf9f537b16fce19ad545907a8ae595e2a64aa020da2af43367469d2ef3308d59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-13T14:31:57.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 podman[123000]: 2024-09-13 14:31:57.176728918 +0000 UTC m=+0.469329792 container start bf9f537b16fce19ad545907a8ae595e2a64aa020da2af43367469d2ef3308d59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-13T14:31:57.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 podman[123000]: 2024-09-13 14:31:57.26748629 +0000 UTC m=+0.560087162 container attach bf9f537b16fce19ad545907a8ae595e2a64aa020da2af43367469d2ef3308d59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD) 2024-09-13T14:31:57.656 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 15 7318 7303 228.2 213.125 0.00406504 0.00437275 2024-09-13T14:31:57.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:57.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:57.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:57.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:57.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:31:57.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:31:57.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:57.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:57.934 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:57.934 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:31:57.934 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:31:57.934 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:31:57.934 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-09-13T14:31:57.934 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-09-13T14:31:58.359 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:31:58.359 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:31:58.359 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:31:58.359 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:31:58.360 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-13T14:31:58.360 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-13T14:31:58.360 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:31:58.360 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:31:58.360 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate[123031]: --> ceph-volume lvm activate successful for osd ID: 1 2024-09-13T14:31:58.360 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:57 smithi003 bash[123000]: --> ceph-volume lvm activate successful for osd ID: 1 2024-09-13T14:31:58.360 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:58 smithi003 conmon[123031]: conmon bf9f537b16fce19ad545 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-bf9f537b16fce19ad545907a8ae595e2a64aa020da2af43367469d2ef3308d59.scope/memory.events 2024-09-13T14:31:58.360 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:58 smithi003 podman[123000]: 2024-09-13 14:31:58.01112929 +0000 UTC m=+1.303730161 container died bf9f537b16fce19ad545907a8ae595e2a64aa020da2af43367469d2ef3308d59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True) 2024-09-13T14:31:58.656 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 11040 11024 229.647 232.562 0.00463422 0.00434781 2024-09-13T14:31:58.932 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:58 smithi003 podman[123200]: 2024-09-13 14:31:58.656263954 +0000 UTC m=+0.634476379 container remove bf9f537b16fce19ad545907a8ae595e2a64aa020da2af43367469d2ef3308d59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-13T14:31:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[107278]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[107278]: pgmap v155: 89 pgs: 12 active+undersized, 26 peering, 51 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[107278]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[104007]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[104007]: pgmap v155: 89 pgs: 12 active+undersized, 26 peering, 51 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:58 smithi003 ceph-mon[104007]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:58 smithi019 ceph-mon[91296]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:58 smithi019 ceph-mon[91296]: pgmap v155: 89 pgs: 12 active+undersized, 26 peering, 51 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:31:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:31:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:58 smithi019 ceph-mon[91296]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:59.358 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:58 smithi003 podman[123255]: 2024-09-13 14:31:58.870747812 +0000 UTC m=+0.033582444 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:31:59.611 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:59 smithi003 podman[123255]: 2024-09-13 14:31:59.357832165 +0000 UTC m=+0.520666782 container create 3126fff5816571e7e11301859cc4536af2765da1cf451a39bf3254ac08c5c207 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True) 2024-09-13T14:31:59.656 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 14295 14279 223.09 203.438 0.00764761 0.00447575 2024-09-13T14:31:59.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:59 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 3636/30588 objects degraded (11.887%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:59.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:31:59 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-13T14:31:59.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:59 smithi003 podman[123255]: 2024-09-13 14:31:59.610603124 +0000 UTC m=+0.773437741 container init 3126fff5816571e7e11301859cc4536af2765da1cf451a39bf3254ac08c5c207 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-13T14:31:59.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:59 smithi003 podman[123255]: 2024-09-13 14:31:59.619188518 +0000 UTC m=+0.782023130 container start 3126fff5816571e7e11301859cc4536af2765da1cf451a39bf3254ac08c5c207 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:31:59.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:59 smithi003 ceph-osd[123291]: -- 172.21.15.3:0/3168610013 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557d583d7a40 con 0x557d591b0400 2024-09-13T14:31:59.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:59 smithi003 bash[123255]: 3126fff5816571e7e11301859cc4536af2765da1cf451a39bf3254ac08c5c207 2024-09-13T14:31:59.933 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:31:59 smithi003 systemd[1]: Started Ceph osd.1 for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:31:59.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:59 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 3636/30588 objects degraded (11.887%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:31:59.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:31:59 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-13T14:32:00.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:59 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 3636/30588 objects degraded (11.887%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:00.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:31:59 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-13T14:32:00.658 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 17004 16988 212.256 169.312 0.00430308 0.00470389 2024-09-13T14:32:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:32:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:32:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:01.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[104007]: pgmap v156: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 636 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 3636/30588 objects degraded (11.887%) 2024-09-13T14:32:01.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:01.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:01.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:01.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2285438328' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:01.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[107278]: pgmap v156: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 636 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 3636/30588 objects degraded (11.887%) 2024-09-13T14:32:01.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:01.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:01.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:01.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2285438328' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:01.088 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:32:00 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[123268]: 2024-09-13T14:32:00.765+0000 7f59c06a2640 -1 Falling back to public interface 2024-09-13T14:32:01.088 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:00 smithi019 ceph-mon[91296]: pgmap v156: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 636 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 3636/30588 objects degraded (11.887%) 2024-09-13T14:32:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:00 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:00 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:00 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:01.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:00 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2285438328' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:01.658 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 19716 19700 205.13 169.5 0.00221421 0.00486772 2024-09-13T14:32:02.658 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 22406 22390 199.842 168.125 0.00515096 0.00499813 2024-09-13T14:32:02.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:02 smithi003 ceph-mon[104007]: from='client.44444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:02.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:02 smithi003 ceph-mon[104007]: pgmap v157: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 636 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 3636/30588 objects degraded (11.887%) 2024-09-13T14:32:02.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:02 smithi003 ceph-mon[104007]: from='client.34557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:02 smithi003 ceph-mon[107278]: from='client.44444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:02 smithi003 ceph-mon[107278]: pgmap v157: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 636 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 3636/30588 objects degraded (11.887%) 2024-09-13T14:32:02.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:02 smithi003 ceph-mon[107278]: from='client.34557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:03.097 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:02 smithi019 ceph-mon[91296]: from='client.44444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:03.097 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:02 smithi019 ceph-mon[91296]: pgmap v157: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 636 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 3636/30588 objects degraded (11.887%) 2024-09-13T14:32:03.097 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:02 smithi019 ceph-mon[91296]: from='client.34557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:03.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 24953 24937 194.731 159.188 0.00421868 0.00512907 2024-09-13T14:32:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:32:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:32:04] "GET /metrics HTTP/1.1" 200 36353 "" "Prometheus/2.51.0" 2024-09-13T14:32:04.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 27087 27071 187.915 133.375 0.00294544 0.00531439 2024-09-13T14:32:04.810 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:04 smithi019 ceph-mon[91296]: pgmap v158: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 866 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 5438/41607 objects degraded (13.070%) 2024-09-13T14:32:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:04 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2080285331' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:05.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:04 smithi003 ceph-mon[104007]: pgmap v158: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 866 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 5438/41607 objects degraded (13.070%) 2024-09-13T14:32:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:04 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2080285331' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:04 smithi003 ceph-mon[107278]: pgmap v158: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 866 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 5438/41607 objects degraded (13.070%) 2024-09-13T14:32:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:04 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2080285331' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:05.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 29144 29128 181.98 128.562 0.00446693 0.00548981 2024-09-13T14:32:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:05 smithi003 ceph-mon[104007]: from='client.34563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5438/41607 objects degraded (13.070%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:05 smithi003 ceph-mon[107278]: from='client.34563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5438/41607 objects degraded (13.070%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:06.114 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:05 smithi019 ceph-mon[91296]: from='client.34563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:06.114 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5438/41607 objects degraded (13.070%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:06.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 31269 31253 177.51 132.812 0.00878707 0.00562767 2024-09-13T14:32:06.848 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[123268]: 2024-09-13T14:32:06.472+0000 7f59c06a2640 -1 osd.1 93 log_to_monitors true 2024-09-13T14:32:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:06 smithi019 ceph-mon[91296]: from='client.54397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:06 smithi019 ceph-mon[91296]: pgmap v159: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.5 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s; 8885/73191 objects degraded (12.139%) 2024-09-13T14:32:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:06 smithi019 ceph-mon[91296]: from='osd.1 [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:32:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:06 smithi019 ceph-mon[91296]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:32:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[104007]: from='client.54397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[104007]: pgmap v159: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.5 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s; 8885/73191 objects degraded (12.139%) 2024-09-13T14:32:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[104007]: from='osd.1 [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:32:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[104007]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:32:07.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[107278]: from='client.54397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:07.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[107278]: pgmap v159: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.5 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s; 8885/73191 objects degraded (12.139%) 2024-09-13T14:32:07.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:07.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:07.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[107278]: from='osd.1 [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:32:07.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:06 smithi003 ceph-mon[107278]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:32:07.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 33474 33458 174.202 137.812 0.00682741 0.00573424 2024-09-13T14:32:07.870 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 14:32:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[123268]: 2024-09-13T14:32:07.538+0000 7f59b844c640 -1 osd.1 93 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:32:08.541 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:08.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 34948 34932 167.889 92.125 0.00249941 0.00589112 2024-09-13T14:32:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:08 smithi019 ceph-mon[91296]: pgmap v160: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.5 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s; 8885/73191 objects degraded (12.139%) 2024-09-13T14:32:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:08 smithi019 ceph-mon[91296]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:32:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:08 smithi019 ceph-mon[91296]: osdmap e96: 8 total, 7 up, 8 in 2024-09-13T14:32:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:08 smithi019 ceph-mon[91296]: from='osd.1 [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:32:08.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:08 smithi019 ceph-mon[91296]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:32:08.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:08 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3000873920' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:08.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:08.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:08.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[107278]: pgmap v160: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.5 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s; 8885/73191 objects degraded (12.139%) 2024-09-13T14:32:08.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[107278]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:32:08.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[107278]: osdmap e96: 8 total, 7 up, 8 in 2024-09-13T14:32:08.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[107278]: from='osd.1 [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[107278]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3000873920' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[104007]: pgmap v160: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.5 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s; 8885/73191 objects degraded (12.139%) 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[104007]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[104007]: osdmap e96: 8 total, 7 up, 8 in 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[104007]: from='osd.1 [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[104007]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3000873920' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:08.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:08.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:09.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 34948 34932 155.9 0 - 0.00589112 2024-09-13T14:32:09.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[107278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:32:09.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[107278]: osd.1 [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243] boot 2024-09-13T14:32:09.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[107278]: osdmap e97: 8 total, 8 up, 8 in 2024-09-13T14:32:09.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:32:09.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[107278]: from='client.54412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:09.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[107278]: from='client.44480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:09.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:09.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:09.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[107278]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T14:32:09.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[104007]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:32:09.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[104007]: osd.1 [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243] boot 2024-09-13T14:32:09.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[104007]: osdmap e97: 8 total, 8 up, 8 in 2024-09-13T14:32:09.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:32:09.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[104007]: from='client.54412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:09.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[104007]: from='client.44480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:09.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:09.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:09.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:09 smithi003 ceph-mon[104007]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T14:32:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:09 smithi019 ceph-mon[91296]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:32:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:09 smithi019 ceph-mon[91296]: osd.1 [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243] boot 2024-09-13T14:32:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:09 smithi019 ceph-mon[91296]: osdmap e97: 8 total, 8 up, 8 in 2024-09-13T14:32:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:32:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:09 smithi019 ceph-mon[91296]: from='client.54412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:09.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:09 smithi019 ceph-mon[91296]: from='client.44480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:09.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:09.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:09.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:09 smithi019 ceph-mon[91296]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T14:32:10.497 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:32:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:32:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:10.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 34948 34932 145.508 0 - 0.00589112 2024-09-13T14:32:10.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:10 smithi019 ceph-mon[91296]: pgmap v163: 89 pgs: 7 peering, 22 active+undersized, 9 active+undersized+degraded, 51 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 191 MiB/s wr, 3.06k op/s; 10600/104001 objects degraded (10.192%) 2024-09-13T14:32:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10600/104001 objects degraded (10.192%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:10 smithi019 ceph-mon[91296]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:32:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:10 smithi003 ceph-mon[107278]: pgmap v163: 89 pgs: 7 peering, 22 active+undersized, 9 active+undersized+degraded, 51 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 191 MiB/s wr, 3.06k op/s; 10600/104001 objects degraded (10.192%) 2024-09-13T14:32:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10600/104001 objects degraded (10.192%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:10 smithi003 ceph-mon[107278]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:32:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:10 smithi003 ceph-mon[104007]: pgmap v163: 89 pgs: 7 peering, 22 active+undersized, 9 active+undersized+degraded, 51 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 191 MiB/s wr, 3.06k op/s; 10600/104001 objects degraded (10.192%) 2024-09-13T14:32:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10600/104001 objects degraded (10.192%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:10 smithi003 ceph-mon[104007]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:32:11.505 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:11.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 35061 35045 136.857 2.35417 0.00782932 0.00729929 2024-09-13T14:32:11.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3133018496' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:11.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3133018496' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3133018496' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:12.660 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 36507 36491 134.122 90.375 0.00813939 0.00744892 2024-09-13T14:32:12.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:12 smithi019 ceph-mon[91296]: pgmap v166: 89 pgs: 7 peering, 22 active+undersized, 9 active+undersized+degraded, 51 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 160 MiB/s wr, 2.57k op/s; 10600/104001 objects degraded (10.192%) 2024-09-13T14:32:12.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:12 smithi019 ceph-mon[91296]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:12.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:12 smithi003 ceph-mon[104007]: pgmap v166: 89 pgs: 7 peering, 22 active+undersized, 9 active+undersized+degraded, 51 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 160 MiB/s wr, 2.57k op/s; 10600/104001 objects degraded (10.192%) 2024-09-13T14:32:12.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:12 smithi003 ceph-mon[104007]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:12.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:12 smithi003 ceph-mon[107278]: pgmap v166: 89 pgs: 7 peering, 22 active+undersized, 9 active+undersized+degraded, 51 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 160 MiB/s wr, 2.57k op/s; 10600/104001 objects degraded (10.192%) 2024-09-13T14:32:12.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:12 smithi003 ceph-mon[107278]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:13.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 38000 37984 131.855 93.3125 0.0070541 0.00757675 2024-09-13T14:32:14.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:13 smithi003 ceph-mon[107278]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:14.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:14.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:14.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:13 smithi003 ceph-mon[104007]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:14.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:14.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:13 smithi019 ceph-mon[91296]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:14.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:32:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:32:14] "GET /metrics HTTP/1.1" 200 36413 "" "Prometheus/2.51.0" 2024-09-13T14:32:14.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 39369 39353 129.419 85.5625 0.00682496 0.00771864 2024-09-13T14:32:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:14 smithi003 ceph-mon[107278]: pgmap v167: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 peering, 10 active+undersized, 6 active+undersized+degraded, 64 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s; 10706/106263 objects degraded (10.075%) 2024-09-13T14:32:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/709142079' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:14 smithi003 ceph-mon[104007]: pgmap v167: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 peering, 10 active+undersized, 6 active+undersized+degraded, 64 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s; 10706/106263 objects degraded (10.075%) 2024-09-13T14:32:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/709142079' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:14 smithi019 ceph-mon[91296]: pgmap v167: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 peering, 10 active+undersized, 6 active+undersized+degraded, 64 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s; 10706/106263 objects degraded (10.075%) 2024-09-13T14:32:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/709142079' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:15.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:32:15.661026+0000 min lat: 0.00111463 max lat: 3.1215 avg lat: 0.00787269 2024-09-13T14:32:15.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:32:15.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 40643 40627 126.929 79.625 0.0153069 0.00787269 2024-09-13T14:32:15.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:32:15 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:32:15.306+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-13T14:32:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:32:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:32:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:32:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:32:16.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:32:16.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:32:16.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:32:16.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-13T14:32:16.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: pgmap v168: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 55 MiB/s wr, 874 op/s; 12802/113064 objects degraded (11.323%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:32:16.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10706/106263 objects degraded (10.075%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:16.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:16 smithi019 ceph-mon[91296]: from='client.54442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: pgmap v168: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 55 MiB/s wr, 874 op/s; 12802/113064 objects degraded (11.323%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:32:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10706/106263 objects degraded (10.075%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[107278]: from='client.54442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:32:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:32:16.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:32:16.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:32:16.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-13T14:32:16.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: pgmap v168: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 55 MiB/s wr, 874 op/s; 12802/113064 objects degraded (11.323%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:32:16.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10706/106263 objects degraded (10.075%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:16.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:16 smithi003 ceph-mon[104007]: from='client.54442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 42142 42126 125.346 93.6875 0.0114455 0.00796859 2024-09-13T14:32:17.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 15 43665 43650 123.977 95.25 0.0131491 0.00806003 2024-09-13T14:32:18.417 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:18 smithi019 ceph-mon[91296]: pgmap v169: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 24 MiB/s wr, 377 op/s; 12802/113064 objects degraded (11.323%); 1016 KiB/s, 15 objects/s recovering 2024-09-13T14:32:18.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 45012 44996 122.245 84.125 0.00788727 0.00817405 2024-09-13T14:32:18.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/571717398' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:18.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:18 smithi003 ceph-mon[104007]: pgmap v169: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 24 MiB/s wr, 377 op/s; 12802/113064 objects degraded (11.323%); 1016 KiB/s, 15 objects/s recovering 2024-09-13T14:32:18.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/571717398' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:18.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:18 smithi003 ceph-mon[107278]: pgmap v169: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 24 MiB/s wr, 377 op/s; 12802/113064 objects degraded (11.323%); 1016 KiB/s, 15 objects/s recovering 2024-09-13T14:32:18.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/571717398' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:18.974 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:19.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 46299 46283 120.503 80.4375 0.0163309 0.00829331 2024-09-13T14:32:20.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 47723 47707 119.242 89 0.00537033 0.00838021 2024-09-13T14:32:20.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:32:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:32:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:20 smithi019 ceph-mon[91296]: from='client.44525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:20 smithi019 ceph-mon[91296]: pgmap v170: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 12442/134166 objects degraded (9.274%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:20 smithi019 ceph-mon[91296]: from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 12442/134166 objects degraded (9.274%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:20.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:20 smithi003 ceph-mon[107278]: from='client.44525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:20.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:20 smithi003 ceph-mon[107278]: pgmap v170: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 12442/134166 objects degraded (9.274%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:20.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:20 smithi003 ceph-mon[107278]: from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:20.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 12442/134166 objects degraded (9.274%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:20 smithi003 ceph-mon[104007]: from='client.44525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:20 smithi003 ceph-mon[104007]: pgmap v170: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 12442/134166 objects degraded (9.274%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:20 smithi003 ceph-mon[104007]: from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 12442/134166 objects degraded (9.274%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:21.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 49129 49113 118.036 87.875 0.00777735 0.00846567 2024-09-13T14:32:22.661 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 50358 50342 116.509 76.8125 0.00957469 0.00857696 2024-09-13T14:32:22.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:22 smithi019 ceph-mon[91296]: pgmap v171: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 58 MiB/s wr, 923 op/s; 12442/134166 objects degraded (9.274%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:32:22.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/531508077' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:22.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:22 smithi003 ceph-mon[107278]: pgmap v171: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 58 MiB/s wr, 923 op/s; 12442/134166 objects degraded (9.274%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:32:22.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/531508077' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:22.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:22 smithi003 ceph-mon[104007]: pgmap v171: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 58 MiB/s wr, 923 op/s; 12442/134166 objects degraded (9.274%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:32:22.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/531508077' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:22.700 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:23.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 51639 51623 115.207 80.0625 0.0186176 0.00867318 2024-09-13T14:32:23.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:23 smithi019 ceph-mon[91296]: from='client.34629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:23.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:23 smithi003 ceph-mon[107278]: from='client.34629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:23.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:23 smithi003 ceph-mon[104007]: from='client.34629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:32:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:32:24] "GET /metrics HTTP/1.1" 200 36413 "" "Prometheus/2.51.0" 2024-09-13T14:32:24.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 52996 52980 114.159 84.8125 0.00410688 0.00875388 2024-09-13T14:32:24.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:24 smithi019 ceph-mon[91296]: from='client.44543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:24.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:24 smithi019 ceph-mon[91296]: pgmap v172: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 61 MiB/s wr, 979 op/s; 12442/139263 objects degraded (8.934%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:32:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:24 smithi003 ceph-mon[107278]: from='client.44543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:24 smithi003 ceph-mon[107278]: pgmap v172: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 61 MiB/s wr, 979 op/s; 12442/139263 objects degraded (8.934%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:32:24.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:24 smithi003 ceph-mon[104007]: from='client.44543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:24.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:24 smithi003 ceph-mon[104007]: pgmap v172: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 61 MiB/s wr, 979 op/s; 12442/139263 objects degraded (8.934%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:32:25.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 54272 54256 113.012 79.75 0.0172933 0.00884258 2024-09-13T14:32:25.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 12442/139263 objects degraded (8.934%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:25.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 12442/139263 objects degraded (8.934%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:25.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 12442/139263 objects degraded (8.934%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:26.420 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:26.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 55430 55414 111.701 72.375 0.00840519 0.00894586 2024-09-13T14:32:26.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:26 smithi003 ceph-mon[107278]: pgmap v173: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 12065/154200 objects degraded (7.824%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:32:26.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1171638475' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:26.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:26 smithi003 ceph-mon[104007]: pgmap v173: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 12065/154200 objects degraded (7.824%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:32:26.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1171638475' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:26 smithi019 ceph-mon[91296]: pgmap v173: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 12065/154200 objects degraded (7.824%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:32:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1171638475' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:27.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 56684 56668 110.659 78.375 0.00477676 0.00902978 2024-09-13T14:32:27.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:27 smithi019 ceph-mon[91296]: from='client.54472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:27.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:27 smithi003 ceph-mon[104007]: from='client.54472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:27.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:27 smithi003 ceph-mon[107278]: from='client.54472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:28.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 57955 57939 109.713 79.4375 0.0040835 0.00910827 2024-09-13T14:32:28.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:28 smithi019 ceph-mon[91296]: from='client.54478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:28 smithi019 ceph-mon[91296]: pgmap v174: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 12065/154200 objects degraded (7.824%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:32:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:28 smithi003 ceph-mon[107278]: from='client.54478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:28 smithi003 ceph-mon[107278]: pgmap v174: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 12065/154200 objects degraded (7.824%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:32:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:28 smithi003 ceph-mon[104007]: from='client.54478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:28 smithi003 ceph-mon[104007]: pgmap v174: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 12065/154200 objects degraded (7.824%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:32:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:29.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 59133 59117 108.651 73.625 0.00922819 0.00919806 2024-09-13T14:32:30.118 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:30.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:32:30 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:32:30.325+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:32:30.480 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:32:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:32:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:30.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 60223 60207 107.493 68.125 0.00450056 0.00929701 2024-09-13T14:32:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:30 smithi019 ceph-mon[91296]: pgmap v175: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s; 11641/173160 objects degraded (6.723%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:32:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:30 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/526911874' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:32:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11641/173160 objects degraded (6.723%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:30.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:30 smithi003 ceph-mon[107278]: pgmap v175: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s; 11641/173160 objects degraded (6.723%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:32:30.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:30 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/526911874' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:30.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:32:30.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11641/173160 objects degraded (6.723%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:30 smithi003 ceph-mon[104007]: pgmap v175: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s; 11641/173160 objects degraded (6.723%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:32:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:30 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/526911874' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:32:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11641/173160 objects degraded (6.723%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:31.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 61465 61449 106.664 77.625 0.00906858 0.00936976 2024-09-13T14:32:31.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:31 smithi019 ceph-mon[91296]: from='client.44567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:31 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:32:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:31 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:32:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:31 smithi019 ceph-mon[91296]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:31.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:31 smithi003 ceph-mon[107278]: from='client.44567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:31.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:31 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:32:31.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:31 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:32:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:31 smithi003 ceph-mon[107278]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:31 smithi003 ceph-mon[104007]: from='client.44567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:31 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:32:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:31 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:32:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:31 smithi003 ceph-mon[104007]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:32.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 62696 62680 105.86 76.9375 0.00773059 0.00943958 2024-09-13T14:32:32.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:32 smithi019 ceph-mon[91296]: pgmap v176: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 11641/173160 objects degraded (6.723%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:32:32.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:32 smithi003 ceph-mon[104007]: pgmap v176: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 11641/173160 objects degraded (6.723%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:32:32.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:32 smithi003 ceph-mon[107278]: pgmap v176: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 11641/173160 objects degraded (6.723%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:32:33.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 63746 63730 104.801 65.625 0.0131284 0.00953547 2024-09-13T14:32:33.867 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:33.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:33 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3127815605' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:33.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:33 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3127815605' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:33.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:33 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3127815605' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:34.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:32:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:32:34] "GET /metrics HTTP/1.1" 200 36434 "" "Prometheus/2.51.0" 2024-09-13T14:32:34.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 64844 64828 103.873 68.625 0.00987908 0.00962076 2024-09-13T14:32:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:34 smithi019 ceph-mon[91296]: pgmap v177: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s; 11641/177651 objects degraded (6.553%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:32:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:34 smithi019 ceph-mon[91296]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:34.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:34 smithi003 ceph-mon[107278]: pgmap v177: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s; 11641/177651 objects degraded (6.553%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:32:34.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:34 smithi003 ceph-mon[107278]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:34.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:34 smithi003 ceph-mon[104007]: pgmap v177: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s; 11641/177651 objects degraded (6.553%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:32:34.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:34 smithi003 ceph-mon[104007]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:35.662 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:32:35.662866+0000 min lat: 0.00111463 max lat: 3.1215 avg lat: 0.00968695 2024-09-13T14:32:35.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:32:35.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 66050 66034 103.161 75.375 0.00583503 0.00968695 2024-09-13T14:32:35.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:35 smithi019 ceph-mon[91296]: from='client.44582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:35.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11641/177651 objects degraded (6.553%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:35.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:35 smithi003 ceph-mon[107278]: from='client.44582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:35.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11641/177651 objects degraded (6.553%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:35.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:35 smithi003 ceph-mon[104007]: from='client.44582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:35.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11641/177651 objects degraded (6.553%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:36.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 67222 67206 102.431 73.25 0.0110474 0.00975569 2024-09-13T14:32:36.832 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:36 smithi019 ceph-mon[91296]: pgmap v178: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s; 11183/190671 objects degraded (5.865%); 6.6 MiB/s, 104 objects/s recovering 2024-09-13T14:32:36.833 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:36 smithi003 ceph-mon[107278]: pgmap v178: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s; 11183/190671 objects degraded (5.865%); 6.6 MiB/s, 104 objects/s recovering 2024-09-13T14:32:36.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:36 smithi003 ceph-mon[104007]: pgmap v178: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s; 11183/190671 objects degraded (5.865%); 6.6 MiB/s, 104 objects/s recovering 2024-09-13T14:32:37.610 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:37.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 68113 68097 101.318 55.6875 0.00899405 0.00986068 2024-09-13T14:32:37.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/778831727' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:37.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/778831727' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:37.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/778831727' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:38.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 68368 68352 99.3328 15.9375 0.00924327 0.0100604 2024-09-13T14:32:38.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:38 smithi019 ceph-mon[91296]: pgmap v179: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 11183/190671 objects degraded (5.865%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:32:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:38 smithi019 ceph-mon[91296]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T14:32:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:38 smithi019 ceph-mon[91296]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:38.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:38 smithi003 ceph-mon[107278]: pgmap v179: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 11183/190671 objects degraded (5.865%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:32:38.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:38 smithi003 ceph-mon[107278]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T14:32:38.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:38 smithi003 ceph-mon[107278]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:38.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:38 smithi003 ceph-mon[104007]: pgmap v179: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 11183/190671 objects degraded (5.865%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:32:38.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:38 smithi003 ceph-mon[104007]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T14:32:38.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:38 smithi003 ceph-mon[104007]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:39.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 69514 69498 98.7029 71.625 0.0133486 0.0101254 2024-09-13T14:32:39.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:39 smithi019 ceph-mon[91296]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:39.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:39 smithi019 ceph-mon[91296]: osdmap e101: 8 total, 8 up, 8 in 2024-09-13T14:32:39.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:39 smithi003 ceph-mon[107278]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:39.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:39 smithi003 ceph-mon[107278]: osdmap e101: 8 total, 8 up, 8 in 2024-09-13T14:32:39.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:39 smithi003 ceph-mon[104007]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:39.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:39 smithi003 ceph-mon[104007]: osdmap e101: 8 total, 8 up, 8 in 2024-09-13T14:32:40.580 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:32:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:32:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:40.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 70564 70548 97.9678 65.625 0.0188453 0.0102007 2024-09-13T14:32:40.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:40 smithi019 ceph-mon[91296]: pgmap v182: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 10783/205188 objects degraded (5.255%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:32:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10783/205188 objects degraded (5.255%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:40.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:40 smithi003 ceph-mon[107278]: pgmap v182: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 10783/205188 objects degraded (5.255%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:32:40.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10783/205188 objects degraded (5.255%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:40.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:40 smithi003 ceph-mon[104007]: pgmap v182: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 10783/205188 objects degraded (5.255%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:32:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10783/205188 objects degraded (5.255%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:41.202 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:41.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 71544 71528 97.1695 61.25 0.00594574 0.0102835 2024-09-13T14:32:41.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:41 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1711794179' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:41.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:41 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1711794179' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:41.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:41 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1711794179' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:42.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 72655 72639 96.5793 69.4375 0.0106775 0.0103479 2024-09-13T14:32:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:42 smithi019 ceph-mon[91296]: from='client.44603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:42 smithi019 ceph-mon[91296]: pgmap v183: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 10783/205188 objects degraded (5.255%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:32:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:42 smithi019 ceph-mon[91296]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:42.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:42 smithi003 ceph-mon[107278]: from='client.44603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:42.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:42 smithi003 ceph-mon[107278]: pgmap v183: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 10783/205188 objects degraded (5.255%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:32:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:42 smithi003 ceph-mon[107278]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:42 smithi003 ceph-mon[104007]: from='client.44603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:42 smithi003 ceph-mon[104007]: pgmap v183: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 10783/205188 objects degraded (5.255%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:32:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:42 smithi003 ceph-mon[104007]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:43.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 73730 73714 95.9669 67.1875 0.0111379 0.010414 2024-09-13T14:32:43.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:43.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:43.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:32:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:32:44] "GET /metrics HTTP/1.1" 200 36437 "" "Prometheus/2.51.0" 2024-09-13T14:32:44.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 74718 74702 95.2685 61.75 0.0264837 0.0104908 2024-09-13T14:32:44.924 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:44.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:44 smithi003 ceph-mon[104007]: pgmap v184: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 47 MiB/s wr, 755 op/s; 10507/208797 objects degraded (5.032%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:32:44.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/275203237' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:44.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:44 smithi003 ceph-mon[107278]: pgmap v184: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 47 MiB/s wr, 755 op/s; 10507/208797 objects degraded (5.032%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:32:44.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/275203237' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:44.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:44 smithi019 ceph-mon[91296]: pgmap v184: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 47 MiB/s wr, 755 op/s; 10507/208797 objects degraded (5.032%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:32:44.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/275203237' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:45.614 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:32:45 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:32:45.346+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:32:45.663 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 75705 75689 94.5969 61.6875 0.0131961 0.010565 2024-09-13T14:32:45.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:45 smithi019 ceph-mon[91296]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:45 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:32:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10507/208797 objects degraded (5.032%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:45.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:45 smithi003 ceph-mon[107278]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:45.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:45 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:32:45.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10507/208797 objects degraded (5.032%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:45.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:45 smithi003 ceph-mon[104007]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:45.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:45 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:32:45.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10507/208797 objects degraded (5.032%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:46.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 76788 76772 94.0691 67.6875 0.0057936 0.0106246 2024-09-13T14:32:46.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:46 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:32:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:46 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:32:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:46 smithi019 ceph-mon[91296]: pgmap v185: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s; 10507/220713 objects degraded (4.760%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:32:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:46 smithi019 ceph-mon[91296]: from='client.54544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:46.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:46 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:32:46.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:46 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:32:46.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:46 smithi003 ceph-mon[107278]: pgmap v185: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s; 10507/220713 objects degraded (4.760%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:32:46.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:46 smithi003 ceph-mon[107278]: from='client.54544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:46 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:32:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:46 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:32:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:46 smithi003 ceph-mon[104007]: pgmap v185: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s; 10507/220713 objects degraded (4.760%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:32:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:46 smithi003 ceph-mon[104007]: from='client.54544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 77852 77836 93.5389 66.5 0.0162805 0.0106849 2024-09-13T14:32:48.645 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:48.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 78775 78759 92.8624 57.6875 0.00895939 0.010761 2024-09-13T14:32:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:48 smithi019 ceph-mon[91296]: pgmap v186: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 10507/220713 objects degraded (4.760%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:32:48.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1438716676' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:48.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:48 smithi003 ceph-mon[104007]: pgmap v186: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 10507/220713 objects degraded (4.760%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:32:48.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1438716676' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:48.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:48 smithi003 ceph-mon[107278]: pgmap v186: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 10507/220713 objects degraded (4.760%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:32:48.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1438716676' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:49.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 79755 79739 92.2769 61.25 0.00570252 0.01083 2024-09-13T14:32:49.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:49 smithi019 ceph-mon[91296]: from='client.44624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:49.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:49 smithi003 ceph-mon[104007]: from='client.44624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:49 smithi003 ceph-mon[107278]: from='client.44624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:50.640 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:32:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:32:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:50.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 80779 80763 91.7627 64 0.00946903 0.0108907 2024-09-13T14:32:50.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:50 smithi019 ceph-mon[91296]: from='client.44630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:50 smithi019 ceph-mon[91296]: pgmap v187: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 10291/235899 objects degraded (4.362%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:32:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10291/235899 objects degraded (4.362%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:50.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:50 smithi003 ceph-mon[107278]: from='client.44630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:50.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:50 smithi003 ceph-mon[107278]: pgmap v187: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 10291/235899 objects degraded (4.362%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:32:50.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10291/235899 objects degraded (4.362%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:50.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:50 smithi003 ceph-mon[104007]: from='client.44630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:50.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:50 smithi003 ceph-mon[104007]: pgmap v187: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 10291/235899 objects degraded (4.362%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:32:50.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10291/235899 objects degraded (4.362%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:51.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 15 81749 81734 91.2077 60.6875 0.0251697 0.0109571 2024-09-13T14:32:52.361 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:52.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 82660 82644 90.6053 56.875 0.019091 0.0110305 2024-09-13T14:32:52.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:52 smithi019 ceph-mon[91296]: pgmap v188: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 10291/235899 objects degraded (4.362%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:32:52.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:52 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2212251393' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:52.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:52 smithi003 ceph-mon[107278]: pgmap v188: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 10291/235899 objects degraded (4.362%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:32:52.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:52 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2212251393' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:52.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:52 smithi003 ceph-mon[104007]: pgmap v188: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 10291/235899 objects degraded (4.362%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:32:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:52 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2212251393' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:53.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 83638 83622 90.0969 61.125 0.0256256 0.0110919 2024-09-13T14:32:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:53 smithi019 ceph-mon[91296]: from='client.54565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:53 smithi019 ceph-mon[91296]: from='client.54568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:53.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:53 smithi003 ceph-mon[107278]: from='client.54565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:53.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:53 smithi003 ceph-mon[107278]: from='client.54568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:53.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:53 smithi003 ceph-mon[104007]: from='client.54565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:53.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:53 smithi003 ceph-mon[104007]: from='client.54568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:32:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:32:54] "GET /metrics HTTP/1.1" 200 36437 "" "Prometheus/2.51.0" 2024-09-13T14:32:54.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 84651 84635 89.6429 63.3125 0.0325858 0.0111487 2024-09-13T14:32:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:54 smithi019 ceph-mon[91296]: pgmap v189: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 952 op/s; 10087/239511 objects degraded (4.211%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:32:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:54 smithi003 ceph-mon[107278]: pgmap v189: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 952 op/s; 10087/239511 objects degraded (4.211%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:32:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:54 smithi003 ceph-mon[104007]: pgmap v189: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 952 op/s; 10087/239511 objects degraded (4.211%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:32:55.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:32:55.664760+0000 min lat: 0.00111463 max lat: 3.1215 avg lat: 0.0112157 2024-09-13T14:32:55.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:32:55.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 85563 85547 89.0988 57 0.00751635 0.0112157 2024-09-13T14:32:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10087/239511 objects degraded (4.211%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2098340608' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:55.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10087/239511 objects degraded (4.211%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:55.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2098340608' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:55.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10087/239511 objects degraded (4.211%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:55.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2098340608' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:32:56.077 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:32:56.664 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 86469 86453 88.5663 56.625 0.0104787 0.0112849 2024-09-13T14:32:56.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:56 smithi019 ceph-mon[91296]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 10087/250482 objects degraded (4.027%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:32:56.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:56 smithi003 ceph-mon[107278]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 10087/250482 objects degraded (4.027%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:32:56.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:56 smithi003 ceph-mon[104007]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 10087/250482 objects degraded (4.027%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:32:57.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 87472 87456 88.1489 62.6875 0.0168077 0.0113383 2024-09-13T14:32:58.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:57 smithi019 ceph-mon[91296]: from='client.44654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:57 smithi019 ceph-mon[91296]: from='client.54583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:58.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:57 smithi003 ceph-mon[107278]: from='client.44654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:58.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:57 smithi003 ceph-mon[107278]: from='client.54583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:58.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:57 smithi003 ceph-mon[104007]: from='client.44654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:58.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:57 smithi003 ceph-mon[104007]: from='client.54583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:58.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 88455 88439 87.7248 61.4375 0.00514398 0.0113924 2024-09-13T14:32:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:58 smithi019 ceph-mon[91296]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 52 MiB/s wr, 826 op/s; 10087/250482 objects degraded (4.027%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:32:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:58 smithi003 ceph-mon[107278]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 52 MiB/s wr, 826 op/s; 10087/250482 objects degraded (4.027%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:32:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:58 smithi003 ceph-mon[104007]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 52 MiB/s wr, 826 op/s; 10087/250482 objects degraded (4.027%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:32:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:59.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 89319 89303 87.1978 54 0.011775 0.0114617 2024-09-13T14:32:59.789 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:00.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:32:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2159768170' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:00.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:32:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2159768170' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:00.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:32:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2159768170' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:00.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 90217 90201 86.7197 56.125 0.011147 0.0115246 2024-09-13T14:33:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:33:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:33:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:00.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:00 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:33:00.364+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:00 smithi019 ceph-mon[91296]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 9859/264888 objects degraded (3.722%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:33:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:00 smithi019 ceph-mon[91296]: from='client.44666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:00 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9859/264888 objects degraded (3.722%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:01.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:00 smithi003 ceph-mon[104007]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 9859/264888 objects degraded (3.722%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:33:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:00 smithi003 ceph-mon[104007]: from='client.44666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:00 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9859/264888 objects degraded (3.722%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:00 smithi003 ceph-mon[107278]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 9859/264888 objects degraded (3.722%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:33:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:00 smithi003 ceph-mon[107278]: from='client.44666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:00 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9859/264888 objects degraded (3.722%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:01.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 91190 91174 86.3271 60.8125 0.00997472 0.0115775 2024-09-13T14:33:02.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:01 smithi019 ceph-mon[91296]: from='client.54595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:02.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:01 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:02.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:01 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:02.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:01 smithi003 ceph-mon[104007]: from='client.54595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:02.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:01 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:02.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:01 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:02.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:01 smithi003 ceph-mon[107278]: from='client.54595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:02.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:01 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:02.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:01 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:02.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 92147 92131 85.9313 59.8125 0.0273495 0.0116312 2024-09-13T14:33:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:02 smithi019 ceph-mon[91296]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 804 op/s; 9859/264888 objects degraded (3.722%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:33:03.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:02 smithi003 ceph-mon[104007]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 804 op/s; 9859/264888 objects degraded (3.722%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:33:03.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:02 smithi003 ceph-mon[107278]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 804 op/s; 9859/264888 objects degraded (3.722%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:33:03.501 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:03.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 92994 92978 85.4461 52.9375 0.0430345 0.0116967 2024-09-13T14:33:03.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:03 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2171735990' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:03.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:03 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2171735990' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:04.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:03 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2171735990' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:33:04] "GET /metrics HTTP/1.1" 200 36436 "" "Prometheus/2.51.0" 2024-09-13T14:33:04.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 93916 93900 85.0428 57.625 0.0126285 0.0117529 2024-09-13T14:33:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:04 smithi019 ceph-mon[91296]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 56 MiB/s wr, 901 op/s; 9621/268365 objects degraded (3.585%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:33:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:04 smithi019 ceph-mon[91296]: from='client.44678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:05.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:04 smithi003 ceph-mon[107278]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 56 MiB/s wr, 901 op/s; 9621/268365 objects degraded (3.585%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:33:05.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:04 smithi003 ceph-mon[107278]: from='client.44678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:04 smithi003 ceph-mon[104007]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 56 MiB/s wr, 901 op/s; 9621/268365 objects degraded (3.585%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:33:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:04 smithi003 ceph-mon[104007]: from='client.44678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:05.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 15 94839 94824 84.6528 57.75 0.0186651 0.0118074 2024-09-13T14:33:06.082 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:05 smithi019 ceph-mon[91296]: from='client.54610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:06.082 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9621/268365 objects degraded (3.585%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:06.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:05 smithi003 ceph-mon[104007]: from='client.54610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:06.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9621/268365 objects degraded (3.585%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:06.084 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:05 smithi003 ceph-mon[107278]: from='client.54610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:06.084 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9621/268365 objects degraded (3.585%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:06.665 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 95707 95691 84.2237 54.1875 0.0115178 0.0118664 2024-09-13T14:33:07.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:06 smithi019 ceph-mon[91296]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 9621/278673 objects degraded (3.452%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:33:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:06 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1482882622' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:07.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:06 smithi003 ceph-mon[104007]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 9621/278673 objects degraded (3.452%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:33:07.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:06 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1482882622' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:07.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:06 smithi003 ceph-mon[107278]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 9621/278673 objects degraded (3.452%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:33:07.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:06 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1482882622' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:07.191 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:07.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 96543 96527 83.7796 52.25 0.00434016 0.0119289 2024-09-13T14:33:08.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 97500 97484 83.4512 59.8125 0.00990368 0.0119767 2024-09-13T14:33:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:08 smithi019 ceph-mon[91296]: from='client.54619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:08 smithi019 ceph-mon[91296]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 782 op/s; 9621/278673 objects degraded (3.452%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:33:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:08 smithi019 ceph-mon[91296]: from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:09.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:08 smithi003 ceph-mon[104007]: from='client.54619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:09.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:08 smithi003 ceph-mon[104007]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 782 op/s; 9621/278673 objects degraded (3.452%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:33:09.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:08 smithi003 ceph-mon[104007]: from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:09.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:08 smithi003 ceph-mon[107278]: from='client.54619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:09.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:08 smithi003 ceph-mon[107278]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 782 op/s; 9621/278673 objects degraded (3.452%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:33:09.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:08 smithi003 ceph-mon[107278]: from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:09.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 98442 98426 83.119 58.875 0.0164922 0.0120248 2024-09-13T14:33:10.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 99286 99270 82.7141 52.75 0.0077316 0.0120832 2024-09-13T14:33:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:33:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:33:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:10.901 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:10 smithi019 ceph-mon[91296]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 9367/292065 objects degraded (3.207%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:33:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2612679627' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9367/292065 objects degraded (3.207%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:11.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:10 smithi003 ceph-mon[104007]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 9367/292065 objects degraded (3.207%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:33:11.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2612679627' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:11.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9367/292065 objects degraded (3.207%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:11.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:10 smithi003 ceph-mon[107278]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 9367/292065 objects degraded (3.207%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:33:11.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2612679627' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9367/292065 objects degraded (3.207%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:11.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 100114 100098 82.3066 51.75 0.0207981 0.0121437 2024-09-13T14:33:12.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:11 smithi019 ceph-mon[91296]: from='client.34755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:12.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:11 smithi003 ceph-mon[104007]: from='client.34755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:12.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:11 smithi003 ceph-mon[107278]: from='client.34755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:12.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 101039 101023 81.9884 57.8125 0.010731 0.0121906 2024-09-13T14:33:13.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:12 smithi019 ceph-mon[91296]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 754 op/s; 9367/292065 objects degraded (3.207%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:33:13.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:12 smithi019 ceph-mon[91296]: from='client.44711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:13.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:12 smithi003 ceph-mon[104007]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 754 op/s; 9367/292065 objects degraded (3.207%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:33:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:12 smithi003 ceph-mon[104007]: from='client.44711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:13.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:12 smithi003 ceph-mon[107278]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 754 op/s; 9367/292065 objects degraded (3.207%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:33:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:12 smithi003 ceph-mon[107278]: from='client.44711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:13.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 101932 101916 81.6528 55.8125 0.0254554 0.0122409 2024-09-13T14:33:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:33:14] "GET /metrics HTTP/1.1" 200 36435 "" "Prometheus/2.51.0" 2024-09-13T14:33:14.602 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:14.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 102735 102719 81.2544 50.1875 0.0135083 0.0123003 2024-09-13T14:33:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:14 smithi003 ceph-mon[104007]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 53 MiB/s wr, 843 op/s; 9089/295266 objects degraded (3.078%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:33:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/159860733' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:14.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:14 smithi003 ceph-mon[107278]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 53 MiB/s wr, 843 op/s; 9089/295266 objects degraded (3.078%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:33:14.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/159860733' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:14 smithi019 ceph-mon[91296]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 53 MiB/s wr, 843 op/s; 9089/295266 objects degraded (3.078%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:33:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/159860733' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:15.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:33:15.666607+0000 min lat: 0.00111463 max lat: 3.1215 avg lat: 0.0123576 2024-09-13T14:33:15.666 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:33:15.667 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 103557 103541 80.8809 51.375 0.0159783 0.0123576 2024-09-13T14:33:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:15 smithi019 ceph-mon[91296]: from='client.34761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:33:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:15 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9089/295266 objects degraded (3.078%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:16.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:15 smithi003 ceph-mon[104007]: from='client.34761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:16.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:33:16.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:15 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9089/295266 objects degraded (3.078%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:16.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:15 smithi003 ceph-mon[107278]: from='client.34761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:16.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:33:16.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:15 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9089/295266 objects degraded (3.078%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:16.669 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 104376 104360 80.5121 51.1875 0.0292615 0.0124143 2024-09-13T14:33:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:16 smithi019 ceph-mon[91296]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:16 smithi019 ceph-mon[91296]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 9089/305418 objects degraded (2.976%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:33:17.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:16 smithi003 ceph-mon[107278]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:17.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:16 smithi003 ceph-mon[107278]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 9089/305418 objects degraded (2.976%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:33:17.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:16 smithi003 ceph-mon[104007]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:17.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:16 smithi003 ceph-mon[104007]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 9089/305418 objects degraded (2.976%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:33:17.669 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 105160 105144 80.1278 49 0.0242262 0.0124736 2024-09-13T14:33:17.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:17 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/730215762' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:18.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:17 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/730215762' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:18.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:17 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/730215762' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:18.296 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:18.669 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 105917 105901 79.7324 47.3125 0.035475 0.0125352 2024-09-13T14:33:19.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:18 smithi003 ceph-mon[107278]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 742 op/s; 9089/305418 objects degraded (2.976%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:33:19.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:18 smithi003 ceph-mon[104007]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 742 op/s; 9089/305418 objects degraded (2.976%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:33:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:18 smithi019 ceph-mon[91296]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 742 op/s; 9089/305418 objects degraded (2.976%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:33:19.669 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 106760 106744 79.4104 52.6875 0.0197935 0.0125863 2024-09-13T14:33:20.097 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:19 smithi019 ceph-mon[91296]: from='client.54664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:20.097 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:19 smithi019 ceph-mon[91296]: from='client.54667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:20.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:19 smithi003 ceph-mon[107278]: from='client.54664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:20.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:19 smithi003 ceph-mon[107278]: from='client.54667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:20.121 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:19 smithi003 ceph-mon[104007]: from='client.54664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:20.121 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:19 smithi003 ceph-mon[104007]: from='client.54667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:20.376 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:33:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:33:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:20.669 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 107565 107549 79.068 50.3125 0.00771637 0.0126221 2024-09-13T14:33:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:20 smithi019 ceph-mon[91296]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 8828/317442 objects degraded (2.781%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:33:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8828/317442 objects degraded (2.781%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:20 smithi019 ceph-mon[91296]: osdmap e102: 8 total, 8 up, 8 in 2024-09-13T14:33:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:20 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:20 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[104007]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 8828/317442 objects degraded (2.781%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:33:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8828/317442 objects degraded (2.781%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[104007]: osdmap e102: 8 total, 8 up, 8 in 2024-09-13T14:33:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[107278]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 8828/317442 objects degraded (2.781%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:33:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8828/317442 objects degraded (2.781%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[107278]: osdmap e102: 8 total, 8 up, 8 in 2024-09-13T14:33:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:21.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:20 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:21.573 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:21.669 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 107795 107779 78.3158 14.375 0.0135678 0.0127624 2024-09-13T14:33:21.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:33:21.380+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:22.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4259364123' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:21 smithi019 ceph-mon[91296]: osdmap e103: 8 total, 8 up, 8 in 2024-09-13T14:33:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4259364123' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:33:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:33:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[107278]: osdmap e103: 8 total, 8 up, 8 in 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4259364123' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:33:22.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:33:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:33:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:33:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:33:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:22.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:21 smithi003 ceph-mon[104007]: osdmap e103: 8 total, 8 up, 8 in 2024-09-13T14:33:22.669 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 108633 108617 78.0176 52.375 0.0294123 0.0128117 2024-09-13T14:33:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:22 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:22 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:22 smithi019 ceph-mon[91296]: pgmap v204: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 53 MiB/s wr, 845 op/s; 8828/317442 objects degraded (2.781%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:33:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:22 smithi019 ceph-mon[91296]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:23.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:22 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:23.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:22 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:22 smithi003 ceph-mon[107278]: pgmap v204: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 53 MiB/s wr, 845 op/s; 8828/317442 objects degraded (2.781%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:33:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:22 smithi003 ceph-mon[107278]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:22 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:22 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:22 smithi003 ceph-mon[104007]: pgmap v204: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 53 MiB/s wr, 845 op/s; 8828/317442 objects degraded (2.781%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:33:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:22 smithi003 ceph-mon[104007]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:23.669 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 109508 109492 77.7525 54.6875 0.0123082 0.0128538 2024-09-13T14:33:24.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:23 smithi019 ceph-mon[91296]: from='client.44756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:24.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:23 smithi003 ceph-mon[107278]: from='client.44756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:24.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:23 smithi003 ceph-mon[104007]: from='client.44756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:24.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:33:24] "GET /metrics HTTP/1.1" 200 36435 "" "Prometheus/2.51.0" 2024-09-13T14:33:24.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 110353 110337 77.4722 52.8125 0.023403 0.0129017 2024-09-13T14:33:25.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:24 smithi019 ceph-mon[91296]: pgmap v206: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 38 MiB/s wr, 611 op/s; 8684/320088 objects degraded (2.713%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:33:25.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:24 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2150884696' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:25.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:24 smithi003 ceph-mon[104007]: pgmap v206: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 38 MiB/s wr, 611 op/s; 8684/320088 objects degraded (2.713%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:33:25.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:24 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2150884696' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:24 smithi003 ceph-mon[107278]: pgmap v206: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 38 MiB/s wr, 611 op/s; 8684/320088 objects degraded (2.713%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:33:25.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:24 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2150884696' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:25.277 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:25.669 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 90 2 111115 111113 77.1503 48.5 0.0329904 0.0129564 2024-09-13T14:33:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8684/320088 objects degraded (2.713%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:26.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8684/320088 objects degraded (2.713%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:26.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8684/320088 objects degraded (2.713%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0162 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 111115 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 77.1493 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 44.6391 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 243.312 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1234 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 714.238 2024-09-13T14:33:26.670 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3893 2024-09-13T14:33:26.671 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-09-13T14:33:26.671 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0129565 2024-09-13T14:33:26.671 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0405448 2024-09-13T14:33:26.671 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 3.1215 2024-09-13T14:33:26.671 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00111463 2024-09-13T14:33:26.679 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:33:27.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:26 smithi019 ceph-mon[91296]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:27.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:26 smithi019 ceph-mon[91296]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s; 8456/328254 objects degraded (2.576%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:33:27.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:26 smithi019 ceph-mon[91296]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:26 smithi003 ceph-mon[104007]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:26 smithi003 ceph-mon[104007]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s; 8456/328254 objects degraded (2.576%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:33:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:26 smithi003 ceph-mon[104007]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:27.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:26 smithi003 ceph-mon[107278]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:27.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:26 smithi003 ceph-mon[107278]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s; 8456/328254 objects degraded (2.576%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:33:27.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:26 smithi003 ceph-mon[107278]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:28.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:28.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:28.896 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:29.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:28 smithi019 ceph-mon[91296]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 28 MiB/s wr, 450 op/s; 8456/328254 objects degraded (2.576%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:33:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/920585943' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:28 smithi003 ceph-mon[104007]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 28 MiB/s wr, 450 op/s; 8456/328254 objects degraded (2.576%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:33:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/920585943' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:28 smithi003 ceph-mon[107278]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 28 MiB/s wr, 450 op/s; 8456/328254 objects degraded (2.576%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:33:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/920585943' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:30.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:29 smithi019 ceph-mon[91296]: from='client.44774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:30.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:29 smithi003 ceph-mon[107278]: from='client.44774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:30.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:29 smithi003 ceph-mon[104007]: from='client.44774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:30.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:33:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:33:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:30 smithi019 ceph-mon[91296]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 114 B/s rd, 37 MiB/s wr, 770 op/s; 8031/328719 objects degraded (2.443%); 5.6 MiB/s, 88 objects/s recovering 2024-09-13T14:33:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:30 smithi019 ceph-mon[91296]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8031/328719 objects degraded (2.443%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:31.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:30 smithi003 ceph-mon[107278]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 114 B/s rd, 37 MiB/s wr, 770 op/s; 8031/328719 objects degraded (2.443%); 5.6 MiB/s, 88 objects/s recovering 2024-09-13T14:33:31.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:30 smithi003 ceph-mon[107278]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:31.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8031/328719 objects degraded (2.443%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:30 smithi003 ceph-mon[104007]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 114 B/s rd, 37 MiB/s wr, 770 op/s; 8031/328719 objects degraded (2.443%); 5.6 MiB/s, 88 objects/s recovering 2024-09-13T14:33:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:30 smithi003 ceph-mon[104007]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8031/328719 objects degraded (2.443%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:32.577 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:33.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:32 smithi019 ceph-mon[91296]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 102 B/s rd, 33 MiB/s wr, 689 op/s; 8031/328719 objects degraded (2.443%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:33:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3796356921' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:32 smithi003 ceph-mon[107278]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 102 B/s rd, 33 MiB/s wr, 689 op/s; 8031/328719 objects degraded (2.443%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:33:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3796356921' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:32 smithi003 ceph-mon[104007]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 102 B/s rd, 33 MiB/s wr, 689 op/s; 8031/328719 objects degraded (2.443%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:33:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3796356921' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:34.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:33 smithi019 ceph-mon[91296]: from='client.54709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:34.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:33 smithi003 ceph-mon[107278]: from='client.54709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:33:34] "GET /metrics HTTP/1.1" 200 36436 "" "Prometheus/2.51.0" 2024-09-13T14:33:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:33 smithi003 ceph-mon[104007]: from='client.54709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:35.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:34 smithi019 ceph-mon[91296]: from='client.44795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:34 smithi019 ceph-mon[91296]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 85 B/s rd, 28 MiB/s wr, 662 op/s; 8031/325626 objects degraded (2.466%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:33:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:34 smithi003 ceph-mon[107278]: from='client.44795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:34 smithi003 ceph-mon[107278]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 85 B/s rd, 28 MiB/s wr, 662 op/s; 8031/325626 objects degraded (2.466%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:33:35.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:34 smithi003 ceph-mon[104007]: from='client.44795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:34 smithi003 ceph-mon[104007]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 85 B/s rd, 28 MiB/s wr, 662 op/s; 8031/325626 objects degraded (2.466%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:33:36.250 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:36.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8031/325626 objects degraded (2.466%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:36.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:35 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/86478955' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:36.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8031/325626 objects degraded (2.466%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:36.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:35 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/86478955' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:36.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8031/325626 objects degraded (2.466%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:36.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:35 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/86478955' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:36.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:36 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:33:36.396+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:37.265 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:36 smithi003 ceph-mon[104007]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 B/s rd, 23 MiB/s wr, 851 op/s; 7749/316107 objects degraded (2.451%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:33:37.265 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:36 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:37.266 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:36 smithi003 ceph-mon[107278]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 B/s rd, 23 MiB/s wr, 851 op/s; 7749/316107 objects degraded (2.451%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:33:37.266 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:36 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:37.349 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:36 smithi019 ceph-mon[91296]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 B/s rd, 23 MiB/s wr, 851 op/s; 7749/316107 objects degraded (2.451%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:33:37.350 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:36 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:38.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:37 smithi019 ceph-mon[91296]: from='client.34839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:38.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:37 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:38.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:37 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:38.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:37 smithi019 ceph-mon[91296]: from='client.54727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:38.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:37 smithi003 ceph-mon[104007]: from='client.34839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:38.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:37 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:38.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:37 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:38.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:37 smithi003 ceph-mon[104007]: from='client.54727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:38.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:37 smithi003 ceph-mon[107278]: from='client.34839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:38.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:37 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:38.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:37 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:38.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:37 smithi003 ceph-mon[107278]: from='client.54727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:39.282 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:38 smithi003 ceph-mon[107278]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 624 op/s; 7749/316107 objects degraded (2.451%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:39.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:38 smithi003 ceph-mon[104007]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 624 op/s; 7749/316107 objects degraded (2.451%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:38 smithi019 ceph-mon[91296]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 624 op/s; 7749/316107 objects degraded (2.451%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:39.928 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:40.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/890197509' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:40.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:39 smithi003 ceph-mon[107278]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 959 op/s; 7451/304023 objects degraded (2.451%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:33:40.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:39 smithi003 ceph-mon[107278]: from='client.54733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:40.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/890197509' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:40.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:39 smithi003 ceph-mon[104007]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 959 op/s; 7451/304023 objects degraded (2.451%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:33:40.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:39 smithi003 ceph-mon[104007]: from='client.54733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:40.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:33:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:33:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/890197509' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:39 smithi019 ceph-mon[91296]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 959 op/s; 7451/304023 objects degraded (2.451%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:33:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:39 smithi019 ceph-mon[91296]: from='client.54733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:41.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 7451/304023 objects degraded (2.451%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:41.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:40 smithi019 ceph-mon[91296]: from='client.54739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 7451/304023 objects degraded (2.451%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:41 smithi003 ceph-mon[107278]: from='client.54739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:41.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 7451/304023 objects degraded (2.451%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:41.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:40 smithi003 ceph-mon[104007]: from='client.54739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:42.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:41 smithi019 ceph-mon[91296]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 7451/304023 objects degraded (2.451%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:33:42.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:41 smithi003 ceph-mon[107278]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 7451/304023 objects degraded (2.451%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:33:42.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:41 smithi003 ceph-mon[104007]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 7451/304023 objects degraded (2.451%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:33:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:43.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:43.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:43.591 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4105877483' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:44 smithi019 ceph-mon[91296]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 7451/301131 objects degraded (2.474%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:33:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:44 smithi019 ceph-mon[91296]: from='client.44822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4105877483' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:44 smithi003 ceph-mon[107278]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 7451/301131 objects degraded (2.474%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:33:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:44 smithi003 ceph-mon[107278]: from='client.44822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:44.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:33:44] "GET /metrics HTTP/1.1" 200 36441 "" "Prometheus/2.51.0" 2024-09-13T14:33:44.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4105877483' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:44.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:44 smithi003 ceph-mon[104007]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 7451/301131 objects degraded (2.474%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:33:44.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:44 smithi003 ceph-mon[104007]: from='client.44822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:45 smithi019 ceph-mon[91296]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:45.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:45 smithi003 ceph-mon[104007]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:45.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:45 smithi003 ceph-mon[107278]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:46.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:46 smithi019 ceph-mon[91296]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 7131/292089 objects degraded (2.441%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:33:46.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 7451/301131 objects degraded (2.474%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:46.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:46 smithi003 ceph-mon[104007]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 7131/292089 objects degraded (2.441%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:33:46.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 7451/301131 objects degraded (2.474%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:46 smithi003 ceph-mon[107278]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 7131/292089 objects degraded (2.441%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:33:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 7451/301131 objects degraded (2.474%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:47.275 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2504765559' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2504765559' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:47.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2504765559' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:48.314 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:48 smithi019 ceph-mon[91296]: from='client.34875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:48.315 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:48 smithi019 ceph-mon[91296]: pgmap v218: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 7131/292089 objects degraded (2.441%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:33:48.315 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:48 smithi019 ceph-mon[91296]: from='client.44837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:48.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:48 smithi003 ceph-mon[107278]: from='client.34875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:48.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:48 smithi003 ceph-mon[107278]: pgmap v218: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 7131/292089 objects degraded (2.441%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:33:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:48 smithi003 ceph-mon[107278]: from='client.44837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:48 smithi003 ceph-mon[104007]: from='client.34875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:48 smithi003 ceph-mon[104007]: pgmap v218: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 7131/292089 objects degraded (2.441%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:33:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:48 smithi003 ceph-mon[104007]: from='client.44837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:33:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:33:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:50 smithi019 ceph-mon[91296]: pgmap v219: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 6783/280257 objects degraded (2.420%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:33:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:50 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3289407586' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 6783/280257 objects degraded (2.420%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:50.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:50 smithi003 ceph-mon[107278]: pgmap v219: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 6783/280257 objects degraded (2.420%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:33:50.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:50 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3289407586' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:50.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 6783/280257 objects degraded (2.420%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:50.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:50 smithi003 ceph-mon[104007]: pgmap v219: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 6783/280257 objects degraded (2.420%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:33:50.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:50 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3289407586' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:50.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 6783/280257 objects degraded (2.420%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:50.974 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:51.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:51 smithi003 ceph-mon[107278]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:51.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:51 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:51.683 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:33:51.417+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:51 smithi003 ceph-mon[104007]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:51 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:51.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:51 smithi019 ceph-mon[91296]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:51.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:51 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:33:52.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:52 smithi019 ceph-mon[91296]: pgmap v220: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 6783/280257 objects degraded (2.420%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:33:52.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:52 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:52.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:52 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:52.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:52 smithi019 ceph-mon[91296]: from='client.34893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:52.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:52 smithi003 ceph-mon[107278]: pgmap v220: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 6783/280257 objects degraded (2.420%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:33:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:52 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:52 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:52 smithi003 ceph-mon[107278]: from='client.34893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:52 smithi003 ceph-mon[104007]: pgmap v220: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 6783/280257 objects degraded (2.420%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:33:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:52 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:33:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:52 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:33:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:52 smithi003 ceph-mon[104007]: from='client.34893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:53.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:53 smithi019 ceph-mon[91296]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:33:53.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:53 smithi003 ceph-mon[107278]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:33:53.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:53 smithi003 ceph-mon[104007]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:33:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:33:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:33:54] "GET /metrics HTTP/1.1" 200 36441 "" "Prometheus/2.51.0" 2024-09-13T14:33:54.664 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:54 smithi019 ceph-mon[91296]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 6783/277623 objects degraded (2.443%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:33:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:54 smithi019 ceph-mon[91296]: osdmap e105: 8 total, 8 up, 8 in 2024-09-13T14:33:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:54 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2373543252' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:54 smithi003 ceph-mon[107278]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 6783/277623 objects degraded (2.443%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:33:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:54 smithi003 ceph-mon[107278]: osdmap e105: 8 total, 8 up, 8 in 2024-09-13T14:33:54.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:54 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2373543252' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:54 smithi003 ceph-mon[104007]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 6783/277623 objects degraded (2.443%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:33:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:54 smithi003 ceph-mon[104007]: osdmap e105: 8 total, 8 up, 8 in 2024-09-13T14:33:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:54 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2373543252' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:55 smithi019 ceph-mon[91296]: from='client.44858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:55.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 6783/277623 objects degraded (2.443%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:55.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:55 smithi003 ceph-mon[107278]: from='client.44858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:55.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 6783/277623 objects degraded (2.443%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:55.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:55 smithi003 ceph-mon[104007]: from='client.44858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:55.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 6783/277623 objects degraded (2.443%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:56 smithi019 ceph-mon[91296]: from='client.34905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:56 smithi019 ceph-mon[91296]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 6388/269901 objects degraded (2.367%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:33:56.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:56 smithi003 ceph-mon[104007]: from='client.34905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:56.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:56 smithi003 ceph-mon[104007]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 6388/269901 objects degraded (2.367%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:33:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:56 smithi003 ceph-mon[107278]: from='client.34905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:56 smithi003 ceph-mon[107278]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 6388/269901 objects degraded (2.367%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:33:58.359 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:33:58.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:58 smithi003 ceph-mon[104007]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 431 op/s; 6388/269901 objects degraded (2.367%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:33:58.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:58.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:58 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4184890114' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:58.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:58 smithi003 ceph-mon[107278]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 431 op/s; 6388/269901 objects degraded (2.367%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:33:58.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:58.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:58 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4184890114' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:58.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:58 smithi019 ceph-mon[91296]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 431 op/s; 6388/269901 objects degraded (2.367%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:33:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:58 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4184890114' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:33:59.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:59 smithi019 ceph-mon[91296]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:33:59 smithi019 ceph-mon[91296]: from='client.44870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:59.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:59 smithi003 ceph-mon[104007]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:59.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:33:59 smithi003 ceph-mon[104007]: from='client.44870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:59.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:59 smithi003 ceph-mon[107278]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:59.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:33:59 smithi003 ceph-mon[107278]: from='client.44870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:00.515 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:34:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:34:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:00.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:00 smithi019 ceph-mon[91296]: pgmap v226: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 6237/258852 objects degraded (2.409%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:34:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 6237/258852 objects degraded (2.409%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:00.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:00 smithi003 ceph-mon[104007]: pgmap v226: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 6237/258852 objects degraded (2.409%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:34:00.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 6237/258852 objects degraded (2.409%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:00 smithi003 ceph-mon[107278]: pgmap v226: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 6237/258852 objects degraded (2.409%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:34:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 6237/258852 objects degraded (2.409%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:01.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:01 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/42973688' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:01.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:01 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/42973688' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:01.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:01 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/42973688' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:02.051 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:02 smithi019 ceph-mon[91296]: pgmap v227: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 6237/258852 objects degraded (2.409%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:34:02.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:02 smithi003 ceph-mon[104007]: pgmap v227: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 6237/258852 objects degraded (2.409%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:34:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:02 smithi003 ceph-mon[107278]: pgmap v227: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 6237/258852 objects degraded (2.409%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:34:03.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:03 smithi019 ceph-mon[91296]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:03.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:03 smithi019 ceph-mon[91296]: from='client.44882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:03.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:03 smithi003 ceph-mon[104007]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:03.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:03 smithi003 ceph-mon[104007]: from='client.44882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:03.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:03 smithi003 ceph-mon[107278]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:03.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:03 smithi003 ceph-mon[107278]: from='client.44882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:34:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:34:04] "GET /metrics HTTP/1.1" 200 36438 "" "Prometheus/2.51.0" 2024-09-13T14:34:04.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:04 smithi003 ceph-mon[104007]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 6237/255858 objects degraded (2.438%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:34:04.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:04 smithi019 ceph-mon[91296]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 6237/255858 objects degraded (2.438%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:34:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:04 smithi003 ceph-mon[107278]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 6237/255858 objects degraded (2.438%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:34:05.721 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:05.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2627880814' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:05.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 6237/255858 objects degraded (2.438%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:05.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2627880814' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:05.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 6237/255858 objects degraded (2.438%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2627880814' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 6237/255858 objects degraded (2.438%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:06.832 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:06 smithi019 ceph-mon[91296]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 6084/247278 objects degraded (2.460%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:34:06.832 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:06 smithi019 ceph-mon[91296]: from='client.54820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:06.833 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:06.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:06 smithi003 ceph-mon[104007]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 6084/247278 objects degraded (2.460%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:34:06.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:06 smithi003 ceph-mon[104007]: from='client.54820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:06.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:06.833 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:06 smithi003 ceph-mon[107278]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 6084/247278 objects degraded (2.460%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:34:06.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:06 smithi003 ceph-mon[107278]: from='client.54820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:06.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:06.834 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:34:06 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:34:06.437+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:07.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:07 smithi019 ceph-mon[91296]: from='client.54823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:07 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:07 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:07.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:07 smithi003 ceph-mon[104007]: from='client.54823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:07.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:07 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:07.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:07 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:07.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:07 smithi003 ceph-mon[107278]: from='client.54823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:07.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:07 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:07.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:07 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:08.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:08 smithi019 ceph-mon[91296]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 628 op/s; 6084/247278 objects degraded (2.460%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:34:08.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:08 smithi003 ceph-mon[107278]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 628 op/s; 6084/247278 objects degraded (2.460%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:34:08.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:08 smithi003 ceph-mon[104007]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 628 op/s; 6084/247278 objects degraded (2.460%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:34:09.401 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:09.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2480226391' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2480226391' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:09.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2480226391' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:10.611 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:34:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:34:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:10 smithi019 ceph-mon[91296]: from='client.34947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:10 smithi019 ceph-mon[91296]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 5927/235986 objects degraded (2.512%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:34:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:10 smithi019 ceph-mon[91296]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:10.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5927/235986 objects degraded (2.512%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:10.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:10 smithi003 ceph-mon[107278]: from='client.34947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:10.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:10 smithi003 ceph-mon[107278]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 5927/235986 objects degraded (2.512%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:34:10.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:10 smithi003 ceph-mon[107278]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:10.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5927/235986 objects degraded (2.512%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:10 smithi003 ceph-mon[104007]: from='client.34947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:10.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:10 smithi003 ceph-mon[104007]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 5927/235986 objects degraded (2.512%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:34:10.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:10 smithi003 ceph-mon[104007]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:10.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5927/235986 objects degraded (2.512%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:12.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:12 smithi019 ceph-mon[91296]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 5927/235986 objects degraded (2.512%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:34:12.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:12 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4242729585' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:12.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:12 smithi003 ceph-mon[107278]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 5927/235986 objects degraded (2.512%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:34:12.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:12 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4242729585' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:12.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:12 smithi003 ceph-mon[104007]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 5927/235986 objects degraded (2.512%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:34:12.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:12 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4242729585' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:13.067 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:13.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:13.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:13.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:34:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:34:14] "GET /metrics HTTP/1.1" 200 36439 "" "Prometheus/2.51.0" 2024-09-13T14:34:14.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:14 smithi019 ceph-mon[91296]: from='client.54838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:14 smithi019 ceph-mon[91296]: pgmap v233: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 5927/233175 objects degraded (2.542%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:34:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:14 smithi019 ceph-mon[91296]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:14 smithi003 ceph-mon[104007]: from='client.54838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:14 smithi003 ceph-mon[104007]: pgmap v233: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 5927/233175 objects degraded (2.542%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:34:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:14 smithi003 ceph-mon[104007]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:14.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:14 smithi003 ceph-mon[107278]: from='client.54838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:14.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:14 smithi003 ceph-mon[107278]: pgmap v233: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 5927/233175 objects degraded (2.542%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:34:14.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:14 smithi003 ceph-mon[107278]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:15.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:15 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5927/233175 objects degraded (2.542%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:15.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:15 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5927/233175 objects degraded (2.542%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:15 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5927/233175 objects degraded (2.542%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:16.767 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:16.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:16 smithi019 ceph-mon[91296]: pgmap v234: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 5732/225030 objects degraded (2.547%); 2.1 MiB/s, 42 objects/s recovering 2024-09-13T14:34:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:16 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2314489501' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:16.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:16 smithi003 ceph-mon[104007]: pgmap v234: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 5732/225030 objects degraded (2.547%); 2.1 MiB/s, 42 objects/s recovering 2024-09-13T14:34:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:16 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2314489501' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:16 smithi003 ceph-mon[107278]: pgmap v234: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 5732/225030 objects degraded (2.547%); 2.1 MiB/s, 42 objects/s recovering 2024-09-13T14:34:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:16 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2314489501' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:17.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:17 smithi019 ceph-mon[91296]: from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:17.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:17 smithi003 ceph-mon[107278]: from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:17.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:17 smithi003 ceph-mon[104007]: from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:18.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:18 smithi019 ceph-mon[91296]: from='client.54853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:18.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:18 smithi019 ceph-mon[91296]: pgmap v235: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 5732/225030 objects degraded (2.547%); 1.3 MiB/s, 29 objects/s recovering 2024-09-13T14:34:18.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:18 smithi003 ceph-mon[107278]: from='client.54853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:18.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:18 smithi003 ceph-mon[107278]: pgmap v235: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 5732/225030 objects degraded (2.547%); 1.3 MiB/s, 29 objects/s recovering 2024-09-13T14:34:18.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:18 smithi003 ceph-mon[104007]: from='client.54853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:18.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:18 smithi003 ceph-mon[104007]: pgmap v235: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 5732/225030 objects degraded (2.547%); 1.3 MiB/s, 29 objects/s recovering 2024-09-13T14:34:20.433 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:20.655 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:34:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:34:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:20.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:20 smithi019 ceph-mon[91296]: pgmap v236: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 5541/213660 objects degraded (2.593%); 1.3 MiB/s, 45 objects/s recovering 2024-09-13T14:34:20.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:20 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/663089844' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:20.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5541/213660 objects degraded (2.593%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:20 smithi003 ceph-mon[107278]: pgmap v236: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 5541/213660 objects degraded (2.593%); 1.3 MiB/s, 45 objects/s recovering 2024-09-13T14:34:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:20 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/663089844' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:20.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5541/213660 objects degraded (2.593%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:20.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:20 smithi003 ceph-mon[104007]: pgmap v236: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 5541/213660 objects degraded (2.593%); 1.3 MiB/s, 45 objects/s recovering 2024-09-13T14:34:20.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:20 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/663089844' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:20.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5541/213660 objects degraded (2.593%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:21.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:21 smithi019 ceph-mon[91296]: from='client.54856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:21.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:21 smithi019 ceph-mon[91296]: from='client.44960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:21.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:21.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:21 smithi003 ceph-mon[104007]: from='client.54856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:21.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:21 smithi003 ceph-mon[104007]: from='client.44960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:21.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:21.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:21 smithi003 ceph-mon[107278]: from='client.54856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:21.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:21 smithi003 ceph-mon[107278]: from='client.44960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:21.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:22.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:22 smithi019 ceph-mon[91296]: pgmap v237: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 5541/213660 objects degraded (2.593%); 533 KiB/s, 32 objects/s recovering 2024-09-13T14:34:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:22 smithi003 ceph-mon[107278]: pgmap v237: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 5541/213660 objects degraded (2.593%); 533 KiB/s, 32 objects/s recovering 2024-09-13T14:34:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:22 smithi003 ceph-mon[104007]: pgmap v237: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 5541/213660 objects degraded (2.593%); 533 KiB/s, 32 objects/s recovering 2024-09-13T14:34:23.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:23 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/800616012' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:23.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:23 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/800616012' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:23.959 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:23 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/800616012' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:24.134 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:34:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:34:24] "GET /metrics HTTP/1.1" 200 36439 "" "Prometheus/2.51.0" 2024-09-13T14:34:24.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:24 smithi019 ceph-mon[91296]: pgmap v238: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 5541/211044 objects degraded (2.626%); 533 KiB/s, 32 objects/s recovering 2024-09-13T14:34:24.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:24 smithi003 ceph-mon[107278]: pgmap v238: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 5541/211044 objects degraded (2.626%); 533 KiB/s, 32 objects/s recovering 2024-09-13T14:34:24.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:24 smithi003 ceph-mon[104007]: pgmap v238: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 5541/211044 objects degraded (2.626%); 533 KiB/s, 32 objects/s recovering 2024-09-13T14:34:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:25 smithi019 ceph-mon[91296]: from='client.54868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:25 smithi019 ceph-mon[91296]: from='client.44978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5541/211044 objects degraded (2.626%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:26.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:25 smithi003 ceph-mon[104007]: from='client.54868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:26.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:25 smithi003 ceph-mon[104007]: from='client.44978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:26.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5541/211044 objects degraded (2.626%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:26.053 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:25 smithi003 ceph-mon[107278]: from='client.54868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:26.053 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:25 smithi003 ceph-mon[107278]: from='client.44978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:26.053 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5541/211044 objects degraded (2.626%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:26 smithi019 ceph-mon[91296]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 5355/203166 objects degraded (2.636%); 533 KiB/s, 47 objects/s recovering 2024-09-13T14:34:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:26 smithi003 ceph-mon[104007]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 5355/203166 objects degraded (2.636%); 533 KiB/s, 47 objects/s recovering 2024-09-13T14:34:27.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:27.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:27.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:26 smithi003 ceph-mon[107278]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 5355/203166 objects degraded (2.636%); 533 KiB/s, 47 objects/s recovering 2024-09-13T14:34:27.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:27.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:27.869 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:28.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:27 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2711291079' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:28.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:28.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:28.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:27 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2711291079' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:28.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:28.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:27 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2711291079' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:28.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:34:28.103+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: pgmap v240: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 5355/203166 objects degraded (2.636%); 0 B/s, 31 objects/s recovering 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='client.54886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: pgmap v240: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 5355/203166 objects degraded (2.636%); 0 B/s, 31 objects/s recovering 2024-09-13T14:34:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='client.54886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:34:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:34:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: pgmap v240: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 5355/203166 objects degraded (2.636%); 0 B/s, 31 objects/s recovering 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='client.54886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:34:29.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:34:29.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:29.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:29.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:29.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:29.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:30.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:29 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:29 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:29 smithi019 ceph-mon[91296]: from='client.44993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:30.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:29 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:30.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:29 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:30.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:29 smithi003 ceph-mon[104007]: from='client.44993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:30.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:29 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:30.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:29 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:30.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:29 smithi003 ceph-mon[107278]: from='client.44993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:30.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:34:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:34:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:31.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:30 smithi019 ceph-mon[91296]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 890 op/s; 5141/192960 objects degraded (2.664%); 0 B/s, 49 objects/s recovering 2024-09-13T14:34:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5141/192960 objects degraded (2.664%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:31.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:30 smithi003 ceph-mon[104007]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 890 op/s; 5141/192960 objects degraded (2.664%); 0 B/s, 49 objects/s recovering 2024-09-13T14:34:31.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5141/192960 objects degraded (2.664%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:31.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:30 smithi003 ceph-mon[107278]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 890 op/s; 5141/192960 objects degraded (2.664%); 0 B/s, 49 objects/s recovering 2024-09-13T14:34:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5141/192960 objects degraded (2.664%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:31.567 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:31.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2571482984' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2571482984' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:32.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2571482984' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:33.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:32 smithi019 ceph-mon[91296]: pgmap v242: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 5141/192960 objects degraded (2.664%); 0 B/s, 33 objects/s recovering 2024-09-13T14:34:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:32 smithi019 ceph-mon[91296]: from='client.54898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:32 smithi003 ceph-mon[107278]: pgmap v242: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 5141/192960 objects degraded (2.664%); 0 B/s, 33 objects/s recovering 2024-09-13T14:34:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:32 smithi003 ceph-mon[107278]: from='client.54898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:32 smithi003 ceph-mon[104007]: pgmap v242: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 5141/192960 objects degraded (2.664%); 0 B/s, 33 objects/s recovering 2024-09-13T14:34:33.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:32 smithi003 ceph-mon[104007]: from='client.54898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.094 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:33 smithi003 ceph-mon[107278]: from='client.54904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:33 smithi003 ceph-mon[104007]: from='client.54904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:33 smithi019 ceph-mon[91296]: from='client.54904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:34:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:34:34] "GET /metrics HTTP/1.1" 200 36425 "" "Prometheus/2.51.0" 2024-09-13T14:34:35.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:34 smithi019 ceph-mon[91296]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 5141/192960 objects degraded (2.664%); 0 B/s, 33 objects/s recovering 2024-09-13T14:34:35.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:34 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1983219772' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:34 smithi003 ceph-mon[107278]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 5141/192960 objects degraded (2.664%); 0 B/s, 33 objects/s recovering 2024-09-13T14:34:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:34 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1983219772' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:34 smithi003 ceph-mon[104007]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 5141/192960 objects degraded (2.664%); 0 B/s, 33 objects/s recovering 2024-09-13T14:34:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:34 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1983219772' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:35.254 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:36.095 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4906/181827 objects degraded (2.698%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:36.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4906/181827 objects degraded (2.698%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:36.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4906/181827 objects degraded (2.698%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:36 smithi019 ceph-mon[91296]: from='client.45011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:36 smithi019 ceph-mon[91296]: pgmap v244: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 4906/181827 objects degraded (2.698%); 0 B/s, 52 objects/s recovering 2024-09-13T14:34:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:36 smithi019 ceph-mon[91296]: from='client.45014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:36 smithi003 ceph-mon[104007]: from='client.45011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:36 smithi003 ceph-mon[104007]: pgmap v244: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 4906/181827 objects degraded (2.698%); 0 B/s, 52 objects/s recovering 2024-09-13T14:34:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:36 smithi003 ceph-mon[104007]: from='client.45014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:36 smithi003 ceph-mon[107278]: from='client.45011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:36 smithi003 ceph-mon[107278]: pgmap v244: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 4906/181827 objects degraded (2.698%); 0 B/s, 52 objects/s recovering 2024-09-13T14:34:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:36 smithi003 ceph-mon[107278]: from='client.45014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:38.987 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:39.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:38 smithi019 ceph-mon[91296]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 592 op/s; 4906/181827 objects degraded (2.698%); 0 B/s, 37 objects/s recovering 2024-09-13T14:34:39.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1930122654' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:38 smithi003 ceph-mon[104007]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 592 op/s; 4906/181827 objects degraded (2.698%); 0 B/s, 37 objects/s recovering 2024-09-13T14:34:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1930122654' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:39.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:38 smithi003 ceph-mon[107278]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 592 op/s; 4906/181827 objects degraded (2.698%); 0 B/s, 37 objects/s recovering 2024-09-13T14:34:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1930122654' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:40.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:39 smithi019 ceph-mon[91296]: from='client.54922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:40.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:39 smithi003 ceph-mon[104007]: from='client.54922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:40.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:39 smithi003 ceph-mon[107278]: from='client.54922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:40.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:34:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:34:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:41.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:40 smithi019 ceph-mon[91296]: pgmap v246: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 4682/171309 objects degraded (2.733%); 0 B/s, 56 objects/s recovering 2024-09-13T14:34:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:40 smithi019 ceph-mon[91296]: from='client.45029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:41.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:40 smithi003 ceph-mon[104007]: pgmap v246: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 4682/171309 objects degraded (2.733%); 0 B/s, 56 objects/s recovering 2024-09-13T14:34:41.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:40 smithi003 ceph-mon[104007]: from='client.45029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:41.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:40 smithi003 ceph-mon[107278]: pgmap v246: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 4682/171309 objects degraded (2.733%); 0 B/s, 56 objects/s recovering 2024-09-13T14:34:41.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:40 smithi003 ceph-mon[107278]: from='client.45029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:42.699 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:43.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:42 smithi003 ceph-mon[104007]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 601 op/s; 4682/171309 objects degraded (2.733%); 0 B/s, 38 objects/s recovering 2024-09-13T14:34:43.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:42 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2598335838' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:43.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:43.126 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:42 smithi003 ceph-mon[107278]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 601 op/s; 4682/171309 objects degraded (2.733%); 0 B/s, 38 objects/s recovering 2024-09-13T14:34:43.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:42 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2598335838' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:43.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:43.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:42 smithi019 ceph-mon[91296]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 601 op/s; 4682/171309 objects degraded (2.733%); 0 B/s, 38 objects/s recovering 2024-09-13T14:34:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:42 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2598335838' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:43.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:34:43 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:34:43.124+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:44.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:43 smithi019 ceph-mon[91296]: from='client.45035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:44.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:44.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:43 smithi003 ceph-mon[107278]: from='client.45035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:44.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:44.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:43 smithi003 ceph-mon[104007]: from='client.45035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:44.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:44.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:34:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:34:44] "GET /metrics HTTP/1.1" 200 36420 "" "Prometheus/2.51.0" 2024-09-13T14:34:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:44 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:44 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:44 smithi019 ceph-mon[91296]: from='client.35055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:44 smithi019 ceph-mon[91296]: pgmap v248: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 601 op/s; 4682/171309 objects degraded (2.733%); 0 B/s, 38 objects/s recovering 2024-09-13T14:34:45.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:44 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:45.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:44 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:45.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:44 smithi003 ceph-mon[107278]: from='client.35055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:45.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:44 smithi003 ceph-mon[107278]: pgmap v248: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 601 op/s; 4682/171309 objects degraded (2.733%); 0 B/s, 38 objects/s recovering 2024-09-13T14:34:45.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:44 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:45.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:44 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-13T14:34:45.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:44 smithi003 ceph-mon[104007]: from='client.35055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:45.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:44 smithi003 ceph-mon[104007]: pgmap v248: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 601 op/s; 4682/171309 objects degraded (2.733%); 0 B/s, 38 objects/s recovering 2024-09-13T14:34:46.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4682/171309 objects degraded (2.733%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:45 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1827859221' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:46.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4682/171309 objects degraded (2.733%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:46.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:45 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1827859221' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:46.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4682/171309 objects degraded (2.733%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:46.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:45 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1827859221' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:46.431 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:47.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:46 smithi019 ceph-mon[91296]: pgmap v249: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 898 op/s; 4436/160611 objects degraded (2.762%); 0 B/s, 58 objects/s recovering 2024-09-13T14:34:47.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:46 smithi003 ceph-mon[107278]: pgmap v249: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 898 op/s; 4436/160611 objects degraded (2.762%); 0 B/s, 58 objects/s recovering 2024-09-13T14:34:47.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:46 smithi003 ceph-mon[104007]: pgmap v249: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 898 op/s; 4436/160611 objects degraded (2.762%); 0 B/s, 58 objects/s recovering 2024-09-13T14:34:48.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:47 smithi019 ceph-mon[91296]: from='client.54946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:48.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:47 smithi019 ceph-mon[91296]: osdmap e106: 8 total, 8 up, 8 in 2024-09-13T14:34:48.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:47 smithi019 ceph-mon[91296]: from='client.54952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:48.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:47 smithi003 ceph-mon[104007]: from='client.54946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:48.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:47 smithi003 ceph-mon[104007]: osdmap e106: 8 total, 8 up, 8 in 2024-09-13T14:34:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:47 smithi003 ceph-mon[104007]: from='client.54952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:47 smithi003 ceph-mon[107278]: from='client.54946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:47 smithi003 ceph-mon[107278]: osdmap e106: 8 total, 8 up, 8 in 2024-09-13T14:34:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:47 smithi003 ceph-mon[107278]: from='client.54952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:49.359 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:48 smithi003 ceph-mon[107278]: pgmap v251: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 4436/160611 objects degraded (2.762%); 0 B/s, 46 objects/s recovering 2024-09-13T14:34:49.359 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:48 smithi003 ceph-mon[107278]: osdmap e107: 8 total, 8 up, 8 in 2024-09-13T14:34:49.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:48 smithi003 ceph-mon[104007]: pgmap v251: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 4436/160611 objects degraded (2.762%); 0 B/s, 46 objects/s recovering 2024-09-13T14:34:49.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:48 smithi003 ceph-mon[104007]: osdmap e107: 8 total, 8 up, 8 in 2024-09-13T14:34:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:48 smithi019 ceph-mon[91296]: pgmap v251: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 4436/160611 objects degraded (2.762%); 0 B/s, 46 objects/s recovering 2024-09-13T14:34:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:48 smithi019 ceph-mon[91296]: osdmap e107: 8 total, 8 up, 8 in 2024-09-13T14:34:50.124 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:50.212 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2485852309' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:50.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2485852309' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2485852309' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:34:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:34:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:50 smithi019 ceph-mon[91296]: pgmap v253: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 3909/151176 objects degraded (2.586%); 0 B/s, 96 objects/s recovering 2024-09-13T14:34:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3909/151176 objects degraded (2.586%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:51.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:50 smithi003 ceph-mon[107278]: pgmap v253: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 3909/151176 objects degraded (2.586%); 0 B/s, 96 objects/s recovering 2024-09-13T14:34:51.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3909/151176 objects degraded (2.586%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:51.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:50 smithi003 ceph-mon[104007]: pgmap v253: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 3909/151176 objects degraded (2.586%); 0 B/s, 96 objects/s recovering 2024-09-13T14:34:51.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3909/151176 objects degraded (2.586%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:52.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:51 smithi019 ceph-mon[91296]: from='client.54961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:51 smithi019 ceph-mon[91296]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:52.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:51 smithi003 ceph-mon[104007]: from='client.54961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:52.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:51 smithi003 ceph-mon[104007]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:52.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:51 smithi003 ceph-mon[107278]: from='client.54961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:52.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:51 smithi003 ceph-mon[107278]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:52 smithi019 ceph-mon[91296]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 3909/151176 objects degraded (2.586%); 0 B/s, 96 objects/s recovering 2024-09-13T14:34:53.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:52 smithi003 ceph-mon[104007]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 3909/151176 objects degraded (2.586%); 0 B/s, 96 objects/s recovering 2024-09-13T14:34:53.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:52 smithi003 ceph-mon[107278]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 3909/151176 objects degraded (2.586%); 0 B/s, 96 objects/s recovering 2024-09-13T14:34:53.829 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:54.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1412979684' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:54.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1412979684' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:54.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1412979684' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:34:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:34:54] "GET /metrics HTTP/1.1" 200 36420 "" "Prometheus/2.51.0" 2024-09-13T14:34:55.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:54 smithi019 ceph-mon[91296]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 393 op/s; 3909/151176 objects degraded (2.586%); 0 B/s, 65 objects/s recovering 2024-09-13T14:34:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:54 smithi019 ceph-mon[91296]: from='client.54976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:54 smithi003 ceph-mon[104007]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 393 op/s; 3909/151176 objects degraded (2.586%); 0 B/s, 65 objects/s recovering 2024-09-13T14:34:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:54 smithi003 ceph-mon[104007]: from='client.54976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:54 smithi003 ceph-mon[107278]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 393 op/s; 3909/151176 objects degraded (2.586%); 0 B/s, 65 objects/s recovering 2024-09-13T14:34:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:54 smithi003 ceph-mon[107278]: from='client.54976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:56.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:55 smithi019 ceph-mon[91296]: from='client.45083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:56.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:55 smithi003 ceph-mon[104007]: from='client.45083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:56.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:55 smithi003 ceph-mon[107278]: from='client.45083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:56 smithi019 ceph-mon[91296]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 3677/140289 objects degraded (2.621%); 0 B/s, 89 objects/s recovering 2024-09-13T14:34:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3677/140289 objects degraded (2.621%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1840290814' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:57.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:56 smithi003 ceph-mon[104007]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 3677/140289 objects degraded (2.621%); 0 B/s, 89 objects/s recovering 2024-09-13T14:34:57.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3677/140289 objects degraded (2.621%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1840290814' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:56 smithi003 ceph-mon[107278]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 3677/140289 objects degraded (2.621%); 0 B/s, 89 objects/s recovering 2024-09-13T14:34:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3677/140289 objects degraded (2.621%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:34:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1840290814' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:34:57.473 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:34:58.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:58.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:58.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[104007]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 3677/140289 objects degraded (2.621%); 0 B/s, 75 objects/s recovering 2024-09-13T14:34:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[104007]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[104007]: from='client.54994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[104007]: Upgrade: osd.2 is safe to restart 2024-09-13T14:34:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[107278]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 3677/140289 objects degraded (2.621%); 0 B/s, 75 objects/s recovering 2024-09-13T14:34:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[107278]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:59.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[107278]: from='client.54994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:59.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:59.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:59.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:34:59 smithi003 ceph-mon[107278]: Upgrade: osd.2 is safe to restart 2024-09-13T14:34:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:59 smithi019 ceph-mon[91296]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 3677/140289 objects degraded (2.621%); 0 B/s, 75 objects/s recovering 2024-09-13T14:34:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:59 smithi019 ceph-mon[91296]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:59 smithi019 ceph-mon[91296]: from='client.54994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:59 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:34:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:59 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:34:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:34:59 smithi019 ceph-mon[91296]: Upgrade: osd.2 is safe to restart 2024-09-13T14:35:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:00 smithi003 ceph-mon[107278]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 908 op/s; 3469/129276 objects degraded (2.683%); 0 B/s, 84 objects/s recovering 2024-09-13T14:35:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:00 smithi003 ceph-mon[104007]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 908 op/s; 3469/129276 objects degraded (2.683%); 0 B/s, 84 objects/s recovering 2024-09-13T14:35:00.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:00 smithi019 ceph-mon[91296]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 908 op/s; 3469/129276 objects degraded (2.683%); 0 B/s, 84 objects/s recovering 2024-09-13T14:35:00.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:35:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:35:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:01.177 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:01 smithi019 ceph-mon[91296]: Upgrade: Updating osd.2 2024-09-13T14:35:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:35:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:01 smithi019 ceph-mon[91296]: Deploying daemon osd.2 on smithi003 2024-09-13T14:35:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:01 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/46888293' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[107278]: Upgrade: Updating osd.2 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[107278]: Deploying daemon osd.2 on smithi003 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/46888293' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[104007]: Upgrade: Updating osd.2 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[104007]: Deploying daemon osd.2 on smithi003 2024-09-13T14:35:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:01 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/46888293' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:02.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:02 smithi019 ceph-mon[91296]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:02.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:02 smithi019 ceph-mon[91296]: pgmap v259: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 3469/129276 objects degraded (2.683%); 0 B/s, 36 objects/s recovering 2024-09-13T14:35:02.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:02 smithi019 ceph-mon[91296]: from='client.55006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:02.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:02 smithi003 ceph-mon[104007]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:02.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:02 smithi003 ceph-mon[104007]: pgmap v259: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 3469/129276 objects degraded (2.683%); 0 B/s, 36 objects/s recovering 2024-09-13T14:35:02.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:02 smithi003 ceph-mon[104007]: from='client.55006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:02.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:02 smithi003 ceph-mon[107278]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:02.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:02 smithi003 ceph-mon[107278]: pgmap v259: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 3469/129276 objects degraded (2.683%); 0 B/s, 36 objects/s recovering 2024-09-13T14:35:02.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:02 smithi003 ceph-mon[107278]: from='client.55006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:04.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:35:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:35:04] "GET /metrics HTTP/1.1" 200 36424 "" "Prometheus/2.51.0" 2024-09-13T14:35:04.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:04 smithi003 ceph-mon[107278]: pgmap v260: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 3469/129276 objects degraded (2.683%); 0 B/s, 36 objects/s recovering 2024-09-13T14:35:04.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:04 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1012311184' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:04.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:04 smithi003 ceph-mon[104007]: pgmap v260: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 3469/129276 objects degraded (2.683%); 0 B/s, 36 objects/s recovering 2024-09-13T14:35:04.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:04 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1012311184' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:04.929 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:04.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:04 smithi019 ceph-mon[91296]: pgmap v260: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 3469/129276 objects degraded (2.683%); 0 B/s, 36 objects/s recovering 2024-09-13T14:35:04.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:04 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1012311184' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:05.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:05 smithi003 ceph-mon[104007]: from='client.35115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:05.908 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3469/129276 objects degraded (2.683%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:35:05.908 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:05 smithi003 ceph-mon[107278]: from='client.35115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:05.908 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3469/129276 objects degraded (2.683%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:35:05.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:05 smithi019 ceph-mon[91296]: from='client.35115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:05.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3469/129276 objects degraded (2.683%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:35:06.832 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:06 smithi019 ceph-mon[91296]: pgmap v261: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 3234/118761 objects degraded (2.723%); 0 B/s, 56 objects/s recovering 2024-09-13T14:35:06.832 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:06 smithi019 ceph-mon[91296]: from='client.35121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:06 smithi003 ceph-mon[104007]: pgmap v261: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 3234/118761 objects degraded (2.723%); 0 B/s, 56 objects/s recovering 2024-09-13T14:35:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:06 smithi003 ceph-mon[104007]: from='client.35121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:06 smithi003 ceph-mon[107278]: pgmap v261: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 3234/118761 objects degraded (2.723%); 0 B/s, 56 objects/s recovering 2024-09-13T14:35:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:06 smithi003 ceph-mon[107278]: from='client.35121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:07.433 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:07 smithi003 systemd[1]: Stopping Ceph osd.2 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:35:07.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[58420]: 2024-09-13T14:35:07.514+0000 7fbb7e7c0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:35:07.936 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[58420]: 2024-09-13T14:35:07.514+0000 7fbb7e7c0640 -1 osd.2 107 *** Got signal Terminated *** 2024-09-13T14:35:07.936 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[58420]: 2024-09-13T14:35:07.514+0000 7fbb7e7c0640 -1 osd.2 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:35:08.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:08 smithi003 ceph-mon[104007]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 3234/118761 objects degraded (2.723%); 0 B/s, 36 objects/s recovering 2024-09-13T14:35:08.566 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:08.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:08 smithi019 ceph-mon[91296]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 3234/118761 objects degraded (2.723%); 0 B/s, 36 objects/s recovering 2024-09-13T14:35:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:08 smithi019 ceph-mon[91296]: osd.2 marked itself down and dead 2024-09-13T14:35:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:08 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/907305220' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:08.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:08 smithi003 ceph-mon[107278]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 3234/118761 objects degraded (2.723%); 0 B/s, 36 objects/s recovering 2024-09-13T14:35:08.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:08 smithi003 ceph-mon[107278]: osd.2 marked itself down and dead 2024-09-13T14:35:08.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:08 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/907305220' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:08.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:08 smithi003 ceph-mon[104007]: osd.2 marked itself down and dead 2024-09-13T14:35:08.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:08 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/907305220' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:09.311 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:09 smithi003 podman[132779]: 2024-09-13 14:35:09.017262042 +0000 UTC m=+1.659128651 container died d32f24c0ca7ecfa05cd67278cf87c2eabb1fbc6cb50891f6711633ddc0abecec (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-bbdadfa, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:35:09.678 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:09 smithi003 ceph-mon[107278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:35:09.678 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:09 smithi003 ceph-mon[107278]: osdmap e108: 8 total, 7 up, 8 in 2024-09-13T14:35:09.678 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:09 smithi003 ceph-mon[107278]: from='client.55024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:09.678 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:09 smithi003 ceph-mon[107278]: from='client.35139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:09.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:09 smithi003 ceph-mon[104007]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:35:09.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:09 smithi003 ceph-mon[104007]: osdmap e108: 8 total, 7 up, 8 in 2024-09-13T14:35:09.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:09 smithi003 ceph-mon[104007]: from='client.55024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:09.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:09 smithi003 ceph-mon[104007]: from='client.35139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:09.679 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:09 smithi003 podman[132779]: 2024-09-13 14:35:09.408966638 +0000 UTC m=+2.050833249 container cleanup d32f24c0ca7ecfa05cd67278cf87c2eabb1fbc6cb50891f6711633ddc0abecec (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-13T14:35:09.679 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:09 smithi003 bash[132779]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2 2024-09-13T14:35:09.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:09 smithi019 ceph-mon[91296]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:35:09.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:09 smithi019 ceph-mon[91296]: osdmap e108: 8 total, 7 up, 8 in 2024-09-13T14:35:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:09 smithi019 ceph-mon[91296]: from='client.55024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:09 smithi019 ceph-mon[91296]: from='client.35139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:09.932 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:09 smithi003 podman[132874]: 2024-09-13 14:35:09.677585193 +0000 UTC m=+0.657070133 container remove d32f24c0ca7ecfa05cd67278cf87c2eabb1fbc6cb50891f6711633ddc0abecec (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:35:10.504 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:35:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:35:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:10.616 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:10 smithi003 ceph-mon[104007]: pgmap v264: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 1 active+recovering+undersized+degraded+remapped, 3 stale+active+clean, 75 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 2992/110709 objects degraded (2.703%); 0 B/s, 47 objects/s recovering 2024-09-13T14:35:10.617 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:10 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:35:10.617 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:10 smithi003 ceph-mon[104007]: osdmap e109: 8 total, 7 up, 8 in 2024-09-13T14:35:10.617 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2992/110709 objects degraded (2.703%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:35:10.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:10 smithi003 ceph-mon[107278]: pgmap v264: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 1 active+recovering+undersized+degraded+remapped, 3 stale+active+clean, 75 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 2992/110709 objects degraded (2.703%); 0 B/s, 47 objects/s recovering 2024-09-13T14:35:10.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:10 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:35:10.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:10 smithi003 ceph-mon[107278]: osdmap e109: 8 total, 7 up, 8 in 2024-09-13T14:35:10.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2992/110709 objects degraded (2.703%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:35:10.618 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:10 smithi003 podman[132952]: 2024-09-13 14:35:10.173444432 +0000 UTC m=+0.039733921 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:35:10.618 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:10 smithi003 podman[132952]: 2024-09-13 14:35:10.295437326 +0000 UTC m=+0.161726798 container create b4ec82714f43cf931c8452d345ad990ae4918d9936cebf7879a5edb124e7a6d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-13T14:35:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:10 smithi019 ceph-mon[91296]: pgmap v264: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 1 active+recovering+undersized+degraded+remapped, 3 stale+active+clean, 75 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 2992/110709 objects degraded (2.703%); 0 B/s, 47 objects/s recovering 2024-09-13T14:35:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:10 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:35:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:10 smithi019 ceph-mon[91296]: osdmap e109: 8 total, 7 up, 8 in 2024-09-13T14:35:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2992/110709 objects degraded (2.703%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:35:10.932 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:10 smithi003 podman[132952]: 2024-09-13 14:35:10.616070984 +0000 UTC m=+0.482360457 container init b4ec82714f43cf931c8452d345ad990ae4918d9936cebf7879a5edb124e7a6d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True) 2024-09-13T14:35:10.932 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:10 smithi003 podman[132952]: 2024-09-13 14:35:10.629325637 +0000 UTC m=+0.495615109 container start b4ec82714f43cf931c8452d345ad990ae4918d9936cebf7879a5edb124e7a6d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-13T14:35:10.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:10 smithi003 podman[132952]: 2024-09-13 14:35:10.732543473 +0000 UTC m=+0.598832942 container attach b4ec82714f43cf931c8452d345ad990ae4918d9936cebf7879a5edb124e7a6d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:35:11.182 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:10 smithi003 conmon[132964]: conmon b4ec82714f43cf931c84 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-b4ec82714f43cf931c8452d345ad990ae4918d9936cebf7879a5edb124e7a6d0.scope/memory.events 2024-09-13T14:35:11.183 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:10 smithi003 podman[132952]: 2024-09-13 14:35:10.933397151 +0000 UTC m=+0.799686628 container died b4ec82714f43cf931c8452d345ad990ae4918d9936cebf7879a5edb124e7a6d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-13T14:35:11.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/568108661' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:11.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/568108661' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:11.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/568108661' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:11.932 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:11 smithi003 podman[132993]: 2024-09-13 14:35:11.612695973 +0000 UTC m=+0.668980504 container remove b4ec82714f43cf931c8452d345ad990ae4918d9936cebf7879a5edb124e7a6d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:35:11.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:11 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-13T14:35:11.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:11 smithi003 systemd[1]: Stopped Ceph osd.2 for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:35:11.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:11 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.2.service: Consumed 19min 6.315s CPU time. 2024-09-13T14:35:11.977 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:12.340 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:11 smithi003 systemd[1]: Starting Ceph osd.2 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:35:12.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:12 smithi003 ceph-mon[107278]: pgmap v266: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 1 active+recovering+undersized+degraded+remapped, 3 stale+active+clean, 75 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 2992/110709 objects degraded (2.703%); 0 B/s, 59 objects/s recovering 2024-09-13T14:35:12.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:12 smithi003 ceph-mon[107278]: from='client.45140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:12 smithi003 ceph-mon[104007]: pgmap v266: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 1 active+recovering+undersized+degraded+remapped, 3 stale+active+clean, 75 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 2992/110709 objects degraded (2.703%); 0 B/s, 59 objects/s recovering 2024-09-13T14:35:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:12 smithi003 ceph-mon[104007]: from='client.45140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:12.683 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 podman[133151]: 2024-09-13 14:35:12.245555435 +0000 UTC m=+0.048452902 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:35:12.683 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 podman[133151]: 2024-09-13 14:35:12.389125372 +0000 UTC m=+0.192022837 container create c7aed8d16bbe2669730d057ff64257b660800994feb08982663eb9ae93af5914 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:35:12.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:12 smithi019 ceph-mon[91296]: pgmap v266: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 1 active+recovering+undersized+degraded+remapped, 3 stale+active+clean, 75 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 2992/110709 objects degraded (2.703%); 0 B/s, 59 objects/s recovering 2024-09-13T14:35:12.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:12 smithi019 ceph-mon[91296]: from='client.45140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:12.945 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 podman[133151]: 2024-09-13 14:35:12.683586197 +0000 UTC m=+0.486483662 container init c7aed8d16bbe2669730d057ff64257b660800994feb08982663eb9ae93af5914 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-13T14:35:12.945 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 podman[133151]: 2024-09-13 14:35:12.700984515 +0000 UTC m=+0.503881981 container start c7aed8d16bbe2669730d057ff64257b660800994feb08982663eb9ae93af5914 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2) 2024-09-13T14:35:12.945 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 podman[133151]: 2024-09-13 14:35:12.783960268 +0000 UTC m=+0.586857751 container attach c7aed8d16bbe2669730d057ff64257b660800994feb08982663eb9ae93af5914 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True) 2024-09-13T14:35:12.945 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:35:12.945 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 bash[133151]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:35:12.945 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:35:12.945 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 bash[133151]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:35:13.197 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:35:13.197 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 bash[133151]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:35:13.197 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:35:13.197 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 bash[133151]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:35:13.197 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:35:13.197 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:12 smithi003 bash[133151]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:35:13.197 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:35:13.197 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 bash[133151]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 bash[133151]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 bash[133151]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 bash[133151]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 bash[133151]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:35:13.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 bash[133151]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:35:13.606 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate[133167]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:35:13.606 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 bash[133151]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:35:13.606 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 podman[133151]: 2024-09-13 14:35:13.234307533 +0000 UTC m=+1.037205003 container died c7aed8d16bbe2669730d057ff64257b660800994feb08982663eb9ae93af5914 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:35:13.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-mon[107278]: from='client.55045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:13.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:13.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:13.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-mon[104007]: from='client.55045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:13.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:13.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:13 smithi019 ceph-mon[91296]: from='client.55045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:14.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:14.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:14.182 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:13 smithi003 podman[133311]: 2024-09-13 14:35:13.924089531 +0000 UTC m=+0.678602312 container remove c7aed8d16bbe2669730d057ff64257b660800994feb08982663eb9ae93af5914 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-activate, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2) 2024-09-13T14:35:14.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:35:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:35:14] "GET /metrics HTTP/1.1" 200 36351 "" "Prometheus/2.51.0" 2024-09-13T14:35:14.682 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:14 smithi003 podman[133364]: 2024-09-13 14:35:14.166985815 +0000 UTC m=+0.036580376 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:35:14.682 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:14 smithi003 podman[133364]: 2024-09-13 14:35:14.29907726 +0000 UTC m=+0.168671808 container create 357adcd35fd784ab04273b17f6138d403b09e5ca9d33f115b977c117e5021ca9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:35:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:14 smithi019 ceph-mon[91296]: pgmap v267: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 5 active+undersized, 1 active+recovering+undersized+degraded+remapped, 2 stale+active+clean, 71 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 335 op/s; 2992/110709 objects degraded (2.703%); 0 B/s, 30 objects/s recovering 2024-09-13T14:35:15.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:14 smithi003 ceph-mon[107278]: pgmap v267: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 5 active+undersized, 1 active+recovering+undersized+degraded+remapped, 2 stale+active+clean, 71 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 335 op/s; 2992/110709 objects degraded (2.703%); 0 B/s, 30 objects/s recovering 2024-09-13T14:35:15.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:14 smithi003 ceph-mon[104007]: pgmap v267: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 5 active+undersized, 1 active+recovering+undersized+degraded+remapped, 2 stale+active+clean, 71 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 335 op/s; 2992/110709 objects degraded (2.703%); 0 B/s, 30 objects/s recovering 2024-09-13T14:35:15.183 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:14 smithi003 podman[133364]: 2024-09-13 14:35:14.697346707 +0000 UTC m=+0.566941256 container init 357adcd35fd784ab04273b17f6138d403b09e5ca9d33f115b977c117e5021ca9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731) 2024-09-13T14:35:15.183 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:14 smithi003 podman[133364]: 2024-09-13 14:35:14.711429108 +0000 UTC m=+0.581023660 container start 357adcd35fd784ab04273b17f6138d403b09e5ca9d33f115b977c117e5021ca9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-13T14:35:15.184 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:14 smithi003 bash[133364]: 357adcd35fd784ab04273b17f6138d403b09e5ca9d33f115b977c117e5021ca9 2024-09-13T14:35:15.184 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:14 smithi003 systemd[1]: Started Ceph osd.2 for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:35:15.670 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2295336093' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2295336093' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2295336093' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:16.433 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:16 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[133375]: 2024-09-13T14:35:16.145+0000 7efece472640 -1 Falling back to public interface 2024-09-13T14:35:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:16 smithi019 ceph-mon[91296]: pgmap v268: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 822 op/s; 14717/99009 objects degraded (14.864%); 0 B/s, 111 objects/s recovering 2024-09-13T14:35:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:16 smithi019 ceph-mon[91296]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 14717/99009 objects degraded (14.864%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:35:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:16 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:35:17.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:16 smithi003 ceph-mon[107278]: pgmap v268: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 822 op/s; 14717/99009 objects degraded (14.864%); 0 B/s, 111 objects/s recovering 2024-09-13T14:35:17.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:16 smithi003 ceph-mon[107278]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:17.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 14717/99009 objects degraded (14.864%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:35:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:16 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:35:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:16 smithi003 ceph-mon[104007]: pgmap v268: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 822 op/s; 14717/99009 objects degraded (14.864%); 0 B/s, 111 objects/s recovering 2024-09-13T14:35:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:16 smithi003 ceph-mon[104007]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 14717/99009 objects degraded (14.864%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:35:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:16 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:35:18.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:17 smithi003 ceph-mon[107278]: from='client.55063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:18.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:17 smithi003 ceph-mon[107278]: osdmap e110: 8 total, 7 up, 8 in 2024-09-13T14:35:18.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:17 smithi003 ceph-mon[104007]: from='client.55063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:18.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:17 smithi003 ceph-mon[104007]: osdmap e110: 8 total, 7 up, 8 in 2024-09-13T14:35:18.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:17 smithi019 ceph-mon[91296]: from='client.55063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:18.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:17 smithi019 ceph-mon[91296]: osdmap e110: 8 total, 7 up, 8 in 2024-09-13T14:35:19.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:18 smithi003 ceph-mon[104007]: pgmap v270: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 487 op/s; 14717/99009 objects degraded (14.864%); 0 B/s, 81 objects/s recovering 2024-09-13T14:35:19.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:18 smithi003 ceph-mon[104007]: osdmap e111: 8 total, 7 up, 8 in 2024-09-13T14:35:19.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1572842874' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:19.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:18 smithi003 ceph-mon[107278]: pgmap v270: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 487 op/s; 14717/99009 objects degraded (14.864%); 0 B/s, 81 objects/s recovering 2024-09-13T14:35:19.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:18 smithi003 ceph-mon[107278]: osdmap e111: 8 total, 7 up, 8 in 2024-09-13T14:35:19.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1572842874' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:19.340 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:18 smithi019 ceph-mon[91296]: pgmap v270: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 487 op/s; 14717/99009 objects degraded (14.864%); 0 B/s, 81 objects/s recovering 2024-09-13T14:35:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:18 smithi019 ceph-mon[91296]: osdmap e111: 8 total, 7 up, 8 in 2024-09-13T14:35:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1572842874' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:20.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:35:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:35:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:20 smithi019 ceph-mon[91296]: from='client.55072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:20 smithi019 ceph-mon[91296]: pgmap v272: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 919 op/s; 13079/88632 objects degraded (14.757%); 0 B/s, 126 objects/s recovering 2024-09-13T14:35:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:20 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:20 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:20 smithi019 ceph-mon[91296]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[107278]: from='client.55072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[107278]: pgmap v272: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 919 op/s; 13079/88632 objects degraded (14.757%); 0 B/s, 126 objects/s recovering 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[107278]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[104007]: from='client.55072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[104007]: pgmap v272: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 919 op/s; 13079/88632 objects degraded (14.757%); 0 B/s, 126 objects/s recovering 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:20 smithi003 ceph-mon[104007]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:22.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:22.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:22.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:22.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:22.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:23.042 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:23.060 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:22 smithi003 ceph-mon[107278]: pgmap v273: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 919 op/s; 13079/88632 objects degraded (14.757%); 0 B/s, 126 objects/s recovering 2024-09-13T14:35:23.060 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/847389050' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:23.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:22 smithi003 ceph-mon[104007]: pgmap v273: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 919 op/s; 13079/88632 objects degraded (14.757%); 0 B/s, 126 objects/s recovering 2024-09-13T14:35:23.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/847389050' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:22 smithi019 ceph-mon[91296]: pgmap v273: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 62 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 919 op/s; 13079/88632 objects degraded (14.757%); 0 B/s, 126 objects/s recovering 2024-09-13T14:35:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/847389050' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:24.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:23 smithi019 ceph-mon[91296]: from='client.55084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:24.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:23 smithi003 ceph-mon[104007]: from='client.55084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:24.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:23 smithi003 ceph-mon[107278]: from='client.55084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:35:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:35:24] "GET /metrics HTTP/1.1" 200 36351 "" "Prometheus/2.51.0" 2024-09-13T14:35:25.309 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:24 smithi003 ceph-mon[107278]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 460 op/s; 13079/87960 objects degraded (14.869%); 0 B/s, 45 objects/s recovering 2024-09-13T14:35:25.309 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:24 smithi003 ceph-mon[107278]: from='client.55087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:25.309 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:24 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 13079/87960 objects degraded (14.869%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:24 smithi003 ceph-mon[104007]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 460 op/s; 13079/87960 objects degraded (14.869%); 0 B/s, 45 objects/s recovering 2024-09-13T14:35:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:24 smithi003 ceph-mon[104007]: from='client.55087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:24 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 13079/87960 objects degraded (14.869%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:25.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:24 smithi019 ceph-mon[91296]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 460 op/s; 13079/87960 objects degraded (14.869%); 0 B/s, 45 objects/s recovering 2024-09-13T14:35:25.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:24 smithi019 ceph-mon[91296]: from='client.55087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:24 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 13079/87960 objects degraded (14.869%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:26.757 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:26.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:35:26.523+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-13T14:35:27.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 11512/77673 objects degraded (14.821%); 0 B/s, 61 objects/s recovering 2024-09-13T14:35:27.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3721874897' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:27.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:27.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:27.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:27.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:35:27.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:27.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:27.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:35:27.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:35:27.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:35:27.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:35:27.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 11512/77673 objects degraded (14.821%); 0 B/s, 61 objects/s recovering 2024-09-13T14:35:27.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3721874897' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:35:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:35:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 11512/77673 objects degraded (14.821%); 0 B/s, 61 objects/s recovering 2024-09-13T14:35:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3721874897' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:35:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:35:27.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:35:27.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:35:27.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:35:27.683 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[133375]: 2024-09-13T14:35:27.236+0000 7efece472640 -1 osd.2 107 log_to_monitors true 2024-09-13T14:35:28.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:27 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:35:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:27 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-13T14:35:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:27 smithi019 ceph-mon[91296]: from='client.45188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:27 smithi019 ceph-mon[91296]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:35:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:28.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[107278]: from='client.45188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[107278]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[104007]: from='client.45188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[104007]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:35:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:28.932 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[133375]: 2024-09-13T14:35:28.521+0000 7efec2214640 -1 osd.2 107 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:35:29.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:28 smithi019 ceph-mon[91296]: from='client.35187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:28 smithi019 ceph-mon[91296]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 11512/77673 objects degraded (14.821%); 0 B/s, 52 objects/s recovering 2024-09-13T14:35:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:28 smithi019 ceph-mon[91296]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:35:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:28 smithi019 ceph-mon[91296]: osdmap e112: 8 total, 7 up, 8 in 2024-09-13T14:35:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:28 smithi019 ceph-mon[91296]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:35:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:28 smithi019 ceph-mon[91296]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' 2024-09-13T14:35:29.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[107278]: from='client.35187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:29.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[107278]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 11512/77673 objects degraded (14.821%); 0 B/s, 52 objects/s recovering 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[107278]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[107278]: osdmap e112: 8 total, 7 up, 8 in 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[107278]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[107278]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[104007]: from='client.35187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[104007]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 11512/77673 objects degraded (14.821%); 0 B/s, 52 objects/s recovering 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[104007]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[104007]: osdmap e112: 8 total, 7 up, 8 in 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[104007]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:35:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:28 smithi003 ceph-mon[104007]: from='osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691]' entity='osd.2' 2024-09-13T14:35:30.135 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:30.212 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:29 smithi019 ceph-mon[91296]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:35:30.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:29 smithi019 ceph-mon[91296]: osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691] boot 2024-09-13T14:35:30.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:29 smithi019 ceph-mon[91296]: osdmap e113: 8 total, 8 up, 8 in 2024-09-13T14:35:30.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:35:30.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:29 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2717306145' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:30.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[104007]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:35:30.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[104007]: osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691] boot 2024-09-13T14:35:30.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[104007]: osdmap e113: 8 total, 8 up, 8 in 2024-09-13T14:35:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:35:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2717306145' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[107278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:35:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[107278]: osd.2 [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691] boot 2024-09-13T14:35:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[107278]: osdmap e113: 8 total, 8 up, 8 in 2024-09-13T14:35:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:35:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:29 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2717306145' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:35:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:35:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:30 smithi019 ceph-mon[91296]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 9896/66453 objects degraded (14.892%); 0 B/s, 33 objects/s recovering 2024-09-13T14:35:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9896/66453 objects degraded (14.892%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:30 smithi019 ceph-mon[91296]: osdmap e114: 8 total, 8 up, 8 in 2024-09-13T14:35:31.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:30 smithi003 ceph-mon[104007]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 9896/66453 objects degraded (14.892%); 0 B/s, 33 objects/s recovering 2024-09-13T14:35:31.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9896/66453 objects degraded (14.892%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:31.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:30 smithi003 ceph-mon[104007]: osdmap e114: 8 total, 8 up, 8 in 2024-09-13T14:35:31.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:30 smithi003 ceph-mon[107278]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 9896/66453 objects degraded (14.892%); 0 B/s, 33 objects/s recovering 2024-09-13T14:35:31.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9896/66453 objects degraded (14.892%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:31.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:30 smithi003 ceph-mon[107278]: osdmap e114: 8 total, 8 up, 8 in 2024-09-13T14:35:32.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:31 smithi019 ceph-mon[91296]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:31 smithi019 ceph-mon[91296]: from='client.45212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:31 smithi019 ceph-mon[91296]: osdmap e115: 8 total, 8 up, 8 in 2024-09-13T14:35:32.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:31 smithi003 ceph-mon[107278]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:32.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:31 smithi003 ceph-mon[107278]: from='client.45212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:32.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:31 smithi003 ceph-mon[107278]: osdmap e115: 8 total, 8 up, 8 in 2024-09-13T14:35:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:31 smithi003 ceph-mon[104007]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:31 smithi003 ceph-mon[104007]: from='client.45212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:31 smithi003 ceph-mon[104007]: osdmap e115: 8 total, 8 up, 8 in 2024-09-13T14:35:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:32 smithi019 ceph-mon[91296]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 9896/66453 objects degraded (14.892%); 0 B/s, 28 objects/s recovering 2024-09-13T14:35:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1601943957' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:33.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:32 smithi003 ceph-mon[107278]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 9896/66453 objects degraded (14.892%); 0 B/s, 28 objects/s recovering 2024-09-13T14:35:33.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1601943957' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:33.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:32 smithi003 ceph-mon[104007]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 9 active+undersized+degraded, 63 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 9896/66453 objects degraded (14.892%); 0 B/s, 28 objects/s recovering 2024-09-13T14:35:33.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1601943957' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:33.551 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:35:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:35:34] "GET /metrics HTTP/1.1" 200 36420 "" "Prometheus/2.51.0" 2024-09-13T14:35:35.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:34 smithi019 ceph-mon[91296]: pgmap v283: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+undersized, 9 active+undersized+degraded, 69 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 9896/66099 objects degraded (14.971%); 0 B/s, 30 objects/s recovering 2024-09-13T14:35:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:34 smithi019 ceph-mon[91296]: from='client.35217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:34 smithi003 ceph-mon[107278]: pgmap v283: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+undersized, 9 active+undersized+degraded, 69 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 9896/66099 objects degraded (14.971%); 0 B/s, 30 objects/s recovering 2024-09-13T14:35:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:34 smithi003 ceph-mon[107278]: from='client.35217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:34 smithi003 ceph-mon[104007]: pgmap v283: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+undersized, 9 active+undersized+degraded, 69 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 9896/66099 objects degraded (14.971%); 0 B/s, 30 objects/s recovering 2024-09-13T14:35:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:34 smithi003 ceph-mon[104007]: from='client.35217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:36.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:35 smithi019 ceph-mon[91296]: from='client.55126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:36.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9896/66099 objects degraded (14.971%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:36.333 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:35 smithi003 ceph-mon[107278]: from='client.55126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:36.333 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9896/66099 objects degraded (14.971%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:36.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:35 smithi003 ceph-mon[104007]: from='client.55126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:36.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9896/66099 objects degraded (14.971%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:37.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:36 smithi003 ceph-mon[107278]: pgmap v284: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 348 op/s; 5824/60180 objects degraded (9.678%); 0 B/s, 117 objects/s recovering 2024-09-13T14:35:37.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:36 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/40516926' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:37.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:36 smithi003 ceph-mon[104007]: pgmap v284: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 348 op/s; 5824/60180 objects degraded (9.678%); 0 B/s, 117 objects/s recovering 2024-09-13T14:35:37.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:36 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/40516926' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:37.285 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:37.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:36 smithi019 ceph-mon[91296]: pgmap v284: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 348 op/s; 5824/60180 objects degraded (9.678%); 0 B/s, 117 objects/s recovering 2024-09-13T14:35:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:36 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/40516926' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:38 smithi019 ceph-mon[91296]: from='client.35232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:38 smithi019 ceph-mon[91296]: pgmap v285: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 264 op/s; 5824/60180 objects degraded (9.678%); 0 B/s, 89 objects/s recovering 2024-09-13T14:35:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:38 smithi019 ceph-mon[91296]: from='client.55135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:39.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:38 smithi003 ceph-mon[107278]: from='client.35232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:39.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:38 smithi003 ceph-mon[107278]: pgmap v285: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 264 op/s; 5824/60180 objects degraded (9.678%); 0 B/s, 89 objects/s recovering 2024-09-13T14:35:39.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:38 smithi003 ceph-mon[107278]: from='client.55135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:39.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:38 smithi003 ceph-mon[104007]: from='client.35232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:39.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:38 smithi003 ceph-mon[104007]: pgmap v285: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 264 op/s; 5824/60180 objects degraded (9.678%); 0 B/s, 89 objects/s recovering 2024-09-13T14:35:39.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:38 smithi003 ceph-mon[104007]: from='client.55135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:40.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:35:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:35:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:40.982 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:41.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:41 smithi003 ceph-mon[107278]: pgmap v286: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 543 op/s; 5637/51822 objects degraded (10.878%); 0 B/s, 136 objects/s recovering 2024-09-13T14:35:41.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:41 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3649379524' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:41.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5637/51822 objects degraded (10.878%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:41.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:41 smithi019 ceph-mon[91296]: pgmap v286: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 543 op/s; 5637/51822 objects degraded (10.878%); 0 B/s, 136 objects/s recovering 2024-09-13T14:35:41.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:41 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3649379524' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:41.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5637/51822 objects degraded (10.878%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:41.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:41 smithi003 ceph-mon[104007]: pgmap v286: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 543 op/s; 5637/51822 objects degraded (10.878%); 0 B/s, 136 objects/s recovering 2024-09-13T14:35:41.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:41 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3649379524' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:41.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5637/51822 objects degraded (10.878%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:41.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:35:41 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:35:41.543+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-13T14:35:42.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:42 smithi019 ceph-mon[91296]: from='client.55147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:35:42.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:42 smithi003 ceph-mon[107278]: from='client.55147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:42.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:35:42.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:42 smithi003 ceph-mon[104007]: from='client.55147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:42.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:35:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:43 smithi019 ceph-mon[91296]: pgmap v287: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.0 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 5637/49971 objects degraded (11.281%); 0 B/s, 122 objects/s recovering 2024-09-13T14:35:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:43 smithi019 ceph-mon[91296]: from='client.55153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:43 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:35:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:43 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-13T14:35:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:43.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[107278]: pgmap v287: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.0 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 5637/49971 objects degraded (11.281%); 0 B/s, 122 objects/s recovering 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[107278]: from='client.55153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[104007]: pgmap v287: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.0 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 5637/49971 objects degraded (11.281%); 0 B/s, 122 objects/s recovering 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[104007]: from='client.55153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:35:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:35:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:35:44] "GET /metrics HTTP/1.1" 200 36414 "" "Prometheus/2.51.0" 2024-09-13T14:35:44.697 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:45 smithi019 ceph-mon[91296]: pgmap v288: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 477 op/s; 5637/49308 objects degraded (11.432%); 0 B/s, 102 objects/s recovering 2024-09-13T14:35:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:45 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1503087477' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:45.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:45 smithi003 ceph-mon[107278]: pgmap v288: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 477 op/s; 5637/49308 objects degraded (11.432%); 0 B/s, 102 objects/s recovering 2024-09-13T14:35:45.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:45 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1503087477' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:45.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:45 smithi003 ceph-mon[104007]: pgmap v288: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 477 op/s; 5637/49308 objects degraded (11.432%); 0 B/s, 102 objects/s recovering 2024-09-13T14:35:45.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:45 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1503087477' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:46.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:46 smithi019 ceph-mon[91296]: from='client.55156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:46.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5637/49308 objects degraded (11.432%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:46.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:46 smithi003 ceph-mon[107278]: from='client.55156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:46.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5637/49308 objects degraded (11.432%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:46.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:46 smithi003 ceph-mon[104007]: from='client.55156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:46.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5637/49308 objects degraded (11.432%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:47 smithi019 ceph-mon[91296]: from='client.35259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:47 smithi019 ceph-mon[91296]: pgmap v289: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 814 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 5408/39120 objects degraded (13.824%); 0 B/s, 150 objects/s recovering 2024-09-13T14:35:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:47 smithi019 ceph-mon[91296]: osdmap e116: 8 total, 8 up, 8 in 2024-09-13T14:35:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:47 smithi019 ceph-mon[91296]: osdmap e117: 8 total, 8 up, 8 in 2024-09-13T14:35:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:47 smithi003 ceph-mon[107278]: from='client.35259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:47 smithi003 ceph-mon[107278]: pgmap v289: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 814 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 5408/39120 objects degraded (13.824%); 0 B/s, 150 objects/s recovering 2024-09-13T14:35:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:47 smithi003 ceph-mon[107278]: osdmap e116: 8 total, 8 up, 8 in 2024-09-13T14:35:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:47 smithi003 ceph-mon[107278]: osdmap e117: 8 total, 8 up, 8 in 2024-09-13T14:35:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:47 smithi003 ceph-mon[104007]: from='client.35259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:47 smithi003 ceph-mon[104007]: pgmap v289: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 814 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 5408/39120 objects degraded (13.824%); 0 B/s, 150 objects/s recovering 2024-09-13T14:35:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:47 smithi003 ceph-mon[104007]: osdmap e116: 8 total, 8 up, 8 in 2024-09-13T14:35:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:47 smithi003 ceph-mon[104007]: osdmap e117: 8 total, 8 up, 8 in 2024-09-13T14:35:48.381 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:48.382 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:48 smithi003 ceph-mon[107278]: pgmap v292: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 814 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 528 op/s; 5408/39120 objects degraded (13.824%); 0 B/s, 72 objects/s recovering 2024-09-13T14:35:48.382 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/430798263' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:48 smithi003 ceph-mon[104007]: pgmap v292: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 814 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 528 op/s; 5408/39120 objects degraded (13.824%); 0 B/s, 72 objects/s recovering 2024-09-13T14:35:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/430798263' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:48.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:48 smithi019 ceph-mon[91296]: pgmap v292: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 814 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 528 op/s; 5408/39120 objects degraded (13.824%); 0 B/s, 72 objects/s recovering 2024-09-13T14:35:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/430798263' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:49 smithi019 ceph-mon[91296]: from='client.55168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:49 smithi019 ceph-mon[91296]: from='client.55171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:49.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:49 smithi003 ceph-mon[107278]: from='client.55168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:49.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:49 smithi003 ceph-mon[107278]: from='client.55171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:49.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:49 smithi003 ceph-mon[104007]: from='client.55168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:49.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:49 smithi003 ceph-mon[104007]: from='client.55171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:50.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:35:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:35:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:50.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:50 smithi019 ceph-mon[91296]: pgmap v293: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 665 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 4884/31956 objects degraded (15.284%); 0 B/s, 159 objects/s recovering 2024-09-13T14:35:50.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:50 smithi003 ceph-mon[107278]: pgmap v293: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 665 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 4884/31956 objects degraded (15.284%); 0 B/s, 159 objects/s recovering 2024-09-13T14:35:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:50 smithi003 ceph-mon[104007]: pgmap v293: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 665 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 4884/31956 objects degraded (15.284%); 0 B/s, 159 objects/s recovering 2024-09-13T14:35:51.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4884/31956 objects degraded (15.284%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:51.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4884/31956 objects degraded (15.284%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:51.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4884/31956 objects degraded (15.284%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:51.770 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:52.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:52 smithi019 ceph-mon[91296]: osdmap e118: 8 total, 8 up, 8 in 2024-09-13T14:35:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:52 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2656219107' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:52 smithi019 ceph-mon[91296]: pgmap v295: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 626 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 499 op/s; 4884/30120 objects degraded (16.215%); 0 B/s, 115 objects/s recovering 2024-09-13T14:35:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:52 smithi019 ceph-mon[91296]: from='client.45275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:52.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:52 smithi003 ceph-mon[107278]: osdmap e118: 8 total, 8 up, 8 in 2024-09-13T14:35:52.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:52 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2656219107' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:52.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:52 smithi003 ceph-mon[107278]: pgmap v295: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 626 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 499 op/s; 4884/30120 objects degraded (16.215%); 0 B/s, 115 objects/s recovering 2024-09-13T14:35:52.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:52 smithi003 ceph-mon[107278]: from='client.45275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:52.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:52 smithi003 ceph-mon[104007]: osdmap e118: 8 total, 8 up, 8 in 2024-09-13T14:35:52.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:52 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2656219107' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:52.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:52 smithi003 ceph-mon[104007]: pgmap v295: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 626 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 499 op/s; 4884/30120 objects degraded (16.215%); 0 B/s, 115 objects/s recovering 2024-09-13T14:35:52.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:52 smithi003 ceph-mon[104007]: from='client.45275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:53 smithi019 ceph-mon[91296]: from='client.45281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:53.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:53 smithi019 ceph-mon[91296]: osdmap e119: 8 total, 8 up, 8 in 2024-09-13T14:35:53.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:53 smithi003 ceph-mon[107278]: from='client.45281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:53.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:53 smithi003 ceph-mon[107278]: osdmap e119: 8 total, 8 up, 8 in 2024-09-13T14:35:53.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:53 smithi003 ceph-mon[104007]: from='client.45281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:53.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:53 smithi003 ceph-mon[104007]: osdmap e119: 8 total, 8 up, 8 in 2024-09-13T14:35:54.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:54 smithi019 ceph-mon[91296]: pgmap v297: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 614 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 498 op/s; 4884/29517 objects degraded (16.546%); 0 B/s, 108 objects/s recovering 2024-09-13T14:35:54.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:54 smithi003 ceph-mon[107278]: pgmap v297: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 614 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 498 op/s; 4884/29517 objects degraded (16.546%); 0 B/s, 108 objects/s recovering 2024-09-13T14:35:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:35:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:35:54] "GET /metrics HTTP/1.1" 200 36414 "" "Prometheus/2.51.0" 2024-09-13T14:35:54.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:54 smithi003 ceph-mon[104007]: pgmap v297: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 614 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 498 op/s; 4884/29517 objects degraded (16.546%); 0 B/s, 108 objects/s recovering 2024-09-13T14:35:55.209 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:55.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3053035323' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:55.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3053035323' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3053035323' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:56.419 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:56 smithi019 ceph-mon[91296]: from='client.55195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:56.419 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:56 smithi019 ceph-mon[91296]: pgmap v298: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 421 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 786 op/s; 3818/20244 objects degraded (18.860%); 0 B/s, 152 objects/s recovering 2024-09-13T14:35:56.419 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4884/29517 objects degraded (16.546%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:56.419 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:56 smithi019 ceph-mon[91296]: from='client.45290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:56.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:56 smithi003 ceph-mon[104007]: from='client.55195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:56.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:56 smithi003 ceph-mon[104007]: pgmap v298: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 421 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 786 op/s; 3818/20244 objects degraded (18.860%); 0 B/s, 152 objects/s recovering 2024-09-13T14:35:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4884/29517 objects degraded (16.546%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:56 smithi003 ceph-mon[104007]: from='client.45290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:56 smithi003 ceph-mon[107278]: from='client.55195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:56 smithi003 ceph-mon[107278]: pgmap v298: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 421 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 786 op/s; 3818/20244 objects degraded (18.860%); 0 B/s, 152 objects/s recovering 2024-09-13T14:35:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4884/29517 objects degraded (16.546%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:35:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:56 smithi003 ceph-mon[107278]: from='client.45290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:56.681 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:35:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=cleanup t=2024-09-13T14:35:56.419074829Z level=info msg="Completed cleanup jobs" duration=13.21551ms 2024-09-13T14:35:56.932 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:35:56 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:35:56.561+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-13T14:35:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:35:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:57 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:35:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:57 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-13T14:35:57.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:35:57.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:57 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:35:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:57 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-13T14:35:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:35:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:57 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:35:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:57 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-13T14:35:58.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:58 smithi019 ceph-mon[91296]: pgmap v299: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 421 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 487 op/s; 3818/20244 objects degraded (18.860%); 0 B/s, 65 objects/s recovering 2024-09-13T14:35:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:58.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:58 smithi003 ceph-mon[107278]: pgmap v299: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 421 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 487 op/s; 3818/20244 objects degraded (18.860%); 0 B/s, 65 objects/s recovering 2024-09-13T14:35:58.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:58.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:58 smithi003 ceph-mon[104007]: pgmap v299: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 421 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 487 op/s; 3818/20244 objects degraded (18.860%); 0 B/s, 65 objects/s recovering 2024-09-13T14:35:58.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:58.905 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:35:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/996288665' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:35:59 smithi003 ceph-mon[104007]: from='client.55204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/996288665' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:35:59 smithi003 ceph-mon[107278]: from='client.55204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/996288665' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:35:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:35:59 smithi019 ceph-mon[91296]: from='client.55204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:00.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:36:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:36:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:00 smithi019 ceph-mon[91296]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:00 smithi019 ceph-mon[91296]: pgmap v300: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 227 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 763 op/s; 3818/10950 objects degraded (34.868%); 0 B/s, 62 objects/s recovering 2024-09-13T14:36:00.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:00 smithi003 ceph-mon[104007]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:00.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:00 smithi003 ceph-mon[104007]: pgmap v300: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 227 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 763 op/s; 3818/10950 objects degraded (34.868%); 0 B/s, 62 objects/s recovering 2024-09-13T14:36:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:00 smithi003 ceph-mon[107278]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:00 smithi003 ceph-mon[107278]: pgmap v300: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 227 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 763 op/s; 3818/10950 objects degraded (34.868%); 0 B/s, 62 objects/s recovering 2024-09-13T14:36:01.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:01 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3818/10950 objects degraded (34.868%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:01.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:01 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3818/10950 objects degraded (34.868%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:01 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3818/10950 objects degraded (34.868%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:02.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:02 smithi019 ceph-mon[91296]: pgmap v301: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 184 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 3818/8874 objects degraded (43.025%); 0 B/s, 80 objects/s recovering 2024-09-13T14:36:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:02 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/663156802' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:02.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:02 smithi003 ceph-mon[104007]: pgmap v301: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 184 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 3818/8874 objects degraded (43.025%); 0 B/s, 80 objects/s recovering 2024-09-13T14:36:02.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:02 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/663156802' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:02 smithi003 ceph-mon[107278]: pgmap v301: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 184 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 3818/8874 objects degraded (43.025%); 0 B/s, 80 objects/s recovering 2024-09-13T14:36:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:02 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/663156802' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:02.605 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:02.621 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 111115 objects 2024-09-13T14:36:02.621 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :155.941 2024-09-13T14:36:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:03 smithi019 ceph-mon[91296]: from='client.35313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:03 smithi019 ceph-mon[91296]: from='client.35319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:03.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:03 smithi003 ceph-mon[107278]: from='client.35313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:03.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:03 smithi003 ceph-mon[107278]: from='client.35319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:03.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:03 smithi003 ceph-mon[104007]: from='client.35313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:03.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:03 smithi003 ceph-mon[104007]: from='client.35319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:04 smithi019 ceph-mon[91296]: osdmap e120: 8 total, 8 up, 8 in 2024-09-13T14:36:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:04 smithi019 ceph-mon[91296]: pgmap v303: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 170 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 3818/8235 objects degraded (46.363%); 0 B/s, 80 objects/s recovering 2024-09-13T14:36:04.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:04 smithi003 ceph-mon[107278]: osdmap e120: 8 total, 8 up, 8 in 2024-09-13T14:36:04.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:04 smithi003 ceph-mon[107278]: pgmap v303: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 170 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 3818/8235 objects degraded (46.363%); 0 B/s, 80 objects/s recovering 2024-09-13T14:36:04.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:04 smithi003 ceph-mon[104007]: osdmap e120: 8 total, 8 up, 8 in 2024-09-13T14:36:04.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:04 smithi003 ceph-mon[104007]: pgmap v303: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 170 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 3818/8235 objects degraded (46.363%); 0 B/s, 80 objects/s recovering 2024-09-13T14:36:04.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:36:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:36:04] "GET /metrics HTTP/1.1" 200 36412 "" "Prometheus/2.51.0" 2024-09-13T14:36:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:05 smithi019 ceph-mon[91296]: osdmap e121: 8 total, 8 up, 8 in 2024-09-13T14:36:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:05 smithi003 ceph-mon[107278]: osdmap e121: 8 total, 8 up, 8 in 2024-09-13T14:36:05.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:05 smithi003 ceph-mon[104007]: osdmap e121: 8 total, 8 up, 8 in 2024-09-13T14:36:05.624 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:05.765 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-13T14:36:05.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-13T14:36:06.077 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:06.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:06 smithi019 ceph-mon[91296]: osdmap e122: 8 total, 8 up, 8 in 2024-09-13T14:36:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:06 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/923699573' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:06 smithi019 ceph-mon[91296]: pgmap v306: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 2840/78 objects degraded (3641.026%); 0 B/s, 148 objects/s recovering 2024-09-13T14:36:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3818/8235 objects degraded (46.363%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:06 smithi019 ceph-mon[91296]: from='client.45326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:06 smithi019 ceph-mon[91296]: from='client.45332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[104007]: osdmap e122: 8 total, 8 up, 8 in 2024-09-13T14:36:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/923699573' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[104007]: pgmap v306: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 2840/78 objects degraded (3641.026%); 0 B/s, 148 objects/s recovering 2024-09-13T14:36:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3818/8235 objects degraded (46.363%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[104007]: from='client.45326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[104007]: from='client.45332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[107278]: osdmap e122: 8 total, 8 up, 8 in 2024-09-13T14:36:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/923699573' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[107278]: pgmap v306: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 2840/78 objects degraded (3641.026%); 0 B/s, 148 objects/s recovering 2024-09-13T14:36:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3818/8235 objects degraded (46.363%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[107278]: from='client.45326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:06 smithi003 ceph-mon[107278]: from='client.45332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:07.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:07 smithi003 ceph-mon[107278]: osdmap e123: 8 total, 8 up, 8 in 2024-09-13T14:36:07.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:07 smithi003 ceph-mon[104007]: osdmap e123: 8 total, 8 up, 8 in 2024-09-13T14:36:07.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:07 smithi019 ceph-mon[91296]: osdmap e123: 8 total, 8 up, 8 in 2024-09-13T14:36:08.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:08 smithi003 ceph-mon[107278]: osdmap e124: 8 total, 8 up, 8 in 2024-09-13T14:36:08.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:08 smithi003 ceph-mon[107278]: pgmap v309: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 2840/78 objects degraded (3641.026%); 0 B/s, 152 objects/s recovering 2024-09-13T14:36:08.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:08 smithi003 ceph-mon[104007]: osdmap e124: 8 total, 8 up, 8 in 2024-09-13T14:36:08.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:08 smithi003 ceph-mon[104007]: pgmap v309: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 2840/78 objects degraded (3641.026%); 0 B/s, 152 objects/s recovering 2024-09-13T14:36:08.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:08 smithi019 ceph-mon[91296]: osdmap e124: 8 total, 8 up, 8 in 2024-09-13T14:36:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:08 smithi019 ceph-mon[91296]: pgmap v309: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 2840/78 objects degraded (3641.026%); 0 B/s, 152 objects/s recovering 2024-09-13T14:36:08.682 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:09 smithi003 ceph-mon[107278]: osdmap e125: 8 total, 8 up, 8 in 2024-09-13T14:36:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1261019987' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:09 smithi003 ceph-mon[107278]: from='client.35349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:09 smithi003 ceph-mon[107278]: from='client.35355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:09.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:09 smithi003 ceph-mon[104007]: osdmap e125: 8 total, 8 up, 8 in 2024-09-13T14:36:09.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1261019987' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:09.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:09 smithi003 ceph-mon[104007]: from='client.35349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:09.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:09 smithi003 ceph-mon[104007]: from='client.35355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:09.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:09 smithi019 ceph-mon[91296]: osdmap e125: 8 total, 8 up, 8 in 2024-09-13T14:36:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1261019987' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:09 smithi019 ceph-mon[91296]: from='client.35349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:09 smithi019 ceph-mon[91296]: from='client.35355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:10.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:10 smithi019 ceph-mon[91296]: pgmap v311: 89 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1867/78 objects degraded (2393.590%); 0 B/s, 146 objects/s recovering 2024-09-13T14:36:10.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:36:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:36:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:10.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:10 smithi003 ceph-mon[107278]: pgmap v311: 89 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1867/78 objects degraded (2393.590%); 0 B/s, 146 objects/s recovering 2024-09-13T14:36:10.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:10 smithi003 ceph-mon[104007]: pgmap v311: 89 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1867/78 objects degraded (2393.590%); 0 B/s, 146 objects/s recovering 2024-09-13T14:36:11.580 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:11 smithi003 ceph-mon[107278]: osdmap e126: 8 total, 8 up, 8 in 2024-09-13T14:36:11.580 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1867/78 objects degraded (2393.590%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:11.580 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:11 smithi003 ceph-mon[104007]: osdmap e126: 8 total, 8 up, 8 in 2024-09-13T14:36:11.580 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1867/78 objects degraded (2393.590%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:11.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:11 smithi019 ceph-mon[91296]: osdmap e126: 8 total, 8 up, 8 in 2024-09-13T14:36:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1867/78 objects degraded (2393.590%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:11.693 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:11.857 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:36:11 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:36:11.576+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-13T14:36:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:12 smithi019 ceph-mon[91296]: osdmap e127: 8 total, 8 up, 8 in 2024-09-13T14:36:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:12 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3462588397' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:12 smithi019 ceph-mon[91296]: pgmap v314: 89 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 peering, 86 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 369 objects/s recovering 2024-09-13T14:36:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:36:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:12 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:36:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:12 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-13T14:36:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:12 smithi019 ceph-mon[91296]: from='client.35367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:12 smithi019 ceph-mon[91296]: from='client.45368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[107278]: osdmap e127: 8 total, 8 up, 8 in 2024-09-13T14:36:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3462588397' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[107278]: pgmap v314: 89 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 peering, 86 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 369 objects/s recovering 2024-09-13T14:36:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:36:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:36:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-13T14:36:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[107278]: from='client.35367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[107278]: from='client.45368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[104007]: osdmap e127: 8 total, 8 up, 8 in 2024-09-13T14:36:12.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3462588397' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:12.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[104007]: pgmap v314: 89 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 peering, 86 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 369 objects/s recovering 2024-09-13T14:36:12.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:36:12.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:36:12.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-13T14:36:12.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[104007]: from='client.35367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:12.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:12 smithi003 ceph-mon[104007]: from='client.45368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_2' removed 2024-09-13T14:36:13.564 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1867/78 objects degraded (2393.590%), 2 pgs degraded) 2024-09-13T14:36:13.565 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:36:13.565 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[107278]: osdmap e128: 8 total, 8 up, 8 in 2024-09-13T14:36:13.565 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1465266331' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:36:13.565 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:36:13.565 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:13.565 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:13.565 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1867/78 objects degraded (2393.590%), 2 pgs degraded) 2024-09-13T14:36:13.565 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:36:13.566 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[104007]: osdmap e128: 8 total, 8 up, 8 in 2024-09-13T14:36:13.566 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1465266331' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:36:13.566 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:36:13.566 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:13.566 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:13 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1867/78 objects degraded (2393.590%), 2 pgs degraded) 2024-09-13T14:36:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:13 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:36:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:13 smithi019 ceph-mon[91296]: osdmap e128: 8 total, 8 up, 8 in 2024-09-13T14:36:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:13 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1465266331' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:36:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:13 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:36:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:13.982 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:36:13.983 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:36:13.983 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-13T14:36:13.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-13T14:36:14.230 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:36:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:36:14] "GET /metrics HTTP/1.1" 200 36403 "" "Prometheus/2.51.0" 2024-09-13T14:36:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:14 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:36:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:14 smithi003 ceph-mon[104007]: osdmap e129: 8 total, 8 up, 8 in 2024-09-13T14:36:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:14 smithi003 ceph-mon[104007]: pgmap v317: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:14.274 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:14.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:14 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:36:14.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:14 smithi003 ceph-mon[107278]: osdmap e129: 8 total, 8 up, 8 in 2024-09-13T14:36:14.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:14 smithi003 ceph-mon[107278]: pgmap v317: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:14.650 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:14 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:36:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:14 smithi019 ceph-mon[91296]: osdmap e129: 8 total, 8 up, 8 in 2024-09-13T14:36:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:14 smithi019 ceph-mon[91296]: pgmap v317: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:15.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1841216552' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:15.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:15 smithi003 ceph-mon[107278]: from='client.35391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:15.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:15 smithi003 ceph-mon[107278]: from='client.45392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:15.407 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1841216552' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:15.407 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:15 smithi003 ceph-mon[104007]: from='client.35391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:15.407 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:15 smithi003 ceph-mon[104007]: from='client.45392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:15.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1841216552' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:15 smithi019 ceph-mon[91296]: from='client.35391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:15 smithi019 ceph-mon[91296]: from='client.45392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:16.244 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_3' created 2024-09-13T14:36:16.606 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:16 smithi003 ceph-mon[107278]: pgmap v318: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:16.606 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:16 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2590179178' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:36:16.606 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:16 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:36:16.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:16 smithi003 ceph-mon[104007]: pgmap v318: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:16.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:16 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2590179178' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:36:16.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:16 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:36:16.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:16 smithi019 ceph-mon[91296]: pgmap v318: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:16 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2590179178' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:36:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:16 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:36:17.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-13T14:36:17.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:17 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T14:36:17.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:17 smithi003 ceph-mon[107278]: osdmap e130: 8 total, 8 up, 8 in 2024-09-13T14:36:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:17 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4067452444' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:17 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T14:36:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:17 smithi003 ceph-mon[104007]: osdmap e130: 8 total, 8 up, 8 in 2024-09-13T14:36:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:17 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4067452444' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:17.441 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:17.568 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:17.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:17 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T14:36:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:17 smithi019 ceph-mon[91296]: osdmap e130: 8 total, 8 up, 8 in 2024-09-13T14:36:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:17 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4067452444' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:18.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:18 smithi003 ceph-mon[107278]: osdmap e131: 8 total, 8 up, 8 in 2024-09-13T14:36:18.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:18 smithi003 ceph-mon[107278]: pgmap v321: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:18.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:18 smithi003 ceph-mon[107278]: from='client.35415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:18 smithi003 ceph-mon[107278]: from='client.45416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:18 smithi003 ceph-mon[104007]: osdmap e131: 8 total, 8 up, 8 in 2024-09-13T14:36:18.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:18 smithi003 ceph-mon[104007]: pgmap v321: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:18.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:18 smithi003 ceph-mon[104007]: from='client.35415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:18 smithi003 ceph-mon[104007]: from='client.45416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:18 smithi019 ceph-mon[91296]: osdmap e131: 8 total, 8 up, 8 in 2024-09-13T14:36:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:18 smithi019 ceph-mon[91296]: pgmap v321: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:18 smithi019 ceph-mon[91296]: from='client.35415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:18 smithi019 ceph-mon[91296]: from='client.45416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:19 smithi003 ceph-mon[104007]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:36:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:19 smithi003 ceph-mon[104007]: osdmap e132: 8 total, 8 up, 8 in 2024-09-13T14:36:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:19 smithi003 ceph-mon[107278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:36:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:19 smithi003 ceph-mon[107278]: osdmap e132: 8 total, 8 up, 8 in 2024-09-13T14:36:19.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:19 smithi019 ceph-mon[91296]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:36:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:19 smithi019 ceph-mon[91296]: osdmap e132: 8 total, 8 up, 8 in 2024-09-13T14:36:20.264 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-13T14:36:20.477 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:20.538 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:20 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3980787337' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:36:20.539 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:20 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:36:20.539 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:20 smithi003 ceph-mon[107278]: pgmap v323: 89 pgs: 3 unknown, 86 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:20.539 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:20 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1900636743' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:20.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:20 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3980787337' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:36:20.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:20 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:36:20.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:20 smithi003 ceph-mon[104007]: pgmap v323: 89 pgs: 3 unknown, 86 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:20.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:20 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1900636743' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:36:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:36:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:20 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3980787337' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:36:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:20 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:36:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:20 smithi019 ceph-mon[91296]: pgmap v323: 89 pgs: 3 unknown, 86 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:20 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1900636743' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:21 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:36:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:21 smithi019 ceph-mon[91296]: osdmap e133: 8 total, 8 up, 8 in 2024-09-13T14:36:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:21 smithi019 ceph-mon[91296]: from='client.45434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:21 smithi019 ceph-mon[91296]: from='client.45440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:21.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:21 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:36:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:21 smithi003 ceph-mon[107278]: osdmap e133: 8 total, 8 up, 8 in 2024-09-13T14:36:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:21 smithi003 ceph-mon[107278]: from='client.45434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:21 smithi003 ceph-mon[107278]: from='client.45440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:21 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:36:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:21 smithi003 ceph-mon[104007]: osdmap e133: 8 total, 8 up, 8 in 2024-09-13T14:36:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:21 smithi003 ceph-mon[104007]: from='client.45434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:21 smithi003 ceph-mon[104007]: from='client.45440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:21.838 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-13T14:36:21.842 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:36:21.842 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:36:21.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T14:36:21.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:36:21.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_139172 2024-09-13T14:36:21.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:36:21.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:36:22.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:22 smithi019 ceph-mon[91296]: pgmap v325: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:22.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:22 smithi003 ceph-mon[104007]: pgmap v325: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:22.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:22 smithi003 ceph-mon[107278]: pgmap v325: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:36:22.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3815 3799 237.424 237.438 0.00262441 0.00419681 2024-09-13T14:36:23.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:23 smithi019 ceph-mon[91296]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:36:23.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:23 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:36:23.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:23 smithi003 ceph-mon[107278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:36:23.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:23 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:36:23.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:23 smithi003 ceph-mon[104007]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:36:23.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:23 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:36:23.897 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 7185 7169 224.018 210.625 0.00456473 0.00445565 2024-09-13T14:36:23.929 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:24.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:36:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:36:24] "GET /metrics HTTP/1.1" 200 36403 "" "Prometheus/2.51.0" 2024-09-13T14:36:24.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:24 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1767083787' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:24 smithi019 ceph-mon[91296]: pgmap v326: 89 pgs: 89 active+clean; 67 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s wr, 171 op/s 2024-09-13T14:36:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:24 smithi019 ceph-mon[91296]: from='client.45458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:24.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:24 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1767083787' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:24.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:24 smithi003 ceph-mon[107278]: pgmap v326: 89 pgs: 89 active+clean; 67 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s wr, 171 op/s 2024-09-13T14:36:24.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:24 smithi003 ceph-mon[107278]: from='client.45458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:24.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:24 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1767083787' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:24 smithi003 ceph-mon[104007]: pgmap v326: 89 pgs: 89 active+clean; 67 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s wr, 171 op/s 2024-09-13T14:36:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:24 smithi003 ceph-mon[104007]: from='client.45458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:24.897 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 10423 10407 216.797 202.375 0.00625573 0.00460528 2024-09-13T14:36:25.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:25 smithi019 ceph-mon[91296]: from='client.45464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:25.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:25 smithi003 ceph-mon[107278]: from='client.45464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:25.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:25 smithi003 ceph-mon[104007]: from='client.45464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:25.897 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 13734 13718 214.328 206.938 0.00559813 0.00465983 2024-09-13T14:36:26.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:26 smithi019 ceph-mon[91296]: pgmap v327: 89 pgs: 89 active+clean; 394 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s 2024-09-13T14:36:26.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:26 smithi003 ceph-mon[107278]: pgmap v327: 89 pgs: 89 active+clean; 394 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s 2024-09-13T14:36:26.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:26 smithi003 ceph-mon[104007]: pgmap v327: 89 pgs: 89 active+clean; 394 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s 2024-09-13T14:36:26.897 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 16613 16597 207.446 179.938 0.00443858 0.00481393 2024-09-13T14:36:27.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:27 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3986877119' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:27.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3986877119' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:27.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3986877119' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:27.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:27.720 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:27.897 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 18836 18820 196.008 138.938 0.00479796 0.00509459 2024-09-13T14:36:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:28 smithi019 ceph-mon[91296]: pgmap v328: 89 pgs: 89 active+clean; 394 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 43 MiB/s wr, 683 op/s 2024-09-13T14:36:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:28 smithi019 ceph-mon[91296]: from='client.55345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:28 smithi003 ceph-mon[107278]: pgmap v328: 89 pgs: 89 active+clean; 394 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 43 MiB/s wr, 683 op/s 2024-09-13T14:36:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:28 smithi003 ceph-mon[107278]: from='client.55345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:28.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:28 smithi003 ceph-mon[104007]: pgmap v328: 89 pgs: 89 active+clean; 394 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 43 MiB/s wr, 683 op/s 2024-09-13T14:36:28.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:28 smithi003 ceph-mon[104007]: from='client.55345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:28.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:28.897 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 21279 21263 189.818 152.688 0.00467475 0.00526035 2024-09-13T14:36:29.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:29 smithi019 ceph-mon[91296]: from='client.55351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:29.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:29 smithi003 ceph-mon[104007]: from='client.55351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:29.616 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:29 smithi003 ceph-mon[107278]: from='client.55351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:29.898 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 23799 23783 185.776 157.5 0.0125576 0.00537693 2024-09-13T14:36:30.323 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:36:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:36:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:30 smithi019 ceph-mon[91296]: pgmap v329: 89 pgs: 89 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:36:30.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:30 smithi003 ceph-mon[104007]: pgmap v329: 89 pgs: 89 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:36:30.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:30 smithi003 ceph-mon[107278]: pgmap v329: 89 pgs: 89 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:36:30.898 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 26062 26046 180.845 141.438 0.0115575 0.00552327 2024-09-13T14:36:31.454 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:31.579 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/32352033' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:31.579 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/32352033' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:31.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/32352033' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:31.898 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 28071 28055 175.316 125.562 0.00577929 0.00569836 2024-09-13T14:36:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:32 smithi019 ceph-mon[91296]: from='client.45491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:32 smithi019 ceph-mon[91296]: pgmap v330: 89 pgs: 89 active+clean; 1.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:36:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:32 smithi019 ceph-mon[91296]: from='client.45494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:32 smithi003 ceph-mon[107278]: from='client.45491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:32 smithi003 ceph-mon[107278]: pgmap v330: 89 pgs: 89 active+clean; 1.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:36:32.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:32 smithi003 ceph-mon[107278]: from='client.45494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:32 smithi003 ceph-mon[104007]: from='client.45491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:32 smithi003 ceph-mon[104007]: pgmap v330: 89 pgs: 89 active+clean; 1.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:36:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:32 smithi003 ceph-mon[104007]: from='client.45494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:32.898 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 30039 30023 170.559 123 0.0067277 0.00585625 2024-09-13T14:36:33.898 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 32144 32128 167.308 131.562 0.00753947 0.0059718 2024-09-13T14:36:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:34.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.434 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:36:34] "GET /metrics HTTP/1.1" 200 36400 "" "Prometheus/2.51.0" 2024-09-13T14:36:34.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:34.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:34.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:34.898 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 33982 33966 163.274 114.875 0.0207041 0.00611849 2024-09-13T14:36:35.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[107278]: pgmap v331: 89 pgs: 89 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T14:36:35.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:35.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:35.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:35.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:35.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:36:35.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1531386945' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:35.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[104007]: pgmap v331: 89 pgs: 89 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T14:36:35.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:35.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:35.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:35.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:35.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:36:35.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:34 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1531386945' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:35.136 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:35.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:34 smithi019 ceph-mon[91296]: pgmap v331: 89 pgs: 89 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T14:36:35.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:36:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:34 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1531386945' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:35.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 35945 35929 160.375 122.688 0.00471036 0.0062302 2024-09-13T14:36:36.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:36 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:36:36.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:36 smithi019 ceph-mon[91296]: Upgrade: osd.3 is safe to restart 2024-09-13T14:36:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:36 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:36:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:36 smithi003 ceph-mon[104007]: Upgrade: osd.3 is safe to restart 2024-09-13T14:36:36.383 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:36 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:36:36.383 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:36 smithi003 ceph-mon[107278]: Upgrade: osd.3 is safe to restart 2024-09-13T14:36:36.898 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 37997 37981 158.232 128.25 0.0144806 0.00631342 2024-09-13T14:36:37.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[107278]: from='client.55366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:37.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[107278]: pgmap v332: 89 pgs: 89 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:36:37.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[107278]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:37.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:37.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:36:37.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:37.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[104007]: from='client.55366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:37.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[104007]: pgmap v332: 89 pgs: 89 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:36:37.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[104007]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:37.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:37.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:36:37.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:37 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:37.347 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:37 smithi019 ceph-mon[91296]: from='client.55366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:37.347 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:37 smithi019 ceph-mon[91296]: pgmap v332: 89 pgs: 89 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:36:37.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:37 smithi019 ceph-mon[91296]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:37.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:37 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:37.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:37 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:36:37.348 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:37 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:37.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 15 39769 39754 155.268 110.812 0.00391043 0.00643431 2024-09-13T14:36:38.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:38 smithi019 ceph-mon[91296]: Upgrade: Updating osd.3 2024-09-13T14:36:38.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:38 smithi019 ceph-mon[91296]: Deploying daemon osd.3 on smithi003 2024-09-13T14:36:38.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:38 smithi003 ceph-mon[107278]: Upgrade: Updating osd.3 2024-09-13T14:36:38.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:38 smithi003 ceph-mon[107278]: Deploying daemon osd.3 on smithi003 2024-09-13T14:36:38.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:38 smithi003 ceph-mon[104007]: Upgrade: Updating osd.3 2024-09-13T14:36:38.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:38 smithi003 ceph-mon[104007]: Deploying daemon osd.3 on smithi003 2024-09-13T14:36:38.835 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:38.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 41213 41197 151.439 90.1875 0.00290325 0.00659477 2024-09-13T14:36:39.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:39 smithi003 ceph-mon[107278]: pgmap v333: 89 pgs: 89 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 132 MiB/s wr, 2.10k op/s 2024-09-13T14:36:39.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4280038296' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:39.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:39 smithi003 ceph-mon[104007]: pgmap v333: 89 pgs: 89 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 132 MiB/s wr, 2.10k op/s 2024-09-13T14:36:39.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4280038296' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:39 smithi019 ceph-mon[91296]: pgmap v333: 89 pgs: 89 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 132 MiB/s wr, 2.10k op/s 2024-09-13T14:36:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4280038296' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:39.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 42747 42731 148.352 95.875 0.00751284 0.00673586 2024-09-13T14:36:40.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:40 smithi019 ceph-mon[91296]: from='client.55381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:40.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:36:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:36:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:40.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:40 smithi003 ceph-mon[104007]: from='client.55381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:40.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:40 smithi003 ceph-mon[107278]: from='client.55381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:40.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 44601 44585 146.642 115.875 0.00844185 0.00681373 2024-09-13T14:36:41.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:41 smithi019 ceph-mon[91296]: from='client.35523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:41.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:41 smithi019 ceph-mon[91296]: pgmap v334: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-09-13T14:36:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:41 smithi003 ceph-mon[107278]: from='client.35523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:41 smithi003 ceph-mon[107278]: pgmap v334: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-09-13T14:36:41.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:41 smithi003 ceph-mon[104007]: from='client.35523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:41.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:41 smithi003 ceph-mon[104007]: pgmap v334: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-09-13T14:36:41.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:36:41.899265+0000 min lat: 0.0011714 max lat: 0.113088 avg lat: 0.00691967 2024-09-13T14:36:41.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:36:41.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 46228 46212 144.394 101.688 0.00410518 0.00691967 2024-09-13T14:36:42.390 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:42 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/262710774' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:42.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:42 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/262710774' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:42.413 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:42.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:42 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/262710774' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:42.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 47886 47870 142.453 103.625 0.00594123 0.00701339 2024-09-13T14:36:43.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:43 smithi003 ceph-mon[107278]: pgmap v335: 89 pgs: 89 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-13T14:36:43.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:43 smithi003 ceph-mon[107278]: from='client.55396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:43.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:43.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:43 smithi003 ceph-mon[107278]: from='client.35541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:43.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:43 smithi003 ceph-mon[104007]: pgmap v335: 89 pgs: 89 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-13T14:36:43.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:43 smithi003 ceph-mon[104007]: from='client.55396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:43.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:43.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:43 smithi003 ceph-mon[104007]: from='client.35541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:43 smithi019 ceph-mon[91296]: pgmap v335: 89 pgs: 89 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-13T14:36:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:43 smithi019 ceph-mon[91296]: from='client.55396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:43 smithi019 ceph-mon[91296]: from='client.35541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:43.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 49628 49612 140.926 108.875 0.00611857 0.00709067 2024-09-13T14:36:44.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:44 smithi003 ceph-mon[107278]: pgmap v336: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-13T14:36:44.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:44 smithi003 ceph-mon[104007]: pgmap v336: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-13T14:36:44.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:44 smithi019 ceph-mon[91296]: pgmap v336: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-13T14:36:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:36:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:36:44] "GET /metrics HTTP/1.1" 200 36409 "" "Prometheus/2.51.0" 2024-09-13T14:36:44.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 51426 51410 139.684 112.375 0.00682544 0.00715385 2024-09-13T14:36:45.432 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:45 smithi003 systemd[1]: Stopping Ceph osd.3 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:36:45.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 52908 52892 137.723 92.625 0.00716048 0.00725567 2024-09-13T14:36:45.932 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:45 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[64478]: 2024-09-13T14:36:45.661+0000 7f90f38fb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:36:45.932 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:45 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[64478]: 2024-09-13T14:36:45.661+0000 7f90f38fb640 -1 osd.3 133 *** Got signal Terminated *** 2024-09-13T14:36:45.933 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:45 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[64478]: 2024-09-13T14:36:45.661+0000 7f90f38fb640 -1 osd.3 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:36:46.125 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:46.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 53964 53948 134.854 66 0.00280092 0.00730675 2024-09-13T14:36:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:46 smithi019 ceph-mon[91296]: pgmap v337: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-13T14:36:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:46 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3832574629' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:46 smithi019 ceph-mon[91296]: osd.3 marked itself down and dead 2024-09-13T14:36:46.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:46 smithi003 ceph-mon[104007]: pgmap v337: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-13T14:36:46.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:46 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3832574629' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:46.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:46 smithi003 ceph-mon[104007]: osd.3 marked itself down and dead 2024-09-13T14:36:46.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:46 smithi003 ceph-mon[107278]: pgmap v337: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-13T14:36:46.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:46 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3832574629' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:46.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:46 smithi003 ceph-mon[107278]: osd.3 marked itself down and dead 2024-09-13T14:36:47.432 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:47 smithi003 podman[140794]: 2024-09-13 14:36:47.161473264 +0000 UTC m=+1.647385686 container died 20984c58cba91d478466910177479d2d37c7f68ee893b6d9efd5e0d4427bebba (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-13T14:36:47.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[104007]: from='client.55408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:47.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[104007]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:36:47.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[104007]: osdmap e134: 8 total, 7 up, 8 in 2024-09-13T14:36:47.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[104007]: from='client.45548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:47.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[104007]: osdmap e135: 8 total, 7 up, 8 in 2024-09-13T14:36:47.800 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[107278]: from='client.55408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:47.800 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[107278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:36:47.800 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[107278]: osdmap e134: 8 total, 7 up, 8 in 2024-09-13T14:36:47.800 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[107278]: from='client.45548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:47.800 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:47 smithi003 ceph-mon[107278]: osdmap e135: 8 total, 7 up, 8 in 2024-09-13T14:36:47.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 54639 54623 131.29 42.1875 0.00813937 0.00761169 2024-09-13T14:36:47.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:47 smithi019 ceph-mon[91296]: from='client.55408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:47.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:47 smithi019 ceph-mon[91296]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:36:47.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:47 smithi019 ceph-mon[91296]: osdmap e134: 8 total, 7 up, 8 in 2024-09-13T14:36:47.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:47 smithi019 ceph-mon[91296]: from='client.45548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:47.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:47 smithi019 ceph-mon[91296]: osdmap e135: 8 total, 7 up, 8 in 2024-09-13T14:36:48.182 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:47 smithi003 podman[140794]: 2024-09-13 14:36:47.990195806 +0000 UTC m=+2.476108231 container cleanup 20984c58cba91d478466910177479d2d37c7f68ee893b6d9efd5e0d4427bebba (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:36:48.182 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:47 smithi003 bash[140794]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3 2024-09-13T14:36:48.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:48 smithi019 ceph-mon[91296]: pgmap v339: 89 pgs: 16 stale+active+clean, 73 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:36:48.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 56563 56547 130.88 120.25 0.00860027 0.00763463 2024-09-13T14:36:48.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:48 smithi003 ceph-mon[104007]: pgmap v339: 89 pgs: 16 stale+active+clean, 73 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:36:48.933 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:48 smithi003 podman[140866]: 2024-09-13 14:36:48.509605806 +0000 UTC m=+1.340583512 container remove 20984c58cba91d478466910177479d2d37c7f68ee893b6d9efd5e0d4427bebba (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:36:48.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:48 smithi003 ceph-mon[107278]: pgmap v339: 89 pgs: 16 stale+active+clean, 73 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:36:49.539 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:49 smithi003 podman[140969]: 2024-09-13 14:36:49.148373595 +0000 UTC m=+0.029452884 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:36:49.539 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:49 smithi003 podman[140969]: 2024-09-13 14:36:49.285980749 +0000 UTC m=+0.167060020 container create 8f416f13b09567eac31f4d131e5d02e691ab4dc637fb1590be3b5e13dd82e7f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-13T14:36:49.746 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:49.798 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1560404022' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:49.798 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:49 smithi003 podman[140969]: 2024-09-13 14:36:49.697814512 +0000 UTC m=+0.578893791 container init 8f416f13b09567eac31f4d131e5d02e691ab4dc637fb1590be3b5e13dd82e7f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-13T14:36:49.799 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:49 smithi003 podman[140969]: 2024-09-13 14:36:49.706235492 +0000 UTC m=+0.587314771 container start 8f416f13b09567eac31f4d131e5d02e691ab4dc637fb1590be3b5e13dd82e7f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-13T14:36:49.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1560404022' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:49.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 15 58309 58294 130.105 109.188 0.00847151 0.00768111 2024-09-13T14:36:49.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1560404022' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:50.182 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:49 smithi003 podman[140969]: 2024-09-13 14:36:49.797935902 +0000 UTC m=+0.679015173 container attach 8f416f13b09567eac31f4d131e5d02e691ab4dc637fb1590be3b5e13dd82e7f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:36:50.182 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:50 smithi003 podman[140969]: 2024-09-13 14:36:50.022666345 +0000 UTC m=+0.903745628 container died 8f416f13b09567eac31f4d131e5d02e691ab4dc637fb1590be3b5e13dd82e7f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-13T14:36:50.578 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:36:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:36:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:50.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 60139 60123 129.56 114.312 0.00481145 0.00771273 2024-09-13T14:36:50.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:50 smithi019 ceph-mon[91296]: pgmap v341: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 116 MiB/s wr, 1.85k op/s; 20800/167142 objects degraded (12.445%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:36:50.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:50 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 20800/167142 objects degraded (12.445%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:50 smithi019 ceph-mon[91296]: from='client.45560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:50 smithi003 ceph-mon[104007]: pgmap v341: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 116 MiB/s wr, 1.85k op/s; 20800/167142 objects degraded (12.445%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:36:50.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:50 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 20800/167142 objects degraded (12.445%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:50.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:50 smithi003 ceph-mon[104007]: from='client.45560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:50 smithi003 ceph-mon[107278]: pgmap v341: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 116 MiB/s wr, 1.85k op/s; 20800/167142 objects degraded (12.445%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:36:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:50 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 20800/167142 objects degraded (12.445%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:50 smithi003 ceph-mon[107278]: from='client.45560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:51.550 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:51 smithi003 podman[141059]: 2024-09-13 14:36:51.212181415 +0000 UTC m=+1.180385922 container remove 8f416f13b09567eac31f4d131e5d02e691ab4dc637fb1590be3b5e13dd82e7f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:36:51.550 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:51 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-13T14:36:51.551 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:51 smithi003 systemd[1]: Stopped Ceph osd.3 for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:36:51.551 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:51 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.3.service: Consumed 21min 53.021s CPU time. 2024-09-13T14:36:51.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:51 smithi003 ceph-mon[107278]: from='client.45566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:51.862 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:51 smithi003 systemd[1]: Starting Ceph osd.3 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T14:36:51.862 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:51 smithi003 ceph-mon[104007]: from='client.45566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:51.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 61975 61959 129.066 114.75 0.00618077 0.00774297 2024-09-13T14:36:51.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:51 smithi019 ceph-mon[91296]: from='client.45566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:52.131 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:51 smithi003 podman[141161]: 2024-09-13 14:36:51.767905907 +0000 UTC m=+0.029230565 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:36:52.131 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:51 smithi003 podman[141161]: 2024-09-13 14:36:51.996575662 +0000 UTC m=+0.257900308 container create 351354980da7bbef5b1ec2b36905348604607a3813486b6e01d980567c363fdc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:36:52.392 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 podman[141161]: 2024-09-13 14:36:52.266780629 +0000 UTC m=+0.528105270 container init 351354980da7bbef5b1ec2b36905348604607a3813486b6e01d980567c363fdc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:36:52.392 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 podman[141161]: 2024-09-13 14:36:52.275353398 +0000 UTC m=+0.536678041 container start 351354980da7bbef5b1ec2b36905348604607a3813486b6e01d980567c363fdc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:36:52.683 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 podman[141161]: 2024-09-13 14:36:52.391640903 +0000 UTC m=+0.652965549 container attach 351354980da7bbef5b1ec2b36905348604607a3813486b6e01d980567c363fdc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-13T14:36:52.683 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:36:52.683 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 bash[141161]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:36:52.683 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:36:52.684 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 bash[141161]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:36:52.684 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:36:52.684 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:36:52.684 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 bash[141161]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:36:52.684 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 bash[141161]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:36:52.684 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:36:52.684 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 bash[141161]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:36:52.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:52 smithi003 ceph-mon[107278]: pgmap v342: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s; 20800/167142 objects degraded (12.445%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:36:52.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:52 smithi003 ceph-mon[104007]: pgmap v342: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s; 20800/167142 objects degraded (12.445%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:36:52.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 63710 63694 128.401 108.438 0.011086 0.00778326 2024-09-13T14:36:52.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:52 smithi019 ceph-mon[91296]: pgmap v342: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s; 20800/167142 objects degraded (12.445%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:36:53.019 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:36:53.019 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 bash[141161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:36:53.019 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-13T14:36:53.019 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:52 smithi003 bash[141161]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-13T14:36:53.019 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:36:53.342 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 bash[141161]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:36:53.342 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:36:53.342 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 bash[141161]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:36:53.343 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-13T14:36:53.343 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 bash[141161]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-13T14:36:53.343 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:36:53.343 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 bash[141161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:36:53.343 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate[141173]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-13T14:36:53.343 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 bash[141161]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-13T14:36:53.343 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 podman[141161]: 2024-09-13 14:36:53.048846032 +0000 UTC m=+1.310170671 container died 351354980da7bbef5b1ec2b36905348604607a3813486b6e01d980567c363fdc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:36:53.343 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:53.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 65337 65321 127.566 101.688 0.0125001 0.00783415 2024-09-13T14:36:53.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2605345714' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:53.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2605345714' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:53.932 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:53 smithi003 podman[141360]: 2024-09-13 14:36:53.834181575 +0000 UTC m=+0.771869075 container remove 351354980da7bbef5b1ec2b36905348604607a3813486b6e01d980567c363fdc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2) 2024-09-13T14:36:53.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2605345714' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:54.396 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:54 smithi003 podman[141441]: 2024-09-13 14:36:54.038716491 +0000 UTC m=+0.029225968 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:36:54.396 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:54 smithi003 podman[141441]: 2024-09-13 14:36:54.240653935 +0000 UTC m=+0.231163389 container create c85652aac1b4ceb4fe56e8e8255e4b021ea3aef4b343a4bd711dfd45d4c796ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-13T14:36:54.396 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:36:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:36:54] "GET /metrics HTTP/1.1" 200 36409 "" "Prometheus/2.51.0" 2024-09-13T14:36:54.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:54 smithi003 ceph-mon[107278]: from='client.35577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:54.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:54 smithi003 ceph-mon[107278]: pgmap v343: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s; 21935/175644 objects degraded (12.488%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:36:54.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:54 smithi003 ceph-mon[107278]: from='client.35583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:54.683 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:54 smithi003 podman[141441]: 2024-09-13 14:36:54.554197913 +0000 UTC m=+0.544707378 container init c85652aac1b4ceb4fe56e8e8255e4b021ea3aef4b343a4bd711dfd45d4c796ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-13T14:36:54.683 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:54 smithi003 podman[141441]: 2024-09-13 14:36:54.564185958 +0000 UTC m=+0.554695415 container start c85652aac1b4ceb4fe56e8e8255e4b021ea3aef4b343a4bd711dfd45d4c796ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:36:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:54 smithi003 ceph-mon[104007]: from='client.35577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:54 smithi003 ceph-mon[104007]: pgmap v343: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s; 21935/175644 objects degraded (12.488%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:36:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:54 smithi003 ceph-mon[104007]: from='client.35583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:54.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 67067 67051 126.976 108.125 0.00800452 0.00787016 2024-09-13T14:36:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:54 smithi019 ceph-mon[91296]: from='client.35577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:54 smithi019 ceph-mon[91296]: pgmap v343: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s; 21935/175644 objects degraded (12.488%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:36:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:54 smithi019 ceph-mon[91296]: from='client.35583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:55.183 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:54 smithi003 bash[141441]: c85652aac1b4ceb4fe56e8e8255e4b021ea3aef4b343a4bd711dfd45d4c796ba 2024-09-13T14:36:55.184 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:54 smithi003 systemd[1]: Started Ceph osd.3 for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T14:36:55.901 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 68704 68688 126.25 102.312 0.00376682 0.00791556 2024-09-13T14:36:56.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:56.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:56.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:56.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 21935/175644 objects degraded (12.488%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:56.182 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:36:55 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T14:36:55.704+0000 7f3e84926640 -1 Falling back to public interface 2024-09-13T14:36:56.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:56.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:56.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:56.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 21935/175644 objects degraded (12.488%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 21935/175644 objects degraded (12.488%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:56.813 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:36:56.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 70207 70191 125.327 93.9375 0.0122303 0.00797416 2024-09-13T14:36:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:56 smithi019 ceph-mon[91296]: pgmap v344: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 24059/193728 objects degraded (12.419%); 7.2 KiB/s, 0 objects/s recovering 2024-09-13T14:36:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2694791433' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:57.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:56 smithi003 ceph-mon[107278]: pgmap v344: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 24059/193728 objects degraded (12.419%); 7.2 KiB/s, 0 objects/s recovering 2024-09-13T14:36:57.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2694791433' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:57.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:56 smithi003 ceph-mon[104007]: pgmap v344: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 24059/193728 objects degraded (12.419%); 7.2 KiB/s, 0 objects/s recovering 2024-09-13T14:36:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2694791433' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:36:57.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 71764 71748 124.549 97.3125 0.00636072 0.00802376 2024-09-13T14:36:57.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:57 smithi003 ceph-mon[107278]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:57.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:57.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:57.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:57 smithi003 ceph-mon[104007]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:57.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:57.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:57 smithi019 ceph-mon[91296]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:36:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:58.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 73342 73326 123.848 98.625 0.0144227 0.0080688 2024-09-13T14:36:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:58 smithi019 ceph-mon[91296]: from='client.35604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:58 smithi019 ceph-mon[91296]: pgmap v345: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s; 24059/193728 objects degraded (12.419%); 6.4 KiB/s, 0 objects/s recovering 2024-09-13T14:36:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:58 smithi003 ceph-mon[104007]: from='client.35604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:58 smithi003 ceph-mon[104007]: pgmap v345: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s; 24059/193728 objects degraded (12.419%); 6.4 KiB/s, 0 objects/s recovering 2024-09-13T14:36:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:58 smithi003 ceph-mon[107278]: from='client.35604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:58 smithi003 ceph-mon[107278]: pgmap v345: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s; 24059/193728 objects degraded (12.419%); 6.4 KiB/s, 0 objects/s recovering 2024-09-13T14:36:59.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 74822 74806 123.02 92.5 0.0123658 0.0081231 2024-09-13T14:37:00.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:36:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2099971742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:00.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:36:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2099971742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:00.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:36:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2099971742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:00.396 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:37:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:37:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:37:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:00.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 76109 76093 121.928 80.4375 0.0142751 0.0081966 2024-09-13T14:37:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:00 smithi019 ceph-mon[91296]: pgmap v346: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 122 MiB/s wr, 1.96k op/s; 27014/217839 objects degraded (12.401%); 5.4 KiB/s, 0 objects/s recovering 2024-09-13T14:37:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 27014/217839 objects degraded (12.401%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:00 smithi003 ceph-mon[107278]: pgmap v346: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 122 MiB/s wr, 1.96k op/s; 27014/217839 objects degraded (12.401%); 5.4 KiB/s, 0 objects/s recovering 2024-09-13T14:37:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 27014/217839 objects degraded (12.401%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:01.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:00 smithi003 ceph-mon[104007]: pgmap v346: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 122 MiB/s wr, 1.96k op/s; 27014/217839 objects degraded (12.401%); 5.4 KiB/s, 0 objects/s recovering 2024-09-13T14:37:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 27014/217839 objects degraded (12.401%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:01.901 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:37:01.901866+0000 min lat: 0.0011714 max lat: 1.04333 avg lat: 0.00824501 2024-09-13T14:37:01.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:37:01.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 77604 77588 121.216 93.4375 0.0112211 0.00824501 2024-09-13T14:37:02.161 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:01 smithi003 ceph-mon[104007]: from='client.35607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:02.161 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:01 smithi003 ceph-mon[104007]: from='client.45611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:02.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:01 smithi003 ceph-mon[107278]: from='client.35607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:02.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:01 smithi003 ceph-mon[107278]: from='client.45611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:02.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:01 smithi019 ceph-mon[91296]: from='client.35607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:02.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:01 smithi019 ceph-mon[91296]: from='client.45611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:02.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 79087 79071 120.52 92.6875 0.00967469 0.00829247 2024-09-13T14:37:03.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:02 smithi003 ceph-mon[104007]: pgmap v347: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 27014/217839 objects degraded (12.401%) 2024-09-13T14:37:03.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:02 smithi003 ceph-mon[107278]: pgmap v347: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 27014/217839 objects degraded (12.401%) 2024-09-13T14:37:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:02 smithi019 ceph-mon[91296]: pgmap v347: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 27014/217839 objects degraded (12.401%) 2024-09-13T14:37:03.903 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 80318 80302 119.482 76.9375 0.010166 0.0083648 2024-09-13T14:37:04.126 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:37:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:37:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:37:04] "GET /metrics HTTP/1.1" 200 36423 "" "Prometheus/2.51.0" 2024-09-13T14:37:04.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 81484 81468 118.398 72.875 0.00597628 0.00844095 2024-09-13T14:37:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:04 smithi019 ceph-mon[91296]: pgmap v348: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s; 27900/224646 objects degraded (12.420%) 2024-09-13T14:37:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:04 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1627765935' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:04.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:04 smithi003 ceph-mon[104007]: pgmap v348: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s; 27900/224646 objects degraded (12.420%) 2024-09-13T14:37:04.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:04.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:04.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:04 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1627765935' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:04 smithi003 ceph-mon[107278]: pgmap v348: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s; 27900/224646 objects degraded (12.420%) 2024-09-13T14:37:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:04.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:04 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1627765935' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:05.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 82859 82843 117.66 85.9375 0.015777 0.00849419 2024-09-13T14:37:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[104007]: from='client.55477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[104007]: from='client.55483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 27900/224646 objects degraded (12.420%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[107278]: from='client.55477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[107278]: from='client.55483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 27900/224646 objects degraded (12.420%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:05 smithi019 ceph-mon[91296]: from='client.55477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:05 smithi019 ceph-mon[91296]: from='client.55483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:05 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:05 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 27900/224646 objects degraded (12.420%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:06.716 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:37:06 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T14:37:06.620+0000 7f3e84926640 -1 osd.3 133 log_to_monitors true 2024-09-13T14:37:06.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 84232 84216 116.952 85.8125 0.00432352 0.0085454 2024-09-13T14:37:06.995 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:06 smithi003 ceph-mon[104007]: pgmap v349: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s; 29683/239343 objects degraded (12.402%) 2024-09-13T14:37:06.995 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:06 smithi003 ceph-mon[104007]: from='osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:37:06.995 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:06 smithi003 ceph-mon[107278]: pgmap v349: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s; 29683/239343 objects degraded (12.402%) 2024-09-13T14:37:06.995 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:06 smithi003 ceph-mon[107278]: from='osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:37:06.995 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 14:37:06 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T14:37:06.776+0000 7f3e7c6d0640 -1 osd.3 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:37:07.082 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:06 smithi019 ceph-mon[91296]: pgmap v349: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s; 29683/239343 objects degraded (12.402%) 2024-09-13T14:37:07.082 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:06 smithi019 ceph-mon[91296]: from='osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:37:07.732 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-09-13T14:37:07.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 85407 85391 116.006 73.4375 0.00355797 0.00858938 2024-09-13T14:37:07.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:07 smithi003 ceph-mon[104007]: from='osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:37:07.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:07 smithi003 ceph-mon[104007]: osdmap e136: 8 total, 7 up, 8 in 2024-09-13T14:37:07.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:07 smithi003 ceph-mon[104007]: from='osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:37:07.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:07 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/910357643' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:07.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:07 smithi003 ceph-mon[107278]: from='osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:37:07.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:07 smithi003 ceph-mon[107278]: osdmap e136: 8 total, 7 up, 8 in 2024-09-13T14:37:07.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:07 smithi003 ceph-mon[107278]: from='osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:37:07.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:07 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/910357643' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:07 smithi019 ceph-mon[91296]: from='osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:37:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:07 smithi019 ceph-mon[91296]: osdmap e136: 8 total, 7 up, 8 in 2024-09-13T14:37:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:07 smithi019 ceph-mon[91296]: from='osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-09-13T14:37:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:07 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/910357643' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:08.877 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[107278]: pgmap v351: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s; 29683/239343 objects degraded (12.402%) 2024-09-13T14:37:08.877 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[107278]: from='client.55495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:08.877 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[107278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:37:08.877 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[107278]: osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419] boot 2024-09-13T14:37:08.877 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[107278]: osdmap e137: 8 total, 8 up, 8 in 2024-09-13T14:37:08.877 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:37:08.877 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[107278]: osdmap e138: 8 total, 8 up, 8 in 2024-09-13T14:37:08.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[104007]: pgmap v351: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s; 29683/239343 objects degraded (12.402%) 2024-09-13T14:37:08.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[104007]: from='client.55495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:08.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[104007]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:37:08.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[104007]: osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419] boot 2024-09-13T14:37:08.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[104007]: osdmap e137: 8 total, 8 up, 8 in 2024-09-13T14:37:08.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:37:08.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:08 smithi003 ceph-mon[104007]: osdmap e138: 8 total, 8 up, 8 in 2024-09-13T14:37:08.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 85407 85391 113.538 0 - 0.00858938 2024-09-13T14:37:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:08 smithi019 ceph-mon[91296]: pgmap v351: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s; 29683/239343 objects degraded (12.402%) 2024-09-13T14:37:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:08 smithi019 ceph-mon[91296]: from='client.55495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:08 smithi019 ceph-mon[91296]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:37:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:08 smithi019 ceph-mon[91296]: osd.3 [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419] boot 2024-09-13T14:37:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:08 smithi019 ceph-mon[91296]: osdmap e137: 8 total, 8 up, 8 in 2024-09-13T14:37:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:37:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:08 smithi019 ceph-mon[91296]: osdmap e138: 8 total, 8 up, 8 in 2024-09-13T14:37:09.682 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:37:09.555+0000 7f0f7ea9d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-13T14:37:09.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 85407 85391 111.173 0 - 0.00858938 2024-09-13T14:37:10.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='client.35640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:10.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:10.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:10.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:10.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:10.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:10.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:10.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:10.185 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:10.185 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:10.185 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-13T14:37:10.185 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:09 smithi019 ceph-mon[91296]: osdmap e139: 8 total, 8 up, 8 in 2024-09-13T14:37:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='client.35640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:10.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[107278]: osdmap e139: 8 total, 8 up, 8 in 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='client.35640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:10.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:10.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:10.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:10.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:10.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:10.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:10.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:10.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:10.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-13T14:37:10.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:09 smithi003 ceph-mon[104007]: osdmap e139: 8 total, 8 up, 8 in 2024-09-13T14:37:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 4 2024-09-13T14:37:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:37:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:37:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:10.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 86396 86380 110.165 20.6042 0.0302726 0.00907105 2024-09-13T14:37:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[107278]: pgmap v354: 89 pgs: 1 remapped+peering, 15 peering, 11 active+undersized, 5 active+undersized+degraded, 57 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s; 14888/252975 objects degraded (5.885%) 2024-09-13T14:37:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[107278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-13T14:37:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[107278]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-13T14:37:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 14888/252975 objects degraded (5.885%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:10.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2308308157' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[104007]: pgmap v354: 89 pgs: 1 remapped+peering, 15 peering, 11 active+undersized, 5 active+undersized+degraded, 57 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s; 14888/252975 objects degraded (5.885%) 2024-09-13T14:37:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[104007]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-13T14:37:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[104007]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-13T14:37:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 14888/252975 objects degraded (5.885%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2308308157' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:10 smithi019 ceph-mon[91296]: pgmap v354: 89 pgs: 1 remapped+peering, 15 peering, 11 active+undersized, 5 active+undersized+degraded, 57 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s; 14888/252975 objects degraded (5.885%) 2024-09-13T14:37:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:10 smithi019 ceph-mon[91296]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-13T14:37:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:10 smithi019 ceph-mon[91296]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-13T14:37:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 14888/252975 objects degraded (5.885%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2308308157' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:37:11.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade pause' 2024-09-13T14:37:11.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 87405 87389 109.22 63.0625 0.012119 0.00915031 2024-09-13T14:37:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:12.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 88431 88415 108.336 64.125 0.00880146 0.00922438 2024-09-13T14:37:12.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:12 smithi003 ceph-mon[107278]: pgmap v356: 89 pgs: 1 remapped+peering, 15 peering, 11 active+undersized, 5 active+undersized+degraded, 57 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s; 14888/252975 objects degraded (5.885%) 2024-09-13T14:37:12.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:12 smithi003 ceph-mon[104007]: pgmap v356: 89 pgs: 1 remapped+peering, 15 peering, 11 active+undersized, 5 active+undersized+degraded, 57 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s; 14888/252975 objects degraded (5.885%) 2024-09-13T14:37:13.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:12 smithi019 ceph-mon[91296]: pgmap v356: 89 pgs: 1 remapped+peering, 15 peering, 11 active+undersized, 5 active+undersized+degraded, 57 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s; 14888/252975 objects degraded (5.885%) 2024-09-13T14:37:13.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 89469 89453 107.5 64.875 0.0099497 0.00929646 2024-09-13T14:37:14.083 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:14.084 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:14.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:14.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:37:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:37:14] "GET /metrics HTTP/1.1" 200 36409 "" "Prometheus/2.51.0" 2024-09-13T14:37:14.523 INFO:teuthology.orchestra.run.smithi003.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:37:14.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 90460 90444 106.64 61.9375 0.0200158 0.00937204 2024-09-13T14:37:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:14 smithi019 ceph-mon[91296]: pgmap v357: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 9 peering, 75 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 7656/260541 objects degraded (2.939%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:37:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:14 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:15.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:14 smithi003 ceph-mon[107278]: pgmap v357: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 9 peering, 75 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 7656/260541 objects degraded (2.939%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:37:15.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:14 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:15.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:14 smithi003 ceph-mon[104007]: pgmap v357: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 9 peering, 75 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 7656/260541 objects degraded (2.939%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:37:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:14 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:15.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 91299 91283 105.636 52.4375 0.0188404 0.00946184 2024-09-13T14:37:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:15 smithi019 ceph-mon[91296]: from='client.55513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:15 smithi019 ceph-mon[91296]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:37:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:15 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 7656/260541 objects degraded (2.939%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:16.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:15 smithi003 ceph-mon[104007]: from='client.55513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:16.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:15 smithi003 ceph-mon[104007]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:37:16.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:15 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 7656/260541 objects degraded (2.939%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:16.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:15 smithi003 ceph-mon[107278]: from='client.55513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:16.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:15 smithi003 ceph-mon[107278]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:37:16.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:15 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 7656/260541 objects degraded (2.939%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:16.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 92156 92140 104.689 53.5625 0.00733065 0.00954705 2024-09-13T14:37:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:16 smithi019 ceph-mon[91296]: pgmap v358: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 11563/266925 objects degraded (4.332%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:37:17.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:16 smithi003 ceph-mon[104007]: pgmap v358: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 11563/266925 objects degraded (4.332%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:37:17.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:16 smithi003 ceph-mon[107278]: pgmap v358: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 11563/266925 objects degraded (4.332%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:37:17.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:37:17.905 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 93059 93043 103.826 56.4375 0.0231577 0.00962603 2024-09-13T14:37:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:18.906 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 94075 94059 103.118 63.5 0.0121431 0.00969216 2024-09-13T14:37:18.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:18 smithi003 ceph-mon[104007]: pgmap v359: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 40 MiB/s wr, 636 op/s; 11563/266925 objects degraded (4.332%); 930 KiB/s, 14 objects/s recovering 2024-09-13T14:37:18.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:18 smithi003 ceph-mon[107278]: pgmap v359: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 40 MiB/s wr, 636 op/s; 11563/266925 objects degraded (4.332%); 930 KiB/s, 14 objects/s recovering 2024-09-13T14:37:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:18 smithi019 ceph-mon[91296]: pgmap v359: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 40 MiB/s wr, 636 op/s; 11563/266925 objects degraded (4.332%); 930 KiB/s, 14 objects/s recovering 2024-09-13T14:37:19.906 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 95075 95059 102.418 62.5 0.0181071 0.00975813 2024-09-13T14:37:20.482 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:37:20.482 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (11m) 16s ago 16m 19.9M - 0.27.0 11f11916f8cd b74b601869dc 2024-09-13T14:37:20.482 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi019 *:3000 running (11m) 47s ago 16m 83.9M - 9.4.7 954c08fa6188 1d1343f3e5db 2024-09-13T14:37:20.482 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.zmhiel smithi003 running (15m) 16s ago 15m 21.8M - 17.2.7-1627-gbbdadfa3 a994b00deb3a db8314ccc6fd 2024-09-13T14:37:20.482 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi019.ydtwbg smithi019 running (15m) 47s ago 15m 16.5M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c0fef505f410 2024-09-13T14:37:20.483 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi019 *:8443,9283,8765 running (10m) 47s ago 20m 457M - 19.3.0-4697-gd58f7306 fec07016a2b0 f0b9ab84e7f3 2024-09-13T14:37:20.483 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:8443,9283,8765 running (11m) 16s ago 22m 520M - 19.3.0-4697-gd58f7306 fec07016a2b0 e8e7986695ad 2024-09-13T14:37:20.483 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (10m) 16s ago 22m 64.6M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 265013e6136c 2024-09-13T14:37:20.483 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi019 running (7m) 47s ago 20m 47.4M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 301a92c4f814 2024-09-13T14:37:20.483 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (9m) 16s ago 20m 51.5M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 8539c77bb8fb 2024-09-13T14:37:20.484 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (11m) 16s ago 15m 11.1M - 1.7.0 72c9c2088986 30d257fdd98b 2024-09-13T14:37:20.484 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi019 *:9100 running (11m) 47s ago 15m 10.8M - 1.7.0 72c9c2088986 738f7b07d0d8 2024-09-13T14:37:20.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (6m) 16s ago 20m 837M 2481M 19.3.0-4697-gd58f7306 fec07016a2b0 8e2479752fc3 2024-09-13T14:37:20.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (5m) 16s ago 19m 297M 2481M 19.3.0-4697-gd58f7306 fec07016a2b0 3126fff58165 2024-09-13T14:37:20.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (2m) 16s ago 19m 264M 2481M 19.3.0-4697-gd58f7306 fec07016a2b0 357adcd35fd7 2024-09-13T14:37:20.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (26s) 16s ago 19m 14.6M 2481M 19.3.0-4697-gd58f7306 fec07016a2b0 c85652aac1b4 2024-09-13T14:37:20.485 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi019 running (18m) 47s ago 18m 556M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a e50c61f762b0 2024-09-13T14:37:20.485 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi019 running (18m) 47s ago 18m 1081M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 1cd389248627 2024-09-13T14:37:20.485 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi019 running (17m) 47s ago 17m 693M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a aa95f3146e47 2024-09-13T14:37:20.485 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi019 running (17m) 47s ago 17m 501M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 118a8cf6eadc 2024-09-13T14:37:20.485 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi019 *:9095 running (10m) 47s ago 16m 55.9M - 2.51.0 1d3b7f56885b 0b10348ed2da 2024-09-13T14:37:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:37:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:37:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:20.906 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 96016 96000 101.679 58.8125 0.00732862 0.00982868 2024-09-13T14:37:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:20 smithi019 ceph-mon[91296]: pgmap v360: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 58 MiB/s wr, 927 op/s; 11396/280818 objects degraded (4.058%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:37:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11396/280818 objects degraded (4.058%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:21.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:20 smithi003 ceph-mon[107278]: pgmap v360: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 58 MiB/s wr, 927 op/s; 11396/280818 objects degraded (4.058%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:37:21.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11396/280818 objects degraded (4.058%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:21.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:20 smithi003 ceph-mon[104007]: pgmap v360: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 58 MiB/s wr, 927 op/s; 11396/280818 objects degraded (4.058%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:37:21.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11396/280818 objects degraded (4.058%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:21.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:37:21.900 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:21.907 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:37:21.907864+0000 min lat: 0.0011714 max lat: 2.24071 avg lat: 0.00989873 2024-09-13T14:37:21.907 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:37:21.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 96950 96934 100.954 58.375 0.0321795 0.00989873 2024-09-13T14:37:22.157 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:21 smithi003 ceph-mon[107278]: from='client.45650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:22.158 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:21 smithi003 ceph-mon[104007]: from='client.45650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:22.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:21 smithi019 ceph-mon[91296]: from='client.45650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:22.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 97954 97938 100.328 62.75 0.00743585 0.00996192 2024-09-13T14:37:23.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:22 smithi003 ceph-mon[104007]: pgmap v361: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 790 op/s; 11396/280818 objects degraded (4.058%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:37:23.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:22 smithi003 ceph-mon[107278]: pgmap v361: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 790 op/s; 11396/280818 objects degraded (4.058%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:37:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:22 smithi019 ceph-mon[91296]: pgmap v361: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 790 op/s; 11396/280818 objects degraded (4.058%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:37:23.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 98913 98897 99.6764 59.9375 0.0083156 0.0100271 2024-09-13T14:37:24.286 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (12m) 20s ago 16m 19.9M - 0.27.0 11f11916f8cd b74b601869dc 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi019 *:3000 running (11m) 51s ago 16m 83.9M - 9.4.7 954c08fa6188 1d1343f3e5db 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.zmhiel smithi003 running (15m) 20s ago 15m 21.8M - 17.2.7-1627-gbbdadfa3 a994b00deb3a db8314ccc6fd 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi019.ydtwbg smithi019 running (15m) 51s ago 15m 16.5M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c0fef505f410 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi019 *:8443,9283,8765 running (10m) 51s ago 20m 457M - 19.3.0-4697-gd58f7306 fec07016a2b0 f0b9ab84e7f3 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:8443,9283,8765 running (11m) 20s ago 22m 520M - 19.3.0-4697-gd58f7306 fec07016a2b0 e8e7986695ad 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (10m) 20s ago 22m 64.6M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 265013e6136c 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi019 running (7m) 51s ago 20m 47.4M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 301a92c4f814 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (9m) 20s ago 20m 51.5M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 8539c77bb8fb 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (11m) 20s ago 16m 11.1M - 1.7.0 72c9c2088986 30d257fdd98b 2024-09-13T14:37:24.287 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi019 *:9100 running (11m) 51s ago 15m 10.8M - 1.7.0 72c9c2088986 738f7b07d0d8 2024-09-13T14:37:24.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (6m) 20s ago 20m 837M 2481M 19.3.0-4697-gd58f7306 fec07016a2b0 8e2479752fc3 2024-09-13T14:37:24.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (5m) 20s ago 19m 297M 2481M 19.3.0-4697-gd58f7306 fec07016a2b0 3126fff58165 2024-09-13T14:37:24.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (2m) 20s ago 19m 264M 2481M 19.3.0-4697-gd58f7306 fec07016a2b0 357adcd35fd7 2024-09-13T14:37:24.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (30s) 20s ago 19m 14.6M 2481M 19.3.0-4697-gd58f7306 fec07016a2b0 c85652aac1b4 2024-09-13T14:37:24.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi019 running (18m) 51s ago 18m 556M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a e50c61f762b0 2024-09-13T14:37:24.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi019 running (18m) 51s ago 18m 1081M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 1cd389248627 2024-09-13T14:37:24.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi019 running (17m) 51s ago 17m 693M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a aa95f3146e47 2024-09-13T14:37:24.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi019 running (17m) 51s ago 17m 501M 2481M 17.2.7-1627-gbbdadfa3 a994b00deb3a 118a8cf6eadc 2024-09-13T14:37:24.288 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi019 *:9095 running (10m) 51s ago 16m 55.9M - 2.51.0 1d3b7f56885b 0b10348ed2da 2024-09-13T14:37:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:37:24] "GET /metrics HTTP/1.1" 200 36409 "" "Prometheus/2.51.0" 2024-09-13T14:37:24.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 99825 99809 98.9989 57 0.0100875 0.0100956 2024-09-13T14:37:25.031 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[107278]: pgmap v362: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s; 11204/287370 objects degraded (3.899%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:37:25.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:25.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:25.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:25.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:25.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:25.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:25.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:25.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:25.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[104007]: pgmap v362: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s; 11204/287370 objects degraded (3.899%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:37:25.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:25.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:25.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:25.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:25.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:25.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:25.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:25.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:25.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph versions' 2024-09-13T14:37:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:24 smithi019 ceph-mon[91296]: pgmap v362: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s; 11204/287370 objects degraded (3.899%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:37:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:25.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:25.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:37:25.541 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:25.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 100712 100696 98.3183 55.4375 0.0420802 0.010165 2024-09-13T14:37:26.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:25 smithi003 ceph-mon[104007]: from='client.35649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:26.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11204/287370 objects degraded (3.899%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:26.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:25 smithi003 ceph-mon[107278]: from='client.35649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:26.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11204/287370 objects degraded (3.899%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:25 smithi019 ceph-mon[91296]: from='client.35649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11204/287370 objects degraded (3.899%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:26.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 101688 101672 97.7441 61 0.00444683 0.010225 2024-09-13T14:37:27.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:26 smithi019 ceph-mon[91296]: pgmap v363: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 971 op/s; 11204/295515 objects degraded (3.791%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:37:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:26 smithi003 ceph-mon[104007]: pgmap v363: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 971 op/s; 11204/295515 objects degraded (3.791%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:37:27.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:26 smithi003 ceph-mon[107278]: pgmap v363: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 971 op/s; 11204/295515 objects degraded (3.791%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:37:27.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 102615 102599 97.141 57.9375 0.0184931 0.010289 2024-09-13T14:37:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:28.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:28.404 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:37:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": { 2024-09-13T14:37:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:37:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:37:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": { 2024-09-13T14:37:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": { 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)": 4, 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 4 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": { 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)": 2 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "overall": { 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)": 6, 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 9 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:37:28.405 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:37:28.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 103540 103524 96.5539 57.8125 0.0116997 0.0103517 2024-09-13T14:37:29.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:28 smithi019 ceph-mon[91296]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 50 MiB/s wr, 793 op/s; 11204/295515 objects degraded (3.791%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:37:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3147005301' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:29.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:28 smithi003 ceph-mon[107278]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 50 MiB/s wr, 793 op/s; 11204/295515 objects degraded (3.791%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:37:29.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3147005301' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:28 smithi003 ceph-mon[104007]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 50 MiB/s wr, 793 op/s; 11204/295515 objects degraded (3.791%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:37:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3147005301' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:37:29.576 DEBUG:teuthology.parallel:result is None 2024-09-13T14:37:29.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 104515 104499 96.0301 60.9375 0.00891459 0.0104074 2024-09-13T14:37:30.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:37:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:37:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:30.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 105514 105498 95.5433 62.4375 0.0157606 0.0104612 2024-09-13T14:37:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:30 smithi019 ceph-mon[91296]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 10988/309393 objects degraded (3.551%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:37:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10988/309393 objects degraded (3.551%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:30 smithi003 ceph-mon[104007]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 10988/309393 objects degraded (3.551%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:37:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10988/309393 objects degraded (3.551%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:30 smithi003 ceph-mon[107278]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 10988/309393 objects degraded (3.551%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:37:31.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10988/309393 objects degraded (3.551%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:31.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 106502 106486 95.0605 61.75 0.0280145 0.0105144 2024-09-13T14:37:32.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 107501 107485 94.6009 62.4375 0.0447849 0.0105646 2024-09-13T14:37:33.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:32 smithi019 ceph-mon[91296]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 50 MiB/s wr, 793 op/s; 10988/309393 objects degraded (3.551%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:37:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:32 smithi003 ceph-mon[104007]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 50 MiB/s wr, 793 op/s; 10988/309393 objects degraded (3.551%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:37:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:32 smithi003 ceph-mon[107278]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 50 MiB/s wr, 793 op/s; 10988/309393 objects degraded (3.551%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:37:33.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 108458 108442 94.1177 59.8125 0.00824682 0.010619 2024-09-13T14:37:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:37:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:37:34] "GET /metrics HTTP/1.1" 200 36427 "" "Prometheus/2.51.0" 2024-09-13T14:37:34.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 109432 109416 93.6623 60.875 0.0113268 0.0106717 2024-09-13T14:37:35.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:34 smithi019 ceph-mon[91296]: pgmap v367: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 970 op/s; 10751/315753 objects degraded (3.405%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:37:35.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:34 smithi003 ceph-mon[104007]: pgmap v367: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 970 op/s; 10751/315753 objects degraded (3.405%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:37:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:34 smithi003 ceph-mon[107278]: pgmap v367: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 970 op/s; 10751/315753 objects degraded (3.405%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:37:35.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 110351 110335 93.1728 57.4375 0.0125878 0.0107276 2024-09-13T14:37:36.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10751/315753 objects degraded (3.405%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:36.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10751/315753 objects degraded (3.405%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:36.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10751/315753 objects degraded (3.405%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:36.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 111164 111148 92.6079 50.8125 0.00944979 0.0107921 2024-09-13T14:37:37.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:36 smithi019 ceph-mon[91296]: pgmap v368: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 10751/324195 objects degraded (3.316%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:37:37.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:36 smithi003 ceph-mon[104007]: pgmap v368: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 10751/324195 objects degraded (3.316%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:37:37.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:36 smithi003 ceph-mon[107278]: pgmap v368: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 10751/324195 objects degraded (3.316%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:37:37.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 111999 111983 92.0761 52.1875 0.00688727 0.010855 2024-09-13T14:37:38.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 112922 112906 91.6294 57.6875 0.0216961 0.010908 2024-09-13T14:37:39.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:38 smithi019 ceph-mon[91296]: pgmap v369: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 10751/324195 objects degraded (3.316%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:37:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:38 smithi003 ceph-mon[104007]: pgmap v369: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 10751/324195 objects degraded (3.316%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:37:39.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:38 smithi003 ceph-mon[107278]: pgmap v369: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 10751/324195 objects degraded (3.316%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:37:39.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 113866 113850 91.2111 59 0.013141 0.0109582 2024-09-13T14:37:40.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:37:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:37:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:40.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 114795 114779 90.7914 58.0625 0.0103588 0.0110081 2024-09-13T14:37:41.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:40 smithi019 ceph-mon[91296]: pgmap v370: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 10509/337638 objects degraded (3.113%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:37:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10509/337638 objects degraded (3.113%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:41.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:40 smithi003 ceph-mon[104007]: pgmap v370: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 10509/337638 objects degraded (3.113%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:37:41.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10509/337638 objects degraded (3.113%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:41.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:40 smithi003 ceph-mon[107278]: pgmap v370: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 10509/337638 objects degraded (3.113%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:37:41.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10509/337638 objects degraded (3.113%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:41.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:37:41.909676+0000 min lat: 0.0011714 max lat: 2.24071 avg lat: 0.0110619 2024-09-13T14:37:41.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:37:41.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 115692 115676 90.3573 56.0625 0.0128529 0.0110619 2024-09-13T14:37:42.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 116625 116609 89.9616 58.3125 0.016783 0.0111105 2024-09-13T14:37:43.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:42 smithi019 ceph-mon[91296]: pgmap v371: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 10509/337638 objects degraded (3.113%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:37:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:43.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:42 smithi003 ceph-mon[104007]: pgmap v371: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 10509/337638 objects degraded (3.113%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:37:43.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:43.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:42 smithi003 ceph-mon[107278]: pgmap v371: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 10509/337638 objects degraded (3.113%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:37:43.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:43.909 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 117552 117536 89.571 57.9375 0.0239957 0.0111588 2024-09-13T14:37:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:37:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:37:44] "GET /metrics HTTP/1.1" 200 36428 "" "Prometheus/2.51.0" 2024-09-13T14:37:44.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 118497 118481 89.2034 59.0625 0.0081165 0.0112049 2024-09-13T14:37:45.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:44 smithi019 ceph-mon[91296]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 948 op/s; 10229/343563 objects degraded (2.977%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T14:37:45.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:44 smithi003 ceph-mon[104007]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 948 op/s; 10229/343563 objects degraded (2.977%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T14:37:45.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:44 smithi003 ceph-mon[107278]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 948 op/s; 10229/343563 objects degraded (2.977%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T14:37:45.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 119419 119403 88.8275 57.625 0.00736562 0.0112527 2024-09-13T14:37:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10229/343563 objects degraded (2.977%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:46.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10229/343563 objects degraded (2.977%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:46.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10229/343563 objects degraded (2.977%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:46.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 120309 120293 88.4368 55.625 0.0269909 0.0113021 2024-09-13T14:37:47.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:46 smithi003 ceph-mon[107278]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s; 10229/351540 objects degraded (2.910%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:37:47.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:46 smithi003 ceph-mon[104007]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s; 10229/351540 objects degraded (2.910%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:37:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:46 smithi019 ceph-mon[91296]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s; 10229/351540 objects degraded (2.910%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:37:47.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 121223 121207 88.0727 57.125 0.0130687 0.0113492 2024-09-13T14:37:48.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 15 122114 122099 87.7011 55.75 0.0125705 0.0113966 2024-09-13T14:37:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:48 smithi019 ceph-mon[91296]: pgmap v374: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 47 MiB/s wr, 759 op/s; 10229/351540 objects degraded (2.910%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:37:49.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:48 smithi003 ceph-mon[104007]: pgmap v374: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 47 MiB/s wr, 759 op/s; 10229/351540 objects degraded (2.910%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:37:49.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:48 smithi003 ceph-mon[107278]: pgmap v374: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 47 MiB/s wr, 759 op/s; 10229/351540 objects degraded (2.910%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:37:49.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 123025 123009 87.3508 56.875 0.0158935 0.0114425 2024-09-13T14:37:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:37:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:37:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:50.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 123928 123912 87.0034 56.4375 0.0136609 0.0114873 2024-09-13T14:37:51.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:50 smithi019 ceph-mon[91296]: pgmap v375: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 9949/365301 objects degraded (2.724%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:37:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9949/365301 objects degraded (2.724%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:51.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:50 smithi003 ceph-mon[104007]: pgmap v375: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 9949/365301 objects degraded (2.724%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:37:51.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9949/365301 objects degraded (2.724%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:51.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:50 smithi003 ceph-mon[107278]: pgmap v375: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 9949/365301 objects degraded (2.724%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:37:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9949/365301 objects degraded (2.724%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:51.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.008 2024-09-13T14:37:51.910 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 124790 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 86.652 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 42.7497 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 237.438 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1386 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 684.008 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3799 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0115359 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.028534 2024-09-13T14:37:51.911 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.24071 2024-09-13T14:37:51.912 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.0011714 2024-09-13T14:37:51.915 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:37:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:52 smithi019 ceph-mon[91296]: pgmap v376: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 767 op/s; 9949/365301 objects degraded (2.724%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:37:53.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:52 smithi019 ceph-mon[91296]: osdmap e140: 8 total, 8 up, 8 in 2024-09-13T14:37:53.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:52 smithi003 ceph-mon[104007]: pgmap v376: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 767 op/s; 9949/365301 objects degraded (2.724%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:37:53.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:52 smithi003 ceph-mon[104007]: osdmap e140: 8 total, 8 up, 8 in 2024-09-13T14:37:53.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:52 smithi003 ceph-mon[107278]: pgmap v376: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 767 op/s; 9949/365301 objects degraded (2.724%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:37:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:52 smithi003 ceph-mon[107278]: osdmap e140: 8 total, 8 up, 8 in 2024-09-13T14:37:54.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:53 smithi019 ceph-mon[91296]: osdmap e141: 8 total, 8 up, 8 in 2024-09-13T14:37:54.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:53 smithi003 ceph-mon[107278]: osdmap e141: 8 total, 8 up, 8 in 2024-09-13T14:37:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:37:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:37:54] "GET /metrics HTTP/1.1" 200 36428 "" "Prometheus/2.51.0" 2024-09-13T14:37:54.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:53 smithi003 ceph-mon[104007]: osdmap e141: 8 total, 8 up, 8 in 2024-09-13T14:37:55.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:54 smithi019 ceph-mon[91296]: pgmap v379: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 759 op/s; 9949/368370 objects degraded (2.701%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:37:55.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:54 smithi003 ceph-mon[107278]: pgmap v379: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 759 op/s; 9949/368370 objects degraded (2.701%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:37:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:54 smithi003 ceph-mon[104007]: pgmap v379: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 759 op/s; 9949/368370 objects degraded (2.701%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:37:56.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9949/368370 objects degraded (2.701%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:56.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9949/368370 objects degraded (2.701%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:56.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9949/368370 objects degraded (2.701%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:37:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:56 smithi019 ceph-mon[91296]: pgmap v380: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 58 MiB/s wr, 1.02k op/s; 9735/371499 objects degraded (2.620%); 3.9 MiB/s, 61 objects/s recovering 2024-09-13T14:37:57.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:56 smithi003 ceph-mon[107278]: pgmap v380: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 58 MiB/s wr, 1.02k op/s; 9735/371499 objects degraded (2.620%); 3.9 MiB/s, 61 objects/s recovering 2024-09-13T14:37:57.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:56 smithi003 ceph-mon[104007]: pgmap v380: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 58 MiB/s wr, 1.02k op/s; 9735/371499 objects degraded (2.620%); 3.9 MiB/s, 61 objects/s recovering 2024-09-13T14:37:58.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:58.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:58.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:37:58 smithi019 ceph-mon[91296]: pgmap v381: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 24 MiB/s wr, 506 op/s; 9580/371451 objects degraded (2.579%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:37:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:37:58 smithi003 ceph-mon[104007]: pgmap v381: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 24 MiB/s wr, 506 op/s; 9580/371451 objects degraded (2.579%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:37:59.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:37:59 smithi003 ceph-mon[107278]: pgmap v381: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 24 MiB/s wr, 506 op/s; 9580/371451 objects degraded (2.579%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:38:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:38:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:38:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:01.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:01 smithi019 ceph-mon[91296]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 24 MiB/s wr, 1.04k op/s; 9366/358596 objects degraded (2.612%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:38:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:01 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9366/358596 objects degraded (2.612%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:01.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:01 smithi003 ceph-mon[104007]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 24 MiB/s wr, 1.04k op/s; 9366/358596 objects degraded (2.612%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:38:01.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:01 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9366/358596 objects degraded (2.612%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:01.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:01 smithi003 ceph-mon[107278]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 24 MiB/s wr, 1.04k op/s; 9366/358596 objects degraded (2.612%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:38:01.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:01 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9366/358596 objects degraded (2.612%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:03 smithi019 ceph-mon[91296]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 107 B/s rd, 20 MiB/s wr, 874 op/s; 9366/358596 objects degraded (2.612%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:38:03.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:03 smithi003 ceph-mon[104007]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 107 B/s rd, 20 MiB/s wr, 874 op/s; 9366/358596 objects degraded (2.612%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:38:03.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:03 smithi003 ceph-mon[107278]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 107 B/s rd, 20 MiB/s wr, 874 op/s; 9366/358596 objects degraded (2.612%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:38:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:38:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:38:04] "GET /metrics HTTP/1.1" 200 36427 "" "Prometheus/2.51.0" 2024-09-13T14:38:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:05 smithi019 ceph-mon[91296]: pgmap v384: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 97 B/s rd, 18 MiB/s wr, 979 op/s; 9366/352632 objects degraded (2.656%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:38:05.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:05 smithi003 ceph-mon[104007]: pgmap v384: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 97 B/s rd, 18 MiB/s wr, 979 op/s; 9366/352632 objects degraded (2.656%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:38:05.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:05 smithi003 ceph-mon[107278]: pgmap v384: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 97 B/s rd, 18 MiB/s wr, 979 op/s; 9366/352632 objects degraded (2.656%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:38:06.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9366/352632 objects degraded (2.656%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:06.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9366/352632 objects degraded (2.656%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:06.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9366/352632 objects degraded (2.656%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:07.036 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:07 smithi003 ceph-mon[104007]: pgmap v385: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 9.3 MiB/s wr, 947 op/s; 9141/345018 objects degraded (2.649%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:38:07.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:07 smithi019 ceph-mon[91296]: pgmap v385: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 9.3 MiB/s wr, 947 op/s; 9141/345018 objects degraded (2.649%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:38:07.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:07 smithi003 ceph-mon[107278]: pgmap v385: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 9.3 MiB/s wr, 947 op/s; 9141/345018 objects degraded (2.649%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:38:09.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:09 smithi019 ceph-mon[91296]: pgmap v386: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 MiB/s wr, 793 op/s; 9141/344214 objects degraded (2.656%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:38:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:09 smithi003 ceph-mon[107278]: pgmap v386: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 MiB/s wr, 793 op/s; 9141/344214 objects degraded (2.656%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:38:09.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:09 smithi003 ceph-mon[104007]: pgmap v386: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 MiB/s wr, 793 op/s; 9141/344214 objects degraded (2.656%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:38:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:38:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:38:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:11 smithi019 ceph-mon[91296]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 8914/332370 objects degraded (2.682%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:38:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8914/332370 objects degraded (2.682%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:11 smithi003 ceph-mon[107278]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 8914/332370 objects degraded (2.682%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:38:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8914/332370 objects degraded (2.682%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:11 smithi003 ceph-mon[104007]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 8914/332370 objects degraded (2.682%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:38:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8914/332370 objects degraded (2.682%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:13 smithi019 ceph-mon[91296]: pgmap v388: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 8914/332370 objects degraded (2.682%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:38:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:13.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:13 smithi003 ceph-mon[104007]: pgmap v388: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 8914/332370 objects degraded (2.682%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:38:13.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:13 smithi003 ceph-mon[107278]: pgmap v388: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 8914/332370 objects degraded (2.682%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:38:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:38:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:38:14] "GET /metrics HTTP/1.1" 200 36429 "" "Prometheus/2.51.0" 2024-09-13T14:38:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:15 smithi019 ceph-mon[91296]: pgmap v389: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 8914/326643 objects degraded (2.729%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:38:15.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:15 smithi003 ceph-mon[104007]: pgmap v389: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 8914/326643 objects degraded (2.729%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:38:15.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:15 smithi003 ceph-mon[107278]: pgmap v389: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 8914/326643 objects degraded (2.729%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:38:16.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8914/326643 objects degraded (2.729%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:16.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8914/326643 objects degraded (2.729%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:16.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8914/326643 objects degraded (2.729%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:17 smithi019 ceph-mon[91296]: pgmap v390: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 8670/319020 objects degraded (2.718%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:38:17.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:17 smithi003 ceph-mon[104007]: pgmap v390: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 8670/319020 objects degraded (2.718%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:38:17.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:17 smithi003 ceph-mon[107278]: pgmap v390: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 8670/319020 objects degraded (2.718%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:38:18.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:18 smithi019 ceph-mon[91296]: pgmap v391: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 8670/318180 objects degraded (2.725%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:38:18.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:18 smithi003 ceph-mon[104007]: pgmap v391: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 8670/318180 objects degraded (2.725%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:38:18.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:18 smithi003 ceph-mon[107278]: pgmap v391: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 8670/318180 objects degraded (2.725%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:38:20.562 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:38:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:38:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:20.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:20 smithi019 ceph-mon[91296]: pgmap v392: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 8406/305886 objects degraded (2.748%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:38:20.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8406/305886 objects degraded (2.748%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:20.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:20 smithi003 ceph-mon[104007]: pgmap v392: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 8406/305886 objects degraded (2.748%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:38:20.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8406/305886 objects degraded (2.748%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:20 smithi003 ceph-mon[107278]: pgmap v392: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 8406/305886 objects degraded (2.748%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:38:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8406/305886 objects degraded (2.748%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:22.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:22 smithi019 ceph-mon[91296]: pgmap v393: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 8406/305886 objects degraded (2.748%); 2.6 MiB/s, 42 objects/s recovering 2024-09-13T14:38:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:22 smithi003 ceph-mon[104007]: pgmap v393: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 8406/305886 objects degraded (2.748%); 2.6 MiB/s, 42 objects/s recovering 2024-09-13T14:38:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:22 smithi003 ceph-mon[107278]: pgmap v393: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 8406/305886 objects degraded (2.748%); 2.6 MiB/s, 42 objects/s recovering 2024-09-13T14:38:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:38:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:38:24] "GET /metrics HTTP/1.1" 200 36429 "" "Prometheus/2.51.0" 2024-09-13T14:38:24.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:24 smithi019 ceph-mon[91296]: pgmap v394: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 8406/300129 objects degraded (2.801%); 2.6 MiB/s, 42 objects/s recovering 2024-09-13T14:38:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:24 smithi003 ceph-mon[107278]: pgmap v394: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 8406/300129 objects degraded (2.801%); 2.6 MiB/s, 42 objects/s recovering 2024-09-13T14:38:24.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:24 smithi003 ceph-mon[104007]: pgmap v394: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 8406/300129 objects degraded (2.801%); 2.6 MiB/s, 42 objects/s recovering 2024-09-13T14:38:25.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:25 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:25.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8406/300129 objects degraded (2.801%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:25.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:25.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8406/300129 objects degraded (2.801%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:25.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:25.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8406/300129 objects degraded (2.801%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:26 smithi019 ceph-mon[91296]: pgmap v395: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 8115/292629 objects degraded (2.773%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:38:26.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:26 smithi003 ceph-mon[104007]: pgmap v395: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 8115/292629 objects degraded (2.773%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:38:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:26 smithi003 ceph-mon[107278]: pgmap v395: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 8115/292629 objects degraded (2.773%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:38:28.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:28 smithi019 ceph-mon[91296]: pgmap v396: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 8115/291843 objects degraded (2.781%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:38:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:28 smithi003 ceph-mon[107278]: pgmap v396: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 8115/291843 objects degraded (2.781%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:38:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:28.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:28 smithi003 ceph-mon[104007]: pgmap v396: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 8115/291843 objects degraded (2.781%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:38:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:29.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:29 smithi019 ceph-mon[91296]: osdmap e142: 8 total, 8 up, 8 in 2024-09-13T14:38:29.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:29 smithi003 ceph-mon[107278]: osdmap e142: 8 total, 8 up, 8 in 2024-09-13T14:38:29.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:29 smithi003 ceph-mon[104007]: osdmap e142: 8 total, 8 up, 8 in 2024-09-13T14:38:30.380 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:38:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:38:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:30.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:30 smithi019 ceph-mon[91296]: pgmap v398: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 851 op/s; 7819/280320 objects degraded (2.789%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:38:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:30 smithi019 ceph-mon[91296]: osdmap e143: 8 total, 8 up, 8 in 2024-09-13T14:38:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 7819/280320 objects degraded (2.789%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:30 smithi003 ceph-mon[104007]: pgmap v398: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 851 op/s; 7819/280320 objects degraded (2.789%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:38:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:30 smithi003 ceph-mon[104007]: osdmap e143: 8 total, 8 up, 8 in 2024-09-13T14:38:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 7819/280320 objects degraded (2.789%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:30.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:30 smithi003 ceph-mon[107278]: pgmap v398: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 851 op/s; 7819/280320 objects degraded (2.789%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:38:30.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:30 smithi003 ceph-mon[107278]: osdmap e143: 8 total, 8 up, 8 in 2024-09-13T14:38:30.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 7819/280320 objects degraded (2.789%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:32.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:32.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:32.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:32.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:32.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:33.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:32 smithi019 ceph-mon[91296]: pgmap v400: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 7819/280320 objects degraded (2.789%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:38:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:32 smithi003 ceph-mon[107278]: pgmap v400: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 7819/280320 objects degraded (2.789%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:38:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:32 smithi003 ceph-mon[104007]: pgmap v400: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 7819/280320 objects degraded (2.789%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:38:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:38:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:38:34] "GET /metrics HTTP/1.1" 200 36428 "" "Prometheus/2.51.0" 2024-09-13T14:38:35.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:34 smithi019 ceph-mon[91296]: pgmap v401: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 7819/276045 objects degraded (2.833%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:38:35.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:35.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:34 smithi003 ceph-mon[107278]: pgmap v401: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 7819/276045 objects degraded (2.833%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:38:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:34 smithi003 ceph-mon[104007]: pgmap v401: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 7819/276045 objects degraded (2.833%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:38:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:36.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:35 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:36.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:35 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:36.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:35 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:36.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 7819/276045 objects degraded (2.833%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:36.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:35 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:36.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:35 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:36.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:35 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:36.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 7819/276045 objects degraded (2.833%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:36.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:35 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:36.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:35 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:36.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:35 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:38:36.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 7819/276045 objects degraded (2.833%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:37.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:36 smithi003 ceph-mon[107278]: pgmap v402: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 885 op/s; 7515/270591 objects degraded (2.777%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:38:37.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:36 smithi003 ceph-mon[104007]: pgmap v402: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 885 op/s; 7515/270591 objects degraded (2.777%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:38:37.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:36 smithi019 ceph-mon[91296]: pgmap v402: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 885 op/s; 7515/270591 objects degraded (2.777%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:38:39.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:38 smithi019 ceph-mon[91296]: pgmap v403: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 7515/269787 objects degraded (2.786%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:38:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:38 smithi003 ceph-mon[104007]: pgmap v403: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 7515/269787 objects degraded (2.786%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:38:39.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:38 smithi003 ceph-mon[107278]: pgmap v403: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 7515/269787 objects degraded (2.786%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:38:40.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:38:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:38:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:40 smithi019 ceph-mon[91296]: pgmap v404: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 722 op/s; 7328/258636 objects degraded (2.833%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:38:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 7328/258636 objects degraded (2.833%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:41.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:40 smithi003 ceph-mon[104007]: pgmap v404: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 722 op/s; 7328/258636 objects degraded (2.833%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:38:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 7328/258636 objects degraded (2.833%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:40 smithi003 ceph-mon[107278]: pgmap v404: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 722 op/s; 7328/258636 objects degraded (2.833%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:38:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 7328/258636 objects degraded (2.833%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:42 smithi019 ceph-mon[91296]: pgmap v405: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 7328/258636 objects degraded (2.833%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:38:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:43.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:42 smithi003 ceph-mon[104007]: pgmap v405: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 7328/258636 objects degraded (2.833%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:38:43.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:42 smithi003 ceph-mon[107278]: pgmap v405: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 7328/258636 objects degraded (2.833%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:38:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:44.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:38:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:38:44] "GET /metrics HTTP/1.1" 200 36429 "" "Prometheus/2.51.0" 2024-09-13T14:38:45.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:44 smithi019 ceph-mon[91296]: pgmap v406: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 761 op/s; 7328/252903 objects degraded (2.898%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:38:45.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:44 smithi003 ceph-mon[104007]: pgmap v406: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 761 op/s; 7328/252903 objects degraded (2.898%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:38:45.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:44 smithi003 ceph-mon[107278]: pgmap v406: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 761 op/s; 7328/252903 objects degraded (2.898%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:38:46.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 7328/252903 objects degraded (2.898%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:46.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 7328/252903 objects degraded (2.898%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:46.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 7328/252903 objects degraded (2.898%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:47.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:46 smithi019 ceph-mon[91296]: pgmap v407: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 7135/245547 objects degraded (2.906%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:38:47.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:46 smithi003 ceph-mon[104007]: pgmap v407: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 7135/245547 objects degraded (2.906%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:38:47.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:46 smithi003 ceph-mon[107278]: pgmap v407: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 7135/245547 objects degraded (2.906%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:38:49.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:48 smithi019 ceph-mon[91296]: pgmap v408: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 7135/244725 objects degraded (2.916%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:38:49.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:48 smithi003 ceph-mon[104007]: pgmap v408: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 7135/244725 objects degraded (2.916%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:38:49.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:48 smithi003 ceph-mon[107278]: pgmap v408: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 7135/244725 objects degraded (2.916%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:38:50.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:38:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:38:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:51.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:50 smithi019 ceph-mon[91296]: pgmap v409: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 6924/232554 objects degraded (2.977%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:38:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 6924/232554 objects degraded (2.977%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:51.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:50 smithi003 ceph-mon[104007]: pgmap v409: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 6924/232554 objects degraded (2.977%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:38:51.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 6924/232554 objects degraded (2.977%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:50 smithi003 ceph-mon[107278]: pgmap v409: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 6924/232554 objects degraded (2.977%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:38:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 6924/232554 objects degraded (2.977%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:53.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:52 smithi019 ceph-mon[91296]: pgmap v410: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 6924/232554 objects degraded (2.977%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:38:53.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:52 smithi003 ceph-mon[104007]: pgmap v410: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 6924/232554 objects degraded (2.977%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:38:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:52 smithi003 ceph-mon[107278]: pgmap v410: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 6924/232554 objects degraded (2.977%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:38:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:38:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:38:54] "GET /metrics HTTP/1.1" 200 36429 "" "Prometheus/2.51.0" 2024-09-13T14:38:55.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:54 smithi019 ceph-mon[91296]: pgmap v411: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 6924/231729 objects degraded (2.988%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:38:55.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:54 smithi003 ceph-mon[104007]: pgmap v411: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 6924/231729 objects degraded (2.988%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:38:55.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:54 smithi003 ceph-mon[107278]: pgmap v411: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 6924/231729 objects degraded (2.988%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:38:56.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 6924/231729 objects degraded (2.988%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:56.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 6924/231729 objects degraded (2.988%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:56.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 6924/231729 objects degraded (2.988%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:57.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:56 smithi019 ceph-mon[91296]: pgmap v412: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 6707/219234 objects degraded (3.059%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:38:57.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:56 smithi003 ceph-mon[104007]: pgmap v412: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 6707/219234 objects degraded (3.059%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:38:57.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:56 smithi003 ceph-mon[107278]: pgmap v412: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 6707/219234 objects degraded (3.059%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:38:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:58.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:58.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:38:58 smithi019 ceph-mon[91296]: pgmap v413: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 6707/218505 objects degraded (3.069%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:38:59.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:38:58 smithi003 ceph-mon[107278]: pgmap v413: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 6707/218505 objects degraded (3.069%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:38:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:38:58 smithi003 ceph-mon[104007]: pgmap v413: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 6707/218505 objects degraded (3.069%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:39:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:39:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:39:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:00 smithi019 ceph-mon[91296]: pgmap v414: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 6493/206271 objects degraded (3.148%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:39:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 6493/206271 objects degraded (3.148%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:00 smithi003 ceph-mon[107278]: pgmap v414: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 6493/206271 objects degraded (3.148%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:39:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 6493/206271 objects degraded (3.148%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:00 smithi003 ceph-mon[104007]: pgmap v414: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 6493/206271 objects degraded (3.148%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:39:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 6493/206271 objects degraded (3.148%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:02 smithi019 ceph-mon[91296]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 729 op/s; 6493/206271 objects degraded (3.148%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:39:03.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:02 smithi003 ceph-mon[104007]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 729 op/s; 6493/206271 objects degraded (3.148%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:39:03.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:02 smithi003 ceph-mon[107278]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 729 op/s; 6493/206271 objects degraded (3.148%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:39:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:39:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:39:04] "GET /metrics HTTP/1.1" 200 36425 "" "Prometheus/2.51.0" 2024-09-13T14:39:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:04 smithi019 ceph-mon[91296]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 6493/205449 objects degraded (3.160%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:39:05.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:04 smithi003 ceph-mon[104007]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 6493/205449 objects degraded (3.160%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:39:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:04 smithi003 ceph-mon[107278]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 6493/205449 objects degraded (3.160%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:39:06.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 6493/205449 objects degraded (3.160%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:06.333 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 6493/205449 objects degraded (3.160%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:06.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 6493/205449 objects degraded (3.160%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:07.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:06 smithi019 ceph-mon[91296]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 6255/193008 objects degraded (3.241%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:39:07.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:06 smithi003 ceph-mon[104007]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 6255/193008 objects degraded (3.241%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:39:07.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:06 smithi003 ceph-mon[107278]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 6255/193008 objects degraded (3.241%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:39:09.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:08 smithi019 ceph-mon[91296]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 6255/192243 objects degraded (3.254%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:39:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:08 smithi003 ceph-mon[107278]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 6255/192243 objects degraded (3.254%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:39:09.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:08 smithi003 ceph-mon[104007]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 6255/192243 objects degraded (3.254%); 2.4 MiB/s, 37 objects/s recovering 2024-09-13T14:39:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:39:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:39:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:11.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:10 smithi019 ceph-mon[91296]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 5978/179988 objects degraded (3.321%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:39:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5978/179988 objects degraded (3.321%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:10 smithi019 ceph-mon[91296]: osdmap e144: 8 total, 8 up, 8 in 2024-09-13T14:39:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:10 smithi003 ceph-mon[107278]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 5978/179988 objects degraded (3.321%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:39:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5978/179988 objects degraded (3.321%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:10 smithi003 ceph-mon[107278]: osdmap e144: 8 total, 8 up, 8 in 2024-09-13T14:39:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:10 smithi003 ceph-mon[104007]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 5978/179988 objects degraded (3.321%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:39:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5978/179988 objects degraded (3.321%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:10 smithi003 ceph-mon[104007]: osdmap e144: 8 total, 8 up, 8 in 2024-09-13T14:39:12.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:12 smithi019 ceph-mon[91296]: osdmap e145: 8 total, 8 up, 8 in 2024-09-13T14:39:12.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:12 smithi019 ceph-mon[91296]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 5978/179988 objects degraded (3.321%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:39:12.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:12 smithi003 ceph-mon[107278]: osdmap e145: 8 total, 8 up, 8 in 2024-09-13T14:39:12.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:12 smithi003 ceph-mon[107278]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 5978/179988 objects degraded (3.321%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:39:12.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:12 smithi003 ceph-mon[104007]: osdmap e145: 8 total, 8 up, 8 in 2024-09-13T14:39:12.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:12 smithi003 ceph-mon[104007]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 5978/179988 objects degraded (3.321%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:39:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:13.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:13.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:39:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:39:14] "GET /metrics HTTP/1.1" 200 36429 "" "Prometheus/2.51.0" 2024-09-13T14:39:14.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:14 smithi019 ceph-mon[91296]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 5978/179325 objects degraded (3.334%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:39:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:14 smithi003 ceph-mon[107278]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 5978/179325 objects degraded (3.334%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:39:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:14 smithi003 ceph-mon[104007]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 5978/179325 objects degraded (3.334%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:39:15.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:15 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5978/179325 objects degraded (3.334%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:15.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:15 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5978/179325 objects degraded (3.334%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:15.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:15 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5978/179325 objects degraded (3.334%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:16.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:16 smithi019 ceph-mon[91296]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 987 op/s; 5671/168525 objects degraded (3.365%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:39:16.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:16 smithi003 ceph-mon[104007]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 987 op/s; 5671/168525 objects degraded (3.365%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:39:16.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:16 smithi003 ceph-mon[107278]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 987 op/s; 5671/168525 objects degraded (3.365%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:39:18.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:18 smithi019 ceph-mon[91296]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 508 op/s; 5671/167778 objects degraded (3.380%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:39:18.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:18 smithi003 ceph-mon[107278]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 508 op/s; 5671/167778 objects degraded (3.380%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:39:18.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:18 smithi003 ceph-mon[104007]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 508 op/s; 5671/167778 objects degraded (3.380%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:39:20.573 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:39:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:39:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:20.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:20 smithi019 ceph-mon[91296]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 904 op/s; 5460/155547 objects degraded (3.510%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:39:20.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5460/155547 objects degraded (3.510%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:20.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:20 smithi003 ceph-mon[104007]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 904 op/s; 5460/155547 objects degraded (3.510%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:39:20.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5460/155547 objects degraded (3.510%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:20 smithi003 ceph-mon[107278]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 904 op/s; 5460/155547 objects degraded (3.510%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:39:20.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5460/155547 objects degraded (3.510%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:22.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:22 smithi019 ceph-mon[91296]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 5460/155547 objects degraded (3.510%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:39:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:22 smithi003 ceph-mon[107278]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 5460/155547 objects degraded (3.510%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:39:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:22 smithi003 ceph-mon[104007]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 5460/155547 objects degraded (3.510%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:39:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:39:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:39:24] "GET /metrics HTTP/1.1" 200 36429 "" "Prometheus/2.51.0" 2024-09-13T14:39:24.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:24 smithi019 ceph-mon[91296]: pgmap v428: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 5460/154701 objects degraded (3.529%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:39:24.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:24 smithi003 ceph-mon[104007]: pgmap v428: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 5460/154701 objects degraded (3.529%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:39:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:24 smithi003 ceph-mon[107278]: pgmap v428: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 5460/154701 objects degraded (3.529%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:39:25.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5460/154701 objects degraded (3.529%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:25.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5460/154701 objects degraded (3.529%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:25.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5460/154701 objects degraded (3.529%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:26 smithi019 ceph-mon[91296]: pgmap v429: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 5268/142158 objects degraded (3.706%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:39:26.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:26 smithi003 ceph-mon[107278]: pgmap v429: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 5268/142158 objects degraded (3.706%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:39:26.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:26 smithi003 ceph-mon[104007]: pgmap v429: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 5268/142158 objects degraded (3.706%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:39:28.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:28 smithi019 ceph-mon[91296]: pgmap v430: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 5268/141330 objects degraded (3.727%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:39:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:28 smithi003 ceph-mon[107278]: pgmap v430: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 5268/141330 objects degraded (3.727%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:39:28.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:28 smithi003 ceph-mon[104007]: pgmap v430: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 5268/141330 objects degraded (3.727%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:39:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:30.614 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:39:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:39:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:30 smithi019 ceph-mon[91296]: pgmap v431: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 5062/128874 objects degraded (3.928%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:39:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5062/128874 objects degraded (3.928%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:30.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:30 smithi003 ceph-mon[107278]: pgmap v431: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 5062/128874 objects degraded (3.928%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:39:30.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5062/128874 objects degraded (3.928%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:30 smithi003 ceph-mon[104007]: pgmap v431: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 5062/128874 objects degraded (3.928%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:39:30.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5062/128874 objects degraded (3.928%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:32 smithi019 ceph-mon[91296]: pgmap v432: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 5062/128874 objects degraded (3.928%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:39:32.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:32 smithi003 ceph-mon[107278]: pgmap v432: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 5062/128874 objects degraded (3.928%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:39:32.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:32 smithi003 ceph-mon[104007]: pgmap v432: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 5062/128874 objects degraded (3.928%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:39:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:39:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:39:34] "GET /metrics HTTP/1.1" 200 36427 "" "Prometheus/2.51.0" 2024-09-13T14:39:34.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:34 smithi019 ceph-mon[91296]: pgmap v433: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 5062/128097 objects degraded (3.952%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:39:34.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:34 smithi003 ceph-mon[107278]: pgmap v433: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 5062/128097 objects degraded (3.952%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:39:34.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:34 smithi003 ceph-mon[104007]: pgmap v433: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 5062/128097 objects degraded (3.952%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:39:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5062/128097 objects degraded (3.952%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:35.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5062/128097 objects degraded (3.952%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:35.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5062/128097 objects degraded (3.952%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:36.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:36 smithi003 ceph-mon[107278]: pgmap v434: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 4829/115737 objects degraded (4.172%); 2.9 MiB/s, 52 objects/s recovering 2024-09-13T14:39:36.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:36 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:36.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:36 smithi003 ceph-mon[104007]: pgmap v434: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 4829/115737 objects degraded (4.172%); 2.9 MiB/s, 52 objects/s recovering 2024-09-13T14:39:36.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:36 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:36 smithi019 ceph-mon[91296]: pgmap v434: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 4829/115737 objects degraded (4.172%); 2.9 MiB/s, 52 objects/s recovering 2024-09-13T14:39:36.937 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:36 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:38 smithi019 ceph-mon[91296]: pgmap v435: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 4829/114960 objects degraded (4.201%); 1.9 MiB/s, 36 objects/s recovering 2024-09-13T14:39:38.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:38 smithi003 ceph-mon[107278]: pgmap v435: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 4829/114960 objects degraded (4.201%); 1.9 MiB/s, 36 objects/s recovering 2024-09-13T14:39:38.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:38 smithi003 ceph-mon[104007]: pgmap v435: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 4829/114960 objects degraded (4.201%); 1.9 MiB/s, 36 objects/s recovering 2024-09-13T14:39:40.513 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:39:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:39:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:40.811 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:40 smithi019 ceph-mon[91296]: pgmap v436: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 4611/103041 objects degraded (4.475%); 1.9 MiB/s, 54 objects/s recovering 2024-09-13T14:39:40.811 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4611/103041 objects degraded (4.475%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:40.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:40 smithi003 ceph-mon[104007]: pgmap v436: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 4611/103041 objects degraded (4.475%); 1.9 MiB/s, 54 objects/s recovering 2024-09-13T14:39:40.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4611/103041 objects degraded (4.475%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:40 smithi003 ceph-mon[107278]: pgmap v436: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 4611/103041 objects degraded (4.475%); 1.9 MiB/s, 54 objects/s recovering 2024-09-13T14:39:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4611/103041 objects degraded (4.475%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:42.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:42 smithi019 ceph-mon[91296]: pgmap v437: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 4611/103041 objects degraded (4.475%); 880 KiB/s, 37 objects/s recovering 2024-09-13T14:39:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:42.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:42 smithi003 ceph-mon[104007]: pgmap v437: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 4611/103041 objects degraded (4.475%); 880 KiB/s, 37 objects/s recovering 2024-09-13T14:39:42.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:42.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:42.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:42 smithi003 ceph-mon[107278]: pgmap v437: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 4611/103041 objects degraded (4.475%); 880 KiB/s, 37 objects/s recovering 2024-09-13T14:39:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:43.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:43.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:43.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:43.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:43.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:43.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:43.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:43.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:43.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:43.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:43.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:43.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:43.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:43.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:39:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:39:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:39:44] "GET /metrics HTTP/1.1" 200 36418 "" "Prometheus/2.51.0" 2024-09-13T14:39:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:44 smithi019 ceph-mon[91296]: pgmap v438: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 4611/102192 objects degraded (4.512%); 880 KiB/s, 37 objects/s recovering 2024-09-13T14:39:44.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:44 smithi003 ceph-mon[104007]: pgmap v438: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 4611/102192 objects degraded (4.512%); 880 KiB/s, 37 objects/s recovering 2024-09-13T14:39:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:44 smithi003 ceph-mon[107278]: pgmap v438: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 4611/102192 objects degraded (4.512%); 880 KiB/s, 37 objects/s recovering 2024-09-13T14:39:45.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4611/102192 objects degraded (4.512%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:45.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4611/102192 objects degraded (4.512%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:45.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4611/102192 objects degraded (4.512%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:46.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:46 smithi003 ceph-mon[104007]: pgmap v439: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 4374/90663 objects degraded (4.824%); 880 KiB/s, 57 objects/s recovering 2024-09-13T14:39:46.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:46 smithi003 ceph-mon[107278]: pgmap v439: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 4374/90663 objects degraded (4.824%); 880 KiB/s, 57 objects/s recovering 2024-09-13T14:39:47.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:46 smithi019 ceph-mon[91296]: pgmap v439: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 4374/90663 objects degraded (4.824%); 880 KiB/s, 57 objects/s recovering 2024-09-13T14:39:49.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:48 smithi019 ceph-mon[91296]: pgmap v440: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 4374/89925 objects degraded (4.864%); 0 B/s, 37 objects/s recovering 2024-09-13T14:39:49.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:48 smithi003 ceph-mon[107278]: pgmap v440: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 4374/89925 objects degraded (4.864%); 0 B/s, 37 objects/s recovering 2024-09-13T14:39:49.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:48 smithi003 ceph-mon[104007]: pgmap v440: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 4374/89925 objects degraded (4.864%); 0 B/s, 37 objects/s recovering 2024-09-13T14:39:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:39:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:39:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:51.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:50 smithi019 ceph-mon[91296]: pgmap v441: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 4048/77748 objects degraded (5.207%); 0 B/s, 65 objects/s recovering 2024-09-13T14:39:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4048/77748 objects degraded (5.207%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:50 smithi003 ceph-mon[107278]: pgmap v441: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 4048/77748 objects degraded (5.207%); 0 B/s, 65 objects/s recovering 2024-09-13T14:39:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4048/77748 objects degraded (5.207%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:51.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:50 smithi003 ceph-mon[104007]: pgmap v441: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 4048/77748 objects degraded (5.207%); 0 B/s, 65 objects/s recovering 2024-09-13T14:39:51.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4048/77748 objects degraded (5.207%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:52.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:51 smithi019 ceph-mon[91296]: osdmap e146: 8 total, 8 up, 8 in 2024-09-13T14:39:52.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:51 smithi003 ceph-mon[107278]: osdmap e146: 8 total, 8 up, 8 in 2024-09-13T14:39:52.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:51 smithi003 ceph-mon[104007]: osdmap e146: 8 total, 8 up, 8 in 2024-09-13T14:39:53.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:52 smithi019 ceph-mon[91296]: pgmap v443: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 4048/77748 objects degraded (5.207%); 0 B/s, 56 objects/s recovering 2024-09-13T14:39:53.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:52 smithi019 ceph-mon[91296]: osdmap e147: 8 total, 8 up, 8 in 2024-09-13T14:39:53.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:52 smithi003 ceph-mon[107278]: pgmap v443: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 4048/77748 objects degraded (5.207%); 0 B/s, 56 objects/s recovering 2024-09-13T14:39:53.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:52 smithi003 ceph-mon[107278]: osdmap e147: 8 total, 8 up, 8 in 2024-09-13T14:39:53.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:52 smithi003 ceph-mon[104007]: pgmap v443: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 4048/77748 objects degraded (5.207%); 0 B/s, 56 objects/s recovering 2024-09-13T14:39:53.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:52 smithi003 ceph-mon[104007]: osdmap e147: 8 total, 8 up, 8 in 2024-09-13T14:39:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:39:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:39:54] "GET /metrics HTTP/1.1" 200 36418 "" "Prometheus/2.51.0" 2024-09-13T14:39:55.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:54 smithi019 ceph-mon[91296]: pgmap v445: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 567 op/s; 4048/77046 objects degraded (5.254%); 0 B/s, 40 objects/s recovering 2024-09-13T14:39:55.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:54 smithi003 ceph-mon[107278]: pgmap v445: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 567 op/s; 4048/77046 objects degraded (5.254%); 0 B/s, 40 objects/s recovering 2024-09-13T14:39:55.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:54 smithi003 ceph-mon[104007]: pgmap v445: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 567 op/s; 4048/77046 objects degraded (5.254%); 0 B/s, 40 objects/s recovering 2024-09-13T14:39:56.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4048/77046 objects degraded (5.254%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:56.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4048/77046 objects degraded (5.254%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:56.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4048/77046 objects degraded (5.254%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:56 smithi019 ceph-mon[91296]: pgmap v446: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 973 op/s; 3557/66549 objects degraded (5.345%); 0 B/s, 102 objects/s recovering 2024-09-13T14:39:57.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:56 smithi003 ceph-mon[107278]: pgmap v446: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 973 op/s; 3557/66549 objects degraded (5.345%); 0 B/s, 102 objects/s recovering 2024-09-13T14:39:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:56 smithi003 ceph-mon[104007]: pgmap v446: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 973 op/s; 3557/66549 objects degraded (5.345%); 0 B/s, 102 objects/s recovering 2024-09-13T14:39:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:58 smithi019 ceph-mon[91296]: pgmap v447: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 499 op/s; 3557/65748 objects degraded (5.410%); 0 B/s, 61 objects/s recovering 2024-09-13T14:39:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:39:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:58 smithi003 ceph-mon[104007]: pgmap v447: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 499 op/s; 3557/65748 objects degraded (5.410%); 0 B/s, 61 objects/s recovering 2024-09-13T14:39:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:39:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:58 smithi003 ceph-mon[107278]: pgmap v447: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 499 op/s; 3557/65748 objects degraded (5.410%); 0 B/s, 61 objects/s recovering 2024-09-13T14:39:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:39:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:40:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:40:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:01.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:00 smithi019 ceph-mon[91296]: pgmap v448: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 3324/53226 objects degraded (6.245%); 0 B/s, 81 objects/s recovering 2024-09-13T14:40:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:00 smithi019 ceph-mon[91296]: Health detail: HEALTH_WARN Degraded data redundancy: 3324/53226 objects degraded (6.245%), 2 pgs degraded, 2 pgs undersized 2024-09-13T14:40:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:00 smithi019 ceph-mon[91296]: [WRN] PG_DEGRADED: Degraded data redundancy: 3324/53226 objects degraded (6.245%), 2 pgs degraded, 2 pgs undersized 2024-09-13T14:40:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:00 smithi019 ceph-mon[91296]: pg 8.0 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [1,5] 2024-09-13T14:40:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:00 smithi019 ceph-mon[91296]: pg 8.f is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,2] 2024-09-13T14:40:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3324/53226 objects degraded (6.245%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:01.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[104007]: pgmap v448: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 3324/53226 objects degraded (6.245%); 0 B/s, 81 objects/s recovering 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[104007]: Health detail: HEALTH_WARN Degraded data redundancy: 3324/53226 objects degraded (6.245%), 2 pgs degraded, 2 pgs undersized 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[104007]: [WRN] PG_DEGRADED: Degraded data redundancy: 3324/53226 objects degraded (6.245%), 2 pgs degraded, 2 pgs undersized 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[104007]: pg 8.0 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [1,5] 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[104007]: pg 8.f is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,2] 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3324/53226 objects degraded (6.245%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[107278]: pgmap v448: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 3324/53226 objects degraded (6.245%); 0 B/s, 81 objects/s recovering 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[107278]: Health detail: HEALTH_WARN Degraded data redundancy: 3324/53226 objects degraded (6.245%), 2 pgs degraded, 2 pgs undersized 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[107278]: [WRN] PG_DEGRADED: Degraded data redundancy: 3324/53226 objects degraded (6.245%), 2 pgs degraded, 2 pgs undersized 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[107278]: pg 8.0 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [1,5] 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[107278]: pg 8.f is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,2] 2024-09-13T14:40:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3324/53226 objects degraded (6.245%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:02 smithi019 ceph-mon[91296]: pgmap v449: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 817 op/s; 3324/53226 objects degraded (6.245%); 0 B/s, 72 objects/s recovering 2024-09-13T14:40:03.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:02 smithi003 ceph-mon[104007]: pgmap v449: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 817 op/s; 3324/53226 objects degraded (6.245%); 0 B/s, 72 objects/s recovering 2024-09-13T14:40:03.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:02 smithi003 ceph-mon[107278]: pgmap v449: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 817 op/s; 3324/53226 objects degraded (6.245%); 0 B/s, 72 objects/s recovering 2024-09-13T14:40:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:40:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:40:04] "GET /metrics HTTP/1.1" 200 36406 "" "Prometheus/2.51.0" 2024-09-13T14:40:05.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:04 smithi019 ceph-mon[91296]: pgmap v450: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 3324/52362 objects degraded (6.348%); 0 B/s, 61 objects/s recovering 2024-09-13T14:40:05.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:04 smithi003 ceph-mon[104007]: pgmap v450: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 3324/52362 objects degraded (6.348%); 0 B/s, 61 objects/s recovering 2024-09-13T14:40:05.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:04 smithi003 ceph-mon[107278]: pgmap v450: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 3324/52362 objects degraded (6.348%); 0 B/s, 61 objects/s recovering 2024-09-13T14:40:06.083 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3324/52362 objects degraded (6.348%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:06.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3324/52362 objects degraded (6.348%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:06.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3324/52362 objects degraded (6.348%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:06.929 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:06 smithi003 ceph-mon[107278]: pgmap v451: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 829 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3086/39840 objects degraded (7.746%); 0 B/s, 80 objects/s recovering 2024-09-13T14:40:06.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:06 smithi003 ceph-mon[104007]: pgmap v451: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 829 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3086/39840 objects degraded (7.746%); 0 B/s, 80 objects/s recovering 2024-09-13T14:40:07.082 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:06 smithi019 ceph-mon[91296]: pgmap v451: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 829 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3086/39840 objects degraded (7.746%); 0 B/s, 80 objects/s recovering 2024-09-13T14:40:09.149 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:08 smithi019 ceph-mon[91296]: pgmap v452: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 814 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 3086/39102 objects degraded (7.892%); 0 B/s, 39 objects/s recovering 2024-09-13T14:40:09.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:08 smithi003 ceph-mon[104007]: pgmap v452: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 814 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 3086/39102 objects degraded (7.892%); 0 B/s, 39 objects/s recovering 2024-09-13T14:40:09.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:08 smithi003 ceph-mon[107278]: pgmap v452: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 814 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 3086/39102 objects degraded (7.892%); 0 B/s, 39 objects/s recovering 2024-09-13T14:40:10.536 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:40:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:40:10.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:10 smithi019 ceph-mon[91296]: pgmap v453: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 563 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 2851/27054 objects degraded (10.538%); 0 B/s, 58 objects/s recovering 2024-09-13T14:40:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2851/27054 objects degraded (10.538%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:11.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:10 smithi003 ceph-mon[104007]: pgmap v453: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 563 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 2851/27054 objects degraded (10.538%); 0 B/s, 58 objects/s recovering 2024-09-13T14:40:11.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2851/27054 objects degraded (10.538%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:10 smithi003 ceph-mon[107278]: pgmap v453: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 563 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 2851/27054 objects degraded (10.538%); 0 B/s, 58 objects/s recovering 2024-09-13T14:40:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2851/27054 objects degraded (10.538%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:13.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:12 smithi019 ceph-mon[91296]: pgmap v454: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 563 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 2851/27054 objects degraded (10.538%); 0 B/s, 39 objects/s recovering 2024-09-13T14:40:13.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:12 smithi003 ceph-mon[104007]: pgmap v454: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 563 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 2851/27054 objects degraded (10.538%); 0 B/s, 39 objects/s recovering 2024-09-13T14:40:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:12 smithi003 ceph-mon[107278]: pgmap v454: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 563 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 2851/27054 objects degraded (10.538%); 0 B/s, 39 objects/s recovering 2024-09-13T14:40:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:40:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:40:14] "GET /metrics HTTP/1.1" 200 36410 "" "Prometheus/2.51.0" 2024-09-13T14:40:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:14 smithi019 ceph-mon[91296]: pgmap v455: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 546 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 2851/26238 objects degraded (10.866%); 0 B/s, 39 objects/s recovering 2024-09-13T14:40:15.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:14 smithi003 ceph-mon[104007]: pgmap v455: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 546 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 2851/26238 objects degraded (10.866%); 0 B/s, 39 objects/s recovering 2024-09-13T14:40:15.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:14 smithi003 ceph-mon[107278]: pgmap v455: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 546 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 2851/26238 objects degraded (10.866%); 0 B/s, 39 objects/s recovering 2024-09-13T14:40:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:15 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2851/26238 objects degraded (10.866%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:16.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:15 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2851/26238 objects degraded (10.866%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:16.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:15 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2851/26238 objects degraded (10.866%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:16 smithi019 ceph-mon[91296]: pgmap v456: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 299 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2532/14403 objects degraded (17.580%); 0 B/s, 65 objects/s recovering 2024-09-13T14:40:17.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:16 smithi003 ceph-mon[104007]: pgmap v456: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 299 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2532/14403 objects degraded (17.580%); 0 B/s, 65 objects/s recovering 2024-09-13T14:40:17.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:16 smithi003 ceph-mon[107278]: pgmap v456: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 299 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2532/14403 objects degraded (17.580%); 0 B/s, 65 objects/s recovering 2024-09-13T14:40:19.112 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 124790 objects 2024-09-13T14:40:19.112 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :147.197 2024-09-13T14:40:19.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:18 smithi019 ceph-mon[91296]: pgmap v457: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 283 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 2532/13617 objects degraded (18.594%); 0 B/s, 46 objects/s recovering 2024-09-13T14:40:19.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:18 smithi003 ceph-mon[104007]: pgmap v457: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 283 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 2532/13617 objects degraded (18.594%); 0 B/s, 46 objects/s recovering 2024-09-13T14:40:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:18 smithi003 ceph-mon[107278]: pgmap v457: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 283 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 2532/13617 objects degraded (18.594%); 0 B/s, 46 objects/s recovering 2024-09-13T14:40:19.796 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-13T14:40:19.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-13T14:40:20.125 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:20.613 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:40:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:40:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:20.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:20 smithi019 ceph-mon[91296]: pgmap v458: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 23 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2244/1167 objects degraded (192.288%); 0 B/s, 70 objects/s recovering 2024-09-13T14:40:20.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:20 smithi019 ceph-mon[91296]: osdmap e148: 8 total, 8 up, 8 in 2024-09-13T14:40:20.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2244/1167 objects degraded (192.288%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:21.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:20 smithi003 ceph-mon[107278]: pgmap v458: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 23 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2244/1167 objects degraded (192.288%); 0 B/s, 70 objects/s recovering 2024-09-13T14:40:21.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:20 smithi003 ceph-mon[107278]: osdmap e148: 8 total, 8 up, 8 in 2024-09-13T14:40:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2244/1167 objects degraded (192.288%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:20 smithi003 ceph-mon[104007]: pgmap v458: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 23 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2244/1167 objects degraded (192.288%); 0 B/s, 70 objects/s recovering 2024-09-13T14:40:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:20 smithi003 ceph-mon[104007]: osdmap e148: 8 total, 8 up, 8 in 2024-09-13T14:40:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2244/1167 objects degraded (192.288%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:22.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:21 smithi019 ceph-mon[91296]: osdmap e149: 8 total, 8 up, 8 in 2024-09-13T14:40:22.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:21 smithi003 ceph-mon[107278]: osdmap e149: 8 total, 8 up, 8 in 2024-09-13T14:40:22.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:21 smithi003 ceph-mon[104007]: osdmap e149: 8 total, 8 up, 8 in 2024-09-13T14:40:22.825 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_3' removed 2024-09-13T14:40:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:22 smithi003 ceph-mon[107278]: pgmap v461: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 23 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2244/1167 objects degraded (192.288%); 0 B/s, 75 objects/s recovering 2024-09-13T14:40:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1446820347' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:40:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:22 smithi003 ceph-mon[104007]: pgmap v461: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 23 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2244/1167 objects degraded (192.288%); 0 B/s, 75 objects/s recovering 2024-09-13T14:40:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1446820347' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:40:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:22 smithi019 ceph-mon[91296]: pgmap v461: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 23 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2244/1167 objects degraded (192.288%); 0 B/s, 75 objects/s recovering 2024-09-13T14:40:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1446820347' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:40:23.612 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T14:40:23.612 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-13T14:40:23.612 DEBUG:teuthology.parallel:result is None 2024-09-13T14:40:23.613 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:40:23.624 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-09-13T14:40:23.624 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-13T14:40:23.675 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:40:23.676 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:40:23.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:40:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:24.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:23 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1446820347' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:40:24.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:23 smithi003 ceph-mon[107278]: osdmap e150: 8 total, 8 up, 8 in 2024-09-13T14:40:24.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:23 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1446820347' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:40:24.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:23 smithi003 ceph-mon[104007]: osdmap e150: 8 total, 8 up, 8 in 2024-09-13T14:40:24.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:23 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1446820347' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:40:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:23 smithi019 ceph-mon[91296]: osdmap e150: 8 total, 8 up, 8 in 2024-09-13T14:40:24.355 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:40:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:40:24] "GET /metrics HTTP/1.1" 200 36410 "" "Prometheus/2.51.0" 2024-09-13T14:40:25.042 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:24 smithi003 ceph-mon[107278]: pgmap v463: 73 pgs: 73 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:25.042 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:24 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2244/1167 objects degraded (192.288%), 2 pgs degraded, 2 pgs undersized) 2024-09-13T14:40:25.042 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:24 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:40:25.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:24 smithi003 ceph-mon[104007]: pgmap v463: 73 pgs: 73 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:25.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:24 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2244/1167 objects degraded (192.288%), 2 pgs degraded, 2 pgs undersized) 2024-09-13T14:40:25.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:24 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:40:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:24 smithi019 ceph-mon[91296]: pgmap v463: 73 pgs: 73 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:24 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2244/1167 objects degraded (192.288%), 2 pgs degraded, 2 pgs undersized) 2024-09-13T14:40:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:24 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:40:25.749 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:40:26.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:25 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/865344470' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:40:26.065 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:25 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/865344470' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:40:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:25 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/865344470' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:40:26.469 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":39,"flags":0,"active_gid":34107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":202288793},{"type":"v1","addr":"172.21.15.3:6801","nonce":202288793}]},"active_addr":"172.21.15.3:6801/202288793","active_change":"2024-09-13T14:27:27.242282+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":25240,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":78,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":4289745153}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2388840808}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":649893210}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":1730889541}]}]} 2024-09-13T14:40:26.474 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:40:26.474 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:40:26.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:40:26.741 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:27.110 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:26 smithi003 ceph-mon[107278]: pgmap v464: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:27.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:26 smithi003 ceph-mon[104007]: pgmap v464: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:27.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:26 smithi019 ceph-mon[91296]: pgmap v464: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:28 smithi019 ceph-mon[91296]: pgmap v465: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:28 smithi003 ceph-mon[107278]: pgmap v465: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:28 smithi003 ceph-mon[104007]: pgmap v465: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:40:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:40:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:30.979 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:40:30.980 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:30 smithi003 ceph-mon[104007]: pgmap v466: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:30.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:30 smithi003 ceph-mon[107278]: pgmap v466: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:30.980 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":150,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:15:00.204452+0000","modified":"2024-09-13T14:40:22.823465+0000","last_up_change":"2024-09-13T14:37:07.731579+0000","last_in_change":"2024-09-13T14:19:42.900010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:18:02.574675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:20:35.870236+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:21:51.432728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:21:52.047402+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a153ec29-6e01-4063-aa90-01d1de00e53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":79,"up_from":83,"up_thru":142,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6803","nonce":2281086808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6805","nonce":2281086808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6809","nonce":2281086808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6807","nonce":2281086808}]},"public_addr":"172.21.15.3:6803/2281086808","cluster_addr":"172.21.15.3:6805/2281086808","heartbeat_back_addr":"172.21.15.3:6809/2281086808","heartbeat_front_addr":"172.21.15.3:6807/2281086808","state":["exists","up"]},{"osd":1,"uuid":"9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":93,"up_from":97,"up_thru":147,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6811","nonce":3208418243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6813","nonce":3208418243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6817","nonce":3208418243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6815","nonce":3208418243}]},"public_addr":"172.21.15.3:6811/3208418243","cluster_addr":"172.21.15.3:6813/3208418243","heartbeat_back_addr":"172.21.15.3:6817/3208418243","heartbeat_front_addr":"172.21.15.3:6815/3208418243","state":["exists","up"]},{"osd":2,"uuid":"676ec1e5-071c-47ba-86ae-2193f448fe8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":107,"up_from":113,"up_thru":137,"down_at":108,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6819","nonce":1260576691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6821","nonce":1260576691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6825","nonce":1260576691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6823","nonce":1260576691}]},"public_addr":"172.21.15.3:6819/1260576691","cluster_addr":"172.21.15.3:6821/1260576691","heartbeat_back_addr":"172.21.15.3:6825/1260576691","heartbeat_front_addr":"172.21.15.3:6823/1260576691","state":["exists","up"]},{"osd":3,"uuid":"7ec0ad16-0df3-4319-b475-72cfaaa5edae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":133,"up_from":137,"up_thru":148,"down_at":134,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6827","nonce":1196691419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6829","nonce":1196691419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6833","nonce":1196691419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6831","nonce":1196691419}]},"public_addr":"172.21.15.3:6827/1196691419","cluster_addr":"172.21.15.3:6829/1196691419","heartbeat_back_addr":"172.21.15.3:6833/1196691419","heartbeat_front_addr":"172.21.15.3:6831/1196691419","state":["exists","up"]},{"osd":4,"uuid":"0ef54d09-4000-4695-942b-1b1cf9659983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6801","nonce":4204660131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6803","nonce":4204660131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6807","nonce":4204660131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6805","nonce":4204660131}]},"public_addr":"172.21.15.19:6801/4204660131","cluster_addr":"172.21.15.19:6803/4204660131","heartbeat_back_addr":"172.21.15.19:6807/4204660131","heartbeat_front_addr":"172.21.15.19:6805/4204660131","state":["exists","up"]},{"osd":5,"uuid":"6dfd8fce-4d3d-4a8b-90ac-556447045378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6809","nonce":3591244104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6811","nonce":3591244104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6815","nonce":3591244104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6813","nonce":3591244104}]},"public_addr":"172.21.15.19:6809/3591244104","cluster_addr":"172.21.15.19:6811/3591244104","heartbeat_back_addr":"172.21.15.19:6815/3591244104","heartbeat_front_addr":"172.21.15.19:6813/3591244104","state":["exists","up"]},{"osd":6,"uuid":"17e27318-c0ce-413d-b660-d756731ea743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6817","nonce":2162145215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6819","nonce":2162145215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6823","nonce":2162145215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6821","nonce":2162145215}]},"public_addr":"172.21.15.19:6817/2162145215","cluster_addr":"172.21.15.19:6819/2162145215","heartbeat_back_addr":"172.21.15.19:6823/2162145215","heartbeat_front_addr":"172.21.15.19:6821/2162145215","state":["exists","up"]},{"osd":7,"uuid":"1edab2d1-dad5-45c1-977c-6911541c282e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6825","nonce":2814206175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6827","nonce":2814206175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6831","nonce":2814206175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6829","nonce":2814206175}]},"public_addr":"172.21.15.19:6825/2814206175","cluster_addr":"172.21.15.19:6827/2814206175","heartbeat_back_addr":"172.21.15.19:6831/2814206175","heartbeat_front_addr":"172.21.15.19:6829/2814206175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:11.323986+0000","dead_epoch":79},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:34.429745+0000","dead_epoch":93},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:59.523592+0000","dead_epoch":107},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:24.423713+0000","dead_epoch":133},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:46.452261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:10.483064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:34.315912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:58.203878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6800/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/2886196962":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/3631040762":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2497820386":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/1688297948":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/1330279387":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1118901319":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/2143031132":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/2723658197":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/3309089360":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6801/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/416475212":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6800/3309089360":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6801/70918773":"2024-09-14T14:24:43.914607+0000","172.21.15.19:0/3762108460":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6800/70918773":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/1018276975":"2024-09-14T14:20:33.329796+0000","172.21.15.19:6834/2089501540":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/253916983":"2024-09-14T14:15:52.081305+0000","172.21.15.19:6835/2089501540":"2024-09-14T14:26:31.587235+0000","172.21.15.19:0/255629335":"2024-09-14T14:26:31.587235+0000","172.21.15.19:0/211301439":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/1087248546":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1863609487":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/2893689034":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/1361669764":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/738145194":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/3529413292":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/3505363388":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2346548909":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/3615653824":"2024-09-14T14:20:33.329796+0000","172.21.15.19:0/1270823610":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6800/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6800/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6801/931806668":"2024-09-14T14:15:28.360446+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:40:31.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:30 smithi019 ceph-mon[91296]: pgmap v466: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:31.565 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:40:31.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:40:31.825 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:31.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1789947184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:40:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1789947184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:40:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1789947184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:40:33.031 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:32 smithi003 ceph-mon[104007]: pgmap v467: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:33.031 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:32 smithi003 ceph-mon[107278]: pgmap v467: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:33.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:32 smithi019 ceph-mon[91296]: pgmap v467: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:40:33.757 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":150,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:15:00.204452+0000","modified":"2024-09-13T14:40:22.823465+0000","last_up_change":"2024-09-13T14:37:07.731579+0000","last_in_change":"2024-09-13T14:19:42.900010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:18:02.574675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:20:35.870236+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:21:51.432728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:21:52.047402+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a153ec29-6e01-4063-aa90-01d1de00e53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":79,"up_from":83,"up_thru":142,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6803","nonce":2281086808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6805","nonce":2281086808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6809","nonce":2281086808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6807","nonce":2281086808}]},"public_addr":"172.21.15.3:6803/2281086808","cluster_addr":"172.21.15.3:6805/2281086808","heartbeat_back_addr":"172.21.15.3:6809/2281086808","heartbeat_front_addr":"172.21.15.3:6807/2281086808","state":["exists","up"]},{"osd":1,"uuid":"9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":93,"up_from":97,"up_thru":147,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6811","nonce":3208418243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6813","nonce":3208418243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6817","nonce":3208418243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6815","nonce":3208418243}]},"public_addr":"172.21.15.3:6811/3208418243","cluster_addr":"172.21.15.3:6813/3208418243","heartbeat_back_addr":"172.21.15.3:6817/3208418243","heartbeat_front_addr":"172.21.15.3:6815/3208418243","state":["exists","up"]},{"osd":2,"uuid":"676ec1e5-071c-47ba-86ae-2193f448fe8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":107,"up_from":113,"up_thru":137,"down_at":108,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6819","nonce":1260576691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6821","nonce":1260576691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6825","nonce":1260576691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6823","nonce":1260576691}]},"public_addr":"172.21.15.3:6819/1260576691","cluster_addr":"172.21.15.3:6821/1260576691","heartbeat_back_addr":"172.21.15.3:6825/1260576691","heartbeat_front_addr":"172.21.15.3:6823/1260576691","state":["exists","up"]},{"osd":3,"uuid":"7ec0ad16-0df3-4319-b475-72cfaaa5edae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":133,"up_from":137,"up_thru":148,"down_at":134,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6827","nonce":1196691419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6829","nonce":1196691419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6833","nonce":1196691419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6831","nonce":1196691419}]},"public_addr":"172.21.15.3:6827/1196691419","cluster_addr":"172.21.15.3:6829/1196691419","heartbeat_back_addr":"172.21.15.3:6833/1196691419","heartbeat_front_addr":"172.21.15.3:6831/1196691419","state":["exists","up"]},{"osd":4,"uuid":"0ef54d09-4000-4695-942b-1b1cf9659983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6801","nonce":4204660131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6803","nonce":4204660131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6807","nonce":4204660131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6805","nonce":4204660131}]},"public_addr":"172.21.15.19:6801/4204660131","cluster_addr":"172.21.15.19:6803/4204660131","heartbeat_back_addr":"172.21.15.19:6807/4204660131","heartbeat_front_addr":"172.21.15.19:6805/4204660131","state":["exists","up"]},{"osd":5,"uuid":"6dfd8fce-4d3d-4a8b-90ac-556447045378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6809","nonce":3591244104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6811","nonce":3591244104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6815","nonce":3591244104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6813","nonce":3591244104}]},"public_addr":"172.21.15.19:6809/3591244104","cluster_addr":"172.21.15.19:6811/3591244104","heartbeat_back_addr":"172.21.15.19:6815/3591244104","heartbeat_front_addr":"172.21.15.19:6813/3591244104","state":["exists","up"]},{"osd":6,"uuid":"17e27318-c0ce-413d-b660-d756731ea743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6817","nonce":2162145215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6819","nonce":2162145215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6823","nonce":2162145215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6821","nonce":2162145215}]},"public_addr":"172.21.15.19:6817/2162145215","cluster_addr":"172.21.15.19:6819/2162145215","heartbeat_back_addr":"172.21.15.19:6823/2162145215","heartbeat_front_addr":"172.21.15.19:6821/2162145215","state":["exists","up"]},{"osd":7,"uuid":"1edab2d1-dad5-45c1-977c-6911541c282e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6825","nonce":2814206175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6827","nonce":2814206175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6831","nonce":2814206175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6829","nonce":2814206175}]},"public_addr":"172.21.15.19:6825/2814206175","cluster_addr":"172.21.15.19:6827/2814206175","heartbeat_back_addr":"172.21.15.19:6831/2814206175","heartbeat_front_addr":"172.21.15.19:6829/2814206175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:11.323986+0000","dead_epoch":79},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:34.429745+0000","dead_epoch":93},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:59.523592+0000","dead_epoch":107},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:24.423713+0000","dead_epoch":133},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:46.452261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:10.483064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:34.315912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:58.203878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6800/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/2886196962":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/3631040762":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/2497820386":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/1688297948":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/1330279387":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1118901319":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/2143031132":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/2723658197":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/3309089360":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6801/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/416475212":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6800/3309089360":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6801/70918773":"2024-09-14T14:24:43.914607+0000","172.21.15.19:0/3762108460":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6800/70918773":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/1018276975":"2024-09-14T14:20:33.329796+0000","172.21.15.19:6834/2089501540":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/253916983":"2024-09-14T14:15:52.081305+0000","172.21.15.19:6835/2089501540":"2024-09-14T14:26:31.587235+0000","172.21.15.19:0/255629335":"2024-09-14T14:26:31.587235+0000","172.21.15.19:0/211301439":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/1087248546":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1863609487":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/2893689034":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/1361669764":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/738145194":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/3529413292":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/3505363388":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2346548909":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/3615653824":"2024-09-14T14:20:33.329796+0000","172.21.15.19:0/1270823610":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6800/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6800/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6801/931806668":"2024-09-14T14:15:28.360446+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:40:33.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:33 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1335788149' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:40:33.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:33 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1335788149' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:40:34.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:33 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1335788149' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:40:34.344 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:40:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:40:34] "GET /metrics HTTP/1.1" 200 34047 "" "Prometheus/2.51.0" 2024-09-13T14:40:34.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:40:34.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:40:34.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:40:34.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:40:34.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:40:34.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:40:34.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:40:34.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:40:34.923 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:34.923 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:34.925 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:34.926 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:34.926 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:34.933 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:34.934 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:35.070 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:34 smithi003 ceph-mon[107278]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:35.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:34 smithi003 ceph-mon[104007]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:35.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:34 smithi019 ceph-mon[91296]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:36.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:36 smithi003 ceph-mon[104007]: pgmap v469: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:36.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:36 smithi003 ceph-mon[107278]: pgmap v469: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:37.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:36 smithi019 ceph-mon[91296]: pgmap v469: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:39.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:38 smithi003 ceph-mon[107278]: pgmap v470: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:39.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:38 smithi003 ceph-mon[104007]: pgmap v470: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:39.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:38 smithi019 ceph-mon[91296]: pgmap v470: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:40.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:40:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:40:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:40.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:40 smithi003 ceph-mon[107278]: pgmap v471: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:40 smithi003 ceph-mon[104007]: pgmap v471: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:41.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:40 smithi019 ceph-mon[91296]: pgmap v471: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:43.100 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:42 smithi003 ceph-mon[107278]: pgmap v472: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:43.100 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:43.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:42 smithi003 ceph-mon[104007]: pgmap v472: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:43.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:43.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:42 smithi019 ceph-mon[91296]: pgmap v472: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:43.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:43.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:44.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:44.284 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:40:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:40:44] "GET /metrics HTTP/1.1" 200 34046 "" "Prometheus/2.51.0" 2024-09-13T14:40:44.577 INFO:teuthology.orchestra.run.smithi003.stdout:485331304513 2024-09-13T14:40:44.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:40:45.073 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:44 smithi003 ceph-mon[104007]: pgmap v473: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:45.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:44 smithi003 ceph-mon[107278]: pgmap v473: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:45.137 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:44 smithi019 ceph-mon[91296]: pgmap v473: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:45.789 INFO:teuthology.orchestra.run.smithi003.stdout:416611827817 2024-09-13T14:40:45.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:40:46.900 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:46 smithi003 ceph-mon[107278]: pgmap v474: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:46.901 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:46 smithi003 ceph-mon[104007]: pgmap v474: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:46 smithi019 ceph-mon[91296]: pgmap v474: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:47.292 INFO:teuthology.orchestra.run.smithi003.stdout:188978561281 2024-09-13T14:40:47.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:40:47.684 INFO:teuthology.orchestra.run.smithi003.stdout:214748365052 2024-09-13T14:40:47.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:40:49.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:49 smithi003 ceph-mon[107278]: pgmap v475: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:49.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:49 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:49.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:49 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:49.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:49 smithi003 ceph-mon[104007]: pgmap v475: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:49.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:49 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:49.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:49 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:49.424 INFO:teuthology.orchestra.run.smithi003.stdout:588410519597 2024-09-13T14:40:49.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:40:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:49 smithi019 ceph-mon[91296]: pgmap v475: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:49 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:49 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:40:50.139 INFO:teuthology.orchestra.run.smithi003.stdout:133143986442 2024-09-13T14:40:50.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:40:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:40:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:40:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:51 smithi003 ceph-mon[107278]: pgmap v476: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:51.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:51 smithi003 ceph-mon[104007]: pgmap v476: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:51.302 INFO:teuthology.orchestra.run.smithi003.stdout:158913790215 2024-09-13T14:40:51.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:40:51.348 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:51.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:51 smithi019 ceph-mon[91296]: pgmap v476: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:52.293 INFO:teuthology.orchestra.run.smithi003.stdout:356482285682 2024-09-13T14:40:52.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:40:52.316 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:52.316 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:52.331 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:53.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:53 smithi003 ceph-mon[107278]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:53.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:53 smithi003 ceph-mon[104007]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:53 smithi019 ceph-mon[91296]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:54.286 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:40:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:40:54] "GET /metrics HTTP/1.1" 200 34046 "" "Prometheus/2.51.0" 2024-09-13T14:40:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:55 smithi019 ceph-mon[91296]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:55.182 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:40:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=sqlstore.transactions t=2024-09-13T14:40:54.844418589Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:40:55.422 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:55 smithi003 ceph-mon[107278]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:55.422 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:55 smithi003 ceph-mon[104007]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:57.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:57 smithi003 ceph-mon[104007]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:57.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:57 smithi003 ceph-mon[107278]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:57 smithi019 ceph-mon[91296]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:57.484 INFO:teuthology.orchestra.run.smithi003.stdout:188978561284 2024-09-13T14:40:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:58.323 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:58 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2978765659' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:40:58.323 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:58.324 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:58 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2978765659' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:40:58.324 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:58.411 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:58.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:58 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2978765659' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:40:58.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:59.304 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561281 got 188978561284 for osd.6 2024-09-13T14:40:59.304 DEBUG:teuthology.parallel:result is None 2024-09-13T14:40:59.371 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:40:59 smithi003 ceph-mon[104007]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:59.371 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:40:59 smithi003 ceph-mon[107278]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:40:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:40:59 smithi019 ceph-mon[91296]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:41:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:41:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:00.959 INFO:teuthology.orchestra.run.smithi003.stdout:416611827820 2024-09-13T14:41:01.171 INFO:teuthology.orchestra.run.smithi003.stdout:588410519600 2024-09-13T14:41:01.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:01 smithi003 ceph-mon[107278]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:01.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:01 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4167291842' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:41:01.172 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:01 smithi003 ceph-mon[104007]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:01.172 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:01 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4167291842' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:41:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:01 smithi019 ceph-mon[91296]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:01 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4167291842' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:41:01.780 INFO:tasks.cephadm.ceph_manager.ceph:need seq 588410519597 got 588410519600 for osd.3 2024-09-13T14:41:01.781 DEBUG:teuthology.parallel:result is None 2024-09-13T14:41:02.196 INFO:teuthology.orchestra.run.smithi003.stdout:485331304516 2024-09-13T14:41:02.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:02 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2238435240' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:41:02.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:02 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2238435240' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:41:02.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:02 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2238435240' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:41:03.005 INFO:tasks.cephadm.ceph_manager.ceph:need seq 485331304513 got 485331304516 for osd.2 2024-09-13T14:41:03.005 DEBUG:teuthology.parallel:result is None 2024-09-13T14:41:03.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:03 smithi003 ceph-mon[107278]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:03.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:03 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1551354243' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:41:03.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:03 smithi003 ceph-mon[104007]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:03.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:03 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1551354243' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:41:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:03 smithi019 ceph-mon[91296]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:03 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1551354243' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:41:03.984 INFO:tasks.cephadm.ceph_manager.ceph:need seq 416611827817 got 416611827820 for osd.1 2024-09-13T14:41:03.984 DEBUG:teuthology.parallel:result is None 2024-09-13T14:41:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:41:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:41:04] "GET /metrics HTTP/1.1" 200 34046 "" "Prometheus/2.51.0" 2024-09-13T14:41:04.816 INFO:teuthology.orchestra.run.smithi003.stdout:214748365055 2024-09-13T14:41:05.340 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:05 smithi003 ceph-mon[107278]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:05.340 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/961263170' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:41:05.340 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:05 smithi003 ceph-mon[104007]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:05.340 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/961263170' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:41:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:05 smithi019 ceph-mon[91296]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/961263170' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:41:05.964 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365052 got 214748365055 for osd.7 2024-09-13T14:41:05.964 DEBUG:teuthology.parallel:result is None 2024-09-13T14:41:07.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:07 smithi003 ceph-mon[107278]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:07.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:07 smithi003 ceph-mon[104007]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:07 smithi019 ceph-mon[91296]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:07.622 INFO:teuthology.orchestra.run.smithi003.stdout:133143986446 2024-09-13T14:41:07.824 INFO:teuthology.orchestra.run.smithi003.stdout:158913790218 2024-09-13T14:41:08.355 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:08 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2342802997' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:41:08.355 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:08 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3494986084' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:41:08.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:08 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2342802997' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:41:08.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:08 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3494986084' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:41:08.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:08 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2342802997' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:41:08.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:08 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3494986084' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:41:08.476 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790215 got 158913790218 for osd.5 2024-09-13T14:41:08.476 DEBUG:teuthology.parallel:result is None 2024-09-13T14:41:08.847 INFO:teuthology.orchestra.run.smithi003.stdout:356482285685 2024-09-13T14:41:09.083 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986442 got 133143986446 for osd.4 2024-09-13T14:41:09.083 DEBUG:teuthology.parallel:result is None 2024-09-13T14:41:09.394 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:09 smithi003 ceph-mon[107278]: pgmap v485: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:09.395 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3776858162' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:41:09.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:09 smithi003 ceph-mon[104007]: pgmap v485: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:09.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3776858162' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:41:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:09 smithi019 ceph-mon[91296]: pgmap v485: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3776858162' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:41:09.780 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285682 got 356482285685 for osd.0 2024-09-13T14:41:09.780 DEBUG:teuthology.parallel:result is None 2024-09-13T14:41:09.780 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:41:09.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:10.664 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:41:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:41:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:11 smithi019 ceph-mon[91296]: pgmap v486: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:11 smithi003 ceph-mon[107278]: pgmap v486: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:11 smithi003 ceph-mon[104007]: pgmap v486: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:41:12.832 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:41:13.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:13 smithi019 ceph-mon[91296]: pgmap v487: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:13.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:13 smithi003 ceph-mon[104007]: pgmap v487: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:13.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:13 smithi003 ceph-mon[107278]: pgmap v487: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:13.544 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-13T14:41:11.561830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":213,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3336920,"kb_used_data":15268,"kb_used_omap":29,"kb_used_meta":3321506,"kb_avail":746624296,"statfs":{"total":767960285184,"available":764543279104,"internally_reserved":0,"allocated":15634432,"data_stored":9787760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":3401222670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002642"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.972038+0000","last_change":"2024-09-13T14:37:08.782889+0000","last_active":"2024-09-13T14:40:22.972038+0000","last_peered":"2024-09-13T14:40:22.972038+0000","last_clean":"2024-09-13T14:40:22.972038+0000","last_became_active":"2024-09-13T14:37:08.781502+0000","last_became_peered":"2024-09-13T14:37:08.781502+0000","last_unstale":"2024-09-13T14:40:22.972038+0000","last_undegraded":"2024-09-13T14:40:22.972038+0000","last_fullsized":"2024-09-13T14:40:22.972038+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:22.769013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":116,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330969+0000","last_change":"2024-09-13T14:35:30.492165+0000","last_active":"2024-09-13T14:40:21.330969+0000","last_peered":"2024-09-13T14:40:21.330969+0000","last_clean":"2024-09-13T14:40:21.330969+0000","last_became_active":"2024-09-13T14:35:30.491844+0000","last_became_peered":"2024-09-13T14:35:30.491844+0000","last_unstale":"2024-09-13T14:40:21.330969+0000","last_undegraded":"2024-09-13T14:40:21.330969+0000","last_fullsized":"2024-09-13T14:40:21.330969+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:22.876890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":111,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822194+0000","last_change":"2024-09-13T14:37:08.755784+0000","last_active":"2024-09-13T14:40:20.822194+0000","last_peered":"2024-09-13T14:40:20.822194+0000","last_clean":"2024-09-13T14:40:20.822194+0000","last_became_active":"2024-09-13T14:37:08.755300+0000","last_became_peered":"2024-09-13T14:37:08.755300+0000","last_unstale":"2024-09-13T14:40:20.822194+0000","last_undegraded":"2024-09-13T14:40:20.822194+0000","last_fullsized":"2024-09-13T14:40:20.822194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:10.881654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827254+0000","last_change":"2024-09-13T14:37:08.748084+0000","last_active":"2024-09-13T14:40:22.827254+0000","last_peered":"2024-09-13T14:40:22.827254+0000","last_clean":"2024-09-13T14:40:22.827254+0000","last_became_active":"2024-09-13T14:37:08.746801+0000","last_became_peered":"2024-09-13T14:37:08.746801+0000","last_unstale":"2024-09-13T14:40:22.827254+0000","last_undegraded":"2024-09-13T14:40:22.827254+0000","last_fullsized":"2024-09-13T14:40:22.827254+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:13.495421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833418+0000","last_change":"2024-09-13T14:37:08.750965+0000","last_active":"2024-09-13T14:40:22.833418+0000","last_peered":"2024-09-13T14:40:22.833418+0000","last_clean":"2024-09-13T14:40:22.833418+0000","last_became_active":"2024-09-13T14:37:08.750129+0000","last_became_peered":"2024-09-13T14:37:08.750129+0000","last_unstale":"2024-09-13T14:40:22.833418+0000","last_undegraded":"2024-09-13T14:40:22.833418+0000","last_fullsized":"2024-09-13T14:40:22.833418+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:48.359985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":97,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330708+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:40:21.330708+0000","last_peered":"2024-09-13T14:40:21.330708+0000","last_clean":"2024-09-13T14:40:21.330708+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:40:21.330708+0000","last_undegraded":"2024-09-13T14:40:21.330708+0000","last_fullsized":"2024-09-13T14:40:21.330708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971111+0000","last_change":"2024-09-13T14:37:08.755312+0000","last_active":"2024-09-13T14:40:22.971111+0000","last_peered":"2024-09-13T14:40:22.971111+0000","last_clean":"2024-09-13T14:40:22.971111+0000","last_became_active":"2024-09-13T14:37:08.753186+0000","last_became_peered":"2024-09-13T14:37:08.753186+0000","last_unstale":"2024-09-13T14:40:22.971111+0000","last_undegraded":"2024-09-13T14:40:22.971111+0000","last_fullsized":"2024-09-13T14:40:22.971111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:57.687749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":121,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576521+0000","last_change":"2024-09-13T14:31:29.946716+0000","last_active":"2024-09-13T14:40:21.576521+0000","last_peered":"2024-09-13T14:40:21.576521+0000","last_clean":"2024-09-13T14:40:21.576521+0000","last_became_active":"2024-09-13T14:31:29.946366+0000","last_became_peered":"2024-09-13T14:31:29.946366+0000","last_unstale":"2024-09-13T14:40:21.576521+0000","last_undegraded":"2024-09-13T14:40:21.576521+0000","last_fullsized":"2024-09-13T14:40:21.576521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:29.686747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":115,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331328+0000","last_change":"2024-09-13T14:31:29.950983+0000","last_active":"2024-09-13T14:40:21.331328+0000","last_peered":"2024-09-13T14:40:21.331328+0000","last_clean":"2024-09-13T14:40:21.331328+0000","last_became_active":"2024-09-13T14:31:29.950612+0000","last_became_peered":"2024-09-13T14:31:29.950612+0000","last_unstale":"2024-09-13T14:40:21.331328+0000","last_undegraded":"2024-09-13T14:40:21.331328+0000","last_fullsized":"2024-09-13T14:40:21.331328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:35.833747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":94,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822196+0000","last_change":"2024-09-13T14:21:52.983025+0000","last_active":"2024-09-13T14:40:20.822196+0000","last_peered":"2024-09-13T14:40:20.822196+0000","last_clean":"2024-09-13T14:40:20.822196+0000","last_became_active":"2024-09-13T14:21:52.982611+0000","last_became_peered":"2024-09-13T14:21:52.982611+0000","last_unstale":"2024-09-13T14:40:20.822196+0000","last_undegraded":"2024-09-13T14:40:20.822196+0000","last_fullsized":"2024-09-13T14:40:20.822196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:00.851531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":116,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331353+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:40:21.331353+0000","last_peered":"2024-09-13T14:40:21.331353+0000","last_clean":"2024-09-13T14:40:21.331353+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:40:21.331353+0000","last_undegraded":"2024-09-13T14:40:21.331353+0000","last_fullsized":"2024-09-13T14:40:21.331353+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576188+0000","last_change":"2024-09-13T14:37:08.752050+0000","last_active":"2024-09-13T14:40:21.576188+0000","last_peered":"2024-09-13T14:40:21.576188+0000","last_clean":"2024-09-13T14:40:21.576188+0000","last_became_active":"2024-09-13T14:37:08.751650+0000","last_became_peered":"2024-09-13T14:37:08.751650+0000","last_unstale":"2024-09-13T14:40:21.576188+0000","last_undegraded":"2024-09-13T14:40:21.576188+0000","last_fullsized":"2024-09-13T14:40:21.576188+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:17.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":98,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331346+0000","last_change":"2024-09-13T14:21:52.610490+0000","last_active":"2024-09-13T14:40:21.331346+0000","last_peered":"2024-09-13T14:40:21.331346+0000","last_clean":"2024-09-13T14:40:21.331346+0000","last_became_active":"2024-09-13T14:21:52.610192+0000","last_became_peered":"2024-09-13T14:21:52.610192+0000","last_unstale":"2024-09-13T14:40:21.331346+0000","last_undegraded":"2024-09-13T14:40:21.331346+0000","last_fullsized":"2024-09-13T14:40:21.331346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:09.564393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":105,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576705+0000","last_change":"2024-09-13T14:22:34.901360+0000","last_active":"2024-09-13T14:40:21.576705+0000","last_peered":"2024-09-13T14:40:21.576705+0000","last_clean":"2024-09-13T14:40:21.576705+0000","last_became_active":"2024-09-13T14:22:34.901010+0000","last_became_peered":"2024-09-13T14:22:34.901010+0000","last_unstale":"2024-09-13T14:40:21.576705+0000","last_undegraded":"2024-09-13T14:40:21.576705+0000","last_fullsized":"2024-09-13T14:40:21.576705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:55.897004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822246+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:40:20.822246+0000","last_peered":"2024-09-13T14:40:20.822246+0000","last_clean":"2024-09-13T14:40:20.822246+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:40:20.822246+0000","last_undegraded":"2024-09-13T14:40:20.822246+0000","last_fullsized":"2024-09-13T14:40:20.822246+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827184+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:40:22.827184+0000","last_peered":"2024-09-13T14:40:22.827184+0000","last_clean":"2024-09-13T14:40:22.827184+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:40:22.827184+0000","last_undegraded":"2024-09-13T14:40:22.827184+0000","last_fullsized":"2024-09-13T14:40:22.827184+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970902+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:40:22.970902+0000","last_peered":"2024-09-13T14:40:22.970902+0000","last_clean":"2024-09-13T14:40:22.970902+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:40:22.970902+0000","last_undegraded":"2024-09-13T14:40:22.970902+0000","last_fullsized":"2024-09-13T14:40:22.970902+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576160+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:40:21.576160+0000","last_peered":"2024-09-13T14:40:21.576160+0000","last_clean":"2024-09-13T14:40:21.576160+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:40:21.576160+0000","last_undegraded":"2024-09-13T14:40:21.576160+0000","last_fullsized":"2024-09-13T14:40:21.576160+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576852+0000","last_change":"2024-09-13T14:32:09.491728+0000","last_active":"2024-09-13T14:40:21.576852+0000","last_peered":"2024-09-13T14:40:21.576852+0000","last_clean":"2024-09-13T14:40:21.576852+0000","last_became_active":"2024-09-13T14:32:09.490612+0000","last_became_peered":"2024-09-13T14:32:09.490612+0000","last_unstale":"2024-09-13T14:40:21.576852+0000","last_undegraded":"2024-09-13T14:40:21.576852+0000","last_fullsized":"2024-09-13T14:40:21.576852+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:21.947056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576605+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:40:21.576605+0000","last_peered":"2024-09-13T14:40:21.576605+0000","last_clean":"2024-09-13T14:40:21.576605+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:40:21.576605+0000","last_undegraded":"2024-09-13T14:40:21.576605+0000","last_fullsized":"2024-09-13T14:40:21.576605+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576740+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:40:21.576740+0000","last_peered":"2024-09-13T14:40:21.576740+0000","last_clean":"2024-09-13T14:40:21.576740+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:40:21.576740+0000","last_undegraded":"2024-09-13T14:40:21.576740+0000","last_fullsized":"2024-09-13T14:40:21.576740+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576503+0000","last_change":"2024-09-13T14:32:09.487930+0000","last_active":"2024-09-13T14:40:21.576503+0000","last_peered":"2024-09-13T14:40:21.576503+0000","last_clean":"2024-09-13T14:40:21.576503+0000","last_became_active":"2024-09-13T14:32:09.487592+0000","last_became_peered":"2024-09-13T14:32:09.487592+0000","last_unstale":"2024-09-13T14:40:21.576503+0000","last_undegraded":"2024-09-13T14:40:21.576503+0000","last_fullsized":"2024-09-13T14:40:21.576503+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:57.014876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":101,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970778+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:40:22.970778+0000","last_peered":"2024-09-13T14:40:22.970778+0000","last_clean":"2024-09-13T14:40:22.970778+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:40:22.970778+0000","last_undegraded":"2024-09-13T14:40:22.970778+0000","last_fullsized":"2024-09-13T14:40:22.970778+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.973048+0000","last_change":"2024-09-13T14:37:08.758818+0000","last_active":"2024-09-13T14:40:22.973048+0000","last_peered":"2024-09-13T14:40:22.973048+0000","last_clean":"2024-09-13T14:40:22.973048+0000","last_became_active":"2024-09-13T14:37:08.758225+0000","last_became_peered":"2024-09-13T14:37:08.758225+0000","last_unstale":"2024-09-13T14:40:22.973048+0000","last_undegraded":"2024-09-13T14:40:22.973048+0000","last_fullsized":"2024-09-13T14:40:22.973048+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:18.890207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":230,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330916+0000","last_change":"2024-09-13T14:31:29.950741+0000","last_active":"2024-09-13T14:40:21.330916+0000","last_peered":"2024-09-13T14:40:21.330916+0000","last_clean":"2024-09-13T14:40:21.330916+0000","last_became_active":"2024-09-13T14:31:29.950338+0000","last_became_peered":"2024-09-13T14:31:29.950338+0000","last_unstale":"2024-09-13T14:40:21.330916+0000","last_undegraded":"2024-09-13T14:40:21.330916+0000","last_fullsized":"2024-09-13T14:40:21.330916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:09.370711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":124,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.821932+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:40:20.821932+0000","last_peered":"2024-09-13T14:40:20.821932+0000","last_clean":"2024-09-13T14:40:20.821932+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:40:20.821932+0000","last_undegraded":"2024-09-13T14:40:20.821932+0000","last_fullsized":"2024-09-13T14:40:20.821932+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":90,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970672+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:40:22.970672+0000","last_peered":"2024-09-13T14:40:22.970672+0000","last_clean":"2024-09-13T14:40:22.970672+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:40:22.970672+0000","last_undegraded":"2024-09-13T14:40:22.970672+0000","last_fullsized":"2024-09-13T14:40:22.970672+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":95,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577239+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:40:21.577239+0000","last_peered":"2024-09-13T14:40:21.577239+0000","last_clean":"2024-09-13T14:40:21.577239+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:40:21.577239+0000","last_undegraded":"2024-09-13T14:40:21.577239+0000","last_fullsized":"2024-09-13T14:40:21.577239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":142,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330870+0000","last_change":"2024-09-13T14:32:09.482227+0000","last_active":"2024-09-13T14:40:21.330870+0000","last_peered":"2024-09-13T14:40:21.330870+0000","last_clean":"2024-09-13T14:40:21.330870+0000","last_became_active":"2024-09-13T14:32:09.481843+0000","last_became_peered":"2024-09-13T14:32:09.481843+0000","last_unstale":"2024-09-13T14:40:21.330870+0000","last_undegraded":"2024-09-13T14:40:21.330870+0000","last_fullsized":"2024-09-13T14:40:21.330870+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:56.929686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.828113+0000","last_change":"2024-09-13T14:37:08.748298+0000","last_active":"2024-09-13T14:40:22.828113+0000","last_peered":"2024-09-13T14:40:22.828113+0000","last_clean":"2024-09-13T14:40:22.828113+0000","last_became_active":"2024-09-13T14:37:08.747638+0000","last_became_peered":"2024-09-13T14:37:08.747638+0000","last_unstale":"2024-09-13T14:40:22.828113+0000","last_undegraded":"2024-09-13T14:40:22.828113+0000","last_fullsized":"2024-09-13T14:40:22.828113+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:46.778883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":182,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833211+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:40:22.833211+0000","last_peered":"2024-09-13T14:40:22.833211+0000","last_clean":"2024-09-13T14:40:22.833211+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:40:22.833211+0000","last_undegraded":"2024-09-13T14:40:22.833211+0000","last_fullsized":"2024-09-13T14:40:22.833211+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":158,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968128+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:40:22.968128+0000","last_peered":"2024-09-13T14:40:22.968128+0000","last_clean":"2024-09-13T14:40:22.968128+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:40:22.968128+0000","last_undegraded":"2024-09-13T14:40:22.968128+0000","last_fullsized":"2024-09-13T14:40:22.968128+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967202+0000","last_change":"2024-09-13T14:35:30.473739+0000","last_active":"2024-09-13T14:40:22.967202+0000","last_peered":"2024-09-13T14:40:22.967202+0000","last_clean":"2024-09-13T14:40:22.967202+0000","last_became_active":"2024-09-13T14:35:30.473448+0000","last_became_peered":"2024-09-13T14:35:30.473448+0000","last_unstale":"2024-09-13T14:40:22.967202+0000","last_undegraded":"2024-09-13T14:40:22.967202+0000","last_fullsized":"2024-09-13T14:40:22.967202+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:52.239598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":169,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.972926+0000","last_change":"2024-09-13T14:32:09.485178+0000","last_active":"2024-09-13T14:40:22.972926+0000","last_peered":"2024-09-13T14:40:22.972926+0000","last_clean":"2024-09-13T14:40:22.972926+0000","last_became_active":"2024-09-13T14:32:09.484869+0000","last_became_peered":"2024-09-13T14:32:09.484869+0000","last_unstale":"2024-09-13T14:40:22.972926+0000","last_undegraded":"2024-09-13T14:40:22.972926+0000","last_fullsized":"2024-09-13T14:40:22.972926+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:18.863308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.821990+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:40:20.821990+0000","last_peered":"2024-09-13T14:40:20.821990+0000","last_clean":"2024-09-13T14:40:20.821990+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:40:20.821990+0000","last_undegraded":"2024-09-13T14:40:20.821990+0000","last_fullsized":"2024-09-13T14:40:20.821990+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577044+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:40:21.577044+0000","last_peered":"2024-09-13T14:40:21.577044+0000","last_clean":"2024-09-13T14:40:21.577044+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:40:21.577044+0000","last_undegraded":"2024-09-13T14:40:21.577044+0000","last_fullsized":"2024-09-13T14:40:21.577044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":153,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968562+0000","last_change":"2024-09-13T14:37:08.794198+0000","last_active":"2024-09-13T14:40:22.968562+0000","last_peered":"2024-09-13T14:40:22.968562+0000","last_clean":"2024-09-13T14:40:22.968562+0000","last_became_active":"2024-09-13T14:37:08.793689+0000","last_became_peered":"2024-09-13T14:37:08.793689+0000","last_unstale":"2024-09-13T14:40:22.968562+0000","last_undegraded":"2024-09-13T14:40:22.968562+0000","last_fullsized":"2024-09-13T14:40:22.968562+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:19.865901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822288+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:40:20.822288+0000","last_peered":"2024-09-13T14:40:20.822288+0000","last_clean":"2024-09-13T14:40:20.822288+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:40:20.822288+0000","last_undegraded":"2024-09-13T14:40:20.822288+0000","last_fullsized":"2024-09-13T14:40:20.822288+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":181,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968754+0000","last_change":"2024-09-13T14:32:09.488979+0000","last_active":"2024-09-13T14:40:22.968754+0000","last_peered":"2024-09-13T14:40:22.968754+0000","last_clean":"2024-09-13T14:40:22.968754+0000","last_became_active":"2024-09-13T14:32:09.488320+0000","last_became_peered":"2024-09-13T14:32:09.488320+0000","last_unstale":"2024-09-13T14:40:22.968754+0000","last_undegraded":"2024-09-13T14:40:22.968754+0000","last_fullsized":"2024-09-13T14:40:22.968754+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:28.662506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969902+0000","last_change":"2024-09-13T14:32:09.489738+0000","last_active":"2024-09-13T14:40:22.969902+0000","last_peered":"2024-09-13T14:40:22.969902+0000","last_clean":"2024-09-13T14:40:22.969902+0000","last_became_active":"2024-09-13T14:32:09.489304+0000","last_became_peered":"2024-09-13T14:32:09.489304+0000","last_unstale":"2024-09-13T14:40:22.969902+0000","last_undegraded":"2024-09-13T14:40:22.969902+0000","last_fullsized":"2024-09-13T14:40:22.969902+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:16.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968664+0000","last_change":"2024-09-13T14:35:30.476616+0000","last_active":"2024-09-13T14:40:22.968664+0000","last_peered":"2024-09-13T14:40:22.968664+0000","last_clean":"2024-09-13T14:40:22.968664+0000","last_became_active":"2024-09-13T14:35:30.476323+0000","last_became_peered":"2024-09-13T14:35:30.476323+0000","last_unstale":"2024-09-13T14:40:22.968664+0000","last_undegraded":"2024-09-13T14:40:22.968664+0000","last_fullsized":"2024-09-13T14:40:22.968664+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:02.000336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331335+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:40:21.331335+0000","last_peered":"2024-09-13T14:40:21.331335+0000","last_clean":"2024-09-13T14:40:21.331335+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:40:21.331335+0000","last_undegraded":"2024-09-13T14:40:21.331335+0000","last_fullsized":"2024-09-13T14:40:21.331335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":160,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827259+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:40:22.827259+0000","last_peered":"2024-09-13T14:40:22.827259+0000","last_clean":"2024-09-13T14:40:22.827259+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:40:22.827259+0000","last_undegraded":"2024-09-13T14:40:22.827259+0000","last_fullsized":"2024-09-13T14:40:22.827259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971295+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:40:22.971295+0000","last_peered":"2024-09-13T14:40:22.971295+0000","last_clean":"2024-09-13T14:40:22.971295+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:40:22.971295+0000","last_undegraded":"2024-09-13T14:40:22.971295+0000","last_fullsized":"2024-09-13T14:40:22.971295+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969089+0000","last_change":"2024-09-13T14:32:09.490012+0000","last_active":"2024-09-13T14:40:22.969089+0000","last_peered":"2024-09-13T14:40:22.969089+0000","last_clean":"2024-09-13T14:40:22.969089+0000","last_became_active":"2024-09-13T14:32:09.489652+0000","last_became_peered":"2024-09-13T14:32:09.489652+0000","last_unstale":"2024-09-13T14:40:22.969089+0000","last_undegraded":"2024-09-13T14:40:22.969089+0000","last_fullsized":"2024-09-13T14:40:22.969089+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:10.587217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":129,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970677+0000","last_change":"2024-09-13T14:37:08.745773+0000","last_active":"2024-09-13T14:40:22.970677+0000","last_peered":"2024-09-13T14:40:22.970677+0000","last_clean":"2024-09-13T14:40:22.970677+0000","last_became_active":"2024-09-13T14:37:08.745130+0000","last_became_peered":"2024-09-13T14:37:08.745130+0000","last_unstale":"2024-09-13T14:40:22.970677+0000","last_undegraded":"2024-09-13T14:40:22.970677+0000","last_fullsized":"2024-09-13T14:40:22.970677+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:29:44.778588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833465+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:40:22.833465+0000","last_peered":"2024-09-13T14:40:22.833465+0000","last_clean":"2024-09-13T14:40:22.833465+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:40:22.833465+0000","last_undegraded":"2024-09-13T14:40:22.833465+0000","last_fullsized":"2024-09-13T14:40:22.833465+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577258+0000","last_change":"2024-09-13T14:35:30.479324+0000","last_active":"2024-09-13T14:40:21.577258+0000","last_peered":"2024-09-13T14:40:21.577258+0000","last_clean":"2024-09-13T14:40:21.577258+0000","last_became_active":"2024-09-13T14:35:30.479033+0000","last_became_peered":"2024-09-13T14:35:30.479033+0000","last_unstale":"2024-09-13T14:40:21.577258+0000","last_undegraded":"2024-09-13T14:40:21.577258+0000","last_fullsized":"2024-09-13T14:40:21.577258+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:57.606853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":95,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576646+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:40:21.576646+0000","last_peered":"2024-09-13T14:40:21.576646+0000","last_clean":"2024-09-13T14:40:21.576646+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:40:21.576646+0000","last_undegraded":"2024-09-13T14:40:21.576646+0000","last_fullsized":"2024-09-13T14:40:21.576646+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576956+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:40:21.576956+0000","last_peered":"2024-09-13T14:40:21.576956+0000","last_clean":"2024-09-13T14:40:21.576956+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:40:21.576956+0000","last_undegraded":"2024-09-13T14:40:21.576956+0000","last_fullsized":"2024-09-13T14:40:21.576956+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":119,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576296+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:40:21.576296+0000","last_peered":"2024-09-13T14:40:21.576296+0000","last_clean":"2024-09-13T14:40:21.576296+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:40:21.576296+0000","last_undegraded":"2024-09-13T14:40:21.576296+0000","last_fullsized":"2024-09-13T14:40:21.576296+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969738+0000","last_change":"2024-09-13T14:35:30.474794+0000","last_active":"2024-09-13T14:40:22.969738+0000","last_peered":"2024-09-13T14:40:22.969738+0000","last_clean":"2024-09-13T14:40:22.969738+0000","last_became_active":"2024-09-13T14:35:30.474501+0000","last_became_peered":"2024-09-13T14:35:30.474501+0000","last_unstale":"2024-09-13T14:40:22.969738+0000","last_undegraded":"2024-09-13T14:40:22.969738+0000","last_fullsized":"2024-09-13T14:40:22.969738+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:58.326635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":117,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331198+0000","last_change":"2024-09-13T14:37:08.756167+0000","last_active":"2024-09-13T14:40:21.331198+0000","last_peered":"2024-09-13T14:40:21.331198+0000","last_clean":"2024-09-13T14:40:21.331198+0000","last_became_active":"2024-09-13T14:37:08.755657+0000","last_became_peered":"2024-09-13T14:37:08.755657+0000","last_unstale":"2024-09-13T14:40:21.331198+0000","last_undegraded":"2024-09-13T14:40:21.331198+0000","last_fullsized":"2024-09-13T14:40:21.331198+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:20.322084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822309+0000","last_change":"2024-09-13T14:32:09.488238+0000","last_active":"2024-09-13T14:40:20.822309+0000","last_peered":"2024-09-13T14:40:20.822309+0000","last_clean":"2024-09-13T14:40:20.822309+0000","last_became_active":"2024-09-13T14:32:09.487750+0000","last_became_peered":"2024-09-13T14:32:09.487750+0000","last_unstale":"2024-09-13T14:40:20.822309+0000","last_undegraded":"2024-09-13T14:40:20.822309+0000","last_fullsized":"2024-09-13T14:40:20.822309+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:17.258720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":91,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970280+0000","last_change":"2024-09-13T14:37:08.747482+0000","last_active":"2024-09-13T14:40:22.970280+0000","last_peered":"2024-09-13T14:40:22.970280+0000","last_clean":"2024-09-13T14:40:22.970280+0000","last_became_active":"2024-09-13T14:37:08.747176+0000","last_became_peered":"2024-09-13T14:37:08.747176+0000","last_unstale":"2024-09-13T14:40:22.970280+0000","last_undegraded":"2024-09-13T14:40:22.970280+0000","last_fullsized":"2024-09-13T14:40:22.970280+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:18.885448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827804+0000","last_change":"2024-09-13T14:37:08.759355+0000","last_active":"2024-09-13T14:40:22.827804+0000","last_peered":"2024-09-13T14:40:22.827804+0000","last_clean":"2024-09-13T14:40:22.827804+0000","last_became_active":"2024-09-13T14:37:08.759058+0000","last_became_peered":"2024-09-13T14:37:08.759058+0000","last_unstale":"2024-09-13T14:40:22.827804+0000","last_undegraded":"2024-09-13T14:40:22.827804+0000","last_fullsized":"2024-09-13T14:40:22.827804+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:42.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.966945+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:40:22.966945+0000","last_peered":"2024-09-13T14:40:22.966945+0000","last_clean":"2024-09-13T14:40:22.966945+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:40:22.966945+0000","last_undegraded":"2024-09-13T14:40:22.966945+0000","last_fullsized":"2024-09-13T14:40:22.966945+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577265+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:40:21.577265+0000","last_peered":"2024-09-13T14:40:21.577265+0000","last_clean":"2024-09-13T14:40:21.577265+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:40:21.577265+0000","last_undegraded":"2024-09-13T14:40:21.577265+0000","last_fullsized":"2024-09-13T14:40:21.577265+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970673+0000","last_change":"2024-09-13T14:37:08.777939+0000","last_active":"2024-09-13T14:40:22.970673+0000","last_peered":"2024-09-13T14:40:22.970673+0000","last_clean":"2024-09-13T14:40:22.970673+0000","last_became_active":"2024-09-13T14:37:08.776339+0000","last_became_peered":"2024-09-13T14:37:08.776339+0000","last_unstale":"2024-09-13T14:40:22.970673+0000","last_undegraded":"2024-09-13T14:40:22.970673+0000","last_fullsized":"2024-09-13T14:40:22.970673+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:03:31.349551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833307+0000","last_change":"2024-09-13T14:37:08.749705+0000","last_active":"2024-09-13T14:40:22.833307+0000","last_peered":"2024-09-13T14:40:22.833307+0000","last_clean":"2024-09-13T14:40:22.833307+0000","last_became_active":"2024-09-13T14:37:08.749158+0000","last_became_peered":"2024-09-13T14:37:08.749158+0000","last_unstale":"2024-09-13T14:40:22.833307+0000","last_undegraded":"2024-09-13T14:40:22.833307+0000","last_fullsized":"2024-09-13T14:40:22.833307+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:04.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967216+0000","last_change":"2024-09-13T14:37:08.758601+0000","last_active":"2024-09-13T14:40:22.967216+0000","last_peered":"2024-09-13T14:40:22.967216+0000","last_clean":"2024-09-13T14:40:22.967216+0000","last_became_active":"2024-09-13T14:37:08.758031+0000","last_became_peered":"2024-09-13T14:37:08.758031+0000","last_unstale":"2024-09-13T14:40:22.967216+0000","last_undegraded":"2024-09-13T14:40:22.967216+0000","last_fullsized":"2024-09-13T14:40:22.967216+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:12.269600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971366+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:40:22.971366+0000","last_peered":"2024-09-13T14:40:22.971366+0000","last_clean":"2024-09-13T14:40:22.971366+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:40:22.971366+0000","last_undegraded":"2024-09-13T14:40:22.971366+0000","last_fullsized":"2024-09-13T14:40:22.971366+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576345+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:40:21.576345+0000","last_peered":"2024-09-13T14:40:21.576345+0000","last_clean":"2024-09-13T14:40:21.576345+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:40:21.576345+0000","last_undegraded":"2024-09-13T14:40:21.576345+0000","last_fullsized":"2024-09-13T14:40:21.576345+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":158,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967841+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:40:22.967841+0000","last_peered":"2024-09-13T14:40:22.967841+0000","last_clean":"2024-09-13T14:40:22.967841+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:40:22.967841+0000","last_undegraded":"2024-09-13T14:40:22.967841+0000","last_fullsized":"2024-09-13T14:40:22.967841+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":162,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827359+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:40:22.827359+0000","last_peered":"2024-09-13T14:40:22.827359+0000","last_clean":"2024-09-13T14:40:22.827359+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:40:22.827359+0000","last_undegraded":"2024-09-13T14:40:22.827359+0000","last_fullsized":"2024-09-13T14:40:22.827359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":139,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576863+0000","last_change":"2024-09-13T14:37:08.749549+0000","last_active":"2024-09-13T14:40:21.576863+0000","last_peered":"2024-09-13T14:40:21.576863+0000","last_clean":"2024-09-13T14:40:21.576863+0000","last_became_active":"2024-09-13T14:37:08.749013+0000","last_became_peered":"2024-09-13T14:37:08.749013+0000","last_unstale":"2024-09-13T14:40:21.576863+0000","last_undegraded":"2024-09-13T14:40:21.576863+0000","last_fullsized":"2024-09-13T14:40:21.576863+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:35.239989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":114,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822005+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:40:20.822005+0000","last_peered":"2024-09-13T14:40:20.822005+0000","last_clean":"2024-09-13T14:40:20.822005+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:40:20.822005+0000","last_undegraded":"2024-09-13T14:40:20.822005+0000","last_fullsized":"2024-09-13T14:40:20.822005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576063+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:40:21.576063+0000","last_peered":"2024-09-13T14:40:21.576063+0000","last_clean":"2024-09-13T14:40:21.576063+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:40:21.576063+0000","last_undegraded":"2024-09-13T14:40:21.576063+0000","last_fullsized":"2024-09-13T14:40:21.576063+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":97,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822237+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:40:20.822237+0000","last_peered":"2024-09-13T14:40:20.822237+0000","last_clean":"2024-09-13T14:40:20.822237+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:40:20.822237+0000","last_undegraded":"2024-09-13T14:40:20.822237+0000","last_fullsized":"2024-09-13T14:40:20.822237+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":133,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970577+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:40:22.970577+0000","last_peered":"2024-09-13T14:40:22.970577+0000","last_clean":"2024-09-13T14:40:22.970577+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:40:22.970577+0000","last_undegraded":"2024-09-13T14:40:22.970577+0000","last_fullsized":"2024-09-13T14:40:22.970577+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971883+0000","last_change":"2024-09-13T14:37:08.750424+0000","last_active":"2024-09-13T14:40:22.971883+0000","last_peered":"2024-09-13T14:40:22.971883+0000","last_clean":"2024-09-13T14:40:22.971883+0000","last_became_active":"2024-09-13T14:37:08.749577+0000","last_became_peered":"2024-09-13T14:37:08.749577+0000","last_unstale":"2024-09-13T14:40:22.971883+0000","last_undegraded":"2024-09-13T14:40:22.971883+0000","last_fullsized":"2024-09-13T14:40:22.971883+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:16.435452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.575871+0000","last_change":"2024-09-13T14:32:09.484255+0000","last_active":"2024-09-13T14:40:21.575871+0000","last_peered":"2024-09-13T14:40:21.575871+0000","last_clean":"2024-09-13T14:40:21.575871+0000","last_became_active":"2024-09-13T14:32:09.483903+0000","last_became_peered":"2024-09-13T14:32:09.483903+0000","last_unstale":"2024-09-13T14:40:21.575871+0000","last_undegraded":"2024-09-13T14:40:21.575871+0000","last_fullsized":"2024-09-13T14:40:21.575871+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:41.931522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827366+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:40:22.827366+0000","last_peered":"2024-09-13T14:40:22.827366+0000","last_clean":"2024-09-13T14:40:22.827366+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:40:22.827366+0000","last_undegraded":"2024-09-13T14:40:22.827366+0000","last_fullsized":"2024-09-13T14:40:22.827366+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365057,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299648,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445504,"statfs":{"total":95995035648,"available":95688196096,"internally_reserved":0,"allocated":2338816,"data_stored":1595354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561287,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307952,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437200,"statfs":{"total":95995035648,"available":95679692800,"internally_reserved":0,"allocated":2330624,"data_stored":1595177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790219,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898592,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":896894,"kb_avail":92846560,"statfs":{"total":95995035648,"available":95074877440,"internally_reserved":0,"allocated":1724416,"data_stored":1000383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918419800},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986447,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595768,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93149384,"statfs":{"total":95995035648,"available":95384969216,"internally_reserved":0,"allocated":1744896,"data_stored":1005181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519602,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218528,"kb_used_data":1684,"kb_used_omap":4,"kb_used_meta":216827,"kb_avail":93526624,"statfs":{"total":95995035648,"available":95771262976,"internally_reserved":0,"allocated":1724416,"data_stored":1000393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":222031296},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304518,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230376,"kb_used_data":1688,"kb_used_omap":6,"kb_used_meta":228665,"kb_avail":93514776,"statfs":{"total":95995035648,"available":95759130624,"internally_reserved":0,"allocated":1728512,"data_stored":1000907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":234153415},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827822,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":389536,"kb_used_data":1684,"kb_used_omap":5,"kb_used_meta":387834,"kb_avail":93355616,"statfs":{"total":95995035648,"available":95596150784,"internally_reserved":0,"allocated":1724416,"data_stored":1000330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":397142252},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285686,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396520,"kb_used_data":2264,"kb_used_omap":5,"kb_used_meta":394234,"kb_avail":93348632,"statfs":{"total":95995035648,"available":95588999168,"internally_reserved":0,"allocated":2318336,"data_stored":1590035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":403695675},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:13.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:14.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:14 smithi003 ceph-mon[107278]: from='client.45782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:14.147 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:14 smithi003 ceph-mon[104007]: from='client.45782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:14.147 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:41:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:41:14] "GET /metrics HTTP/1.1" 200 34046 "" "Prometheus/2.51.0" 2024-09-13T14:41:14.357 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:14 smithi019 ceph-mon[91296]: from='client.45782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:15.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[107278]: pgmap v488: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:15.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:15.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:15.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:15.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:15.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:15.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[104007]: pgmap v488: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:15.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:15.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:15.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:15.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:15.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:15 smithi019 ceph-mon[91296]: pgmap v488: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:16.373 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:41:16.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:16 smithi003 ceph-mon[104007]: pgmap v489: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:16.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:16 smithi003 ceph-mon[107278]: pgmap v489: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:16.376 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:41:16.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:16 smithi019 ceph-mon[91296]: pgmap v489: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:17.016 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-13T14:41:15.562610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":213,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3336920,"kb_used_data":15268,"kb_used_omap":29,"kb_used_meta":3321506,"kb_avail":746624296,"statfs":{"total":767960285184,"available":764543279104,"internally_reserved":0,"allocated":15634432,"data_stored":9787760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":3401222670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002603"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.972038+0000","last_change":"2024-09-13T14:37:08.782889+0000","last_active":"2024-09-13T14:40:22.972038+0000","last_peered":"2024-09-13T14:40:22.972038+0000","last_clean":"2024-09-13T14:40:22.972038+0000","last_became_active":"2024-09-13T14:37:08.781502+0000","last_became_peered":"2024-09-13T14:37:08.781502+0000","last_unstale":"2024-09-13T14:40:22.972038+0000","last_undegraded":"2024-09-13T14:40:22.972038+0000","last_fullsized":"2024-09-13T14:40:22.972038+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:22.769013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":116,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330969+0000","last_change":"2024-09-13T14:35:30.492165+0000","last_active":"2024-09-13T14:40:21.330969+0000","last_peered":"2024-09-13T14:40:21.330969+0000","last_clean":"2024-09-13T14:40:21.330969+0000","last_became_active":"2024-09-13T14:35:30.491844+0000","last_became_peered":"2024-09-13T14:35:30.491844+0000","last_unstale":"2024-09-13T14:40:21.330969+0000","last_undegraded":"2024-09-13T14:40:21.330969+0000","last_fullsized":"2024-09-13T14:40:21.330969+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:22.876890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":111,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822194+0000","last_change":"2024-09-13T14:37:08.755784+0000","last_active":"2024-09-13T14:40:20.822194+0000","last_peered":"2024-09-13T14:40:20.822194+0000","last_clean":"2024-09-13T14:40:20.822194+0000","last_became_active":"2024-09-13T14:37:08.755300+0000","last_became_peered":"2024-09-13T14:37:08.755300+0000","last_unstale":"2024-09-13T14:40:20.822194+0000","last_undegraded":"2024-09-13T14:40:20.822194+0000","last_fullsized":"2024-09-13T14:40:20.822194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:10.881654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827254+0000","last_change":"2024-09-13T14:37:08.748084+0000","last_active":"2024-09-13T14:40:22.827254+0000","last_peered":"2024-09-13T14:40:22.827254+0000","last_clean":"2024-09-13T14:40:22.827254+0000","last_became_active":"2024-09-13T14:37:08.746801+0000","last_became_peered":"2024-09-13T14:37:08.746801+0000","last_unstale":"2024-09-13T14:40:22.827254+0000","last_undegraded":"2024-09-13T14:40:22.827254+0000","last_fullsized":"2024-09-13T14:40:22.827254+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:13.495421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833418+0000","last_change":"2024-09-13T14:37:08.750965+0000","last_active":"2024-09-13T14:40:22.833418+0000","last_peered":"2024-09-13T14:40:22.833418+0000","last_clean":"2024-09-13T14:40:22.833418+0000","last_became_active":"2024-09-13T14:37:08.750129+0000","last_became_peered":"2024-09-13T14:37:08.750129+0000","last_unstale":"2024-09-13T14:40:22.833418+0000","last_undegraded":"2024-09-13T14:40:22.833418+0000","last_fullsized":"2024-09-13T14:40:22.833418+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:48.359985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":97,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330708+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:40:21.330708+0000","last_peered":"2024-09-13T14:40:21.330708+0000","last_clean":"2024-09-13T14:40:21.330708+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:40:21.330708+0000","last_undegraded":"2024-09-13T14:40:21.330708+0000","last_fullsized":"2024-09-13T14:40:21.330708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971111+0000","last_change":"2024-09-13T14:37:08.755312+0000","last_active":"2024-09-13T14:40:22.971111+0000","last_peered":"2024-09-13T14:40:22.971111+0000","last_clean":"2024-09-13T14:40:22.971111+0000","last_became_active":"2024-09-13T14:37:08.753186+0000","last_became_peered":"2024-09-13T14:37:08.753186+0000","last_unstale":"2024-09-13T14:40:22.971111+0000","last_undegraded":"2024-09-13T14:40:22.971111+0000","last_fullsized":"2024-09-13T14:40:22.971111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:57.687749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":121,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576521+0000","last_change":"2024-09-13T14:31:29.946716+0000","last_active":"2024-09-13T14:40:21.576521+0000","last_peered":"2024-09-13T14:40:21.576521+0000","last_clean":"2024-09-13T14:40:21.576521+0000","last_became_active":"2024-09-13T14:31:29.946366+0000","last_became_peered":"2024-09-13T14:31:29.946366+0000","last_unstale":"2024-09-13T14:40:21.576521+0000","last_undegraded":"2024-09-13T14:40:21.576521+0000","last_fullsized":"2024-09-13T14:40:21.576521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:29.686747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":115,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331328+0000","last_change":"2024-09-13T14:31:29.950983+0000","last_active":"2024-09-13T14:40:21.331328+0000","last_peered":"2024-09-13T14:40:21.331328+0000","last_clean":"2024-09-13T14:40:21.331328+0000","last_became_active":"2024-09-13T14:31:29.950612+0000","last_became_peered":"2024-09-13T14:31:29.950612+0000","last_unstale":"2024-09-13T14:40:21.331328+0000","last_undegraded":"2024-09-13T14:40:21.331328+0000","last_fullsized":"2024-09-13T14:40:21.331328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:35.833747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":94,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822196+0000","last_change":"2024-09-13T14:21:52.983025+0000","last_active":"2024-09-13T14:40:20.822196+0000","last_peered":"2024-09-13T14:40:20.822196+0000","last_clean":"2024-09-13T14:40:20.822196+0000","last_became_active":"2024-09-13T14:21:52.982611+0000","last_became_peered":"2024-09-13T14:21:52.982611+0000","last_unstale":"2024-09-13T14:40:20.822196+0000","last_undegraded":"2024-09-13T14:40:20.822196+0000","last_fullsized":"2024-09-13T14:40:20.822196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:00.851531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":116,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331353+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:40:21.331353+0000","last_peered":"2024-09-13T14:40:21.331353+0000","last_clean":"2024-09-13T14:40:21.331353+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:40:21.331353+0000","last_undegraded":"2024-09-13T14:40:21.331353+0000","last_fullsized":"2024-09-13T14:40:21.331353+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576188+0000","last_change":"2024-09-13T14:37:08.752050+0000","last_active":"2024-09-13T14:40:21.576188+0000","last_peered":"2024-09-13T14:40:21.576188+0000","last_clean":"2024-09-13T14:40:21.576188+0000","last_became_active":"2024-09-13T14:37:08.751650+0000","last_became_peered":"2024-09-13T14:37:08.751650+0000","last_unstale":"2024-09-13T14:40:21.576188+0000","last_undegraded":"2024-09-13T14:40:21.576188+0000","last_fullsized":"2024-09-13T14:40:21.576188+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:17.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":98,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331346+0000","last_change":"2024-09-13T14:21:52.610490+0000","last_active":"2024-09-13T14:40:21.331346+0000","last_peered":"2024-09-13T14:40:21.331346+0000","last_clean":"2024-09-13T14:40:21.331346+0000","last_became_active":"2024-09-13T14:21:52.610192+0000","last_became_peered":"2024-09-13T14:21:52.610192+0000","last_unstale":"2024-09-13T14:40:21.331346+0000","last_undegraded":"2024-09-13T14:40:21.331346+0000","last_fullsized":"2024-09-13T14:40:21.331346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:09.564393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":105,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576705+0000","last_change":"2024-09-13T14:22:34.901360+0000","last_active":"2024-09-13T14:40:21.576705+0000","last_peered":"2024-09-13T14:40:21.576705+0000","last_clean":"2024-09-13T14:40:21.576705+0000","last_became_active":"2024-09-13T14:22:34.901010+0000","last_became_peered":"2024-09-13T14:22:34.901010+0000","last_unstale":"2024-09-13T14:40:21.576705+0000","last_undegraded":"2024-09-13T14:40:21.576705+0000","last_fullsized":"2024-09-13T14:40:21.576705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:55.897004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822246+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:40:20.822246+0000","last_peered":"2024-09-13T14:40:20.822246+0000","last_clean":"2024-09-13T14:40:20.822246+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:40:20.822246+0000","last_undegraded":"2024-09-13T14:40:20.822246+0000","last_fullsized":"2024-09-13T14:40:20.822246+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827184+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:40:22.827184+0000","last_peered":"2024-09-13T14:40:22.827184+0000","last_clean":"2024-09-13T14:40:22.827184+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:40:22.827184+0000","last_undegraded":"2024-09-13T14:40:22.827184+0000","last_fullsized":"2024-09-13T14:40:22.827184+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970902+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:40:22.970902+0000","last_peered":"2024-09-13T14:40:22.970902+0000","last_clean":"2024-09-13T14:40:22.970902+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:40:22.970902+0000","last_undegraded":"2024-09-13T14:40:22.970902+0000","last_fullsized":"2024-09-13T14:40:22.970902+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576160+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:40:21.576160+0000","last_peered":"2024-09-13T14:40:21.576160+0000","last_clean":"2024-09-13T14:40:21.576160+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:40:21.576160+0000","last_undegraded":"2024-09-13T14:40:21.576160+0000","last_fullsized":"2024-09-13T14:40:21.576160+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576852+0000","last_change":"2024-09-13T14:32:09.491728+0000","last_active":"2024-09-13T14:40:21.576852+0000","last_peered":"2024-09-13T14:40:21.576852+0000","last_clean":"2024-09-13T14:40:21.576852+0000","last_became_active":"2024-09-13T14:32:09.490612+0000","last_became_peered":"2024-09-13T14:32:09.490612+0000","last_unstale":"2024-09-13T14:40:21.576852+0000","last_undegraded":"2024-09-13T14:40:21.576852+0000","last_fullsized":"2024-09-13T14:40:21.576852+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:21.947056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576605+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:40:21.576605+0000","last_peered":"2024-09-13T14:40:21.576605+0000","last_clean":"2024-09-13T14:40:21.576605+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:40:21.576605+0000","last_undegraded":"2024-09-13T14:40:21.576605+0000","last_fullsized":"2024-09-13T14:40:21.576605+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576740+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:40:21.576740+0000","last_peered":"2024-09-13T14:40:21.576740+0000","last_clean":"2024-09-13T14:40:21.576740+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:40:21.576740+0000","last_undegraded":"2024-09-13T14:40:21.576740+0000","last_fullsized":"2024-09-13T14:40:21.576740+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576503+0000","last_change":"2024-09-13T14:32:09.487930+0000","last_active":"2024-09-13T14:40:21.576503+0000","last_peered":"2024-09-13T14:40:21.576503+0000","last_clean":"2024-09-13T14:40:21.576503+0000","last_became_active":"2024-09-13T14:32:09.487592+0000","last_became_peered":"2024-09-13T14:32:09.487592+0000","last_unstale":"2024-09-13T14:40:21.576503+0000","last_undegraded":"2024-09-13T14:40:21.576503+0000","last_fullsized":"2024-09-13T14:40:21.576503+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:57.014876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":101,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970778+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:40:22.970778+0000","last_peered":"2024-09-13T14:40:22.970778+0000","last_clean":"2024-09-13T14:40:22.970778+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:40:22.970778+0000","last_undegraded":"2024-09-13T14:40:22.970778+0000","last_fullsized":"2024-09-13T14:40:22.970778+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.973048+0000","last_change":"2024-09-13T14:37:08.758818+0000","last_active":"2024-09-13T14:40:22.973048+0000","last_peered":"2024-09-13T14:40:22.973048+0000","last_clean":"2024-09-13T14:40:22.973048+0000","last_became_active":"2024-09-13T14:37:08.758225+0000","last_became_peered":"2024-09-13T14:37:08.758225+0000","last_unstale":"2024-09-13T14:40:22.973048+0000","last_undegraded":"2024-09-13T14:40:22.973048+0000","last_fullsized":"2024-09-13T14:40:22.973048+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:18.890207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":230,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330916+0000","last_change":"2024-09-13T14:31:29.950741+0000","last_active":"2024-09-13T14:40:21.330916+0000","last_peered":"2024-09-13T14:40:21.330916+0000","last_clean":"2024-09-13T14:40:21.330916+0000","last_became_active":"2024-09-13T14:31:29.950338+0000","last_became_peered":"2024-09-13T14:31:29.950338+0000","last_unstale":"2024-09-13T14:40:21.330916+0000","last_undegraded":"2024-09-13T14:40:21.330916+0000","last_fullsized":"2024-09-13T14:40:21.330916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:09.370711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":124,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.821932+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:40:20.821932+0000","last_peered":"2024-09-13T14:40:20.821932+0000","last_clean":"2024-09-13T14:40:20.821932+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:40:20.821932+0000","last_undegraded":"2024-09-13T14:40:20.821932+0000","last_fullsized":"2024-09-13T14:40:20.821932+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":90,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970672+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:40:22.970672+0000","last_peered":"2024-09-13T14:40:22.970672+0000","last_clean":"2024-09-13T14:40:22.970672+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:40:22.970672+0000","last_undegraded":"2024-09-13T14:40:22.970672+0000","last_fullsized":"2024-09-13T14:40:22.970672+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":95,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577239+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:40:21.577239+0000","last_peered":"2024-09-13T14:40:21.577239+0000","last_clean":"2024-09-13T14:40:21.577239+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:40:21.577239+0000","last_undegraded":"2024-09-13T14:40:21.577239+0000","last_fullsized":"2024-09-13T14:40:21.577239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":142,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330870+0000","last_change":"2024-09-13T14:32:09.482227+0000","last_active":"2024-09-13T14:40:21.330870+0000","last_peered":"2024-09-13T14:40:21.330870+0000","last_clean":"2024-09-13T14:40:21.330870+0000","last_became_active":"2024-09-13T14:32:09.481843+0000","last_became_peered":"2024-09-13T14:32:09.481843+0000","last_unstale":"2024-09-13T14:40:21.330870+0000","last_undegraded":"2024-09-13T14:40:21.330870+0000","last_fullsized":"2024-09-13T14:40:21.330870+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:56.929686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.828113+0000","last_change":"2024-09-13T14:37:08.748298+0000","last_active":"2024-09-13T14:40:22.828113+0000","last_peered":"2024-09-13T14:40:22.828113+0000","last_clean":"2024-09-13T14:40:22.828113+0000","last_became_active":"2024-09-13T14:37:08.747638+0000","last_became_peered":"2024-09-13T14:37:08.747638+0000","last_unstale":"2024-09-13T14:40:22.828113+0000","last_undegraded":"2024-09-13T14:40:22.828113+0000","last_fullsized":"2024-09-13T14:40:22.828113+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:46.778883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":182,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833211+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:40:22.833211+0000","last_peered":"2024-09-13T14:40:22.833211+0000","last_clean":"2024-09-13T14:40:22.833211+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:40:22.833211+0000","last_undegraded":"2024-09-13T14:40:22.833211+0000","last_fullsized":"2024-09-13T14:40:22.833211+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":158,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968128+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:40:22.968128+0000","last_peered":"2024-09-13T14:40:22.968128+0000","last_clean":"2024-09-13T14:40:22.968128+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:40:22.968128+0000","last_undegraded":"2024-09-13T14:40:22.968128+0000","last_fullsized":"2024-09-13T14:40:22.968128+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967202+0000","last_change":"2024-09-13T14:35:30.473739+0000","last_active":"2024-09-13T14:40:22.967202+0000","last_peered":"2024-09-13T14:40:22.967202+0000","last_clean":"2024-09-13T14:40:22.967202+0000","last_became_active":"2024-09-13T14:35:30.473448+0000","last_became_peered":"2024-09-13T14:35:30.473448+0000","last_unstale":"2024-09-13T14:40:22.967202+0000","last_undegraded":"2024-09-13T14:40:22.967202+0000","last_fullsized":"2024-09-13T14:40:22.967202+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:52.239598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":169,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.972926+0000","last_change":"2024-09-13T14:32:09.485178+0000","last_active":"2024-09-13T14:40:22.972926+0000","last_peered":"2024-09-13T14:40:22.972926+0000","last_clean":"2024-09-13T14:40:22.972926+0000","last_became_active":"2024-09-13T14:32:09.484869+0000","last_became_peered":"2024-09-13T14:32:09.484869+0000","last_unstale":"2024-09-13T14:40:22.972926+0000","last_undegraded":"2024-09-13T14:40:22.972926+0000","last_fullsized":"2024-09-13T14:40:22.972926+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:18.863308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.821990+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:40:20.821990+0000","last_peered":"2024-09-13T14:40:20.821990+0000","last_clean":"2024-09-13T14:40:20.821990+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:40:20.821990+0000","last_undegraded":"2024-09-13T14:40:20.821990+0000","last_fullsized":"2024-09-13T14:40:20.821990+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577044+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:40:21.577044+0000","last_peered":"2024-09-13T14:40:21.577044+0000","last_clean":"2024-09-13T14:40:21.577044+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:40:21.577044+0000","last_undegraded":"2024-09-13T14:40:21.577044+0000","last_fullsized":"2024-09-13T14:40:21.577044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":153,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968562+0000","last_change":"2024-09-13T14:37:08.794198+0000","last_active":"2024-09-13T14:40:22.968562+0000","last_peered":"2024-09-13T14:40:22.968562+0000","last_clean":"2024-09-13T14:40:22.968562+0000","last_became_active":"2024-09-13T14:37:08.793689+0000","last_became_peered":"2024-09-13T14:37:08.793689+0000","last_unstale":"2024-09-13T14:40:22.968562+0000","last_undegraded":"2024-09-13T14:40:22.968562+0000","last_fullsized":"2024-09-13T14:40:22.968562+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:19.865901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822288+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:40:20.822288+0000","last_peered":"2024-09-13T14:40:20.822288+0000","last_clean":"2024-09-13T14:40:20.822288+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:40:20.822288+0000","last_undegraded":"2024-09-13T14:40:20.822288+0000","last_fullsized":"2024-09-13T14:40:20.822288+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":181,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968754+0000","last_change":"2024-09-13T14:32:09.488979+0000","last_active":"2024-09-13T14:40:22.968754+0000","last_peered":"2024-09-13T14:40:22.968754+0000","last_clean":"2024-09-13T14:40:22.968754+0000","last_became_active":"2024-09-13T14:32:09.488320+0000","last_became_peered":"2024-09-13T14:32:09.488320+0000","last_unstale":"2024-09-13T14:40:22.968754+0000","last_undegraded":"2024-09-13T14:40:22.968754+0000","last_fullsized":"2024-09-13T14:40:22.968754+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:28.662506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969902+0000","last_change":"2024-09-13T14:32:09.489738+0000","last_active":"2024-09-13T14:40:22.969902+0000","last_peered":"2024-09-13T14:40:22.969902+0000","last_clean":"2024-09-13T14:40:22.969902+0000","last_became_active":"2024-09-13T14:32:09.489304+0000","last_became_peered":"2024-09-13T14:32:09.489304+0000","last_unstale":"2024-09-13T14:40:22.969902+0000","last_undegraded":"2024-09-13T14:40:22.969902+0000","last_fullsized":"2024-09-13T14:40:22.969902+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:16.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968664+0000","last_change":"2024-09-13T14:35:30.476616+0000","last_active":"2024-09-13T14:40:22.968664+0000","last_peered":"2024-09-13T14:40:22.968664+0000","last_clean":"2024-09-13T14:40:22.968664+0000","last_became_active":"2024-09-13T14:35:30.476323+0000","last_became_peered":"2024-09-13T14:35:30.476323+0000","last_unstale":"2024-09-13T14:40:22.968664+0000","last_undegraded":"2024-09-13T14:40:22.968664+0000","last_fullsized":"2024-09-13T14:40:22.968664+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:02.000336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331335+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:40:21.331335+0000","last_peered":"2024-09-13T14:40:21.331335+0000","last_clean":"2024-09-13T14:40:21.331335+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:40:21.331335+0000","last_undegraded":"2024-09-13T14:40:21.331335+0000","last_fullsized":"2024-09-13T14:40:21.331335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":160,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827259+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:40:22.827259+0000","last_peered":"2024-09-13T14:40:22.827259+0000","last_clean":"2024-09-13T14:40:22.827259+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:40:22.827259+0000","last_undegraded":"2024-09-13T14:40:22.827259+0000","last_fullsized":"2024-09-13T14:40:22.827259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971295+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:40:22.971295+0000","last_peered":"2024-09-13T14:40:22.971295+0000","last_clean":"2024-09-13T14:40:22.971295+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:40:22.971295+0000","last_undegraded":"2024-09-13T14:40:22.971295+0000","last_fullsized":"2024-09-13T14:40:22.971295+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969089+0000","last_change":"2024-09-13T14:32:09.490012+0000","last_active":"2024-09-13T14:40:22.969089+0000","last_peered":"2024-09-13T14:40:22.969089+0000","last_clean":"2024-09-13T14:40:22.969089+0000","last_became_active":"2024-09-13T14:32:09.489652+0000","last_became_peered":"2024-09-13T14:32:09.489652+0000","last_unstale":"2024-09-13T14:40:22.969089+0000","last_undegraded":"2024-09-13T14:40:22.969089+0000","last_fullsized":"2024-09-13T14:40:22.969089+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:10.587217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":129,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970677+0000","last_change":"2024-09-13T14:37:08.745773+0000","last_active":"2024-09-13T14:40:22.970677+0000","last_peered":"2024-09-13T14:40:22.970677+0000","last_clean":"2024-09-13T14:40:22.970677+0000","last_became_active":"2024-09-13T14:37:08.745130+0000","last_became_peered":"2024-09-13T14:37:08.745130+0000","last_unstale":"2024-09-13T14:40:22.970677+0000","last_undegraded":"2024-09-13T14:40:22.970677+0000","last_fullsized":"2024-09-13T14:40:22.970677+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:29:44.778588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833465+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:40:22.833465+0000","last_peered":"2024-09-13T14:40:22.833465+0000","last_clean":"2024-09-13T14:40:22.833465+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:40:22.833465+0000","last_undegraded":"2024-09-13T14:40:22.833465+0000","last_fullsized":"2024-09-13T14:40:22.833465+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577258+0000","last_change":"2024-09-13T14:35:30.479324+0000","last_active":"2024-09-13T14:40:21.577258+0000","last_peered":"2024-09-13T14:40:21.577258+0000","last_clean":"2024-09-13T14:40:21.577258+0000","last_became_active":"2024-09-13T14:35:30.479033+0000","last_became_peered":"2024-09-13T14:35:30.479033+0000","last_unstale":"2024-09-13T14:40:21.577258+0000","last_undegraded":"2024-09-13T14:40:21.577258+0000","last_fullsized":"2024-09-13T14:40:21.577258+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:57.606853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":95,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576646+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:40:21.576646+0000","last_peered":"2024-09-13T14:40:21.576646+0000","last_clean":"2024-09-13T14:40:21.576646+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:40:21.576646+0000","last_undegraded":"2024-09-13T14:40:21.576646+0000","last_fullsized":"2024-09-13T14:40:21.576646+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576956+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:40:21.576956+0000","last_peered":"2024-09-13T14:40:21.576956+0000","last_clean":"2024-09-13T14:40:21.576956+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:40:21.576956+0000","last_undegraded":"2024-09-13T14:40:21.576956+0000","last_fullsized":"2024-09-13T14:40:21.576956+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":119,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576296+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:40:21.576296+0000","last_peered":"2024-09-13T14:40:21.576296+0000","last_clean":"2024-09-13T14:40:21.576296+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:40:21.576296+0000","last_undegraded":"2024-09-13T14:40:21.576296+0000","last_fullsized":"2024-09-13T14:40:21.576296+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969738+0000","last_change":"2024-09-13T14:35:30.474794+0000","last_active":"2024-09-13T14:40:22.969738+0000","last_peered":"2024-09-13T14:40:22.969738+0000","last_clean":"2024-09-13T14:40:22.969738+0000","last_became_active":"2024-09-13T14:35:30.474501+0000","last_became_peered":"2024-09-13T14:35:30.474501+0000","last_unstale":"2024-09-13T14:40:22.969738+0000","last_undegraded":"2024-09-13T14:40:22.969738+0000","last_fullsized":"2024-09-13T14:40:22.969738+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:58.326635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":117,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331198+0000","last_change":"2024-09-13T14:37:08.756167+0000","last_active":"2024-09-13T14:40:21.331198+0000","last_peered":"2024-09-13T14:40:21.331198+0000","last_clean":"2024-09-13T14:40:21.331198+0000","last_became_active":"2024-09-13T14:37:08.755657+0000","last_became_peered":"2024-09-13T14:37:08.755657+0000","last_unstale":"2024-09-13T14:40:21.331198+0000","last_undegraded":"2024-09-13T14:40:21.331198+0000","last_fullsized":"2024-09-13T14:40:21.331198+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:20.322084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822309+0000","last_change":"2024-09-13T14:32:09.488238+0000","last_active":"2024-09-13T14:40:20.822309+0000","last_peered":"2024-09-13T14:40:20.822309+0000","last_clean":"2024-09-13T14:40:20.822309+0000","last_became_active":"2024-09-13T14:32:09.487750+0000","last_became_peered":"2024-09-13T14:32:09.487750+0000","last_unstale":"2024-09-13T14:40:20.822309+0000","last_undegraded":"2024-09-13T14:40:20.822309+0000","last_fullsized":"2024-09-13T14:40:20.822309+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:17.258720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":91,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970280+0000","last_change":"2024-09-13T14:37:08.747482+0000","last_active":"2024-09-13T14:40:22.970280+0000","last_peered":"2024-09-13T14:40:22.970280+0000","last_clean":"2024-09-13T14:40:22.970280+0000","last_became_active":"2024-09-13T14:37:08.747176+0000","last_became_peered":"2024-09-13T14:37:08.747176+0000","last_unstale":"2024-09-13T14:40:22.970280+0000","last_undegraded":"2024-09-13T14:40:22.970280+0000","last_fullsized":"2024-09-13T14:40:22.970280+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:18.885448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827804+0000","last_change":"2024-09-13T14:37:08.759355+0000","last_active":"2024-09-13T14:40:22.827804+0000","last_peered":"2024-09-13T14:40:22.827804+0000","last_clean":"2024-09-13T14:40:22.827804+0000","last_became_active":"2024-09-13T14:37:08.759058+0000","last_became_peered":"2024-09-13T14:37:08.759058+0000","last_unstale":"2024-09-13T14:40:22.827804+0000","last_undegraded":"2024-09-13T14:40:22.827804+0000","last_fullsized":"2024-09-13T14:40:22.827804+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:42.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.966945+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:40:22.966945+0000","last_peered":"2024-09-13T14:40:22.966945+0000","last_clean":"2024-09-13T14:40:22.966945+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:40:22.966945+0000","last_undegraded":"2024-09-13T14:40:22.966945+0000","last_fullsized":"2024-09-13T14:40:22.966945+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577265+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:40:21.577265+0000","last_peered":"2024-09-13T14:40:21.577265+0000","last_clean":"2024-09-13T14:40:21.577265+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:40:21.577265+0000","last_undegraded":"2024-09-13T14:40:21.577265+0000","last_fullsized":"2024-09-13T14:40:21.577265+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970673+0000","last_change":"2024-09-13T14:37:08.777939+0000","last_active":"2024-09-13T14:40:22.970673+0000","last_peered":"2024-09-13T14:40:22.970673+0000","last_clean":"2024-09-13T14:40:22.970673+0000","last_became_active":"2024-09-13T14:37:08.776339+0000","last_became_peered":"2024-09-13T14:37:08.776339+0000","last_unstale":"2024-09-13T14:40:22.970673+0000","last_undegraded":"2024-09-13T14:40:22.970673+0000","last_fullsized":"2024-09-13T14:40:22.970673+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:03:31.349551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833307+0000","last_change":"2024-09-13T14:37:08.749705+0000","last_active":"2024-09-13T14:40:22.833307+0000","last_peered":"2024-09-13T14:40:22.833307+0000","last_clean":"2024-09-13T14:40:22.833307+0000","last_became_active":"2024-09-13T14:37:08.749158+0000","last_became_peered":"2024-09-13T14:37:08.749158+0000","last_unstale":"2024-09-13T14:40:22.833307+0000","last_undegraded":"2024-09-13T14:40:22.833307+0000","last_fullsized":"2024-09-13T14:40:22.833307+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:04.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967216+0000","last_change":"2024-09-13T14:37:08.758601+0000","last_active":"2024-09-13T14:40:22.967216+0000","last_peered":"2024-09-13T14:40:22.967216+0000","last_clean":"2024-09-13T14:40:22.967216+0000","last_became_active":"2024-09-13T14:37:08.758031+0000","last_became_peered":"2024-09-13T14:37:08.758031+0000","last_unstale":"2024-09-13T14:40:22.967216+0000","last_undegraded":"2024-09-13T14:40:22.967216+0000","last_fullsized":"2024-09-13T14:40:22.967216+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:12.269600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971366+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:40:22.971366+0000","last_peered":"2024-09-13T14:40:22.971366+0000","last_clean":"2024-09-13T14:40:22.971366+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:40:22.971366+0000","last_undegraded":"2024-09-13T14:40:22.971366+0000","last_fullsized":"2024-09-13T14:40:22.971366+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576345+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:40:21.576345+0000","last_peered":"2024-09-13T14:40:21.576345+0000","last_clean":"2024-09-13T14:40:21.576345+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:40:21.576345+0000","last_undegraded":"2024-09-13T14:40:21.576345+0000","last_fullsized":"2024-09-13T14:40:21.576345+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":158,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967841+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:40:22.967841+0000","last_peered":"2024-09-13T14:40:22.967841+0000","last_clean":"2024-09-13T14:40:22.967841+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:40:22.967841+0000","last_undegraded":"2024-09-13T14:40:22.967841+0000","last_fullsized":"2024-09-13T14:40:22.967841+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":162,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827359+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:40:22.827359+0000","last_peered":"2024-09-13T14:40:22.827359+0000","last_clean":"2024-09-13T14:40:22.827359+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:40:22.827359+0000","last_undegraded":"2024-09-13T14:40:22.827359+0000","last_fullsized":"2024-09-13T14:40:22.827359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":139,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576863+0000","last_change":"2024-09-13T14:37:08.749549+0000","last_active":"2024-09-13T14:40:21.576863+0000","last_peered":"2024-09-13T14:40:21.576863+0000","last_clean":"2024-09-13T14:40:21.576863+0000","last_became_active":"2024-09-13T14:37:08.749013+0000","last_became_peered":"2024-09-13T14:37:08.749013+0000","last_unstale":"2024-09-13T14:40:21.576863+0000","last_undegraded":"2024-09-13T14:40:21.576863+0000","last_fullsized":"2024-09-13T14:40:21.576863+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:35.239989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":114,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822005+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:40:20.822005+0000","last_peered":"2024-09-13T14:40:20.822005+0000","last_clean":"2024-09-13T14:40:20.822005+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:40:20.822005+0000","last_undegraded":"2024-09-13T14:40:20.822005+0000","last_fullsized":"2024-09-13T14:40:20.822005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576063+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:40:21.576063+0000","last_peered":"2024-09-13T14:40:21.576063+0000","last_clean":"2024-09-13T14:40:21.576063+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:40:21.576063+0000","last_undegraded":"2024-09-13T14:40:21.576063+0000","last_fullsized":"2024-09-13T14:40:21.576063+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":97,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822237+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:40:20.822237+0000","last_peered":"2024-09-13T14:40:20.822237+0000","last_clean":"2024-09-13T14:40:20.822237+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:40:20.822237+0000","last_undegraded":"2024-09-13T14:40:20.822237+0000","last_fullsized":"2024-09-13T14:40:20.822237+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":133,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970577+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:40:22.970577+0000","last_peered":"2024-09-13T14:40:22.970577+0000","last_clean":"2024-09-13T14:40:22.970577+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:40:22.970577+0000","last_undegraded":"2024-09-13T14:40:22.970577+0000","last_fullsized":"2024-09-13T14:40:22.970577+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971883+0000","last_change":"2024-09-13T14:37:08.750424+0000","last_active":"2024-09-13T14:40:22.971883+0000","last_peered":"2024-09-13T14:40:22.971883+0000","last_clean":"2024-09-13T14:40:22.971883+0000","last_became_active":"2024-09-13T14:37:08.749577+0000","last_became_peered":"2024-09-13T14:37:08.749577+0000","last_unstale":"2024-09-13T14:40:22.971883+0000","last_undegraded":"2024-09-13T14:40:22.971883+0000","last_fullsized":"2024-09-13T14:40:22.971883+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:16.435452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.575871+0000","last_change":"2024-09-13T14:32:09.484255+0000","last_active":"2024-09-13T14:40:21.575871+0000","last_peered":"2024-09-13T14:40:21.575871+0000","last_clean":"2024-09-13T14:40:21.575871+0000","last_became_active":"2024-09-13T14:32:09.483903+0000","last_became_peered":"2024-09-13T14:32:09.483903+0000","last_unstale":"2024-09-13T14:40:21.575871+0000","last_undegraded":"2024-09-13T14:40:21.575871+0000","last_fullsized":"2024-09-13T14:40:21.575871+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:41.931522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827366+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:40:22.827366+0000","last_peered":"2024-09-13T14:40:22.827366+0000","last_clean":"2024-09-13T14:40:22.827366+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:40:22.827366+0000","last_undegraded":"2024-09-13T14:40:22.827366+0000","last_fullsized":"2024-09-13T14:40:22.827366+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365058,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299648,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445504,"statfs":{"total":95995035648,"available":95688196096,"internally_reserved":0,"allocated":2338816,"data_stored":1595354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561288,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307952,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437200,"statfs":{"total":95995035648,"available":95679692800,"internally_reserved":0,"allocated":2330624,"data_stored":1595177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790220,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898592,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":896894,"kb_avail":92846560,"statfs":{"total":95995035648,"available":95074877440,"internally_reserved":0,"allocated":1724416,"data_stored":1000383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918419800},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986448,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595768,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93149384,"statfs":{"total":95995035648,"available":95384969216,"internally_reserved":0,"allocated":1744896,"data_stored":1005181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519603,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218528,"kb_used_data":1684,"kb_used_omap":4,"kb_used_meta":216827,"kb_avail":93526624,"statfs":{"total":95995035648,"available":95771262976,"internally_reserved":0,"allocated":1724416,"data_stored":1000393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":222031296},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304519,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230376,"kb_used_data":1688,"kb_used_omap":6,"kb_used_meta":228665,"kb_avail":93514776,"statfs":{"total":95995035648,"available":95759130624,"internally_reserved":0,"allocated":1728512,"data_stored":1000907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":234153415},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827823,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":389536,"kb_used_data":1684,"kb_used_omap":5,"kb_used_meta":387834,"kb_avail":93355616,"statfs":{"total":95995035648,"available":95596150784,"internally_reserved":0,"allocated":1724416,"data_stored":1000330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":397142252},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285687,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396520,"kb_used_data":2264,"kb_used_omap":5,"kb_used_meta":394234,"kb_avail":93348632,"statfs":{"total":95995035648,"available":95588999168,"internally_reserved":0,"allocated":2318336,"data_stored":1590035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":403695675},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:17.019 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:41:17.019 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:41:17.020 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:41:17.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:41:17.282 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:17.389 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:17 smithi003 ceph-mon[107278]: from='client.45788 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:17.389 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:17 smithi003 ceph-mon[104007]: from='client.45788 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:17.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:17 smithi019 ceph-mon[91296]: from='client.45788 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:18.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:18 smithi003 ceph-mon[107278]: pgmap v490: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:18.399 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:18 smithi003 ceph-mon[104007]: pgmap v490: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:18.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:18 smithi019 ceph-mon[91296]: pgmap v490: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:19.279 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:41:19.279 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:41:19.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:19 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1399056223' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:41:19.612 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:19 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1399056223' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:41:19.612 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:19 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1399056223' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:41:19.892 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:41:19.892 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:41:19.902 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-09-13T14:41:19.902 INFO:teuthology.run_tasks:Running task parallel... 2024-09-13T14:41:20.021 INFO:teuthology.task.parallel:starting parallel... 2024-09-13T14:41:20.021 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:41:20.022 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-09-13T14:41:20.025 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T14:41:20.025 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T14:41:20.025 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T14:41:20.025 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T14:41:20.025 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:41:20.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:20.297 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:20.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:20 smithi003 ceph-mon[107278]: pgmap v491: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:20.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:20 smithi003 ceph-mon[104007]: pgmap v491: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:20.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:20 smithi019 ceph-mon[91296]: pgmap v491: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:41:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:41:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:22.213 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:41:22.214 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:41:22.877 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:22 smithi003 ceph-mon[107278]: pgmap v492: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:22.877 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:22 smithi003 ceph-mon[104007]: pgmap v492: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:22.904 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-13T14:41:21.564186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":213,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3336920,"kb_used_data":15268,"kb_used_omap":29,"kb_used_meta":3321506,"kb_avail":746624296,"statfs":{"total":767960285184,"available":764543279104,"internally_reserved":0,"allocated":15634432,"data_stored":9787760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":3401222670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002882"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.972038+0000","last_change":"2024-09-13T14:37:08.782889+0000","last_active":"2024-09-13T14:40:22.972038+0000","last_peered":"2024-09-13T14:40:22.972038+0000","last_clean":"2024-09-13T14:40:22.972038+0000","last_became_active":"2024-09-13T14:37:08.781502+0000","last_became_peered":"2024-09-13T14:37:08.781502+0000","last_unstale":"2024-09-13T14:40:22.972038+0000","last_undegraded":"2024-09-13T14:40:22.972038+0000","last_fullsized":"2024-09-13T14:40:22.972038+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:22.769013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":116,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330969+0000","last_change":"2024-09-13T14:35:30.492165+0000","last_active":"2024-09-13T14:40:21.330969+0000","last_peered":"2024-09-13T14:40:21.330969+0000","last_clean":"2024-09-13T14:40:21.330969+0000","last_became_active":"2024-09-13T14:35:30.491844+0000","last_became_peered":"2024-09-13T14:35:30.491844+0000","last_unstale":"2024-09-13T14:40:21.330969+0000","last_undegraded":"2024-09-13T14:40:21.330969+0000","last_fullsized":"2024-09-13T14:40:21.330969+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:22.876890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":111,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822194+0000","last_change":"2024-09-13T14:37:08.755784+0000","last_active":"2024-09-13T14:40:20.822194+0000","last_peered":"2024-09-13T14:40:20.822194+0000","last_clean":"2024-09-13T14:40:20.822194+0000","last_became_active":"2024-09-13T14:37:08.755300+0000","last_became_peered":"2024-09-13T14:37:08.755300+0000","last_unstale":"2024-09-13T14:40:20.822194+0000","last_undegraded":"2024-09-13T14:40:20.822194+0000","last_fullsized":"2024-09-13T14:40:20.822194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:10.881654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827254+0000","last_change":"2024-09-13T14:37:08.748084+0000","last_active":"2024-09-13T14:40:22.827254+0000","last_peered":"2024-09-13T14:40:22.827254+0000","last_clean":"2024-09-13T14:40:22.827254+0000","last_became_active":"2024-09-13T14:37:08.746801+0000","last_became_peered":"2024-09-13T14:37:08.746801+0000","last_unstale":"2024-09-13T14:40:22.827254+0000","last_undegraded":"2024-09-13T14:40:22.827254+0000","last_fullsized":"2024-09-13T14:40:22.827254+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:13.495421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833418+0000","last_change":"2024-09-13T14:37:08.750965+0000","last_active":"2024-09-13T14:40:22.833418+0000","last_peered":"2024-09-13T14:40:22.833418+0000","last_clean":"2024-09-13T14:40:22.833418+0000","last_became_active":"2024-09-13T14:37:08.750129+0000","last_became_peered":"2024-09-13T14:37:08.750129+0000","last_unstale":"2024-09-13T14:40:22.833418+0000","last_undegraded":"2024-09-13T14:40:22.833418+0000","last_fullsized":"2024-09-13T14:40:22.833418+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:48.359985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":97,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330708+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:40:21.330708+0000","last_peered":"2024-09-13T14:40:21.330708+0000","last_clean":"2024-09-13T14:40:21.330708+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:40:21.330708+0000","last_undegraded":"2024-09-13T14:40:21.330708+0000","last_fullsized":"2024-09-13T14:40:21.330708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971111+0000","last_change":"2024-09-13T14:37:08.755312+0000","last_active":"2024-09-13T14:40:22.971111+0000","last_peered":"2024-09-13T14:40:22.971111+0000","last_clean":"2024-09-13T14:40:22.971111+0000","last_became_active":"2024-09-13T14:37:08.753186+0000","last_became_peered":"2024-09-13T14:37:08.753186+0000","last_unstale":"2024-09-13T14:40:22.971111+0000","last_undegraded":"2024-09-13T14:40:22.971111+0000","last_fullsized":"2024-09-13T14:40:22.971111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:57.687749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":121,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576521+0000","last_change":"2024-09-13T14:31:29.946716+0000","last_active":"2024-09-13T14:40:21.576521+0000","last_peered":"2024-09-13T14:40:21.576521+0000","last_clean":"2024-09-13T14:40:21.576521+0000","last_became_active":"2024-09-13T14:31:29.946366+0000","last_became_peered":"2024-09-13T14:31:29.946366+0000","last_unstale":"2024-09-13T14:40:21.576521+0000","last_undegraded":"2024-09-13T14:40:21.576521+0000","last_fullsized":"2024-09-13T14:40:21.576521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:29.686747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":115,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331328+0000","last_change":"2024-09-13T14:31:29.950983+0000","last_active":"2024-09-13T14:40:21.331328+0000","last_peered":"2024-09-13T14:40:21.331328+0000","last_clean":"2024-09-13T14:40:21.331328+0000","last_became_active":"2024-09-13T14:31:29.950612+0000","last_became_peered":"2024-09-13T14:31:29.950612+0000","last_unstale":"2024-09-13T14:40:21.331328+0000","last_undegraded":"2024-09-13T14:40:21.331328+0000","last_fullsized":"2024-09-13T14:40:21.331328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:35.833747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":94,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822196+0000","last_change":"2024-09-13T14:21:52.983025+0000","last_active":"2024-09-13T14:40:20.822196+0000","last_peered":"2024-09-13T14:40:20.822196+0000","last_clean":"2024-09-13T14:40:20.822196+0000","last_became_active":"2024-09-13T14:21:52.982611+0000","last_became_peered":"2024-09-13T14:21:52.982611+0000","last_unstale":"2024-09-13T14:40:20.822196+0000","last_undegraded":"2024-09-13T14:40:20.822196+0000","last_fullsized":"2024-09-13T14:40:20.822196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:00.851531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":116,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331353+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:40:21.331353+0000","last_peered":"2024-09-13T14:40:21.331353+0000","last_clean":"2024-09-13T14:40:21.331353+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:40:21.331353+0000","last_undegraded":"2024-09-13T14:40:21.331353+0000","last_fullsized":"2024-09-13T14:40:21.331353+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576188+0000","last_change":"2024-09-13T14:37:08.752050+0000","last_active":"2024-09-13T14:40:21.576188+0000","last_peered":"2024-09-13T14:40:21.576188+0000","last_clean":"2024-09-13T14:40:21.576188+0000","last_became_active":"2024-09-13T14:37:08.751650+0000","last_became_peered":"2024-09-13T14:37:08.751650+0000","last_unstale":"2024-09-13T14:40:21.576188+0000","last_undegraded":"2024-09-13T14:40:21.576188+0000","last_fullsized":"2024-09-13T14:40:21.576188+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:17.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":98,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331346+0000","last_change":"2024-09-13T14:21:52.610490+0000","last_active":"2024-09-13T14:40:21.331346+0000","last_peered":"2024-09-13T14:40:21.331346+0000","last_clean":"2024-09-13T14:40:21.331346+0000","last_became_active":"2024-09-13T14:21:52.610192+0000","last_became_peered":"2024-09-13T14:21:52.610192+0000","last_unstale":"2024-09-13T14:40:21.331346+0000","last_undegraded":"2024-09-13T14:40:21.331346+0000","last_fullsized":"2024-09-13T14:40:21.331346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:09.564393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":105,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576705+0000","last_change":"2024-09-13T14:22:34.901360+0000","last_active":"2024-09-13T14:40:21.576705+0000","last_peered":"2024-09-13T14:40:21.576705+0000","last_clean":"2024-09-13T14:40:21.576705+0000","last_became_active":"2024-09-13T14:22:34.901010+0000","last_became_peered":"2024-09-13T14:22:34.901010+0000","last_unstale":"2024-09-13T14:40:21.576705+0000","last_undegraded":"2024-09-13T14:40:21.576705+0000","last_fullsized":"2024-09-13T14:40:21.576705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:55.897004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822246+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:40:20.822246+0000","last_peered":"2024-09-13T14:40:20.822246+0000","last_clean":"2024-09-13T14:40:20.822246+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:40:20.822246+0000","last_undegraded":"2024-09-13T14:40:20.822246+0000","last_fullsized":"2024-09-13T14:40:20.822246+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827184+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:40:22.827184+0000","last_peered":"2024-09-13T14:40:22.827184+0000","last_clean":"2024-09-13T14:40:22.827184+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:40:22.827184+0000","last_undegraded":"2024-09-13T14:40:22.827184+0000","last_fullsized":"2024-09-13T14:40:22.827184+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970902+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:40:22.970902+0000","last_peered":"2024-09-13T14:40:22.970902+0000","last_clean":"2024-09-13T14:40:22.970902+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:40:22.970902+0000","last_undegraded":"2024-09-13T14:40:22.970902+0000","last_fullsized":"2024-09-13T14:40:22.970902+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576160+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:40:21.576160+0000","last_peered":"2024-09-13T14:40:21.576160+0000","last_clean":"2024-09-13T14:40:21.576160+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:40:21.576160+0000","last_undegraded":"2024-09-13T14:40:21.576160+0000","last_fullsized":"2024-09-13T14:40:21.576160+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576852+0000","last_change":"2024-09-13T14:32:09.491728+0000","last_active":"2024-09-13T14:40:21.576852+0000","last_peered":"2024-09-13T14:40:21.576852+0000","last_clean":"2024-09-13T14:40:21.576852+0000","last_became_active":"2024-09-13T14:32:09.490612+0000","last_became_peered":"2024-09-13T14:32:09.490612+0000","last_unstale":"2024-09-13T14:40:21.576852+0000","last_undegraded":"2024-09-13T14:40:21.576852+0000","last_fullsized":"2024-09-13T14:40:21.576852+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:21.947056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576605+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:40:21.576605+0000","last_peered":"2024-09-13T14:40:21.576605+0000","last_clean":"2024-09-13T14:40:21.576605+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:40:21.576605+0000","last_undegraded":"2024-09-13T14:40:21.576605+0000","last_fullsized":"2024-09-13T14:40:21.576605+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576740+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:40:21.576740+0000","last_peered":"2024-09-13T14:40:21.576740+0000","last_clean":"2024-09-13T14:40:21.576740+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:40:21.576740+0000","last_undegraded":"2024-09-13T14:40:21.576740+0000","last_fullsized":"2024-09-13T14:40:21.576740+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576503+0000","last_change":"2024-09-13T14:32:09.487930+0000","last_active":"2024-09-13T14:40:21.576503+0000","last_peered":"2024-09-13T14:40:21.576503+0000","last_clean":"2024-09-13T14:40:21.576503+0000","last_became_active":"2024-09-13T14:32:09.487592+0000","last_became_peered":"2024-09-13T14:32:09.487592+0000","last_unstale":"2024-09-13T14:40:21.576503+0000","last_undegraded":"2024-09-13T14:40:21.576503+0000","last_fullsized":"2024-09-13T14:40:21.576503+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:57.014876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":101,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970778+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:40:22.970778+0000","last_peered":"2024-09-13T14:40:22.970778+0000","last_clean":"2024-09-13T14:40:22.970778+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:40:22.970778+0000","last_undegraded":"2024-09-13T14:40:22.970778+0000","last_fullsized":"2024-09-13T14:40:22.970778+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.973048+0000","last_change":"2024-09-13T14:37:08.758818+0000","last_active":"2024-09-13T14:40:22.973048+0000","last_peered":"2024-09-13T14:40:22.973048+0000","last_clean":"2024-09-13T14:40:22.973048+0000","last_became_active":"2024-09-13T14:37:08.758225+0000","last_became_peered":"2024-09-13T14:37:08.758225+0000","last_unstale":"2024-09-13T14:40:22.973048+0000","last_undegraded":"2024-09-13T14:40:22.973048+0000","last_fullsized":"2024-09-13T14:40:22.973048+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:18.890207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":230,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330916+0000","last_change":"2024-09-13T14:31:29.950741+0000","last_active":"2024-09-13T14:40:21.330916+0000","last_peered":"2024-09-13T14:40:21.330916+0000","last_clean":"2024-09-13T14:40:21.330916+0000","last_became_active":"2024-09-13T14:31:29.950338+0000","last_became_peered":"2024-09-13T14:31:29.950338+0000","last_unstale":"2024-09-13T14:40:21.330916+0000","last_undegraded":"2024-09-13T14:40:21.330916+0000","last_fullsized":"2024-09-13T14:40:21.330916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:09.370711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":124,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.821932+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:40:20.821932+0000","last_peered":"2024-09-13T14:40:20.821932+0000","last_clean":"2024-09-13T14:40:20.821932+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:40:20.821932+0000","last_undegraded":"2024-09-13T14:40:20.821932+0000","last_fullsized":"2024-09-13T14:40:20.821932+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":91,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970711+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:40:22.970711+0000","last_peered":"2024-09-13T14:40:22.970711+0000","last_clean":"2024-09-13T14:40:22.970711+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:40:22.970711+0000","last_undegraded":"2024-09-13T14:40:22.970711+0000","last_fullsized":"2024-09-13T14:40:22.970711+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":95,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577239+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:40:21.577239+0000","last_peered":"2024-09-13T14:40:21.577239+0000","last_clean":"2024-09-13T14:40:21.577239+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:40:21.577239+0000","last_undegraded":"2024-09-13T14:40:21.577239+0000","last_fullsized":"2024-09-13T14:40:21.577239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":142,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330870+0000","last_change":"2024-09-13T14:32:09.482227+0000","last_active":"2024-09-13T14:40:21.330870+0000","last_peered":"2024-09-13T14:40:21.330870+0000","last_clean":"2024-09-13T14:40:21.330870+0000","last_became_active":"2024-09-13T14:32:09.481843+0000","last_became_peered":"2024-09-13T14:32:09.481843+0000","last_unstale":"2024-09-13T14:40:21.330870+0000","last_undegraded":"2024-09-13T14:40:21.330870+0000","last_fullsized":"2024-09-13T14:40:21.330870+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:56.929686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.828113+0000","last_change":"2024-09-13T14:37:08.748298+0000","last_active":"2024-09-13T14:40:22.828113+0000","last_peered":"2024-09-13T14:40:22.828113+0000","last_clean":"2024-09-13T14:40:22.828113+0000","last_became_active":"2024-09-13T14:37:08.747638+0000","last_became_peered":"2024-09-13T14:37:08.747638+0000","last_unstale":"2024-09-13T14:40:22.828113+0000","last_undegraded":"2024-09-13T14:40:22.828113+0000","last_fullsized":"2024-09-13T14:40:22.828113+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:46.778883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":182,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833211+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:40:22.833211+0000","last_peered":"2024-09-13T14:40:22.833211+0000","last_clean":"2024-09-13T14:40:22.833211+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:40:22.833211+0000","last_undegraded":"2024-09-13T14:40:22.833211+0000","last_fullsized":"2024-09-13T14:40:22.833211+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":158,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968128+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:40:22.968128+0000","last_peered":"2024-09-13T14:40:22.968128+0000","last_clean":"2024-09-13T14:40:22.968128+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:40:22.968128+0000","last_undegraded":"2024-09-13T14:40:22.968128+0000","last_fullsized":"2024-09-13T14:40:22.968128+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967202+0000","last_change":"2024-09-13T14:35:30.473739+0000","last_active":"2024-09-13T14:40:22.967202+0000","last_peered":"2024-09-13T14:40:22.967202+0000","last_clean":"2024-09-13T14:40:22.967202+0000","last_became_active":"2024-09-13T14:35:30.473448+0000","last_became_peered":"2024-09-13T14:35:30.473448+0000","last_unstale":"2024-09-13T14:40:22.967202+0000","last_undegraded":"2024-09-13T14:40:22.967202+0000","last_fullsized":"2024-09-13T14:40:22.967202+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:52.239598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":169,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.972926+0000","last_change":"2024-09-13T14:32:09.485178+0000","last_active":"2024-09-13T14:40:22.972926+0000","last_peered":"2024-09-13T14:40:22.972926+0000","last_clean":"2024-09-13T14:40:22.972926+0000","last_became_active":"2024-09-13T14:32:09.484869+0000","last_became_peered":"2024-09-13T14:32:09.484869+0000","last_unstale":"2024-09-13T14:40:22.972926+0000","last_undegraded":"2024-09-13T14:40:22.972926+0000","last_fullsized":"2024-09-13T14:40:22.972926+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:18.863308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.821990+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:40:20.821990+0000","last_peered":"2024-09-13T14:40:20.821990+0000","last_clean":"2024-09-13T14:40:20.821990+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:40:20.821990+0000","last_undegraded":"2024-09-13T14:40:20.821990+0000","last_fullsized":"2024-09-13T14:40:20.821990+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577044+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:40:21.577044+0000","last_peered":"2024-09-13T14:40:21.577044+0000","last_clean":"2024-09-13T14:40:21.577044+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:40:21.577044+0000","last_undegraded":"2024-09-13T14:40:21.577044+0000","last_fullsized":"2024-09-13T14:40:21.577044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":153,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968562+0000","last_change":"2024-09-13T14:37:08.794198+0000","last_active":"2024-09-13T14:40:22.968562+0000","last_peered":"2024-09-13T14:40:22.968562+0000","last_clean":"2024-09-13T14:40:22.968562+0000","last_became_active":"2024-09-13T14:37:08.793689+0000","last_became_peered":"2024-09-13T14:37:08.793689+0000","last_unstale":"2024-09-13T14:40:22.968562+0000","last_undegraded":"2024-09-13T14:40:22.968562+0000","last_fullsized":"2024-09-13T14:40:22.968562+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:19.865901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822288+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:40:20.822288+0000","last_peered":"2024-09-13T14:40:20.822288+0000","last_clean":"2024-09-13T14:40:20.822288+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:40:20.822288+0000","last_undegraded":"2024-09-13T14:40:20.822288+0000","last_fullsized":"2024-09-13T14:40:20.822288+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":181,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968754+0000","last_change":"2024-09-13T14:32:09.488979+0000","last_active":"2024-09-13T14:40:22.968754+0000","last_peered":"2024-09-13T14:40:22.968754+0000","last_clean":"2024-09-13T14:40:22.968754+0000","last_became_active":"2024-09-13T14:32:09.488320+0000","last_became_peered":"2024-09-13T14:32:09.488320+0000","last_unstale":"2024-09-13T14:40:22.968754+0000","last_undegraded":"2024-09-13T14:40:22.968754+0000","last_fullsized":"2024-09-13T14:40:22.968754+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:28.662506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969902+0000","last_change":"2024-09-13T14:32:09.489738+0000","last_active":"2024-09-13T14:40:22.969902+0000","last_peered":"2024-09-13T14:40:22.969902+0000","last_clean":"2024-09-13T14:40:22.969902+0000","last_became_active":"2024-09-13T14:32:09.489304+0000","last_became_peered":"2024-09-13T14:32:09.489304+0000","last_unstale":"2024-09-13T14:40:22.969902+0000","last_undegraded":"2024-09-13T14:40:22.969902+0000","last_fullsized":"2024-09-13T14:40:22.969902+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:16.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968664+0000","last_change":"2024-09-13T14:35:30.476616+0000","last_active":"2024-09-13T14:40:22.968664+0000","last_peered":"2024-09-13T14:40:22.968664+0000","last_clean":"2024-09-13T14:40:22.968664+0000","last_became_active":"2024-09-13T14:35:30.476323+0000","last_became_peered":"2024-09-13T14:35:30.476323+0000","last_unstale":"2024-09-13T14:40:22.968664+0000","last_undegraded":"2024-09-13T14:40:22.968664+0000","last_fullsized":"2024-09-13T14:40:22.968664+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:02.000336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331335+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:40:21.331335+0000","last_peered":"2024-09-13T14:40:21.331335+0000","last_clean":"2024-09-13T14:40:21.331335+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:40:21.331335+0000","last_undegraded":"2024-09-13T14:40:21.331335+0000","last_fullsized":"2024-09-13T14:40:21.331335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":160,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827259+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:40:22.827259+0000","last_peered":"2024-09-13T14:40:22.827259+0000","last_clean":"2024-09-13T14:40:22.827259+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:40:22.827259+0000","last_undegraded":"2024-09-13T14:40:22.827259+0000","last_fullsized":"2024-09-13T14:40:22.827259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971295+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:40:22.971295+0000","last_peered":"2024-09-13T14:40:22.971295+0000","last_clean":"2024-09-13T14:40:22.971295+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:40:22.971295+0000","last_undegraded":"2024-09-13T14:40:22.971295+0000","last_fullsized":"2024-09-13T14:40:22.971295+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969089+0000","last_change":"2024-09-13T14:32:09.490012+0000","last_active":"2024-09-13T14:40:22.969089+0000","last_peered":"2024-09-13T14:40:22.969089+0000","last_clean":"2024-09-13T14:40:22.969089+0000","last_became_active":"2024-09-13T14:32:09.489652+0000","last_became_peered":"2024-09-13T14:32:09.489652+0000","last_unstale":"2024-09-13T14:40:22.969089+0000","last_undegraded":"2024-09-13T14:40:22.969089+0000","last_fullsized":"2024-09-13T14:40:22.969089+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:10.587217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":129,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970677+0000","last_change":"2024-09-13T14:37:08.745773+0000","last_active":"2024-09-13T14:40:22.970677+0000","last_peered":"2024-09-13T14:40:22.970677+0000","last_clean":"2024-09-13T14:40:22.970677+0000","last_became_active":"2024-09-13T14:37:08.745130+0000","last_became_peered":"2024-09-13T14:37:08.745130+0000","last_unstale":"2024-09-13T14:40:22.970677+0000","last_undegraded":"2024-09-13T14:40:22.970677+0000","last_fullsized":"2024-09-13T14:40:22.970677+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:29:44.778588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833465+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:40:22.833465+0000","last_peered":"2024-09-13T14:40:22.833465+0000","last_clean":"2024-09-13T14:40:22.833465+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:40:22.833465+0000","last_undegraded":"2024-09-13T14:40:22.833465+0000","last_fullsized":"2024-09-13T14:40:22.833465+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577258+0000","last_change":"2024-09-13T14:35:30.479324+0000","last_active":"2024-09-13T14:40:21.577258+0000","last_peered":"2024-09-13T14:40:21.577258+0000","last_clean":"2024-09-13T14:40:21.577258+0000","last_became_active":"2024-09-13T14:35:30.479033+0000","last_became_peered":"2024-09-13T14:35:30.479033+0000","last_unstale":"2024-09-13T14:40:21.577258+0000","last_undegraded":"2024-09-13T14:40:21.577258+0000","last_fullsized":"2024-09-13T14:40:21.577258+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:57.606853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":95,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576646+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:40:21.576646+0000","last_peered":"2024-09-13T14:40:21.576646+0000","last_clean":"2024-09-13T14:40:21.576646+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:40:21.576646+0000","last_undegraded":"2024-09-13T14:40:21.576646+0000","last_fullsized":"2024-09-13T14:40:21.576646+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576956+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:40:21.576956+0000","last_peered":"2024-09-13T14:40:21.576956+0000","last_clean":"2024-09-13T14:40:21.576956+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:40:21.576956+0000","last_undegraded":"2024-09-13T14:40:21.576956+0000","last_fullsized":"2024-09-13T14:40:21.576956+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":119,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576296+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:40:21.576296+0000","last_peered":"2024-09-13T14:40:21.576296+0000","last_clean":"2024-09-13T14:40:21.576296+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:40:21.576296+0000","last_undegraded":"2024-09-13T14:40:21.576296+0000","last_fullsized":"2024-09-13T14:40:21.576296+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969738+0000","last_change":"2024-09-13T14:35:30.474794+0000","last_active":"2024-09-13T14:40:22.969738+0000","last_peered":"2024-09-13T14:40:22.969738+0000","last_clean":"2024-09-13T14:40:22.969738+0000","last_became_active":"2024-09-13T14:35:30.474501+0000","last_became_peered":"2024-09-13T14:35:30.474501+0000","last_unstale":"2024-09-13T14:40:22.969738+0000","last_undegraded":"2024-09-13T14:40:22.969738+0000","last_fullsized":"2024-09-13T14:40:22.969738+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:58.326635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":117,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331198+0000","last_change":"2024-09-13T14:37:08.756167+0000","last_active":"2024-09-13T14:40:21.331198+0000","last_peered":"2024-09-13T14:40:21.331198+0000","last_clean":"2024-09-13T14:40:21.331198+0000","last_became_active":"2024-09-13T14:37:08.755657+0000","last_became_peered":"2024-09-13T14:37:08.755657+0000","last_unstale":"2024-09-13T14:40:21.331198+0000","last_undegraded":"2024-09-13T14:40:21.331198+0000","last_fullsized":"2024-09-13T14:40:21.331198+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:20.322084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822309+0000","last_change":"2024-09-13T14:32:09.488238+0000","last_active":"2024-09-13T14:40:20.822309+0000","last_peered":"2024-09-13T14:40:20.822309+0000","last_clean":"2024-09-13T14:40:20.822309+0000","last_became_active":"2024-09-13T14:32:09.487750+0000","last_became_peered":"2024-09-13T14:32:09.487750+0000","last_unstale":"2024-09-13T14:40:20.822309+0000","last_undegraded":"2024-09-13T14:40:20.822309+0000","last_fullsized":"2024-09-13T14:40:20.822309+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:17.258720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":91,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970280+0000","last_change":"2024-09-13T14:37:08.747482+0000","last_active":"2024-09-13T14:40:22.970280+0000","last_peered":"2024-09-13T14:40:22.970280+0000","last_clean":"2024-09-13T14:40:22.970280+0000","last_became_active":"2024-09-13T14:37:08.747176+0000","last_became_peered":"2024-09-13T14:37:08.747176+0000","last_unstale":"2024-09-13T14:40:22.970280+0000","last_undegraded":"2024-09-13T14:40:22.970280+0000","last_fullsized":"2024-09-13T14:40:22.970280+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:18.885448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827804+0000","last_change":"2024-09-13T14:37:08.759355+0000","last_active":"2024-09-13T14:40:22.827804+0000","last_peered":"2024-09-13T14:40:22.827804+0000","last_clean":"2024-09-13T14:40:22.827804+0000","last_became_active":"2024-09-13T14:37:08.759058+0000","last_became_peered":"2024-09-13T14:37:08.759058+0000","last_unstale":"2024-09-13T14:40:22.827804+0000","last_undegraded":"2024-09-13T14:40:22.827804+0000","last_fullsized":"2024-09-13T14:40:22.827804+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:42.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.966945+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:40:22.966945+0000","last_peered":"2024-09-13T14:40:22.966945+0000","last_clean":"2024-09-13T14:40:22.966945+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:40:22.966945+0000","last_undegraded":"2024-09-13T14:40:22.966945+0000","last_fullsized":"2024-09-13T14:40:22.966945+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577265+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:40:21.577265+0000","last_peered":"2024-09-13T14:40:21.577265+0000","last_clean":"2024-09-13T14:40:21.577265+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:40:21.577265+0000","last_undegraded":"2024-09-13T14:40:21.577265+0000","last_fullsized":"2024-09-13T14:40:21.577265+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970673+0000","last_change":"2024-09-13T14:37:08.777939+0000","last_active":"2024-09-13T14:40:22.970673+0000","last_peered":"2024-09-13T14:40:22.970673+0000","last_clean":"2024-09-13T14:40:22.970673+0000","last_became_active":"2024-09-13T14:37:08.776339+0000","last_became_peered":"2024-09-13T14:37:08.776339+0000","last_unstale":"2024-09-13T14:40:22.970673+0000","last_undegraded":"2024-09-13T14:40:22.970673+0000","last_fullsized":"2024-09-13T14:40:22.970673+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:03:31.349551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833307+0000","last_change":"2024-09-13T14:37:08.749705+0000","last_active":"2024-09-13T14:40:22.833307+0000","last_peered":"2024-09-13T14:40:22.833307+0000","last_clean":"2024-09-13T14:40:22.833307+0000","last_became_active":"2024-09-13T14:37:08.749158+0000","last_became_peered":"2024-09-13T14:37:08.749158+0000","last_unstale":"2024-09-13T14:40:22.833307+0000","last_undegraded":"2024-09-13T14:40:22.833307+0000","last_fullsized":"2024-09-13T14:40:22.833307+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:04.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967216+0000","last_change":"2024-09-13T14:37:08.758601+0000","last_active":"2024-09-13T14:40:22.967216+0000","last_peered":"2024-09-13T14:40:22.967216+0000","last_clean":"2024-09-13T14:40:22.967216+0000","last_became_active":"2024-09-13T14:37:08.758031+0000","last_became_peered":"2024-09-13T14:37:08.758031+0000","last_unstale":"2024-09-13T14:40:22.967216+0000","last_undegraded":"2024-09-13T14:40:22.967216+0000","last_fullsized":"2024-09-13T14:40:22.967216+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:12.269600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971366+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:40:22.971366+0000","last_peered":"2024-09-13T14:40:22.971366+0000","last_clean":"2024-09-13T14:40:22.971366+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:40:22.971366+0000","last_undegraded":"2024-09-13T14:40:22.971366+0000","last_fullsized":"2024-09-13T14:40:22.971366+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576345+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:40:21.576345+0000","last_peered":"2024-09-13T14:40:21.576345+0000","last_clean":"2024-09-13T14:40:21.576345+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:40:21.576345+0000","last_undegraded":"2024-09-13T14:40:21.576345+0000","last_fullsized":"2024-09-13T14:40:21.576345+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":158,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967841+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:40:22.967841+0000","last_peered":"2024-09-13T14:40:22.967841+0000","last_clean":"2024-09-13T14:40:22.967841+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:40:22.967841+0000","last_undegraded":"2024-09-13T14:40:22.967841+0000","last_fullsized":"2024-09-13T14:40:22.967841+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":162,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827359+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:40:22.827359+0000","last_peered":"2024-09-13T14:40:22.827359+0000","last_clean":"2024-09-13T14:40:22.827359+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:40:22.827359+0000","last_undegraded":"2024-09-13T14:40:22.827359+0000","last_fullsized":"2024-09-13T14:40:22.827359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":139,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576863+0000","last_change":"2024-09-13T14:37:08.749549+0000","last_active":"2024-09-13T14:40:21.576863+0000","last_peered":"2024-09-13T14:40:21.576863+0000","last_clean":"2024-09-13T14:40:21.576863+0000","last_became_active":"2024-09-13T14:37:08.749013+0000","last_became_peered":"2024-09-13T14:37:08.749013+0000","last_unstale":"2024-09-13T14:40:21.576863+0000","last_undegraded":"2024-09-13T14:40:21.576863+0000","last_fullsized":"2024-09-13T14:40:21.576863+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:35.239989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":114,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822005+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:40:20.822005+0000","last_peered":"2024-09-13T14:40:20.822005+0000","last_clean":"2024-09-13T14:40:20.822005+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:40:20.822005+0000","last_undegraded":"2024-09-13T14:40:20.822005+0000","last_fullsized":"2024-09-13T14:40:20.822005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576063+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:40:21.576063+0000","last_peered":"2024-09-13T14:40:21.576063+0000","last_clean":"2024-09-13T14:40:21.576063+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:40:21.576063+0000","last_undegraded":"2024-09-13T14:40:21.576063+0000","last_fullsized":"2024-09-13T14:40:21.576063+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":97,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822237+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:40:20.822237+0000","last_peered":"2024-09-13T14:40:20.822237+0000","last_clean":"2024-09-13T14:40:20.822237+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:40:20.822237+0000","last_undegraded":"2024-09-13T14:40:20.822237+0000","last_fullsized":"2024-09-13T14:40:20.822237+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":133,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970577+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:40:22.970577+0000","last_peered":"2024-09-13T14:40:22.970577+0000","last_clean":"2024-09-13T14:40:22.970577+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:40:22.970577+0000","last_undegraded":"2024-09-13T14:40:22.970577+0000","last_fullsized":"2024-09-13T14:40:22.970577+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971883+0000","last_change":"2024-09-13T14:37:08.750424+0000","last_active":"2024-09-13T14:40:22.971883+0000","last_peered":"2024-09-13T14:40:22.971883+0000","last_clean":"2024-09-13T14:40:22.971883+0000","last_became_active":"2024-09-13T14:37:08.749577+0000","last_became_peered":"2024-09-13T14:37:08.749577+0000","last_unstale":"2024-09-13T14:40:22.971883+0000","last_undegraded":"2024-09-13T14:40:22.971883+0000","last_fullsized":"2024-09-13T14:40:22.971883+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:16.435452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.575871+0000","last_change":"2024-09-13T14:32:09.484255+0000","last_active":"2024-09-13T14:40:21.575871+0000","last_peered":"2024-09-13T14:40:21.575871+0000","last_clean":"2024-09-13T14:40:21.575871+0000","last_became_active":"2024-09-13T14:32:09.483903+0000","last_became_peered":"2024-09-13T14:32:09.483903+0000","last_unstale":"2024-09-13T14:40:21.575871+0000","last_undegraded":"2024-09-13T14:40:21.575871+0000","last_fullsized":"2024-09-13T14:40:21.575871+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:41.931522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827366+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:40:22.827366+0000","last_peered":"2024-09-13T14:40:22.827366+0000","last_clean":"2024-09-13T14:40:22.827366+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:40:22.827366+0000","last_undegraded":"2024-09-13T14:40:22.827366+0000","last_fullsized":"2024-09-13T14:40:22.827366+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365059,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299648,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445504,"statfs":{"total":95995035648,"available":95688196096,"internally_reserved":0,"allocated":2338816,"data_stored":1595354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561289,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307952,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437200,"statfs":{"total":95995035648,"available":95679692800,"internally_reserved":0,"allocated":2330624,"data_stored":1595177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790221,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898592,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":896894,"kb_avail":92846560,"statfs":{"total":95995035648,"available":95074877440,"internally_reserved":0,"allocated":1724416,"data_stored":1000383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918419800},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986449,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595768,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93149384,"statfs":{"total":95995035648,"available":95384969216,"internally_reserved":0,"allocated":1744896,"data_stored":1005181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519604,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218528,"kb_used_data":1684,"kb_used_omap":4,"kb_used_meta":216827,"kb_avail":93526624,"statfs":{"total":95995035648,"available":95771262976,"internally_reserved":0,"allocated":1724416,"data_stored":1000393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":222031296},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304520,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230376,"kb_used_data":1688,"kb_used_omap":6,"kb_used_meta":228665,"kb_avail":93514776,"statfs":{"total":95995035648,"available":95759130624,"internally_reserved":0,"allocated":1728512,"data_stored":1000907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":234153415},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827824,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":389536,"kb_used_data":1684,"kb_used_omap":5,"kb_used_meta":387834,"kb_avail":93355616,"statfs":{"total":95995035648,"available":95596150784,"internally_reserved":0,"allocated":1724416,"data_stored":1000330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":397142252},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285688,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396520,"kb_used_data":2264,"kb_used_omap":5,"kb_used_meta":394234,"kb_avail":93348632,"statfs":{"total":95995035648,"available":95588999168,"internally_reserved":0,"allocated":2318336,"data_stored":1590035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":403695675},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:22.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:22.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:22 smithi019 ceph-mon[91296]: pgmap v492: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:23.166 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:23.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:23 smithi019 ceph-mon[91296]: from='client.45800 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:23.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:23 smithi003 ceph-mon[107278]: from='client.45800 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:23.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:23 smithi003 ceph-mon[104007]: from='client.45800 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:24.393 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:41:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:41:24] "GET /metrics HTTP/1.1" 200 34046 "" "Prometheus/2.51.0" 2024-09-13T14:41:24.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:24 smithi003 ceph-mon[104007]: pgmap v493: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:24.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:24 smithi003 ceph-mon[107278]: pgmap v493: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:24.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:24 smithi019 ceph-mon[91296]: pgmap v493: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:25.078 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:41:25.079 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:41:25.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:25 smithi003 ceph-mon[107278]: from='client.45806 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:25.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:25 smithi003 ceph-mon[104007]: from='client.45806 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:25.737 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-13T14:41:23.564521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":213,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3336920,"kb_used_data":15268,"kb_used_omap":29,"kb_used_meta":3321506,"kb_avail":746624296,"statfs":{"total":767960285184,"available":764543279104,"internally_reserved":0,"allocated":15634432,"data_stored":9787760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":3401222670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002690"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.972038+0000","last_change":"2024-09-13T14:37:08.782889+0000","last_active":"2024-09-13T14:40:22.972038+0000","last_peered":"2024-09-13T14:40:22.972038+0000","last_clean":"2024-09-13T14:40:22.972038+0000","last_became_active":"2024-09-13T14:37:08.781502+0000","last_became_peered":"2024-09-13T14:37:08.781502+0000","last_unstale":"2024-09-13T14:40:22.972038+0000","last_undegraded":"2024-09-13T14:40:22.972038+0000","last_fullsized":"2024-09-13T14:40:22.972038+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:22.769013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":116,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330969+0000","last_change":"2024-09-13T14:35:30.492165+0000","last_active":"2024-09-13T14:40:21.330969+0000","last_peered":"2024-09-13T14:40:21.330969+0000","last_clean":"2024-09-13T14:40:21.330969+0000","last_became_active":"2024-09-13T14:35:30.491844+0000","last_became_peered":"2024-09-13T14:35:30.491844+0000","last_unstale":"2024-09-13T14:40:21.330969+0000","last_undegraded":"2024-09-13T14:40:21.330969+0000","last_fullsized":"2024-09-13T14:40:21.330969+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:22.876890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":111,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822194+0000","last_change":"2024-09-13T14:37:08.755784+0000","last_active":"2024-09-13T14:40:20.822194+0000","last_peered":"2024-09-13T14:40:20.822194+0000","last_clean":"2024-09-13T14:40:20.822194+0000","last_became_active":"2024-09-13T14:37:08.755300+0000","last_became_peered":"2024-09-13T14:37:08.755300+0000","last_unstale":"2024-09-13T14:40:20.822194+0000","last_undegraded":"2024-09-13T14:40:20.822194+0000","last_fullsized":"2024-09-13T14:40:20.822194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:10.881654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827254+0000","last_change":"2024-09-13T14:37:08.748084+0000","last_active":"2024-09-13T14:40:22.827254+0000","last_peered":"2024-09-13T14:40:22.827254+0000","last_clean":"2024-09-13T14:40:22.827254+0000","last_became_active":"2024-09-13T14:37:08.746801+0000","last_became_peered":"2024-09-13T14:37:08.746801+0000","last_unstale":"2024-09-13T14:40:22.827254+0000","last_undegraded":"2024-09-13T14:40:22.827254+0000","last_fullsized":"2024-09-13T14:40:22.827254+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:13.495421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833418+0000","last_change":"2024-09-13T14:37:08.750965+0000","last_active":"2024-09-13T14:40:22.833418+0000","last_peered":"2024-09-13T14:40:22.833418+0000","last_clean":"2024-09-13T14:40:22.833418+0000","last_became_active":"2024-09-13T14:37:08.750129+0000","last_became_peered":"2024-09-13T14:37:08.750129+0000","last_unstale":"2024-09-13T14:40:22.833418+0000","last_undegraded":"2024-09-13T14:40:22.833418+0000","last_fullsized":"2024-09-13T14:40:22.833418+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:48.359985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":97,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330708+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:40:21.330708+0000","last_peered":"2024-09-13T14:40:21.330708+0000","last_clean":"2024-09-13T14:40:21.330708+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:40:21.330708+0000","last_undegraded":"2024-09-13T14:40:21.330708+0000","last_fullsized":"2024-09-13T14:40:21.330708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971111+0000","last_change":"2024-09-13T14:37:08.755312+0000","last_active":"2024-09-13T14:40:22.971111+0000","last_peered":"2024-09-13T14:40:22.971111+0000","last_clean":"2024-09-13T14:40:22.971111+0000","last_became_active":"2024-09-13T14:37:08.753186+0000","last_became_peered":"2024-09-13T14:37:08.753186+0000","last_unstale":"2024-09-13T14:40:22.971111+0000","last_undegraded":"2024-09-13T14:40:22.971111+0000","last_fullsized":"2024-09-13T14:40:22.971111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:57.687749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":121,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576521+0000","last_change":"2024-09-13T14:31:29.946716+0000","last_active":"2024-09-13T14:40:21.576521+0000","last_peered":"2024-09-13T14:40:21.576521+0000","last_clean":"2024-09-13T14:40:21.576521+0000","last_became_active":"2024-09-13T14:31:29.946366+0000","last_became_peered":"2024-09-13T14:31:29.946366+0000","last_unstale":"2024-09-13T14:40:21.576521+0000","last_undegraded":"2024-09-13T14:40:21.576521+0000","last_fullsized":"2024-09-13T14:40:21.576521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:29.686747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":115,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331328+0000","last_change":"2024-09-13T14:31:29.950983+0000","last_active":"2024-09-13T14:40:21.331328+0000","last_peered":"2024-09-13T14:40:21.331328+0000","last_clean":"2024-09-13T14:40:21.331328+0000","last_became_active":"2024-09-13T14:31:29.950612+0000","last_became_peered":"2024-09-13T14:31:29.950612+0000","last_unstale":"2024-09-13T14:40:21.331328+0000","last_undegraded":"2024-09-13T14:40:21.331328+0000","last_fullsized":"2024-09-13T14:40:21.331328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:35.833747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":94,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822196+0000","last_change":"2024-09-13T14:21:52.983025+0000","last_active":"2024-09-13T14:40:20.822196+0000","last_peered":"2024-09-13T14:40:20.822196+0000","last_clean":"2024-09-13T14:40:20.822196+0000","last_became_active":"2024-09-13T14:21:52.982611+0000","last_became_peered":"2024-09-13T14:21:52.982611+0000","last_unstale":"2024-09-13T14:40:20.822196+0000","last_undegraded":"2024-09-13T14:40:20.822196+0000","last_fullsized":"2024-09-13T14:40:20.822196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:00.851531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":116,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331353+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:40:21.331353+0000","last_peered":"2024-09-13T14:40:21.331353+0000","last_clean":"2024-09-13T14:40:21.331353+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:40:21.331353+0000","last_undegraded":"2024-09-13T14:40:21.331353+0000","last_fullsized":"2024-09-13T14:40:21.331353+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576188+0000","last_change":"2024-09-13T14:37:08.752050+0000","last_active":"2024-09-13T14:40:21.576188+0000","last_peered":"2024-09-13T14:40:21.576188+0000","last_clean":"2024-09-13T14:40:21.576188+0000","last_became_active":"2024-09-13T14:37:08.751650+0000","last_became_peered":"2024-09-13T14:37:08.751650+0000","last_unstale":"2024-09-13T14:40:21.576188+0000","last_undegraded":"2024-09-13T14:40:21.576188+0000","last_fullsized":"2024-09-13T14:40:21.576188+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:17.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":98,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331346+0000","last_change":"2024-09-13T14:21:52.610490+0000","last_active":"2024-09-13T14:40:21.331346+0000","last_peered":"2024-09-13T14:40:21.331346+0000","last_clean":"2024-09-13T14:40:21.331346+0000","last_became_active":"2024-09-13T14:21:52.610192+0000","last_became_peered":"2024-09-13T14:21:52.610192+0000","last_unstale":"2024-09-13T14:40:21.331346+0000","last_undegraded":"2024-09-13T14:40:21.331346+0000","last_fullsized":"2024-09-13T14:40:21.331346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:09.564393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":105,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576705+0000","last_change":"2024-09-13T14:22:34.901360+0000","last_active":"2024-09-13T14:40:21.576705+0000","last_peered":"2024-09-13T14:40:21.576705+0000","last_clean":"2024-09-13T14:40:21.576705+0000","last_became_active":"2024-09-13T14:22:34.901010+0000","last_became_peered":"2024-09-13T14:22:34.901010+0000","last_unstale":"2024-09-13T14:40:21.576705+0000","last_undegraded":"2024-09-13T14:40:21.576705+0000","last_fullsized":"2024-09-13T14:40:21.576705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:55.897004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822246+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:40:20.822246+0000","last_peered":"2024-09-13T14:40:20.822246+0000","last_clean":"2024-09-13T14:40:20.822246+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:40:20.822246+0000","last_undegraded":"2024-09-13T14:40:20.822246+0000","last_fullsized":"2024-09-13T14:40:20.822246+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827184+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:40:22.827184+0000","last_peered":"2024-09-13T14:40:22.827184+0000","last_clean":"2024-09-13T14:40:22.827184+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:40:22.827184+0000","last_undegraded":"2024-09-13T14:40:22.827184+0000","last_fullsized":"2024-09-13T14:40:22.827184+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970902+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:40:22.970902+0000","last_peered":"2024-09-13T14:40:22.970902+0000","last_clean":"2024-09-13T14:40:22.970902+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:40:22.970902+0000","last_undegraded":"2024-09-13T14:40:22.970902+0000","last_fullsized":"2024-09-13T14:40:22.970902+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576160+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:40:21.576160+0000","last_peered":"2024-09-13T14:40:21.576160+0000","last_clean":"2024-09-13T14:40:21.576160+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:40:21.576160+0000","last_undegraded":"2024-09-13T14:40:21.576160+0000","last_fullsized":"2024-09-13T14:40:21.576160+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576852+0000","last_change":"2024-09-13T14:32:09.491728+0000","last_active":"2024-09-13T14:40:21.576852+0000","last_peered":"2024-09-13T14:40:21.576852+0000","last_clean":"2024-09-13T14:40:21.576852+0000","last_became_active":"2024-09-13T14:32:09.490612+0000","last_became_peered":"2024-09-13T14:32:09.490612+0000","last_unstale":"2024-09-13T14:40:21.576852+0000","last_undegraded":"2024-09-13T14:40:21.576852+0000","last_fullsized":"2024-09-13T14:40:21.576852+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:21.947056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576605+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:40:21.576605+0000","last_peered":"2024-09-13T14:40:21.576605+0000","last_clean":"2024-09-13T14:40:21.576605+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:40:21.576605+0000","last_undegraded":"2024-09-13T14:40:21.576605+0000","last_fullsized":"2024-09-13T14:40:21.576605+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576740+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:40:21.576740+0000","last_peered":"2024-09-13T14:40:21.576740+0000","last_clean":"2024-09-13T14:40:21.576740+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:40:21.576740+0000","last_undegraded":"2024-09-13T14:40:21.576740+0000","last_fullsized":"2024-09-13T14:40:21.576740+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576503+0000","last_change":"2024-09-13T14:32:09.487930+0000","last_active":"2024-09-13T14:40:21.576503+0000","last_peered":"2024-09-13T14:40:21.576503+0000","last_clean":"2024-09-13T14:40:21.576503+0000","last_became_active":"2024-09-13T14:32:09.487592+0000","last_became_peered":"2024-09-13T14:32:09.487592+0000","last_unstale":"2024-09-13T14:40:21.576503+0000","last_undegraded":"2024-09-13T14:40:21.576503+0000","last_fullsized":"2024-09-13T14:40:21.576503+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:57.014876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":101,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970778+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:40:22.970778+0000","last_peered":"2024-09-13T14:40:22.970778+0000","last_clean":"2024-09-13T14:40:22.970778+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:40:22.970778+0000","last_undegraded":"2024-09-13T14:40:22.970778+0000","last_fullsized":"2024-09-13T14:40:22.970778+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.973048+0000","last_change":"2024-09-13T14:37:08.758818+0000","last_active":"2024-09-13T14:40:22.973048+0000","last_peered":"2024-09-13T14:40:22.973048+0000","last_clean":"2024-09-13T14:40:22.973048+0000","last_became_active":"2024-09-13T14:37:08.758225+0000","last_became_peered":"2024-09-13T14:37:08.758225+0000","last_unstale":"2024-09-13T14:40:22.973048+0000","last_undegraded":"2024-09-13T14:40:22.973048+0000","last_fullsized":"2024-09-13T14:40:22.973048+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:18.890207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":230,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330916+0000","last_change":"2024-09-13T14:31:29.950741+0000","last_active":"2024-09-13T14:40:21.330916+0000","last_peered":"2024-09-13T14:40:21.330916+0000","last_clean":"2024-09-13T14:40:21.330916+0000","last_became_active":"2024-09-13T14:31:29.950338+0000","last_became_peered":"2024-09-13T14:31:29.950338+0000","last_unstale":"2024-09-13T14:40:21.330916+0000","last_undegraded":"2024-09-13T14:40:21.330916+0000","last_fullsized":"2024-09-13T14:40:21.330916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:09.370711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":124,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.821932+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:40:20.821932+0000","last_peered":"2024-09-13T14:40:20.821932+0000","last_clean":"2024-09-13T14:40:20.821932+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:40:20.821932+0000","last_undegraded":"2024-09-13T14:40:20.821932+0000","last_fullsized":"2024-09-13T14:40:20.821932+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":91,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970711+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:40:22.970711+0000","last_peered":"2024-09-13T14:40:22.970711+0000","last_clean":"2024-09-13T14:40:22.970711+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:40:22.970711+0000","last_undegraded":"2024-09-13T14:40:22.970711+0000","last_fullsized":"2024-09-13T14:40:22.970711+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":95,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577239+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:40:21.577239+0000","last_peered":"2024-09-13T14:40:21.577239+0000","last_clean":"2024-09-13T14:40:21.577239+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:40:21.577239+0000","last_undegraded":"2024-09-13T14:40:21.577239+0000","last_fullsized":"2024-09-13T14:40:21.577239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":142,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.330870+0000","last_change":"2024-09-13T14:32:09.482227+0000","last_active":"2024-09-13T14:40:21.330870+0000","last_peered":"2024-09-13T14:40:21.330870+0000","last_clean":"2024-09-13T14:40:21.330870+0000","last_became_active":"2024-09-13T14:32:09.481843+0000","last_became_peered":"2024-09-13T14:32:09.481843+0000","last_unstale":"2024-09-13T14:40:21.330870+0000","last_undegraded":"2024-09-13T14:40:21.330870+0000","last_fullsized":"2024-09-13T14:40:21.330870+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:56.929686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.828113+0000","last_change":"2024-09-13T14:37:08.748298+0000","last_active":"2024-09-13T14:40:22.828113+0000","last_peered":"2024-09-13T14:40:22.828113+0000","last_clean":"2024-09-13T14:40:22.828113+0000","last_became_active":"2024-09-13T14:37:08.747638+0000","last_became_peered":"2024-09-13T14:37:08.747638+0000","last_unstale":"2024-09-13T14:40:22.828113+0000","last_undegraded":"2024-09-13T14:40:22.828113+0000","last_fullsized":"2024-09-13T14:40:22.828113+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:46.778883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":182,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833211+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:40:22.833211+0000","last_peered":"2024-09-13T14:40:22.833211+0000","last_clean":"2024-09-13T14:40:22.833211+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:40:22.833211+0000","last_undegraded":"2024-09-13T14:40:22.833211+0000","last_fullsized":"2024-09-13T14:40:22.833211+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":158,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968128+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:40:22.968128+0000","last_peered":"2024-09-13T14:40:22.968128+0000","last_clean":"2024-09-13T14:40:22.968128+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:40:22.968128+0000","last_undegraded":"2024-09-13T14:40:22.968128+0000","last_fullsized":"2024-09-13T14:40:22.968128+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967202+0000","last_change":"2024-09-13T14:35:30.473739+0000","last_active":"2024-09-13T14:40:22.967202+0000","last_peered":"2024-09-13T14:40:22.967202+0000","last_clean":"2024-09-13T14:40:22.967202+0000","last_became_active":"2024-09-13T14:35:30.473448+0000","last_became_peered":"2024-09-13T14:35:30.473448+0000","last_unstale":"2024-09-13T14:40:22.967202+0000","last_undegraded":"2024-09-13T14:40:22.967202+0000","last_fullsized":"2024-09-13T14:40:22.967202+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:52.239598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":169,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.972926+0000","last_change":"2024-09-13T14:32:09.485178+0000","last_active":"2024-09-13T14:40:22.972926+0000","last_peered":"2024-09-13T14:40:22.972926+0000","last_clean":"2024-09-13T14:40:22.972926+0000","last_became_active":"2024-09-13T14:32:09.484869+0000","last_became_peered":"2024-09-13T14:32:09.484869+0000","last_unstale":"2024-09-13T14:40:22.972926+0000","last_undegraded":"2024-09-13T14:40:22.972926+0000","last_fullsized":"2024-09-13T14:40:22.972926+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:18.863308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.821990+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:40:20.821990+0000","last_peered":"2024-09-13T14:40:20.821990+0000","last_clean":"2024-09-13T14:40:20.821990+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:40:20.821990+0000","last_undegraded":"2024-09-13T14:40:20.821990+0000","last_fullsized":"2024-09-13T14:40:20.821990+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":113,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577044+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:40:21.577044+0000","last_peered":"2024-09-13T14:40:21.577044+0000","last_clean":"2024-09-13T14:40:21.577044+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:40:21.577044+0000","last_undegraded":"2024-09-13T14:40:21.577044+0000","last_fullsized":"2024-09-13T14:40:21.577044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":153,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968562+0000","last_change":"2024-09-13T14:37:08.794198+0000","last_active":"2024-09-13T14:40:22.968562+0000","last_peered":"2024-09-13T14:40:22.968562+0000","last_clean":"2024-09-13T14:40:22.968562+0000","last_became_active":"2024-09-13T14:37:08.793689+0000","last_became_peered":"2024-09-13T14:37:08.793689+0000","last_unstale":"2024-09-13T14:40:22.968562+0000","last_undegraded":"2024-09-13T14:40:22.968562+0000","last_fullsized":"2024-09-13T14:40:22.968562+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:19.865901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822288+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:40:20.822288+0000","last_peered":"2024-09-13T14:40:20.822288+0000","last_clean":"2024-09-13T14:40:20.822288+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:40:20.822288+0000","last_undegraded":"2024-09-13T14:40:20.822288+0000","last_fullsized":"2024-09-13T14:40:20.822288+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":181,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968754+0000","last_change":"2024-09-13T14:32:09.488979+0000","last_active":"2024-09-13T14:40:22.968754+0000","last_peered":"2024-09-13T14:40:22.968754+0000","last_clean":"2024-09-13T14:40:22.968754+0000","last_became_active":"2024-09-13T14:32:09.488320+0000","last_became_peered":"2024-09-13T14:32:09.488320+0000","last_unstale":"2024-09-13T14:40:22.968754+0000","last_undegraded":"2024-09-13T14:40:22.968754+0000","last_fullsized":"2024-09-13T14:40:22.968754+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:28.662506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969902+0000","last_change":"2024-09-13T14:32:09.489738+0000","last_active":"2024-09-13T14:40:22.969902+0000","last_peered":"2024-09-13T14:40:22.969902+0000","last_clean":"2024-09-13T14:40:22.969902+0000","last_became_active":"2024-09-13T14:32:09.489304+0000","last_became_peered":"2024-09-13T14:32:09.489304+0000","last_unstale":"2024-09-13T14:40:22.969902+0000","last_undegraded":"2024-09-13T14:40:22.969902+0000","last_fullsized":"2024-09-13T14:40:22.969902+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:16.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.968664+0000","last_change":"2024-09-13T14:35:30.476616+0000","last_active":"2024-09-13T14:40:22.968664+0000","last_peered":"2024-09-13T14:40:22.968664+0000","last_clean":"2024-09-13T14:40:22.968664+0000","last_became_active":"2024-09-13T14:35:30.476323+0000","last_became_peered":"2024-09-13T14:35:30.476323+0000","last_unstale":"2024-09-13T14:40:22.968664+0000","last_undegraded":"2024-09-13T14:40:22.968664+0000","last_fullsized":"2024-09-13T14:40:22.968664+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:02.000336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331335+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:40:21.331335+0000","last_peered":"2024-09-13T14:40:21.331335+0000","last_clean":"2024-09-13T14:40:21.331335+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:40:21.331335+0000","last_undegraded":"2024-09-13T14:40:21.331335+0000","last_fullsized":"2024-09-13T14:40:21.331335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":160,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827259+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:40:22.827259+0000","last_peered":"2024-09-13T14:40:22.827259+0000","last_clean":"2024-09-13T14:40:22.827259+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:40:22.827259+0000","last_undegraded":"2024-09-13T14:40:22.827259+0000","last_fullsized":"2024-09-13T14:40:22.827259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971295+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:40:22.971295+0000","last_peered":"2024-09-13T14:40:22.971295+0000","last_clean":"2024-09-13T14:40:22.971295+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:40:22.971295+0000","last_undegraded":"2024-09-13T14:40:22.971295+0000","last_fullsized":"2024-09-13T14:40:22.971295+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969089+0000","last_change":"2024-09-13T14:32:09.490012+0000","last_active":"2024-09-13T14:40:22.969089+0000","last_peered":"2024-09-13T14:40:22.969089+0000","last_clean":"2024-09-13T14:40:22.969089+0000","last_became_active":"2024-09-13T14:32:09.489652+0000","last_became_peered":"2024-09-13T14:32:09.489652+0000","last_unstale":"2024-09-13T14:40:22.969089+0000","last_undegraded":"2024-09-13T14:40:22.969089+0000","last_fullsized":"2024-09-13T14:40:22.969089+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:10.587217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":129,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970677+0000","last_change":"2024-09-13T14:37:08.745773+0000","last_active":"2024-09-13T14:40:22.970677+0000","last_peered":"2024-09-13T14:40:22.970677+0000","last_clean":"2024-09-13T14:40:22.970677+0000","last_became_active":"2024-09-13T14:37:08.745130+0000","last_became_peered":"2024-09-13T14:37:08.745130+0000","last_unstale":"2024-09-13T14:40:22.970677+0000","last_undegraded":"2024-09-13T14:40:22.970677+0000","last_fullsized":"2024-09-13T14:40:22.970677+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:29:44.778588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833465+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:40:22.833465+0000","last_peered":"2024-09-13T14:40:22.833465+0000","last_clean":"2024-09-13T14:40:22.833465+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:40:22.833465+0000","last_undegraded":"2024-09-13T14:40:22.833465+0000","last_fullsized":"2024-09-13T14:40:22.833465+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577258+0000","last_change":"2024-09-13T14:35:30.479324+0000","last_active":"2024-09-13T14:40:21.577258+0000","last_peered":"2024-09-13T14:40:21.577258+0000","last_clean":"2024-09-13T14:40:21.577258+0000","last_became_active":"2024-09-13T14:35:30.479033+0000","last_became_peered":"2024-09-13T14:35:30.479033+0000","last_unstale":"2024-09-13T14:40:21.577258+0000","last_undegraded":"2024-09-13T14:40:21.577258+0000","last_fullsized":"2024-09-13T14:40:21.577258+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:57.606853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":95,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576646+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:40:21.576646+0000","last_peered":"2024-09-13T14:40:21.576646+0000","last_clean":"2024-09-13T14:40:21.576646+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:40:21.576646+0000","last_undegraded":"2024-09-13T14:40:21.576646+0000","last_fullsized":"2024-09-13T14:40:21.576646+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576956+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:40:21.576956+0000","last_peered":"2024-09-13T14:40:21.576956+0000","last_clean":"2024-09-13T14:40:21.576956+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:40:21.576956+0000","last_undegraded":"2024-09-13T14:40:21.576956+0000","last_fullsized":"2024-09-13T14:40:21.576956+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":119,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576296+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:40:21.576296+0000","last_peered":"2024-09-13T14:40:21.576296+0000","last_clean":"2024-09-13T14:40:21.576296+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:40:21.576296+0000","last_undegraded":"2024-09-13T14:40:21.576296+0000","last_fullsized":"2024-09-13T14:40:21.576296+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.969738+0000","last_change":"2024-09-13T14:35:30.474794+0000","last_active":"2024-09-13T14:40:22.969738+0000","last_peered":"2024-09-13T14:40:22.969738+0000","last_clean":"2024-09-13T14:40:22.969738+0000","last_became_active":"2024-09-13T14:35:30.474501+0000","last_became_peered":"2024-09-13T14:35:30.474501+0000","last_unstale":"2024-09-13T14:40:22.969738+0000","last_undegraded":"2024-09-13T14:40:22.969738+0000","last_fullsized":"2024-09-13T14:40:22.969738+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:58.326635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":117,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.331198+0000","last_change":"2024-09-13T14:37:08.756167+0000","last_active":"2024-09-13T14:40:21.331198+0000","last_peered":"2024-09-13T14:40:21.331198+0000","last_clean":"2024-09-13T14:40:21.331198+0000","last_became_active":"2024-09-13T14:37:08.755657+0000","last_became_peered":"2024-09-13T14:37:08.755657+0000","last_unstale":"2024-09-13T14:40:21.331198+0000","last_undegraded":"2024-09-13T14:40:21.331198+0000","last_fullsized":"2024-09-13T14:40:21.331198+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:20.322084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822309+0000","last_change":"2024-09-13T14:32:09.488238+0000","last_active":"2024-09-13T14:40:20.822309+0000","last_peered":"2024-09-13T14:40:20.822309+0000","last_clean":"2024-09-13T14:40:20.822309+0000","last_became_active":"2024-09-13T14:32:09.487750+0000","last_became_peered":"2024-09-13T14:32:09.487750+0000","last_unstale":"2024-09-13T14:40:20.822309+0000","last_undegraded":"2024-09-13T14:40:20.822309+0000","last_fullsized":"2024-09-13T14:40:20.822309+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:17.258720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":91,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970280+0000","last_change":"2024-09-13T14:37:08.747482+0000","last_active":"2024-09-13T14:40:22.970280+0000","last_peered":"2024-09-13T14:40:22.970280+0000","last_clean":"2024-09-13T14:40:22.970280+0000","last_became_active":"2024-09-13T14:37:08.747176+0000","last_became_peered":"2024-09-13T14:37:08.747176+0000","last_unstale":"2024-09-13T14:40:22.970280+0000","last_undegraded":"2024-09-13T14:40:22.970280+0000","last_fullsized":"2024-09-13T14:40:22.970280+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:18.885448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827804+0000","last_change":"2024-09-13T14:37:08.759355+0000","last_active":"2024-09-13T14:40:22.827804+0000","last_peered":"2024-09-13T14:40:22.827804+0000","last_clean":"2024-09-13T14:40:22.827804+0000","last_became_active":"2024-09-13T14:37:08.759058+0000","last_became_peered":"2024-09-13T14:37:08.759058+0000","last_unstale":"2024-09-13T14:40:22.827804+0000","last_undegraded":"2024-09-13T14:40:22.827804+0000","last_fullsized":"2024-09-13T14:40:22.827804+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:42.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.966945+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:40:22.966945+0000","last_peered":"2024-09-13T14:40:22.966945+0000","last_clean":"2024-09-13T14:40:22.966945+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:40:22.966945+0000","last_undegraded":"2024-09-13T14:40:22.966945+0000","last_fullsized":"2024-09-13T14:40:22.966945+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.577265+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:40:21.577265+0000","last_peered":"2024-09-13T14:40:21.577265+0000","last_clean":"2024-09-13T14:40:21.577265+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:40:21.577265+0000","last_undegraded":"2024-09-13T14:40:21.577265+0000","last_fullsized":"2024-09-13T14:40:21.577265+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970673+0000","last_change":"2024-09-13T14:37:08.777939+0000","last_active":"2024-09-13T14:40:22.970673+0000","last_peered":"2024-09-13T14:40:22.970673+0000","last_clean":"2024-09-13T14:40:22.970673+0000","last_became_active":"2024-09-13T14:37:08.776339+0000","last_became_peered":"2024-09-13T14:37:08.776339+0000","last_unstale":"2024-09-13T14:40:22.970673+0000","last_undegraded":"2024-09-13T14:40:22.970673+0000","last_fullsized":"2024-09-13T14:40:22.970673+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:03:31.349551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":171,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.833307+0000","last_change":"2024-09-13T14:37:08.749705+0000","last_active":"2024-09-13T14:40:22.833307+0000","last_peered":"2024-09-13T14:40:22.833307+0000","last_clean":"2024-09-13T14:40:22.833307+0000","last_became_active":"2024-09-13T14:37:08.749158+0000","last_became_peered":"2024-09-13T14:37:08.749158+0000","last_unstale":"2024-09-13T14:40:22.833307+0000","last_undegraded":"2024-09-13T14:40:22.833307+0000","last_fullsized":"2024-09-13T14:40:22.833307+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:04.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967216+0000","last_change":"2024-09-13T14:37:08.758601+0000","last_active":"2024-09-13T14:40:22.967216+0000","last_peered":"2024-09-13T14:40:22.967216+0000","last_clean":"2024-09-13T14:40:22.967216+0000","last_became_active":"2024-09-13T14:37:08.758031+0000","last_became_peered":"2024-09-13T14:37:08.758031+0000","last_unstale":"2024-09-13T14:40:22.967216+0000","last_undegraded":"2024-09-13T14:40:22.967216+0000","last_fullsized":"2024-09-13T14:40:22.967216+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:12.269600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971366+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:40:22.971366+0000","last_peered":"2024-09-13T14:40:22.971366+0000","last_clean":"2024-09-13T14:40:22.971366+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:40:22.971366+0000","last_undegraded":"2024-09-13T14:40:22.971366+0000","last_fullsized":"2024-09-13T14:40:22.971366+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576345+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:40:21.576345+0000","last_peered":"2024-09-13T14:40:21.576345+0000","last_clean":"2024-09-13T14:40:21.576345+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:40:21.576345+0000","last_undegraded":"2024-09-13T14:40:21.576345+0000","last_fullsized":"2024-09-13T14:40:21.576345+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":158,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.967841+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:40:22.967841+0000","last_peered":"2024-09-13T14:40:22.967841+0000","last_clean":"2024-09-13T14:40:22.967841+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:40:22.967841+0000","last_undegraded":"2024-09-13T14:40:22.967841+0000","last_fullsized":"2024-09-13T14:40:22.967841+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":162,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827359+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:40:22.827359+0000","last_peered":"2024-09-13T14:40:22.827359+0000","last_clean":"2024-09-13T14:40:22.827359+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:40:22.827359+0000","last_undegraded":"2024-09-13T14:40:22.827359+0000","last_fullsized":"2024-09-13T14:40:22.827359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":139,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576863+0000","last_change":"2024-09-13T14:37:08.749549+0000","last_active":"2024-09-13T14:40:21.576863+0000","last_peered":"2024-09-13T14:40:21.576863+0000","last_clean":"2024-09-13T14:40:21.576863+0000","last_became_active":"2024-09-13T14:37:08.749013+0000","last_became_peered":"2024-09-13T14:37:08.749013+0000","last_unstale":"2024-09-13T14:40:21.576863+0000","last_undegraded":"2024-09-13T14:40:21.576863+0000","last_fullsized":"2024-09-13T14:40:21.576863+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:35.239989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":114,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822005+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:40:20.822005+0000","last_peered":"2024-09-13T14:40:20.822005+0000","last_clean":"2024-09-13T14:40:20.822005+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:40:20.822005+0000","last_undegraded":"2024-09-13T14:40:20.822005+0000","last_fullsized":"2024-09-13T14:40:20.822005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":129,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.576063+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:40:21.576063+0000","last_peered":"2024-09-13T14:40:21.576063+0000","last_clean":"2024-09-13T14:40:21.576063+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:40:21.576063+0000","last_undegraded":"2024-09-13T14:40:21.576063+0000","last_fullsized":"2024-09-13T14:40:21.576063+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":97,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:20.822237+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:40:20.822237+0000","last_peered":"2024-09-13T14:40:20.822237+0000","last_clean":"2024-09-13T14:40:20.822237+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:40:20.822237+0000","last_undegraded":"2024-09-13T14:40:20.822237+0000","last_fullsized":"2024-09-13T14:40:20.822237+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":133,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.970577+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:40:22.970577+0000","last_peered":"2024-09-13T14:40:22.970577+0000","last_clean":"2024-09-13T14:40:22.970577+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:40:22.970577+0000","last_undegraded":"2024-09-13T14:40:22.970577+0000","last_fullsized":"2024-09-13T14:40:22.970577+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.971883+0000","last_change":"2024-09-13T14:37:08.750424+0000","last_active":"2024-09-13T14:40:22.971883+0000","last_peered":"2024-09-13T14:40:22.971883+0000","last_clean":"2024-09-13T14:40:22.971883+0000","last_became_active":"2024-09-13T14:37:08.749577+0000","last_became_peered":"2024-09-13T14:37:08.749577+0000","last_unstale":"2024-09-13T14:40:22.971883+0000","last_undegraded":"2024-09-13T14:40:22.971883+0000","last_fullsized":"2024-09-13T14:40:22.971883+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:16.435452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":112,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:40:21.575871+0000","last_change":"2024-09-13T14:32:09.484255+0000","last_active":"2024-09-13T14:40:21.575871+0000","last_peered":"2024-09-13T14:40:21.575871+0000","last_clean":"2024-09-13T14:40:21.575871+0000","last_became_active":"2024-09-13T14:32:09.483903+0000","last_became_peered":"2024-09-13T14:32:09.483903+0000","last_unstale":"2024-09-13T14:40:21.575871+0000","last_undegraded":"2024-09-13T14:40:21.575871+0000","last_fullsized":"2024-09-13T14:40:21.575871+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:41.931522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:40:22.827366+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:40:22.827366+0000","last_peered":"2024-09-13T14:40:22.827366+0000","last_clean":"2024-09-13T14:40:22.827366+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:40:22.827366+0000","last_undegraded":"2024-09-13T14:40:22.827366+0000","last_fullsized":"2024-09-13T14:40:22.827366+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365059,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299648,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445504,"statfs":{"total":95995035648,"available":95688196096,"internally_reserved":0,"allocated":2338816,"data_stored":1595354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561289,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307952,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437200,"statfs":{"total":95995035648,"available":95679692800,"internally_reserved":0,"allocated":2330624,"data_stored":1595177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790221,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898592,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":896894,"kb_avail":92846560,"statfs":{"total":95995035648,"available":95074877440,"internally_reserved":0,"allocated":1724416,"data_stored":1000383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918419800},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986449,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595768,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93149384,"statfs":{"total":95995035648,"available":95384969216,"internally_reserved":0,"allocated":1744896,"data_stored":1005181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519605,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218528,"kb_used_data":1684,"kb_used_omap":4,"kb_used_meta":216827,"kb_avail":93526624,"statfs":{"total":95995035648,"available":95771262976,"internally_reserved":0,"allocated":1724416,"data_stored":1000393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":222031296},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304521,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230376,"kb_used_data":1688,"kb_used_omap":6,"kb_used_meta":228665,"kb_avail":93514776,"statfs":{"total":95995035648,"available":95759130624,"internally_reserved":0,"allocated":1728512,"data_stored":1000907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":234153415},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827824,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":389536,"kb_used_data":1684,"kb_used_omap":5,"kb_used_meta":387834,"kb_avail":93355616,"statfs":{"total":95995035648,"available":95596150784,"internally_reserved":0,"allocated":1724416,"data_stored":1000330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":397142252},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285688,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396520,"kb_used_data":2264,"kb_used_omap":5,"kb_used_meta":394234,"kb_avail":93348632,"statfs":{"total":95995035648,"available":95588999168,"internally_reserved":0,"allocated":2318336,"data_stored":1590035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":403695675},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:25.741 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:41:25.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-13T14:41:25.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:25 smithi019 ceph-mon[91296]: from='client.45806 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:25.999 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:26.800 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:26 smithi003 ceph-mon[107278]: pgmap v494: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:26.800 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:26 smithi003 ceph-mon[104007]: pgmap v494: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:26 smithi019 ceph-mon[91296]: pgmap v494: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:27.772 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 150 2024-09-13T14:41:27.772 INFO:teuthology.orchestra.run.smithi003.stdout:fsid 77a3f016-71da-11ef-bceb-c7b262605968 2024-09-13T14:41:27.772 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-09-13T14:15:00.204452+0000 2024-09-13T14:41:27.772 INFO:teuthology.orchestra.run.smithi003.stdout:modified 2024-09-13T14:40:22.823465+0000 2024-09-13T14:41:27.772 INFO:teuthology.orchestra.run.smithi003.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T14:41:27.772 INFO:teuthology.orchestra.run.smithi003.stdout:crush_version 22 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:full_ratio 0.95 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:backfillfull_ratio 0.9 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:nearfull_ratio 0.85 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:require_min_compat_client luminous 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:min_compat_client luminous 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:require_osd_release quincy 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:stretch_mode_enabled false 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 59 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-09-13T14:41:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:max_osd 8 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 up in weight 1 up_from 83 up_thru 142 down_at 80 last_clean_interval [8,79) [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808] [v2:172.21.15.3:6804/2281086808,v1:172.21.15.3:6805/2281086808] exists,up a153ec29-6e01-4063-aa90-01d1de00e53d 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 up in weight 1 up_from 97 up_thru 147 down_at 94 last_clean_interval [13,93) [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243] [v2:172.21.15.3:6812/3208418243,v1:172.21.15.3:6813/3208418243] exists,up 9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 up in weight 1 up_from 113 up_thru 137 down_at 108 last_clean_interval [18,107) [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691] [v2:172.21.15.3:6820/1260576691,v1:172.21.15.3:6821/1260576691] exists,up 676ec1e5-071c-47ba-86ae-2193f448fe8d 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 up in weight 1 up_from 137 up_thru 148 down_at 134 last_clean_interval [26,133) [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419] [v2:172.21.15.3:6828/1196691419,v1:172.21.15.3:6829/1196691419] exists,up 7ec0ad16-0df3-4319-b475-72cfaaa5edae 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 up in weight 1 up_from 31 up_thru 137 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131] [v2:172.21.15.19:6802/4204660131,v1:172.21.15.19:6803/4204660131] exists,up 0ef54d09-4000-4695-942b-1b1cf9659983 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 up in weight 1 up_from 37 up_thru 144 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104] [v2:172.21.15.19:6810/3591244104,v1:172.21.15.19:6811/3591244104] exists,up 6dfd8fce-4d3d-4a8b-90ac-556447045378 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 up in weight 1 up_from 44 up_thru 134 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215] [v2:172.21.15.19:6818/2162145215,v1:172.21.15.19:6819/2162145215] exists,up 17e27318-c0ce-413d-b660-d756731ea743 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 up in weight 1 up_from 50 up_thru 137 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175] [v2:172.21.15.19:6826/2814206175,v1:172.21.15.19:6827/2814206175] exists,up 1edab2d1-dad5-45c1-977c-6911541c282e 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.a [1,5] 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.17 [1,2] 2024-09-13T14:41:27.774 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.1b [1,2] 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2886196962 expires 2024-09-14T14:27:27.242023+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1330279387 expires 2024-09-14T14:27:27.242023+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2143031132 expires 2024-09-14T14:27:27.242023+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/3309089360 expires 2024-09-14T14:27:27.242023+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1361669764 expires 2024-09-14T14:27:27.242023+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/738145194 expires 2024-09-14T14:15:28.360446+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2893689034 expires 2024-09-14T14:24:43.914607+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1118901319 expires 2024-09-14T14:24:43.914607+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/3309089360 expires 2024-09-14T14:27:27.242023+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/70918773 expires 2024-09-14T14:24:43.914607+0000 2024-09-13T14:41:27.775 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1087248546 expires 2024-09-14T14:27:27.242023+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1863609487 expires 2024-09-14T14:24:43.914607+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.19:0/211301439 expires 2024-09-14T14:26:31.587235+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1018276975 expires 2024-09-14T14:20:33.329796+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.19:6834/2089501540 expires 2024-09-14T14:26:31.587235+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/253916983 expires 2024-09-14T14:15:52.081305+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3529413292 expires 2024-09-14T14:24:43.914607+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3505363388 expires 2024-09-14T14:15:28.360446+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.19:6835/2089501540 expires 2024-09-14T14:26:31.587235+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/70918773 expires 2024-09-14T14:24:43.914607+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.19:0/3762108460 expires 2024-09-14T14:26:31.587235+0000 2024-09-13T14:41:27.776 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/4142617802 expires 2024-09-14T14:15:52.081305+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.19:0/255629335 expires 2024-09-14T14:26:31.587235+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2346548909 expires 2024-09-14T14:20:33.329796+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3615653824 expires 2024-09-14T14:20:33.329796+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.19:0/1270823610 expires 2024-09-14T14:26:31.587235+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/4142617802 expires 2024-09-14T14:15:52.081305+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/2010213868 expires 2024-09-14T14:20:33.329796+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/931806668 expires 2024-09-14T14:15:28.360446+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/931806668 expires 2024-09-14T14:15:28.360446+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2723658197 expires 2024-09-14T14:15:52.081305+0000 2024-09-13T14:41:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1688297948 expires 2024-09-14T14:20:33.329796+0000 2024-09-13T14:41:27.778 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/416475212 expires 2024-09-14T14:27:27.242023+0000 2024-09-13T14:41:27.778 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/2010213868 expires 2024-09-14T14:20:33.329796+0000 2024-09-13T14:41:27.778 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2497820386 expires 2024-09-14T14:15:28.360446+0000 2024-09-13T14:41:27.778 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3631040762 expires 2024-09-14T14:15:52.081305+0000 2024-09-13T14:41:28.417 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 83 up_thru 142 down_at 80 last_clean_interval [8,79) [v2:172.21.15.3:6802/2281086808,v1:172.21.15.3:6803/2281086808] [v2:172.21.15.3:6804/2281086808,v1:172.21.15.3:6805/2281086808] exists,up a153ec29-6e01-4063-aa90-01d1de00e53d', 'osd.1 up in weight 1 up_from 97 up_thru 147 down_at 94 last_clean_interval [13,93) [v2:172.21.15.3:6810/3208418243,v1:172.21.15.3:6811/3208418243] [v2:172.21.15.3:6812/3208418243,v1:172.21.15.3:6813/3208418243] exists,up 9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70', 'osd.2 up in weight 1 up_from 113 up_thru 137 down_at 108 last_clean_interval [18,107) [v2:172.21.15.3:6818/1260576691,v1:172.21.15.3:6819/1260576691] [v2:172.21.15.3:6820/1260576691,v1:172.21.15.3:6821/1260576691] exists,up 676ec1e5-071c-47ba-86ae-2193f448fe8d', 'osd.3 up in weight 1 up_from 137 up_thru 148 down_at 134 last_clean_interval [26,133) [v2:172.21.15.3:6826/1196691419,v1:172.21.15.3:6827/1196691419] [v2:172.21.15.3:6828/1196691419,v1:172.21.15.3:6829/1196691419] exists,up 7ec0ad16-0df3-4319-b475-72cfaaa5edae', 'osd.4 up in weight 1 up_from 31 up_thru 137 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6800/4204660131,v1:172.21.15.19:6801/4204660131] [v2:172.21.15.19:6802/4204660131,v1:172.21.15.19:6803/4204660131] exists,up 0ef54d09-4000-4695-942b-1b1cf9659983', 'osd.5 up in weight 1 up_from 37 up_thru 144 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6808/3591244104,v1:172.21.15.19:6809/3591244104] [v2:172.21.15.19:6810/3591244104,v1:172.21.15.19:6811/3591244104] exists,up 6dfd8fce-4d3d-4a8b-90ac-556447045378', 'osd.6 up in weight 1 up_from 44 up_thru 134 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6816/2162145215,v1:172.21.15.19:6817/2162145215] [v2:172.21.15.19:6818/2162145215,v1:172.21.15.19:6819/2162145215] exists,up 17e27318-c0ce-413d-b660-d756731ea743', 'osd.7 up in weight 1 up_from 50 up_thru 137 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6824/2814206175,v1:172.21.15.19:6825/2814206175] [v2:172.21.15.19:6826/2814206175,v1:172.21.15.19:6827/2814206175] exists,up 1edab2d1-dad5-45c1-977c-6911541c282e'] 2024-09-13T14:41:28.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-13T14:41:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:28.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:28 smithi003 ceph-mon[107278]: pgmap v495: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:28.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:28.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1160482825' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:41:28.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:28 smithi003 ceph-mon[104007]: pgmap v495: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1160482825' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:41:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:28 smithi019 ceph-mon[91296]: pgmap v495: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1160482825' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:41:30.346 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:41:30.346 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mon.a", 2024-09-13T14:41:30.346 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster": "ceph", 2024-09-13T14:41:30.346 INFO:teuthology.orchestra.run.smithi003.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T14:41:30.346 INFO:teuthology.orchestra.run.smithi003.stdout: "admin_socket_mode": "", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_client_required": "cephx, none", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_cluster_required": "cephx", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_debug": "true", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_service_required": "cephx", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_supported": "", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio": "true", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_reap_max": "16", 2024-09-13T14:41:30.347 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_async_discard_threads": "0", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_block_size": "4096", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio": "false", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_enable_discard": "false", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_flock_retry": "3", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_inject_crash": "0", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring": "false", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring_hipri": "false", 2024-09-13T14:41:30.348 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_type": "", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_allocator": "hybrid", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_buffered_io": "true", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T14:41:30.349 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_replay_recovery": "false", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_sync_write": "false", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T14:41:30.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_allocator": "hybrid", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_async_db_compaction": "true", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T14:41:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_create": "true", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_create": "false", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_path": "", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_size": "0", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_path": "", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_size": "107374182400", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_create": "false", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_path": "", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs": "true", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T14:41:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_autotune": "true", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size": "0", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T14:41:30.353 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_type": "2q", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_clone_cow": "true", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T14:41:30.354 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_mode": "none", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_freelist": "false", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T14:41:30.355 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T14:41:30.356 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fail_eio": "false", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T14:41:30.357 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size": "0", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T14:41:30.359 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_mem": "512", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T14:41:30.360 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_tracing": "false", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T14:41:30.361 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_write_v2": "false", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_write_v2_random": "false", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T14:41:30.362 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_cluster_require_version": "2", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_require_signatures": "false", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_require_version": "2", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_service_require_signatures": "false", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_service_require_version": "2", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_sign_messages": "true", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "chdir": "", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client_acl_type": "", 2024-09-13T14:41:30.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client_asio_thread_count": "2", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_cache_mid": "0.750000", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_cache_size": "16384", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_caps_release_delay": "5", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_check_pool_perm": "true", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_force_sync_read": "false", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_getattr_caps": "false", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_inject_features": "", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_die_on_failed_remount": "false", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_dirsize_rbytes": "true", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_force_lazyio": "false", 2024-09-13T14:41:30.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_fs": "", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_inject_release_failure": "false", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_max_inline_size": "4096", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mds_namespace": "", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_metadata": "", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_gid": "-1", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_timeout": "300", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_uid": "-1", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mountpoint": "/", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_notify_timeout": "10", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc": "true", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T14:41:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_objects": "1000", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_size": "209715200", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_permissions": "true", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_quota": "true", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_quota_df": "true", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_max_bytes": "0", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_max_periods": "4", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_min": "131072", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_reconnect_stale": "false", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_shutdown_timeout": "30", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_snapdir": ".snap", 2024-09-13T14:41:30.366 INFO:teuthology.orchestra.run.smithi003.stdout: "client_tick_interval": "1", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "client_trace": "", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "client_use_faked_inos": "false", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "client_use_random_mds": "false", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog": "false", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog_port": "12201", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_monitors": "default=true", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog": "false", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog_level": "info", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "-", 2024-09-13T14:41:30.367 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_network": "", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_network_interface": "", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_isal": "false", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_level": "5", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zstd_level": "1", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T14:41:30.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location_hook": "", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location_hook_timeout": "10", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "daemonize": "false", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "daos_pool": "tank", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_alienstore": "0/5", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asok": "1/5", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asok_assert_abort": "false", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T14:41:30.369 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_auth": "1/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bdev": "1/3", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bluefs": "1/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bluestore": "1/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_buffer": "0/1", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ceph_dedup": "0/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cephsqlite": "0/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_client": "0/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_compressor": "1/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_context": "0/1", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crush": "1/1", 2024-09-13T14:41:30.370 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crypto": "1/5", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cyanstore": "0/5", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_dpdk": "1/5", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_eventtrace": "1/5", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_filer": "0/1", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_filestore": "1/3", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_finisher": "1/1", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_fuse": "1/5", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_javaclient": "1/5", 2024-09-13T14:41:30.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_journal": "1/3", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_journaler": "0/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_kstore": "1/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_lockdep": "0/1", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mclock": "1/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds": "1/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_balancer": "1/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_locker": "1/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_log": "1/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_migrator": "1/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_memstore": "1/5", 2024-09-13T14:41:30.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mgr": "2/5", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mgrc": "1/5", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mon": "20/20", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_monc": "0/10", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ms": "1/1", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_none": "0/5", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objclass": "0/5", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objectcacher": "0/5", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objecter": "0/1", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_optracker": "0/5", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_osd": "1/5", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_paxos": "20/20", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_perfcounter": "1/5", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_prioritycache": "1/5", 2024-09-13T14:41:30.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rados": "0/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd": "0/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_replay": "0/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_refs": "0/0", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_reserver": "1/1", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw": "1/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_access": "1/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_flight": "1/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T14:41:30.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_notification": "1/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_sync": "1/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rocksdb": "4/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_backref": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_cache": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_device": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_epm": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_journal": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_lba": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_odata": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_omap": "0/5", 2024-09-13T14:41:30.375 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_onode": "0/5", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_t": "0/5", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_tm": "0/5", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_striper": "0/1", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_test": "0/5", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_throttle": "1/1", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_timer": "0/1", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_tp": "0/5", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_trace": "1/5", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "device_failure_prediction_mode": "none", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_graylog": "false", 2024-09-13T14:41:30.376 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_journald": "false", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_stderr": "true", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_syslog": "false", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event_tracing": "false", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_cert_file": "", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_http_port": "9926", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_key_file": "", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_prio_limit": "5", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_sort_metrics": "true", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_stats_period": "5", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "fake_statfs_for_testing": "0", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "fatal_signal_handlers": "true", 2024-09-13T14:41:30.377 INFO:teuthology.orchestra.run.smithi003.stdout: "filer_max_purge_ops": "10", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filer_max_truncate_ops": "128", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_blackhole": "false", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_btrfs_snap": "true", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_caller_concurrency": "10", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_omap_check": "false", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_verify_split": "false", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_dump_file": "", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T14:41:30.378 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fadvise": "true", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fail_eio": "true", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fd_cache_size": "128", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fiemap": "false", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_inject_stall": "0", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_parallel": "false", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_trailing": "false", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_writeahead": "false", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_kill_at": "0", 2024-09-13T14:41:30.379 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T14:41:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_merge_threshold": "-10", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_odsync_write": "false", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_backend_path": "", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_threads": "2", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_punch_hole": "false", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T14:41:30.381 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_ops": "50", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_seek_data_hole": "false", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_sloppy_crc": "false", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_splice": "false", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_split_multiple": "2", 2024-09-13T14:41:30.382 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_split_rand_factor": "20", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_update_to": "1000", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T14:41:30.383 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_xfs_extsize": "false", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_zfs_snap": "false", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fio_dir": "/tmp/fio", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "77a3f016-71da-11ef-bceb-c7b262605968", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_allow_other": "true", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_big_writes": "true", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_debug": "false", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_default_permissions": "false", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_disable_pagecache": "false", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_max_write": "0", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_multithreaded": "true", 2024-09-13T14:41:30.384 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_require_active_mds": "true", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_set_user_groups": "true", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_move": "true", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_read": "true", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_write": "true", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "gss_target_name": "ceph", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_file": "", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_inject_failure": "0", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_interval": "5", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "host": "smithi003", 2024-09-13T14:41:30.385 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "inject_early_sigterm": "false", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "jaeger_agent_port": "6799", 2024-09-13T14:41:30.386 INFO:teuthology.orchestra.run.smithi003.stdout: "jaeger_tracing_enable": "false", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_aio": "true", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_align_min_size": "65536", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_block_align": "true", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_block_size": "4096", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_dio": "true", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_discard": "false", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_force_aio": "false", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_ignore_corruption": "false", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_max_write_entries": "100", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_replay_from": "0", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T14:41:30.387 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_write_header_frequency": "0", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_zero_on_create": "false", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_prefetch_periods": "10", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_prezero_periods": "5", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_write_head_interval": "15", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "keyfile": "", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_backend": "rocksdb", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T14:41:30.388 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_max_bytes": "67108864", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_max_ops": "512", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_onode_map_size": "1024", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_sync_transaction": "false", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "librados_thread_count": "2", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "lockdep": "false", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "lockdep_force_backtrace": "false", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "log_coarse_timestamps": "true", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "log_flush_on_exit": "false", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T14:41:30.389 INFO:teuthology.orchestra.run.smithi003.stdout: "log_graylog_port": "12201", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "log_max_new": "1000", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "log_max_recent": "10000", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "log_stderr_prefix": "", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": "true", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_graylog": "false", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_journald": "false", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_stderr": "false", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_syslog": "false", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_action_on_write_error": "1", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_alternate_name_max": "8192", 2024-09-13T14:41:30.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_asio_thread_count": "2", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_export_pin": "true", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_interval": "10", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_max": "-1", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_max_until": "-1", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_merge_size": "50", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T14:41:30.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_mode": "0", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_bits": "3", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_size": "10000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T14:41:30.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_mid": "0.700000", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_interval": "1", 2024-09-13T14:41:30.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_auth_pins": "false", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_frag": "false", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_scatterstat": "false", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_subtrees": "false", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T14:41:30.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_default_dir_hash": "2", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_defer_session_stale": "true", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_max_entries": "0", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_prefetch": "true", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_early_reply": "true", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_enable_op_tracker": "true", 2024-09-13T14:41:30.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_enforce_unique_name": "true", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T14:41:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_health_dummy": "false", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_join_fs": "", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_journal_format": "1", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_create_at": "0", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_dirfrag_at": "0", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_export_at": "0", 2024-09-13T14:41:30.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_import_at": "0", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_at": "0", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_link_at": "0", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_openc_at": "0", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_rename_at": "0", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_max_events": "-1", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_max_segments": "128", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_pause": "false", 2024-09-13T14:41:30.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_segment_size": "0", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_threshold": "128", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_completed_requests": "100000", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_export_size": "20971520", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_file_recover": "32", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_files": "64", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_ops": "8192", 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_metrics_update_interval": "2", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_min_caps_per_client": "100", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_numa_node": "-1", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_duration": "600", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_size": "20", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T14:41:30.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_log_threshold": "5", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_ping_grace": "15", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_ping_interval": "5", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_caps": "30000", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_replay_interval": "1.000000", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_root_ino_gid": "0", 2024-09-13T14:41:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_root_ino_uid": "0", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_shutdown_check": "0", 2024-09-13T14:41:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_skip_ino": "0", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_min_uid": "0", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_rstat": "false", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_symlink_recovery": "true", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_thrash_exports": "0", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_thrash_fragments": "0", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_tick_interval": "5.000000", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_valgrind_exit": "false", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_verify_backtrace": "1", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_verify_scatter": "false", 2024-09-13T14:41:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_wipe_sessions": "false", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool_debug": "false", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_page_set": "false", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_page_size": "65536", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_bytes": "134217728", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_messages": "512", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T14:41:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_disabled_modules": "", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_max_pg_creating": "1024", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mds_messages": "128", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mon_messages": "128", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_duration": "600", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_size": "20", 2024-09-13T14:41:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_log_threshold": "5", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_osd_messages": "8192", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_pool": "true", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_standby_modules": "true", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_stats_period": "5", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_stats_threshold": "5", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_tick_period": "2", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_allow_pool_delete": "true", 2024-09-13T14:41:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_allow_pool_size_one": "false", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_bytes": "104857600", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_directed_command_retry": "2", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T14:41:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_target_rank": "-1", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_level": "debug", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_journald": "true", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T14:41:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_start": "false", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_trim": "true", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cpu_threads": "4", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_avail_crit": "5", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_avail_warn": "5", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T14:41:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_json": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_extra_checks": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_down_mkfs_grace": "60", 2024-09-13T14:41:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_down_uptime_grace": "60", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_election_default_strategy": "1", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_election_timeout": "5.000000", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_enable_op_tracker": "true", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_fake_pool_delete": "false", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_force_quorum_join": "false", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_detail_to_clog": "true", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_log_update_period": "5", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_max_detail": "50", 2024-09-13T14:41:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog": "true", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_host": "[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_host_override": "", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_initial_members": "", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease": "5.000000", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T14:41:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_full_interval": "50", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_max": "10000", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_max_summary": "50", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_log_epochs": "500", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_osd": "10000", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T14:41:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_skip_sanity": "true", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_memory_autotune": "true", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_memory_target": "2147483648", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_digest_period": "5", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_min_osdmap_epochs": "500", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_complaint_time": "30", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_duration": "600", 2024-09-13T14:41:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_size": "20", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_log_threshold": "5", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_cache_size": "500", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T14:41:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T14:41:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T14:41:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T14:41:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T14:41:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T14:41:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_report_timeout": "900", 2024-09-13T14:41:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T14:41:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T14:41:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T14:41:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T14:41:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T14:41:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_max_osds": "4", 2024-09-13T14:41:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_interval": "86400", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_max_keys": "100", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_timeout": "300", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_session_timeout": "300", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_smart_report_timeout": "5", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T14:41:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_pool_size": "4", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_debug": "false", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_tick_interval": "5", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T14:41:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_older_version": "true", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T14:41:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": "", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T14:41:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_tracing_enabled": "false", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_op_threads": "3", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_cm": "false", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_device_name": "", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T14:41:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_sl": "3", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_type": "ib", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_reap_threshold": "5", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_before_connect": "false", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_ipv4": "true", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_ipv6": "false", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_msgr1": "true", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_msgr2": "true", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_port_max": "7568", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_port_min": "6800", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_retry_count": "3", 2024-09-13T14:41:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_retry_delay": "5", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_client": "false", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mds": "false", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mgr": "false", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mon": "false", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_osd": "false", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_client_mode": "crc secure", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_cluster_type": "", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_compress_secure": "false", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_crc_data": "true", 2024-09-13T14:41:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_crc_header": "true", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_bug": "true", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_old_message": "true", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hugepages": "", 2024-09-13T14:41:30.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_lro": "true", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_pmd": "", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_port_id": "0", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dump_on_send": "false", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_type": "", 2024-09-13T14:41:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_network_congestion": "0", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_socket_failures": "0", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_max_accept_failures": "4", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_max_backoff": "15.000000", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compress_mode": "none", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_pq_min_cost": "65536", 2024-09-13T14:41:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_public_type": "", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_service_mode": "crc secure", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_nodelay": "true", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_type": "async+posix", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "no_config_file": "false", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inflight_ops": "1024", 2024-09-13T14:41:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_timeout": "10.000000", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "objectstore_blackhole": "false", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "openssl_engine_opts": "", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_max_ops": "4", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_slop": "0.020000", 2024-09-13T14:41:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_asio_thread_count": "2", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_async_recovery_min_cost": "100", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_scan_max": "512", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_scan_min": "64", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_peering": "false", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_beacon_report_interval": "300", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_duration": "30", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T14:41:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_blkin_trace_all": "false", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_update_on_start": "true", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_message_cap": "256", 2024-09-13T14:41:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_op_priority": "63", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_watch_timeout": "30", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_command_max_records": "256", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_compact_on_start": "false", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_update_on_start": "true", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T14:41:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_op_order": "false", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_shutdown": "false", 2024-09-13T14:41:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_trim_objects": "false", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T14:41:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_default_notify_timeout": "30", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_ec_partial_reads": "true", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_enable_op_tracker": "true", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_info": "true", 2024-09-13T14:41:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown": "true", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_function_tracing": "false", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_grace": "20", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_interval": "6", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_stale": "600", 2024-09-13T14:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal_size": "5120", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_kill_backfill_at": "0", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_cache_size": "50", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_dedup": "true", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_message_max": "40", 2024-09-13T14:41:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_attr_name_len": "100", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_attr_size": "0", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_backfills": "1", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_markdown_count": "5", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_markdown_period": "600", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_name_len": "2048", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_size": "134217728", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T14:41:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pgls": "1024", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_push_cost": "8388608", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_push_objects": "10", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_scrubs": "3", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_write_size": "90", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T14:41:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_profile": "balanced", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_base": "805306368", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target": "4294967296", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target_autotune": "false", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_min_recovery_priority": "0", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_report_interval": "5", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T14:41:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_num_cache_shards": "32", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_node": "-1", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objecter_finishers": "1", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore": "bluestore", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_fuse": "false", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_tracing": "false", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T14:41:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_duration": "600", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_size": "20", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_log_threshold": "5", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards": "0", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T14:41:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_thread_timeout": "15", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_open_classes_on_start": "true", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_os_flags": "0", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_peering_op_priority": "255", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_delete_priority": "5", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T14:41:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_crimson": "false", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T14:41:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flags": "0", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_min_size": "0", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_size": "3", 2024-09-13T14:41:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_type": "replicated", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_cost": "20971520", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active": "0", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T14:41:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_op_priority": "3", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_priority": "5", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_repair_during_recovery": "false", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T14:41:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_cost": "52428800", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_end_hour": "0", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T14:41:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_priority": "5", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_skip_data_digest": "false", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_smart_report_timeout": "5", 2024-09-13T14:41:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_priority": "5", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_target_transaction_size": "30", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T14:41:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tracing": "false", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_use_stale_snap": "false", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_kill_at": "0", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_max_join_drift": "10", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_min": "500", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_min_wait": "0.050000", 2024-09-13T14:41:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_max": "500", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_min": "250", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_stash_full_interval": "25", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_trim_max": "500", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_trim_min": "250", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "perf": "true", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "pid_file": "", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "-", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrv": "", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "public_bind_addr": "-", 2024-09-13T14:41:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "public_network": "172.21.0.0/20", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "public_network_interface": "", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_enabled": "false", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_mon_op_timeout": "0", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_osd_op_timeout": "0", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_tracing": "false", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_atime_update_interval": "60", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T14:41:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache": "true", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_policy": "writearound", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_size": "33554432", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_compression_hint": "none", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T14:41:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_clone_format": "auto", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_data_pool": "", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_features": "61", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_format": "2", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_map_options": "", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_order": "22", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_pool": "rbd", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_stripe_count": "0", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T14:41:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_io_scheduler": "simple", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_order": "24", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_pool": "", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_splay_width": "4", 2024-09-13T14:41:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T14:41:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T14:41:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_op_threads": "1", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T14:41:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_plugins": "", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T14:41:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T14:41:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_tracing": "false", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_validate_names": "true", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_validate_pool": "true", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "restapi_base_url": "", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "restapi_log_level": "", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_admin_entry": "admin", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_asio_assert_yielding": "false", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_backend_store": "rados", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_barbican_url": "", 2024-09-13T14:41:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_beast_enable_async": "true", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_enabled": "true", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_config_store": "rados", 2024-09-13T14:41:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_content_length_compat": "false", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cross_domain_policy": "", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T14:41:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T14:41:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T14:41:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-13T14:41:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_window": "30", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T14:41:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_disable_s3select": "false", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T14:41:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dns_name": "", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dns_s3website_name": "", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T14:41:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_mdsearch": "true", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_ops_log": "false", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_static_website": "false", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_usage_log": "false", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_expose_bucket": "false", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_extended_http_attrs": "", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_filter": "none", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T14:41:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_objs": "32", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_init_timeout": "300", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T14:41:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_password": "", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_project": "", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_token": "", 2024-09-13T14:41:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_user": "", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_url": "", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T14:41:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_objs": "32", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_worker": "3", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_uri": "", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-13T14:41:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_http_headers": "", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attr_size": "0", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T14:41:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_listing_results": "1000", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_notify_retries": "10", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T14:41:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T14:41:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_num_control_oids": "8", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_numa_node": "-1", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_tracing": "false", 2024-09-13T14:41:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_token": "", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_url": "", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_rados": "false", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T14:41:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_print_continue": "true", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_tracing": "false", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm": "", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm_id": "", 2024-09-13T14:41:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_region": "", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_request_uri": "", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_resolve_cname": "false", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T14:41:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_run_sync_thread": "true", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_script_uri": "", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_service_provider_name": "", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T14:41:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_client_id": "", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_client_secret": "", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_entry": "sts", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_key": "sts", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_auth_url": "", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_custom_header": "", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_need_stats": "true", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_tenant_name": "", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T14:41:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_url": "", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_lease_period": "120", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T14:41:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_thread_pool_size": "512", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_comment": "", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_createby": "", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_encoding": "", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_flag": "false", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_origin": "", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T14:41:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_tracker": "", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_max_shards": "32", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_use_opa_authz": "false", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_counters_cache": "false", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T14:41:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_unique_email": "true", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_verify_ssl": "true", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone": "", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone_id": "", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup": "", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup_id": "", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_block_size": "4096", 2024-09-13T14:41:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T14:41:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_partition_filters": "false", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_perf": "false", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "run_dir": "/var/run/ceph", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_block_create": "true", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T14:41:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_device_size": "53687091200", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_full_integrity_check": "false", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_main_device_type": "SSD", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T14:41:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_segment_size": "67108864", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "set_keepcaps": "false", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "setgroup": "ceph", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "setuser": "ceph", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "setuser_match_path": "", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "thp": "false", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "threadpool_default_timeout": "60", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "throttler_perf_counter": "true", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "tmp_dir": "/tmp", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "uadk_compressor_enabled": "false", 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-13T14:41:30.495 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:41:30.636 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:41:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:41:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:30.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:30 smithi019 ceph-mon[91296]: pgmap v496: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:30.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:30 smithi003 ceph-mon[104007]: pgmap v496: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:30.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:30 smithi003 ceph-mon[107278]: pgmap v496: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:31.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T14:41:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:32.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:32 smithi003 ceph-mon[107278]: pgmap v497: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:32.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:32 smithi003 ceph-mon[104007]: pgmap v497: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:32.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:32 smithi019 ceph-mon[91296]: pgmap v497: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:33.019 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a: {} 2024-09-13T14:41:33.019 INFO:teuthology.orchestra.run.smithi003.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:41:33.029 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b: {} 2024-09-13T14:41:33.029 INFO:teuthology.orchestra.run.smithi003.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:41:33.038 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c: {} 2024-09-13T14:41:33.038 INFO:teuthology.orchestra.run.smithi003.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:41:33.703 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-09-13T14:41:33.703 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-13T14:41:33.703 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:41:33.704 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:41:33.704 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-13T14:41:33.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-13T14:41:33.705 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T14:41:33.705 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:41:33.705 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:41:33.705 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T14:41:33.706 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:33.706 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T14:41:33.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:41:33.707 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T14:41:33.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:41:33.707 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T14:41:33.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:41:33.741 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:33.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:33.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:33.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:33 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/400847624' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:41:33.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:33 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/400847624' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:41:33.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:33 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/400847624' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:41:33.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:34.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:34.049 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:34.050 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:34.050 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:34.056 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:34.109 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.137 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:34.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:34.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:34.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:34.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:34.367 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:41:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:41:34] "GET /metrics HTTP/1.1" 200 34048 "" "Prometheus/2.51.0" 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:34.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:34.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:34.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:34.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:34.742 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:34.769 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:34.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:34.898 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:34 smithi019 ceph-mon[91296]: pgmap v498: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:34.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:34 smithi003 ceph-mon[107278]: pgmap v498: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:34.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:34 smithi003 ceph-mon[104007]: pgmap v498: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:35.000 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:35.026 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:35.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:35.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:35.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:35.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:35.385 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:35.412 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:35.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:35.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:35.539 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:35.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:35.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:35.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:35.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:35.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:36.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:36.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:36.148 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:36.175 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:36.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:36.303 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:36.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:36.428 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:36.530 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:36.557 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:36.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:36.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:36.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:36 smithi003 ceph-mon[107278]: pgmap v499: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:36.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:36 smithi003 ceph-mon[104007]: pgmap v499: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:36.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:36.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:36.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:36.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:36.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:36 smithi019 ceph-mon[91296]: pgmap v499: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:37.032 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:37.059 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:37.161 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:37.188 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:37.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:37.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:37.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:37.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:37.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:37.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:37.656 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:41:37.664 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:37.689 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:37.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:37.798 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1867705442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:37.798 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1867705442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:37.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:37.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:37.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1867705442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:37.944 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:38.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:38.172 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:38.200 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:38.302 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:38.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:38.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:38.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:38.556 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.585 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:38.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:38.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:38 smithi003 ceph-mon[107278]: pgmap v500: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:38.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:38 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:38.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1867705442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:38.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:38 smithi003 ceph-mon[107278]: osdmap e151: 8 total, 8 up, 8 in 2024-09-13T14:41:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:38 smithi003 ceph-mon[104007]: pgmap v500: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:38 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1867705442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:38 smithi003 ceph-mon[104007]: osdmap e151: 8 total, 8 up, 8 in 2024-09-13T14:41:38.687 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:38.713 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:38.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:38.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:38 smithi019 ceph-mon[91296]: pgmap v500: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:38.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:38 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1867705442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:38 smithi019 ceph-mon[91296]: osdmap e151: 8 total, 8 up, 8 in 2024-09-13T14:41:38.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:38.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:39.067 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:39.092 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:39.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:39.221 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:39.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:39.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:39.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:39.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:39.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:39.603 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:39.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:39.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:39.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:39.858 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:39.960 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:39.987 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:40.089 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:40.116 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:40.218 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:40.244 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:40.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:40.372 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:40.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:41:40.474 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:40.499 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:40.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:40.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:40.660 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:41:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:41:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:40.674 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_4' created 2024-09-13T14:41:40.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:40.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:40.862 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:41:40.863 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:40.863 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:40 smithi003 ceph-mon[104007]: pgmap v502: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:40.863 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:40 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1683978123' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:41:40.863 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:40 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:41:40.863 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:40 smithi003 ceph-mon[107278]: pgmap v502: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:40.863 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:40 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1683978123' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:41:40.863 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:40 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:41:40.864 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:41:40.864 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:40.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:40.882 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:41:40.882 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:40.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:40.898 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:41:40.898 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:40.914 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:41:40.915 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:40.931 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:41:40.931 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:40 smithi019 ceph-mon[91296]: pgmap v502: 73 pgs: 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:40 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1683978123' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:41:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:40 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:41:40.947 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:41:40.947 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:40.964 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:41:40.965 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:40.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:41.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:41.063 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:41.063 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:41.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:41.142 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:41.244 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:41.270 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:41.372 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.400 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:41.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:41.501 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:41.528 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:41.630 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:41.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:41.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:41.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:41.883 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:41.910 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:41.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:41 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T14:41:41.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:41 smithi019 ceph-mon[91296]: osdmap e152: 8 total, 8 up, 8 in 2024-09-13T14:41:41.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:41 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2460197044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:41.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:41 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T14:41:41.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:41 smithi003 ceph-mon[107278]: osdmap e152: 8 total, 8 up, 8 in 2024-09-13T14:41:41.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:41 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2460197044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:41.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:41 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T14:41:41.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:41 smithi003 ceph-mon[104007]: osdmap e152: 8 total, 8 up, 8 in 2024-09-13T14:41:41.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:41 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2460197044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:42.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:42.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:41:42.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:42.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:42.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:42.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:42.312 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:42.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:42.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:42.539 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:42.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:42.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:42.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:42.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:42.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:42.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:42.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:42 smithi019 ceph-mon[91296]: pgmap v504: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:42 smithi019 ceph-mon[91296]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:41:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:42 smithi019 ceph-mon[91296]: osdmap e153: 8 total, 8 up, 8 in 2024-09-13T14:41:42.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:42 smithi003 ceph-mon[104007]: pgmap v504: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:42.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:42 smithi003 ceph-mon[104007]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:41:42.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:42 smithi003 ceph-mon[104007]: osdmap e153: 8 total, 8 up, 8 in 2024-09-13T14:41:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:42 smithi003 ceph-mon[107278]: pgmap v504: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:42 smithi003 ceph-mon[107278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:41:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:42 smithi003 ceph-mon[107278]: osdmap e153: 8 total, 8 up, 8 in 2024-09-13T14:41:42.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:43.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-13T14:41:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:43.017 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:43.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:43.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.071 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:43.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:43.172 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:43.197 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:43.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:43.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:43.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:43.425 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:43.452 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:43.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.581 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:43.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:43.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:43.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:43.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:43.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:43.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:43.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:43 smithi003 ceph-mon[104007]: osdmap e154: 8 total, 8 up, 8 in 2024-09-13T14:41:43.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:43.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:43.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:43 smithi003 ceph-mon[107278]: osdmap e154: 8 total, 8 up, 8 in 2024-09-13T14:41:43.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:43.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:43.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:44.054 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:44.081 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:44.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:43 smithi019 ceph-mon[91296]: osdmap e154: 8 total, 8 up, 8 in 2024-09-13T14:41:44.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:44.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:44.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:44.209 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:44.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:44.311 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:44.338 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:41:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:41:44] "GET /metrics HTTP/1.1" 200 36401 "" "Prometheus/2.51.0" 2024-09-13T14:41:44.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:44.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:44.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:41:44.565 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:44.592 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:44.694 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:44.720 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:44.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:44 smithi003 ceph-mon[107278]: pgmap v507: 89 pgs: 12 unknown, 77 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:44.789 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:44 smithi003 ceph-mon[104007]: pgmap v507: 89 pgs: 12 unknown, 77 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:44.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:44.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:44.948 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:44.975 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:44.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:45.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:45.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:45.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:44 smithi019 ceph-mon[91296]: pgmap v507: 89 pgs: 12 unknown, 77 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:45.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:45.228 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:45.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:45.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:45.439 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:45.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:45.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:45.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.612 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:45.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:45.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:45.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:45.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:45.839 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:45.865 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:45.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:45.967 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:45.994 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:46.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:46.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:46.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:46.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:46.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:46.372 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:46.474 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:46.501 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:46.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:46.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:46.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:46.775 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:41:46.853 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:46.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:46 smithi003 ceph-mon[107278]: pgmap v508: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:46.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:46 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1696357335' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:46.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:46 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:46.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:46 smithi003 ceph-mon[104007]: pgmap v508: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:46.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:46 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1696357335' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:46.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:46 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:46.880 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:46.982 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:47.008 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:47.110 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:47.135 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:47.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:46 smithi019 ceph-mon[91296]: pgmap v508: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:47.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:46 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1696357335' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:46 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:47.237 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:47.264 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:47.366 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:47.392 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:47.494 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:47.521 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:47.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:47.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:47.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:47.775 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:47.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:47.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:47.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:48.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:48.029 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:48.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:47 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:48.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:47 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:48.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:47 smithi003 ceph-mon[104007]: osdmap e155: 8 total, 8 up, 8 in 2024-09-13T14:41:48.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:47 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:48.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:47 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:48.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:47 smithi003 ceph-mon[107278]: osdmap e155: 8 total, 8 up, 8 in 2024-09-13T14:41:48.132 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:48.159 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:48.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:47 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:48.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:47 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:48.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:47 smithi019 ceph-mon[91296]: osdmap e155: 8 total, 8 up, 8 in 2024-09-13T14:41:48.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:48.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:48.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:48.515 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:48.542 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:48.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:48.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:48.770 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:48.797 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:48.899 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:48.925 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:48.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:48 smithi003 ceph-mon[107278]: pgmap v510: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:48.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:48 smithi003 ceph-mon[104007]: pgmap v510: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:49.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:49.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:49.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:49.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:49.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:48 smithi019 ceph-mon[91296]: pgmap v510: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:49.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:49.307 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:49.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:49.389 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:49.389 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:49.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:49.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:49.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.561 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:49.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:49.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:49.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:49.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:49.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:49.912 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:49.940 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:50.026 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:41:50.027 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:50.035 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:41:50.035 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:50.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:50.051 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:41:50.052 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:50.068 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:41:50.068 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:50.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:50.084 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:41:50.084 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:50.101 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:41:50.102 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:50.117 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:41:50.117 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:50.133 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:41:50.133 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:50.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:50.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:50.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:50.324 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:50.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:50.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:50.549 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:50.576 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:50.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:41:50.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:41:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:41:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:50.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:50.789 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-13T14:41:50.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:50.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:50.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:50.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:51.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:51.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:51.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:50 smithi019 ceph-mon[91296]: pgmap v511: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:50 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/921461797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:50 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2689929005' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:41:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:50 smithi003 ceph-mon[107278]: pgmap v511: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:50 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/921461797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:50 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2689929005' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:41:51.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:50 smithi003 ceph-mon[104007]: pgmap v511: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:51.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:50 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/921461797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:51.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:50 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2689929005' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:41:51.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:51.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:51.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:51.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:51.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:51.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:51.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:51.713 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:51.740 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:51.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:51.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:51.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:51 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2689929005' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:41:51.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:51 smithi003 ceph-mon[104007]: osdmap e156: 8 total, 8 up, 8 in 2024-09-13T14:41:51.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:51 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2689929005' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:41:51.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:51 smithi003 ceph-mon[107278]: osdmap e156: 8 total, 8 up, 8 in 2024-09-13T14:41:51.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:51.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:52.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:41:52.093 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:52.120 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:52.123 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:52.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:51 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2689929005' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:41:52.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:51 smithi019 ceph-mon[91296]: osdmap e156: 8 total, 8 up, 8 in 2024-09-13T14:41:52.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:52.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:52.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:52.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:52.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:52.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:52.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:52.621 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:52.630 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-13T14:41:52.631 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-09-13T14:41:52.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:52.665 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:41:52.665 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:41:52.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:52.751 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:52.853 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:52.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T14:41:52.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:41:52.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_156985 2024-09-13T14:41:52.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:41:52.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:41:52.895 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:52 smithi003 ceph-mon[107278]: pgmap v513: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:52 smithi003 ceph-mon[107278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:41:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:52 smithi003 ceph-mon[104007]: pgmap v513: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:52 smithi003 ceph-mon[104007]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:41:52.964 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:52.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:53.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:41:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:53.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:53.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:53.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:52 smithi019 ceph-mon[91296]: pgmap v513: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:53.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:52 smithi019 ceph-mon[91296]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:41:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:53.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:53.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:53.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:53.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:53.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:53.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:53.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:53.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:53.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:53.769 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:53.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3503 3487 217.924 217.938 0.00221225 0.00457122 2024-09-13T14:41:53.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:53.908 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:54.010 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:54.046 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:54.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:54.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:54.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:54.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:54.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:54.334 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:54.365 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:41:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:41:54] "GET /metrics HTTP/1.1" 200 36401 "" "Prometheus/2.51.0" 2024-09-13T14:41:54.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:54.437 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:54.471 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:54.573 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:54.609 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:54.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:54.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:54.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:54.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 7176 7160 223.734 229.562 0.00363317 0.00446223 2024-09-13T14:41:54.888 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:54.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:55.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:55.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:54 smithi003 ceph-mon[104007]: pgmap v514: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:55.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:54 smithi003 ceph-mon[107278]: pgmap v514: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:55.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:55.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:55.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:55.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:54 smithi019 ceph-mon[91296]: pgmap v514: 89 pgs: 89 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:41:55.444 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:55.546 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:55.579 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:55.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:55.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:55.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:55.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 10472 10456 217.815 206 0.00597305 0.00458023 2024-09-13T14:41:55.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:55.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:55.994 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:56.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:56.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:56.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:56.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:56.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:56.410 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:56.512 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:56.547 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:56.649 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:56.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:56.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:56.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:56.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 13483 13467 210.403 188.188 0.00516322 0.00474723 2024-09-13T14:41:56.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:56.956 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:56.959 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:41:57.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:57.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:57.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:57.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:57.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:56 smithi003 ceph-mon[107278]: pgmap v515: 89 pgs: 89 active+clean; 170 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 19 MiB/s wr, 307 op/s 2024-09-13T14:41:57.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3030309014' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:57.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:56 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:57.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:56 smithi003 ceph-mon[104007]: pgmap v515: 89 pgs: 89 active+clean; 170 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 19 MiB/s wr, 307 op/s 2024-09-13T14:41:57.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3030309014' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:57.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:56 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:57.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:57.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:56 smithi019 ceph-mon[91296]: pgmap v515: 89 pgs: 89 active+clean; 170 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 19 MiB/s wr, 307 op/s 2024-09-13T14:41:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3030309014' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:56 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:57.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:57.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:57.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:57.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:57.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:57.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:57.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:57.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 15 16270 16255 203.169 174.25 0.00606978 0.00491565 2024-09-13T14:41:57.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:41:57.969 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:41:58.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:58.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:58.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:58.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.203 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:58.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:58.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:58.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:57 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:57 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:41:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:57 smithi019 ceph-mon[91296]: osdmap e157: 8 total, 8 up, 8 in 2024-09-13T14:41:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:58.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:58.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:41:58.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[104007]: osdmap e157: 8 total, 8 up, 8 in 2024-09-13T14:41:58.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:58.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:58.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:58.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:41:58.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[107278]: osdmap e157: 8 total, 8 up, 8 in 2024-09-13T14:41:58.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:41:58.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:58.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:58.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:58.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:58.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:58.620 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:58.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:58.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:58.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:58.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 18337 18321 190.827 129.125 0.250846 0.00510718 2024-09-13T14:41:58.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:58.909 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:58.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:59.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.157 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:59.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:59.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:59.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:59.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:59.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:58 smithi019 ceph-mon[91296]: pgmap v517: 89 pgs: 89 active+clean; 395 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-09-13T14:41:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:41:58 smithi019 ceph-mon[91296]: from='client.55693 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:59.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:58 smithi003 ceph-mon[107278]: pgmap v517: 89 pgs: 89 active+clean; 395 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-09-13T14:41:59.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:41:58 smithi003 ceph-mon[107278]: from='client.55693 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:58 smithi003 ceph-mon[104007]: pgmap v517: 89 pgs: 89 active+clean; 395 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-09-13T14:41:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:41:58 smithi003 ceph-mon[104007]: from='client.55693 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:59.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:59.532 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-13T14:41:57.573031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":414455757,"num_objects":6341,"num_object_clones":0,"num_object_copies":19023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":6528,"num_write_kb":405820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6474,"ondisk_log_size":6474,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3992512,"kb_used_data":675056,"kb_used_omap":29,"kb_used_meta":3317346,"kb_avail":745968704,"statfs":{"total":767960285184,"available":763871952896,"internally_reserved":0,"allocated":691257344,"data_stored":685216440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":3396962830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":413859840,"num_objects":6315,"num_object_clones":0,"num_object_copies":18945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6315,"num_write_kb":404160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002095"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":115,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.971521+0000","last_change":"2024-09-13T14:37:08.782889+0000","last_active":"2024-09-13T14:41:56.971521+0000","last_peered":"2024-09-13T14:41:56.971521+0000","last_clean":"2024-09-13T14:41:56.971521+0000","last_became_active":"2024-09-13T14:37:08.781502+0000","last_became_peered":"2024-09-13T14:37:08.781502+0000","last_unstale":"2024-09-13T14:41:56.971521+0000","last_undegraded":"2024-09-13T14:41:56.971521+0000","last_fullsized":"2024-09-13T14:41:56.971521+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:22.769013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":122,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:46.773959+0000","last_change":"2024-09-13T14:35:30.492165+0000","last_active":"2024-09-13T14:41:46.773959+0000","last_peered":"2024-09-13T14:41:46.773959+0000","last_clean":"2024-09-13T14:41:46.773959+0000","last_became_active":"2024-09-13T14:35:30.491844+0000","last_became_peered":"2024-09-13T14:35:30.491844+0000","last_unstale":"2024-09-13T14:41:46.773959+0000","last_undegraded":"2024-09-13T14:41:46.773959+0000","last_fullsized":"2024-09-13T14:41:46.773959+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:22.876890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":117,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592875+0000","last_change":"2024-09-13T14:37:08.755784+0000","last_active":"2024-09-13T14:41:47.592875+0000","last_peered":"2024-09-13T14:41:47.592875+0000","last_clean":"2024-09-13T14:41:47.592875+0000","last_became_active":"2024-09-13T14:37:08.755300+0000","last_became_peered":"2024-09-13T14:37:08.755300+0000","last_unstale":"2024-09-13T14:41:47.592875+0000","last_undegraded":"2024-09-13T14:41:47.592875+0000","last_fullsized":"2024-09-13T14:41:47.592875+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:10.881654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":182,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791271+0000","last_change":"2024-09-13T14:37:08.748084+0000","last_active":"2024-09-13T14:41:50.791271+0000","last_peered":"2024-09-13T14:41:50.791271+0000","last_clean":"2024-09-13T14:41:50.791271+0000","last_became_active":"2024-09-13T14:37:08.746801+0000","last_became_peered":"2024-09-13T14:37:08.746801+0000","last_unstale":"2024-09-13T14:41:50.791271+0000","last_undegraded":"2024-09-13T14:41:50.791271+0000","last_fullsized":"2024-09-13T14:41:50.791271+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:13.495421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":183,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791827+0000","last_change":"2024-09-13T14:37:08.750965+0000","last_active":"2024-09-13T14:41:50.791827+0000","last_peered":"2024-09-13T14:41:50.791827+0000","last_clean":"2024-09-13T14:41:50.791827+0000","last_became_active":"2024-09-13T14:37:08.750129+0000","last_became_peered":"2024-09-13T14:37:08.750129+0000","last_unstale":"2024-09-13T14:41:50.791827+0000","last_undegraded":"2024-09-13T14:41:50.791827+0000","last_fullsized":"2024-09-13T14:41:50.791827+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:48.359985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":103,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:46.773732+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:41:46.773732+0000","last_peered":"2024-09-13T14:41:46.773732+0000","last_clean":"2024-09-13T14:41:46.773732+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:41:46.773732+0000","last_undegraded":"2024-09-13T14:41:46.773732+0000","last_fullsized":"2024-09-13T14:41:46.773732+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":141,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.968972+0000","last_change":"2024-09-13T14:37:08.755312+0000","last_active":"2024-09-13T14:41:56.968972+0000","last_peered":"2024-09-13T14:41:56.968972+0000","last_clean":"2024-09-13T14:41:56.968972+0000","last_became_active":"2024-09-13T14:37:08.753186+0000","last_became_peered":"2024-09-13T14:37:08.753186+0000","last_unstale":"2024-09-13T14:41:56.968972+0000","last_undegraded":"2024-09-13T14:41:56.968972+0000","last_fullsized":"2024-09-13T14:41:56.968972+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:57.687749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":127,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.445998+0000","last_change":"2024-09-13T14:31:29.946716+0000","last_active":"2024-09-13T14:41:47.445998+0000","last_peered":"2024-09-13T14:41:47.445998+0000","last_clean":"2024-09-13T14:41:47.445998+0000","last_became_active":"2024-09-13T14:31:29.946366+0000","last_became_peered":"2024-09-13T14:31:29.946366+0000","last_unstale":"2024-09-13T14:41:47.445998+0000","last_undegraded":"2024-09-13T14:41:47.445998+0000","last_fullsized":"2024-09-13T14:41:47.445998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:29.686747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"156'284","reported_seq":303,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:54.273664+0000","last_change":"2024-09-13T14:41:41.691435+0000","last_active":"2024-09-13T14:41:54.273664+0000","last_peered":"2024-09-13T14:41:54.273664+0000","last_clean":"2024-09-13T14:41:54.273664+0000","last_became_active":"2024-09-13T14:41:41.691118+0000","last_became_peered":"2024-09-13T14:41:41.691118+0000","last_unstale":"2024-09-13T14:41:54.273664+0000","last_undegraded":"2024-09-13T14:41:54.273664+0000","last_fullsized":"2024-09-13T14:41:54.273664+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:24.634377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18612224,"num_objects":284,"num_object_clones":0,"num_object_copies":852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":284,"num_write_kb":18176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":121,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:46.774466+0000","last_change":"2024-09-13T14:31:29.950983+0000","last_active":"2024-09-13T14:41:46.774466+0000","last_peered":"2024-09-13T14:41:46.774466+0000","last_clean":"2024-09-13T14:41:46.774466+0000","last_became_active":"2024-09-13T14:31:29.950612+0000","last_became_peered":"2024-09-13T14:31:29.950612+0000","last_unstale":"2024-09-13T14:41:46.774466+0000","last_undegraded":"2024-09-13T14:41:46.774466+0000","last_fullsized":"2024-09-13T14:41:46.774466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:35.833747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":100,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592119+0000","last_change":"2024-09-13T14:21:52.983025+0000","last_active":"2024-09-13T14:41:47.592119+0000","last_peered":"2024-09-13T14:41:47.592119+0000","last_clean":"2024-09-13T14:41:47.592119+0000","last_became_active":"2024-09-13T14:21:52.982611+0000","last_became_peered":"2024-09-13T14:21:52.982611+0000","last_unstale":"2024-09-13T14:41:47.592119+0000","last_undegraded":"2024-09-13T14:41:47.592119+0000","last_fullsized":"2024-09-13T14:41:47.592119+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:00.851531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"156'200","reported_seq":219,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.769989+0000","last_change":"2024-09-13T14:41:41.950800+0000","last_active":"2024-09-13T14:41:53.769989+0000","last_peered":"2024-09-13T14:41:53.769989+0000","last_clean":"2024-09-13T14:41:53.769989+0000","last_became_active":"2024-09-13T14:41:41.950404+0000","last_became_peered":"2024-09-13T14:41:41.950404+0000","last_unstale":"2024-09-13T14:41:53.769989+0000","last_undegraded":"2024-09-13T14:41:53.769989+0000","last_fullsized":"2024-09-13T14:41:53.769989+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:14.568324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":12800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":122,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:46.774403+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:41:46.774403+0000","last_peered":"2024-09-13T14:41:46.774403+0000","last_clean":"2024-09-13T14:41:46.774403+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:41:46.774403+0000","last_undegraded":"2024-09-13T14:41:46.774403+0000","last_fullsized":"2024-09-13T14:41:46.774403+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592319+0000","last_change":"2024-09-13T14:37:08.752050+0000","last_active":"2024-09-13T14:41:47.592319+0000","last_peered":"2024-09-13T14:41:47.592319+0000","last_clean":"2024-09-13T14:41:47.592319+0000","last_became_active":"2024-09-13T14:37:08.751650+0000","last_became_peered":"2024-09-13T14:37:08.751650+0000","last_unstale":"2024-09-13T14:41:47.592319+0000","last_undegraded":"2024-09-13T14:41:47.592319+0000","last_fullsized":"2024-09-13T14:41:47.592319+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:17.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"156'252","reported_seq":264,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.855998+0000","last_change":"2024-09-13T14:41:41.691626+0000","last_active":"2024-09-13T14:41:53.855998+0000","last_peered":"2024-09-13T14:41:53.855998+0000","last_clean":"2024-09-13T14:41:53.855998+0000","last_became_active":"2024-09-13T14:41:41.691304+0000","last_became_peered":"2024-09-13T14:41:41.691304+0000","last_unstale":"2024-09-13T14:41:53.855998+0000","last_undegraded":"2024-09-13T14:41:53.855998+0000","last_fullsized":"2024-09-13T14:41:53.855998+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:22.079580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":252,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":252,"num_write_kb":16128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":104,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:46.774292+0000","last_change":"2024-09-13T14:21:52.610490+0000","last_active":"2024-09-13T14:41:46.774292+0000","last_peered":"2024-09-13T14:41:46.774292+0000","last_clean":"2024-09-13T14:41:46.774292+0000","last_became_active":"2024-09-13T14:21:52.610192+0000","last_became_peered":"2024-09-13T14:21:52.610192+0000","last_unstale":"2024-09-13T14:41:46.774292+0000","last_undegraded":"2024-09-13T14:41:46.774292+0000","last_fullsized":"2024-09-13T14:41:46.774292+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:09.564393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":111,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.445779+0000","last_change":"2024-09-13T14:22:34.901360+0000","last_active":"2024-09-13T14:41:47.445779+0000","last_peered":"2024-09-13T14:41:47.445779+0000","last_clean":"2024-09-13T14:41:47.445779+0000","last_became_active":"2024-09-13T14:22:34.901010+0000","last_became_peered":"2024-09-13T14:22:34.901010+0000","last_unstale":"2024-09-13T14:41:47.445779+0000","last_undegraded":"2024-09-13T14:41:47.445779+0000","last_fullsized":"2024-09-13T14:41:47.445779+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:55.897004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"156'238","reported_seq":250,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.863986+0000","last_change":"2024-09-13T14:41:41.689261+0000","last_active":"2024-09-13T14:41:53.863986+0000","last_peered":"2024-09-13T14:41:53.863986+0000","last_clean":"2024-09-13T14:41:53.863986+0000","last_became_active":"2024-09-13T14:41:41.688928+0000","last_became_peered":"2024-09-13T14:41:41.688928+0000","last_unstale":"2024-09-13T14:41:53.863986+0000","last_undegraded":"2024-09-13T14:41:53.863986+0000","last_fullsized":"2024-09-13T14:41:53.863986+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:29.231199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15597568,"num_objects":238,"num_object_clones":0,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":238,"num_write_kb":15232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":118,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592690+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:41:47.592690+0000","last_peered":"2024-09-13T14:41:47.592690+0000","last_clean":"2024-09-13T14:41:47.592690+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:41:47.592690+0000","last_undegraded":"2024-09-13T14:41:47.592690+0000","last_fullsized":"2024-09-13T14:41:47.592690+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":183,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791185+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:41:50.791185+0000","last_peered":"2024-09-13T14:41:50.791185+0000","last_clean":"2024-09-13T14:41:50.791185+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:41:50.791185+0000","last_undegraded":"2024-09-13T14:41:50.791185+0000","last_fullsized":"2024-09-13T14:41:50.791185+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"157'868","reported_seq":888,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:57.114923+0000","last_change":"2024-09-13T14:41:41.821140+0000","last_active":"2024-09-13T14:41:57.114923+0000","last_peered":"2024-09-13T14:41:57.114923+0000","last_clean":"2024-09-13T14:41:57.114923+0000","last_became_active":"2024-09-13T14:41:41.820780+0000","last_became_peered":"2024-09-13T14:41:41.820780+0000","last_unstale":"2024-09-13T14:41:57.114923+0000","last_undegraded":"2024-09-13T14:41:57.114923+0000","last_fullsized":"2024-09-13T14:41:57.114923+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:47.389145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56885248,"num_objects":868,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":868,"num_write_kb":55552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":146,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.968099+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:41:56.968099+0000","last_peered":"2024-09-13T14:41:56.968099+0000","last_clean":"2024-09-13T14:41:56.968099+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:41:56.968099+0000","last_undegraded":"2024-09-13T14:41:56.968099+0000","last_fullsized":"2024-09-13T14:41:56.968099+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":118,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592851+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:41:47.592851+0000","last_peered":"2024-09-13T14:41:47.592851+0000","last_clean":"2024-09-13T14:41:47.592851+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:41:47.592851+0000","last_undegraded":"2024-09-13T14:41:47.592851+0000","last_fullsized":"2024-09-13T14:41:47.592851+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"156'166","reported_seq":185,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.778896+0000","last_change":"2024-09-13T14:41:41.821111+0000","last_active":"2024-09-13T14:41:53.778896+0000","last_peered":"2024-09-13T14:41:53.778896+0000","last_clean":"2024-09-13T14:41:53.778896+0000","last_became_active":"2024-09-13T14:41:41.820744+0000","last_became_peered":"2024-09-13T14:41:41.820744+0000","last_unstale":"2024-09-13T14:41:53.778896+0000","last_undegraded":"2024-09-13T14:41:53.778896+0000","last_fullsized":"2024-09-13T14:41:53.778896+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:28.502140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":166,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":119,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.446070+0000","last_change":"2024-09-13T14:32:09.491728+0000","last_active":"2024-09-13T14:41:47.446070+0000","last_peered":"2024-09-13T14:41:47.446070+0000","last_clean":"2024-09-13T14:41:47.446070+0000","last_became_active":"2024-09-13T14:32:09.490612+0000","last_became_peered":"2024-09-13T14:32:09.490612+0000","last_unstale":"2024-09-13T14:41:47.446070+0000","last_undegraded":"2024-09-13T14:41:47.446070+0000","last_fullsized":"2024-09-13T14:41:47.446070+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:21.947056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592541+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:41:47.592541+0000","last_peered":"2024-09-13T14:41:47.592541+0000","last_clean":"2024-09-13T14:41:47.592541+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:41:47.592541+0000","last_undegraded":"2024-09-13T14:41:47.592541+0000","last_fullsized":"2024-09-13T14:41:47.592541+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"156'175","reported_seq":187,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.650610+0000","last_change":"2024-09-13T14:41:41.689801+0000","last_active":"2024-09-13T14:41:53.650610+0000","last_peered":"2024-09-13T14:41:53.650610+0000","last_clean":"2024-09-13T14:41:53.650610+0000","last_became_active":"2024-09-13T14:41:41.689390+0000","last_became_peered":"2024-09-13T14:41:41.689390+0000","last_unstale":"2024-09-13T14:41:53.650610+0000","last_undegraded":"2024-09-13T14:41:53.650610+0000","last_fullsized":"2024-09-13T14:41:53.650610+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:46:54.202699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11468800,"num_objects":175,"num_object_clones":0,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":175,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":119,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592476+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:41:47.592476+0000","last_peered":"2024-09-13T14:41:47.592476+0000","last_clean":"2024-09-13T14:41:47.592476+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:41:47.592476+0000","last_undegraded":"2024-09-13T14:41:47.592476+0000","last_fullsized":"2024-09-13T14:41:47.592476+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":118,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592943+0000","last_change":"2024-09-13T14:32:09.487930+0000","last_active":"2024-09-13T14:41:47.592943+0000","last_peered":"2024-09-13T14:41:47.592943+0000","last_clean":"2024-09-13T14:41:47.592943+0000","last_became_active":"2024-09-13T14:32:09.487592+0000","last_became_peered":"2024-09-13T14:32:09.487592+0000","last_unstale":"2024-09-13T14:41:47.592943+0000","last_undegraded":"2024-09-13T14:41:47.592943+0000","last_fullsized":"2024-09-13T14:41:47.592943+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:57.014876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"157'911","reported_seq":930,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:57.115968+0000","last_change":"2024-09-13T14:41:41.689036+0000","last_active":"2024-09-13T14:41:57.115968+0000","last_peered":"2024-09-13T14:41:57.115968+0000","last_clean":"2024-09-13T14:41:57.115968+0000","last_became_active":"2024-09-13T14:41:41.688610+0000","last_became_peered":"2024-09-13T14:41:41.688610+0000","last_unstale":"2024-09-13T14:41:57.115968+0000","last_undegraded":"2024-09-13T14:41:57.115968+0000","last_fullsized":"2024-09-13T14:41:57.115968+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:09.285319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59703296,"num_objects":911,"num_object_clones":0,"num_object_copies":2733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":911,"num_write_kb":58304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":116,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.970300+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:41:56.970300+0000","last_peered":"2024-09-13T14:41:56.970300+0000","last_clean":"2024-09-13T14:41:56.970300+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:41:56.970300+0000","last_undegraded":"2024-09-13T14:41:56.970300+0000","last_fullsized":"2024-09-13T14:41:56.970300+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":154,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.643177+0000","last_change":"2024-09-13T14:37:08.758818+0000","last_active":"2024-09-13T14:41:51.643177+0000","last_peered":"2024-09-13T14:41:51.643177+0000","last_clean":"2024-09-13T14:41:51.643177+0000","last_became_active":"2024-09-13T14:37:08.758225+0000","last_became_peered":"2024-09-13T14:37:08.758225+0000","last_unstale":"2024-09-13T14:41:51.643177+0000","last_undegraded":"2024-09-13T14:41:51.643177+0000","last_fullsized":"2024-09-13T14:41:51.643177+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:18.890207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"157'923","reported_seq":943,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:57.111061+0000","last_change":"2024-09-13T14:41:41.690155+0000","last_active":"2024-09-13T14:41:57.111061+0000","last_peered":"2024-09-13T14:41:57.111061+0000","last_clean":"2024-09-13T14:41:57.111061+0000","last_became_active":"2024-09-13T14:41:41.689839+0000","last_became_peered":"2024-09-13T14:41:41.689839+0000","last_unstale":"2024-09-13T14:41:57.111061+0000","last_undegraded":"2024-09-13T14:41:57.111061+0000","last_fullsized":"2024-09-13T14:41:57.111061+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":923,"log_dups_size":0,"ondisk_log_size":923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:32.786358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60489728,"num_objects":923,"num_object_clones":0,"num_object_copies":2769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":923,"num_write_kb":59072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":236,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:46.773750+0000","last_change":"2024-09-13T14:31:29.950741+0000","last_active":"2024-09-13T14:41:46.773750+0000","last_peered":"2024-09-13T14:41:46.773750+0000","last_clean":"2024-09-13T14:41:46.773750+0000","last_became_active":"2024-09-13T14:31:29.950338+0000","last_became_peered":"2024-09-13T14:31:29.950338+0000","last_unstale":"2024-09-13T14:41:46.773750+0000","last_undegraded":"2024-09-13T14:41:46.773750+0000","last_fullsized":"2024-09-13T14:41:46.773750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:09.370711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":130,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592608+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:41:47.592608+0000","last_peered":"2024-09-13T14:41:47.592608+0000","last_clean":"2024-09-13T14:41:47.592608+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:41:47.592608+0000","last_undegraded":"2024-09-13T14:41:47.592608+0000","last_fullsized":"2024-09-13T14:41:47.592608+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":105,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.969000+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:41:56.969000+0000","last_peered":"2024-09-13T14:41:56.969000+0000","last_clean":"2024-09-13T14:41:56.969000+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:41:56.969000+0000","last_undegraded":"2024-09-13T14:41:56.969000+0000","last_fullsized":"2024-09-13T14:41:56.969000+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":101,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.446333+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:41:47.446333+0000","last_peered":"2024-09-13T14:41:47.446333+0000","last_clean":"2024-09-13T14:41:47.446333+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:41:47.446333+0000","last_undegraded":"2024-09-13T14:41:47.446333+0000","last_fullsized":"2024-09-13T14:41:47.446333+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"156'305","reported_seq":324,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:54.266634+0000","last_change":"2024-09-13T14:41:41.821076+0000","last_active":"2024-09-13T14:41:54.266634+0000","last_peered":"2024-09-13T14:41:54.266634+0000","last_clean":"2024-09-13T14:41:54.266634+0000","last_became_active":"2024-09-13T14:41:41.820767+0000","last_became_peered":"2024-09-13T14:41:41.820767+0000","last_unstale":"2024-09-13T14:41:54.266634+0000","last_undegraded":"2024-09-13T14:41:54.266634+0000","last_fullsized":"2024-09-13T14:41:54.266634+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:35.487284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19988480,"num_objects":305,"num_object_clones":0,"num_object_copies":915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":305,"num_write_kb":19520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":148,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:46.773683+0000","last_change":"2024-09-13T14:32:09.482227+0000","last_active":"2024-09-13T14:41:46.773683+0000","last_peered":"2024-09-13T14:41:46.773683+0000","last_clean":"2024-09-13T14:41:46.773683+0000","last_became_active":"2024-09-13T14:32:09.481843+0000","last_became_peered":"2024-09-13T14:32:09.481843+0000","last_unstale":"2024-09-13T14:41:46.773683+0000","last_undegraded":"2024-09-13T14:41:46.773683+0000","last_fullsized":"2024-09-13T14:41:46.773683+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:56.929686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":183,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791059+0000","last_change":"2024-09-13T14:37:08.748298+0000","last_active":"2024-09-13T14:41:50.791059+0000","last_peered":"2024-09-13T14:41:50.791059+0000","last_clean":"2024-09-13T14:41:50.791059+0000","last_became_active":"2024-09-13T14:37:08.747638+0000","last_became_peered":"2024-09-13T14:37:08.747638+0000","last_unstale":"2024-09-13T14:41:50.791059+0000","last_undegraded":"2024-09-13T14:41:50.791059+0000","last_fullsized":"2024-09-13T14:41:50.791059+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:46.778883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791665+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:41:50.791665+0000","last_peered":"2024-09-13T14:41:50.791665+0000","last_clean":"2024-09-13T14:41:50.791665+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:41:50.791665+0000","last_undegraded":"2024-09-13T14:41:50.791665+0000","last_fullsized":"2024-09-13T14:41:50.791665+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"156'213","reported_seq":225,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.846543+0000","last_change":"2024-09-13T14:41:41.688564+0000","last_active":"2024-09-13T14:41:53.846543+0000","last_peered":"2024-09-13T14:41:53.846543+0000","last_clean":"2024-09-13T14:41:53.846543+0000","last_became_active":"2024-09-13T14:41:41.688264+0000","last_became_peered":"2024-09-13T14:41:41.688264+0000","last_unstale":"2024-09-13T14:41:53.846543+0000","last_undegraded":"2024-09-13T14:41:53.846543+0000","last_fullsized":"2024-09-13T14:41:53.846543+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:05:05.338770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":13632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":171,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.789729+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:41:50.789729+0000","last_peered":"2024-09-13T14:41:50.789729+0000","last_clean":"2024-09-13T14:41:50.789729+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:41:50.789729+0000","last_undegraded":"2024-09-13T14:41:50.789729+0000","last_fullsized":"2024-09-13T14:41:50.789729+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":154,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.642018+0000","last_change":"2024-09-13T14:35:30.473739+0000","last_active":"2024-09-13T14:41:51.642018+0000","last_peered":"2024-09-13T14:41:51.642018+0000","last_clean":"2024-09-13T14:41:51.642018+0000","last_became_active":"2024-09-13T14:35:30.473448+0000","last_became_peered":"2024-09-13T14:35:30.473448+0000","last_unstale":"2024-09-13T14:41:51.642018+0000","last_undegraded":"2024-09-13T14:41:51.642018+0000","last_fullsized":"2024-09-13T14:41:51.642018+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:52.239598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":181,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.643114+0000","last_change":"2024-09-13T14:32:09.485178+0000","last_active":"2024-09-13T14:41:51.643114+0000","last_peered":"2024-09-13T14:41:51.643114+0000","last_clean":"2024-09-13T14:41:51.643114+0000","last_became_active":"2024-09-13T14:32:09.484869+0000","last_became_peered":"2024-09-13T14:32:09.484869+0000","last_unstale":"2024-09-13T14:41:51.643114+0000","last_undegraded":"2024-09-13T14:41:51.643114+0000","last_fullsized":"2024-09-13T14:41:51.643114+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:18.863308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"156'319","reported_seq":338,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:54.272610+0000","last_change":"2024-09-13T14:41:41.690374+0000","last_active":"2024-09-13T14:41:54.272610+0000","last_peered":"2024-09-13T14:41:54.272610+0000","last_clean":"2024-09-13T14:41:54.272610+0000","last_became_active":"2024-09-13T14:41:41.690093+0000","last_became_peered":"2024-09-13T14:41:41.690093+0000","last_unstale":"2024-09-13T14:41:54.272610+0000","last_undegraded":"2024-09-13T14:41:54.272610+0000","last_fullsized":"2024-09-13T14:41:54.272610+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:28.392006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20905984,"num_objects":319,"num_object_clones":0,"num_object_copies":957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":319,"num_write_kb":20416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":131,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592380+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:41:47.592380+0000","last_peered":"2024-09-13T14:41:47.592380+0000","last_clean":"2024-09-13T14:41:47.592380+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:41:47.592380+0000","last_undegraded":"2024-09-13T14:41:47.592380+0000","last_fullsized":"2024-09-13T14:41:47.592380+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":119,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.593271+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:41:47.593271+0000","last_peered":"2024-09-13T14:41:47.593271+0000","last_clean":"2024-09-13T14:41:47.593271+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:41:47.593271+0000","last_undegraded":"2024-09-13T14:41:47.593271+0000","last_fullsized":"2024-09-13T14:41:47.593271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":165,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.642716+0000","last_change":"2024-09-13T14:37:08.794198+0000","last_active":"2024-09-13T14:41:51.642716+0000","last_peered":"2024-09-13T14:41:51.642716+0000","last_clean":"2024-09-13T14:41:51.642716+0000","last_became_active":"2024-09-13T14:37:08.793689+0000","last_became_peered":"2024-09-13T14:37:08.793689+0000","last_unstale":"2024-09-13T14:41:51.642716+0000","last_undegraded":"2024-09-13T14:41:51.642716+0000","last_fullsized":"2024-09-13T14:41:51.642716+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:19.865901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"156'212","reported_seq":224,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.864208+0000","last_change":"2024-09-13T14:41:41.691642+0000","last_active":"2024-09-13T14:41:53.864208+0000","last_peered":"2024-09-13T14:41:53.864208+0000","last_clean":"2024-09-13T14:41:53.864208+0000","last_became_active":"2024-09-13T14:41:41.691265+0000","last_became_peered":"2024-09-13T14:41:41.691265+0000","last_unstale":"2024-09-13T14:41:53.864208+0000","last_undegraded":"2024-09-13T14:41:53.864208+0000","last_fullsized":"2024-09-13T14:41:53.864208+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:07.758310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":212,"num_write_kb":13568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.593113+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:41:47.593113+0000","last_peered":"2024-09-13T14:41:47.593113+0000","last_clean":"2024-09-13T14:41:47.593113+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:41:47.593113+0000","last_undegraded":"2024-09-13T14:41:47.593113+0000","last_fullsized":"2024-09-13T14:41:47.593113+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":193,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.642888+0000","last_change":"2024-09-13T14:32:09.488979+0000","last_active":"2024-09-13T14:41:51.642888+0000","last_peered":"2024-09-13T14:41:51.642888+0000","last_clean":"2024-09-13T14:41:51.642888+0000","last_became_active":"2024-09-13T14:32:09.488320+0000","last_became_peered":"2024-09-13T14:32:09.488320+0000","last_unstale":"2024-09-13T14:41:51.642888+0000","last_undegraded":"2024-09-13T14:41:51.642888+0000","last_fullsized":"2024-09-13T14:41:51.642888+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:28.662506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":143,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.642863+0000","last_change":"2024-09-13T14:32:09.489738+0000","last_active":"2024-09-13T14:41:51.642863+0000","last_peered":"2024-09-13T14:41:51.642863+0000","last_clean":"2024-09-13T14:41:51.642863+0000","last_became_active":"2024-09-13T14:32:09.489304+0000","last_became_peered":"2024-09-13T14:32:09.489304+0000","last_unstale":"2024-09-13T14:41:51.642863+0000","last_undegraded":"2024-09-13T14:41:51.642863+0000","last_fullsized":"2024-09-13T14:41:51.642863+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:16.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"156'171","reported_seq":183,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.651078+0000","last_change":"2024-09-13T14:41:41.950035+0000","last_active":"2024-09-13T14:41:53.651078+0000","last_peered":"2024-09-13T14:41:53.651078+0000","last_clean":"2024-09-13T14:41:53.651078+0000","last_became_active":"2024-09-13T14:41:41.949672+0000","last_became_peered":"2024-09-13T14:41:41.949672+0000","last_unstale":"2024-09-13T14:41:53.651078+0000","last_undegraded":"2024-09-13T14:41:53.651078+0000","last_fullsized":"2024-09-13T14:41:53.651078+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:43.973874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":154,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.642889+0000","last_change":"2024-09-13T14:35:30.476616+0000","last_active":"2024-09-13T14:41:51.642889+0000","last_peered":"2024-09-13T14:41:51.642889+0000","last_clean":"2024-09-13T14:41:51.642889+0000","last_became_active":"2024-09-13T14:35:30.476323+0000","last_became_peered":"2024-09-13T14:35:30.476323+0000","last_unstale":"2024-09-13T14:41:51.642889+0000","last_undegraded":"2024-09-13T14:41:51.642889+0000","last_fullsized":"2024-09-13T14:41:51.642889+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:02.000336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":131,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:46.774390+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:41:46.774390+0000","last_peered":"2024-09-13T14:41:46.774390+0000","last_clean":"2024-09-13T14:41:46.774390+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:41:46.774390+0000","last_undegraded":"2024-09-13T14:41:46.774390+0000","last_fullsized":"2024-09-13T14:41:46.774390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":173,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791375+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:41:50.791375+0000","last_peered":"2024-09-13T14:41:50.791375+0000","last_clean":"2024-09-13T14:41:50.791375+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:41:50.791375+0000","last_undegraded":"2024-09-13T14:41:50.791375+0000","last_fullsized":"2024-09-13T14:41:50.791375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"156'172","reported_seq":184,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.652795+0000","last_change":"2024-09-13T14:41:41.690506+0000","last_active":"2024-09-13T14:41:53.652795+0000","last_peered":"2024-09-13T14:41:53.652795+0000","last_clean":"2024-09-13T14:41:53.652795+0000","last_became_active":"2024-09-13T14:41:41.690103+0000","last_became_peered":"2024-09-13T14:41:41.690103+0000","last_unstale":"2024-09-13T14:41:53.652795+0000","last_undegraded":"2024-09-13T14:41:53.652795+0000","last_fullsized":"2024-09-13T14:41:53.652795+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:13:45.272717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11272192,"num_objects":172,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":172,"num_write_kb":11008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":146,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.969021+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:41:56.969021+0000","last_peered":"2024-09-13T14:41:56.969021+0000","last_clean":"2024-09-13T14:41:56.969021+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:41:56.969021+0000","last_undegraded":"2024-09-13T14:41:56.969021+0000","last_fullsized":"2024-09-13T14:41:56.969021+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":153,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.643266+0000","last_change":"2024-09-13T14:32:09.490012+0000","last_active":"2024-09-13T14:41:51.643266+0000","last_peered":"2024-09-13T14:41:51.643266+0000","last_clean":"2024-09-13T14:41:51.643266+0000","last_became_active":"2024-09-13T14:32:09.489652+0000","last_became_peered":"2024-09-13T14:32:09.489652+0000","last_unstale":"2024-09-13T14:41:51.643266+0000","last_undegraded":"2024-09-13T14:41:51.643266+0000","last_fullsized":"2024-09-13T14:41:51.643266+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:10.587217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":143,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.969320+0000","last_change":"2024-09-13T14:37:08.745773+0000","last_active":"2024-09-13T14:41:56.969320+0000","last_peered":"2024-09-13T14:41:56.969320+0000","last_clean":"2024-09-13T14:41:56.969320+0000","last_became_active":"2024-09-13T14:37:08.745130+0000","last_became_peered":"2024-09-13T14:37:08.745130+0000","last_unstale":"2024-09-13T14:41:56.969320+0000","last_undegraded":"2024-09-13T14:41:56.969320+0000","last_fullsized":"2024-09-13T14:41:56.969320+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:29:44.778588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"157'906","reported_seq":927,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:57.105682+0000","last_change":"2024-09-13T14:41:41.950797+0000","last_active":"2024-09-13T14:41:57.105682+0000","last_peered":"2024-09-13T14:41:57.105682+0000","last_clean":"2024-09-13T14:41:57.105682+0000","last_became_active":"2024-09-13T14:41:41.950412+0000","last_became_peered":"2024-09-13T14:41:41.950412+0000","last_unstale":"2024-09-13T14:41:57.105682+0000","last_undegraded":"2024-09-13T14:41:57.105682+0000","last_fullsized":"2024-09-13T14:41:57.105682+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":906,"log_dups_size":0,"ondisk_log_size":906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:38.382869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59375616,"num_objects":906,"num_object_clones":0,"num_object_copies":2718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":906,"num_write_kb":57984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":186,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791911+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:41:50.791911+0000","last_peered":"2024-09-13T14:41:50.791911+0000","last_clean":"2024-09-13T14:41:50.791911+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:41:50.791911+0000","last_undegraded":"2024-09-13T14:41:50.791911+0000","last_fullsized":"2024-09-13T14:41:50.791911+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":131,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.446394+0000","last_change":"2024-09-13T14:35:30.479324+0000","last_active":"2024-09-13T14:41:47.446394+0000","last_peered":"2024-09-13T14:41:47.446394+0000","last_clean":"2024-09-13T14:41:47.446394+0000","last_became_active":"2024-09-13T14:35:30.479033+0000","last_became_peered":"2024-09-13T14:35:30.479033+0000","last_unstale":"2024-09-13T14:41:47.446394+0000","last_undegraded":"2024-09-13T14:41:47.446394+0000","last_fullsized":"2024-09-13T14:41:47.446394+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:57.606853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":101,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592161+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:41:47.592161+0000","last_peered":"2024-09-13T14:41:47.592161+0000","last_clean":"2024-09-13T14:41:47.592161+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:41:47.592161+0000","last_undegraded":"2024-09-13T14:41:47.592161+0000","last_fullsized":"2024-09-13T14:41:47.592161+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":118,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.593228+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:41:47.593228+0000","last_peered":"2024-09-13T14:41:47.593228+0000","last_clean":"2024-09-13T14:41:47.593228+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:41:47.593228+0000","last_undegraded":"2024-09-13T14:41:47.593228+0000","last_fullsized":"2024-09-13T14:41:47.593228+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":125,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.593119+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:41:47.593119+0000","last_peered":"2024-09-13T14:41:47.593119+0000","last_clean":"2024-09-13T14:41:47.593119+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:41:47.593119+0000","last_undegraded":"2024-09-13T14:41:47.593119+0000","last_fullsized":"2024-09-13T14:41:47.593119+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":154,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.642746+0000","last_change":"2024-09-13T14:35:30.474794+0000","last_active":"2024-09-13T14:41:51.642746+0000","last_peered":"2024-09-13T14:41:51.642746+0000","last_clean":"2024-09-13T14:41:51.642746+0000","last_became_active":"2024-09-13T14:35:30.474501+0000","last_became_peered":"2024-09-13T14:35:30.474501+0000","last_unstale":"2024-09-13T14:41:51.642746+0000","last_undegraded":"2024-09-13T14:41:51.642746+0000","last_fullsized":"2024-09-13T14:41:51.642746+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:58.326635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":123,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:46.774190+0000","last_change":"2024-09-13T14:37:08.756167+0000","last_active":"2024-09-13T14:41:46.774190+0000","last_peered":"2024-09-13T14:41:46.774190+0000","last_clean":"2024-09-13T14:41:46.774190+0000","last_became_active":"2024-09-13T14:37:08.755657+0000","last_became_peered":"2024-09-13T14:37:08.755657+0000","last_unstale":"2024-09-13T14:41:46.774190+0000","last_undegraded":"2024-09-13T14:41:46.774190+0000","last_fullsized":"2024-09-13T14:41:46.774190+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:20.322084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":118,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592527+0000","last_change":"2024-09-13T14:32:09.488238+0000","last_active":"2024-09-13T14:41:47.592527+0000","last_peered":"2024-09-13T14:41:47.592527+0000","last_clean":"2024-09-13T14:41:47.592527+0000","last_became_active":"2024-09-13T14:32:09.487750+0000","last_became_peered":"2024-09-13T14:32:09.487750+0000","last_unstale":"2024-09-13T14:41:47.592527+0000","last_undegraded":"2024-09-13T14:41:47.592527+0000","last_fullsized":"2024-09-13T14:41:47.592527+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:17.258720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":106,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.968358+0000","last_change":"2024-09-13T14:37:08.747482+0000","last_active":"2024-09-13T14:41:56.968358+0000","last_peered":"2024-09-13T14:41:56.968358+0000","last_clean":"2024-09-13T14:41:56.968358+0000","last_became_active":"2024-09-13T14:37:08.747176+0000","last_became_peered":"2024-09-13T14:37:08.747176+0000","last_unstale":"2024-09-13T14:41:56.968358+0000","last_undegraded":"2024-09-13T14:41:56.968358+0000","last_fullsized":"2024-09-13T14:41:56.968358+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:18.885448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":182,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791442+0000","last_change":"2024-09-13T14:37:08.759355+0000","last_active":"2024-09-13T14:41:50.791442+0000","last_peered":"2024-09-13T14:41:50.791442+0000","last_clean":"2024-09-13T14:41:50.791442+0000","last_became_active":"2024-09-13T14:37:08.759058+0000","last_became_peered":"2024-09-13T14:37:08.759058+0000","last_unstale":"2024-09-13T14:41:50.791442+0000","last_undegraded":"2024-09-13T14:41:50.791442+0000","last_fullsized":"2024-09-13T14:41:50.791442+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:42.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":115,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.969233+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:41:56.969233+0000","last_peered":"2024-09-13T14:41:56.969233+0000","last_clean":"2024-09-13T14:41:56.969233+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:41:56.969233+0000","last_undegraded":"2024-09-13T14:41:56.969233+0000","last_fullsized":"2024-09-13T14:41:56.969233+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":118,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.446346+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:41:47.446346+0000","last_peered":"2024-09-13T14:41:47.446346+0000","last_clean":"2024-09-13T14:41:47.446346+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:41:47.446346+0000","last_undegraded":"2024-09-13T14:41:47.446346+0000","last_fullsized":"2024-09-13T14:41:47.446346+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":133,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.970348+0000","last_change":"2024-09-13T14:37:08.777939+0000","last_active":"2024-09-13T14:41:56.970348+0000","last_peered":"2024-09-13T14:41:56.970348+0000","last_clean":"2024-09-13T14:41:56.970348+0000","last_became_active":"2024-09-13T14:37:08.776339+0000","last_became_peered":"2024-09-13T14:37:08.776339+0000","last_unstale":"2024-09-13T14:41:56.970348+0000","last_undegraded":"2024-09-13T14:41:56.970348+0000","last_fullsized":"2024-09-13T14:41:56.970348+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:03:31.349551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":184,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791746+0000","last_change":"2024-09-13T14:37:08.749705+0000","last_active":"2024-09-13T14:41:50.791746+0000","last_peered":"2024-09-13T14:41:50.791746+0000","last_clean":"2024-09-13T14:41:50.791746+0000","last_became_active":"2024-09-13T14:37:08.749158+0000","last_became_peered":"2024-09-13T14:37:08.749158+0000","last_unstale":"2024-09-13T14:41:50.791746+0000","last_undegraded":"2024-09-13T14:41:50.791746+0000","last_fullsized":"2024-09-13T14:41:50.791746+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:04.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":154,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.641933+0000","last_change":"2024-09-13T14:37:08.758601+0000","last_active":"2024-09-13T14:41:51.641933+0000","last_peered":"2024-09-13T14:41:51.641933+0000","last_clean":"2024-09-13T14:41:51.641933+0000","last_became_active":"2024-09-13T14:37:08.758031+0000","last_became_peered":"2024-09-13T14:37:08.758031+0000","last_unstale":"2024-09-13T14:41:51.641933+0000","last_undegraded":"2024-09-13T14:41:51.641933+0000","last_fullsized":"2024-09-13T14:41:51.641933+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:12.269600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":114,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.967682+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:41:56.967682+0000","last_peered":"2024-09-13T14:41:56.967682+0000","last_clean":"2024-09-13T14:41:56.967682+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:41:56.967682+0000","last_undegraded":"2024-09-13T14:41:56.967682+0000","last_fullsized":"2024-09-13T14:41:56.967682+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":137,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592509+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:41:47.592509+0000","last_peered":"2024-09-13T14:41:47.592509+0000","last_clean":"2024-09-13T14:41:47.592509+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:41:47.592509+0000","last_undegraded":"2024-09-13T14:41:47.592509+0000","last_fullsized":"2024-09-13T14:41:47.592509+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":170,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.789950+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:41:50.789950+0000","last_peered":"2024-09-13T14:41:50.789950+0000","last_clean":"2024-09-13T14:41:50.789950+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:41:50.789950+0000","last_undegraded":"2024-09-13T14:41:50.789950+0000","last_fullsized":"2024-09-13T14:41:50.789950+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":175,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791426+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:41:50.791426+0000","last_peered":"2024-09-13T14:41:50.791426+0000","last_clean":"2024-09-13T14:41:50.791426+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:41:50.791426+0000","last_undegraded":"2024-09-13T14:41:50.791426+0000","last_fullsized":"2024-09-13T14:41:50.791426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":145,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.593169+0000","last_change":"2024-09-13T14:37:08.749549+0000","last_active":"2024-09-13T14:41:47.593169+0000","last_peered":"2024-09-13T14:41:47.593169+0000","last_clean":"2024-09-13T14:41:47.593169+0000","last_became_active":"2024-09-13T14:37:08.749013+0000","last_became_peered":"2024-09-13T14:37:08.749013+0000","last_unstale":"2024-09-13T14:41:47.593169+0000","last_undegraded":"2024-09-13T14:41:47.593169+0000","last_fullsized":"2024-09-13T14:41:47.593169+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:35.239989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":120,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592585+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:41:47.592585+0000","last_peered":"2024-09-13T14:41:47.592585+0000","last_clean":"2024-09-13T14:41:47.592585+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:41:47.592585+0000","last_undegraded":"2024-09-13T14:41:47.592585+0000","last_fullsized":"2024-09-13T14:41:47.592585+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592347+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:41:47.592347+0000","last_peered":"2024-09-13T14:41:47.592347+0000","last_clean":"2024-09-13T14:41:47.592347+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:41:47.592347+0000","last_undegraded":"2024-09-13T14:41:47.592347+0000","last_fullsized":"2024-09-13T14:41:47.592347+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":103,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592985+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:41:47.592985+0000","last_peered":"2024-09-13T14:41:47.592985+0000","last_clean":"2024-09-13T14:41:47.592985+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:41:47.592985+0000","last_undegraded":"2024-09-13T14:41:47.592985+0000","last_fullsized":"2024-09-13T14:41:47.592985+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":148,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.970597+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:41:56.970597+0000","last_peered":"2024-09-13T14:41:56.970597+0000","last_clean":"2024-09-13T14:41:56.970597+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:41:56.970597+0000","last_undegraded":"2024-09-13T14:41:56.970597+0000","last_fullsized":"2024-09-13T14:41:56.970597+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":133,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:41:56.968098+0000","last_change":"2024-09-13T14:37:08.750424+0000","last_active":"2024-09-13T14:41:56.968098+0000","last_peered":"2024-09-13T14:41:56.968098+0000","last_clean":"2024-09-13T14:41:56.968098+0000","last_became_active":"2024-09-13T14:37:08.749577+0000","last_became_peered":"2024-09-13T14:37:08.749577+0000","last_unstale":"2024-09-13T14:41:56.968098+0000","last_undegraded":"2024-09-13T14:41:56.968098+0000","last_fullsized":"2024-09-13T14:41:56.968098+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:16.435452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":118,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T14:41:47.592065+0000","last_change":"2024-09-13T14:32:09.484255+0000","last_active":"2024-09-13T14:41:47.592065+0000","last_peered":"2024-09-13T14:41:47.592065+0000","last_clean":"2024-09-13T14:41:47.592065+0000","last_became_active":"2024-09-13T14:32:09.483903+0000","last_became_peered":"2024-09-13T14:32:09.483903+0000","last_unstale":"2024-09-13T14:41:47.592065+0000","last_undegraded":"2024-09-13T14:41:47.592065+0000","last_fullsized":"2024-09-13T14:41:47.592065+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:41.931522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":187,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:41:50.791187+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:41:50.791187+0000","last_peered":"2024-09-13T14:41:50.791187+0000","last_clean":"2024-09-13T14:41:50.791187+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:41:50.791187+0000","last_undegraded":"2024-09-13T14:41:50.791187+0000","last_fullsized":"2024-09-13T14:41:50.791187+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":413859840,"num_objects":6315,"num_object_clones":0,"num_object_copies":18945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6315,"num_write_kb":404160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":989724672,"data_stored":989724672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6315,"ondisk_log_size":6315,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365066,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357056,"kb_used_data":55540,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93388096,"statfs":{"total":95995035648,"available":95629410304,"internally_reserved":0,"allocated":56872960,"data_stored":56105728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561296,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336564,"kb_used_data":26732,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93408588,"statfs":{"total":95995035648,"available":95650394112,"internally_reserved":0,"allocated":27373568,"data_stored":26614351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790228,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939492,"kb_used_data":38428,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":92805660,"statfs":{"total":95995035648,"available":95032995840,"internally_reserved":0,"allocated":39350272,"data_stored":38602469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986456,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622332,"kb_used_data":24112,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93122820,"statfs":{"total":95995035648,"available":95357767680,"internally_reserved":0,"allocated":24690688,"data_stored":23927203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519612,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660660,"kb_used_data":456296,"kb_used_omap":4,"kb_used_meta":204347,"kb_avail":93084492,"statfs":{"total":95995035648,"available":95318519808,"internally_reserved":0,"allocated":467247104,"data_stored":466495239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":209251776},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304527,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230444,"kb_used_data":1760,"kb_used_omap":6,"kb_used_meta":228665,"kb_avail":93514708,"statfs":{"total":95995035648,"available":95759060992,"internally_reserved":0,"allocated":1802240,"data_stored":1050865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":234153415},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827831,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":423588,"kb_used_data":31580,"kb_used_omap":5,"kb_used_meta":391994,"kb_avail":93321564,"statfs":{"total":95995035648,"available":95561281536,"internally_reserved":0,"allocated":32337920,"data_stored":31590064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":401402092},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285695,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422376,"kb_used_data":40608,"kb_used_omap":5,"kb_used_meta":381754,"kb_avail":93322776,"statfs":{"total":95995035648,"available":95562522624,"internally_reserved":0,"allocated":41582592,"data_stored":40830521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":390916155},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920000,"data_stored":81920000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116850688,"data_stored":116850688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22609920,"data_stored":22609920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":471924736,"data_stored":471924736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33357824,"data_stored":33357824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":132251648,"data_stored":132251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":29294592,"data_stored":29294592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":101515264,"data_stored":101515264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:59.537 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:41:59.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 26 2024-09-13T14:41:59.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:59.572 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:59.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:41:59.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:59.820 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:59.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:59.856 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 21063 21047 187.9 170.375 0.00545803 0.00531558 2024-09-13T14:41:59.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.993 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:41:59.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:00.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:00.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:00.233 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:00.266 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:00.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:00.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:00.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:00.560 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:00.571 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:42:00.571 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:42:00.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:42:00.571 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:42:00.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.571 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:34913 7.9 7:98405b1e:::benchmark_data_smithi003_122851_object34912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:32:08.523980+0000", 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 592.04280337, 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.039446045, 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 34913 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.572 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.523980+0000", 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.523980+0000", 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.523982+0000", 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.669e-06 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524441+0000", 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00045935600000000001 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524459+0000", 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7655e-05 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524468+0000", 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6029999999999994e-06 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524958+0000", 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00048955699999999995 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524971+0000", 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2619999999999999e-05 2024-09-13T14:42:00.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.472986+0000", 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94801524500000001 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.472999+0000", 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3497e-05 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.554264+0000", 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0812645249999999 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.554434+0000", 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017005500000000001 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.554616+0000", 2024-09-13T14:42:00.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000181967 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.559051+0000", 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044356329999999996 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.559128+0000", 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6605000000000003e-05 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.562349+0000", 2024-09-13T14:42:00.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032210110000000002 2024-09-13T14:42:00.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.562421+0000", 2024-09-13T14:42:00.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1958000000000005e-05 2024-09-13T14:42:00.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.563426+0000", 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001005089 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:34918 7.4 7:203c5878:::benchmark_data_smithi003_122851_object34917:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:32:08.524631+0000", 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 592.04215239999996, 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0576409230000001, 2024-09-13T14:42:00.580 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 34918 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524631+0000", 2024-09-13T14:42:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524631+0000", 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524632+0000", 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2799999999999996e-07 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524653+0000", 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1313000000000001e-05 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524660+0000", 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1570000000000003e-06 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.524666+0000", 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5260000000000001e-06 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.525135+0000", 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00046915500000000002 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.525144+0000", 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5640000000000002e-06 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473660+0000", 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.948515891 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473669+0000", 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9889999999999998e-06 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.569430+0000", 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0957612380000001 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.569571+0000", 2024-09-13T14:42:00.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014031699999999999 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.569750+0000", 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017882600000000001 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.574171+0000", 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044219209999999997 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.575933+0000", 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017618200000000001 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.580801+0000", 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048680620000000003 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.580879+0000", 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7381999999999999e-05 2024-09-13T14:42:00.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.582272+0000", 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013932339999999999 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:34926 7.4 7:24a90a1c:::benchmark_data_smithi003_122851_object34925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:32:08.537149+0000", 2024-09-13T14:42:00.588 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 592.02963395500001, 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0461130230000002, 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 34926 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.537149+0000", 2024-09-13T14:42:00.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.537149+0000", 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.537152+0000", 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3300000000000001e-06 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.537192+0000", 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9981999999999998e-05 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.537216+0000", 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3782e-05 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.537229+0000", 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3054000000000001e-05 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.537306+0000", 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7095000000000006e-05 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.537319+0000", 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3368e-05 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473694+0000", 2024-09-13T14:42:00.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93637445900000005 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473701+0000", 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6690000000000006e-06 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.572392+0000", 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0986911340000001 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.572630+0000", 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023729400000000001 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.572775+0000", 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014567900000000001 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.578533+0000", 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005757552 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.578586+0000", 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2821999999999997e-05 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.582327+0000", 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037415069999999998 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.582384+0000", 2024-09-13T14:42:00.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6891000000000001e-05 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.583262+0000", 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00087840499999999996 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:34934 7.9 7:9c1fbeeb:::benchmark_data_smithi003_122851_object34933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T14:42:00.596 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:32:08.540535+0000", 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 592.02624856499995, 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0236921140000002, 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 34934 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.540535+0000", 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.540535+0000", 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.540536+0000", 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6499999999999998e-07 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.540559+0000", 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3095000000000002e-05 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.540570+0000", 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1085e-05 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.540577+0000", 2024-09-13T14:42:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7479999999999996e-06 2024-09-13T14:42:00.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.540704+0000", 2024-09-13T14:42:00.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012731099999999999 2024-09-13T14:42:00.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.540713+0000", 2024-09-13T14:42:00.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0559999999999994e-06 2024-09-13T14:42:00.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473131+0000", 2024-09-13T14:42:00.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93241785300000002 2024-09-13T14:42:00.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473142+0000", 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1259e-05 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.557174+0000", 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0840314850000001 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.557371+0000", 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019776200000000001 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.557572+0000", 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020096799999999999 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.560966+0000", 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033937720000000002 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.561021+0000", 2024-09-13T14:42:00.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4460999999999999e-05 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.563484+0000", 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024629919999999998 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.563542+0000", 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8189000000000001e-05 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.564227+0000", 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00068521299999999995 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:34936 7.9 7:98851f9a:::benchmark_data_smithi003_122851_object34935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:32:08.543873+0000", 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 592.02291002899995, 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.021456937, 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 34936 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.543873+0000", 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.543873+0000", 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.543875+0000", 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4780000000000001e-06 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.543936+0000", 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1631000000000007e-05 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.543954+0000", 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7739000000000001e-05 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.543964+0000", 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6069999999999997e-06 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.544036+0000", 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1841000000000006e-05 2024-09-13T14:42:00.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.544046+0000", 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0178e-05 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473791+0000", 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92974508199999994 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473799+0000", 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8860000000000004e-06 2024-09-13T14:42:00.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.559167+0000", 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.085368538 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.559339+0000", 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017169200000000001 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.559560+0000", 2024-09-13T14:42:00.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022122600000000001 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.561157+0000", 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015971539999999999 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.561308+0000", 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000150667 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.564293+0000", 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029852210000000001 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.564467+0000", 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000173654 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.565330+0000", 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00086334300000000003 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:34940 7.4 7:2ee13b57:::benchmark_data_smithi003_122851_object34939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:32:08.545772+0000", 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 592.02101134999998, 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0419830960000001, 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.613 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 34940 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.545772+0000", 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.545772+0000", 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.545774+0000", 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.44e-06 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.545804+0000", 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0688000000000002e-05 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.545823+0000", 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9236e-05 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.545833+0000", 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5240000000000006e-06 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.545945+0000", 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111978 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.545955+0000", 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0526000000000001e-05 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473726+0000", 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92777091 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473734+0000", 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2710000000000003e-06 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.574210+0000", 2024-09-13T14:42:00.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1004763720000001 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.574343+0000", 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133399 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.574497+0000", 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015344800000000001 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.578598+0000", 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041013519999999999 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.578775+0000", 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017681700000000001 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.586264+0000", 2024-09-13T14:42:00.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074892279999999997 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.586326+0000", 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1258000000000002e-05 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.587755+0000", 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014296490000000001 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:34948 7.4 7:24ac5b6d:::benchmark_data_smithi003_122851_object34947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:32:08.554279+0000", 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 592.01250413699995, 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0351117990000001, 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.622 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 34948 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.554279+0000", 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.554279+0000", 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.554280+0000", 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.074e-06 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.554309+0000", 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8311999999999999e-05 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.554324+0000", 2024-09-13T14:42:00.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5051999999999999e-05 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.554332+0000", 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0020000000000006e-06 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.554406+0000", 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4176000000000003e-05 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:08.554419+0000", 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3307e-05 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473756+0000", 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91933712899999998 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:09.473764+0000", 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5089999999999996e-06 2024-09-13T14:42:00.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.575965+0000", 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.102200769 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.576114+0000", 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000148966 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.576276+0000", 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000162201 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.578604+0000", 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023278330000000001 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.580754+0000", 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002150332 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.587987+0000", 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072326719999999999 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.588177+0000", 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019010500000000001 2024-09-13T14:42:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:32:11.589391+0000", 2024-09-13T14:42:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00121436 2024-09-13T14:42:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200914 7.e 7:7560cf08:::benchmark_data_smithi003_122851_object89796:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:30.472473+0000", 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 390.09431076099997, 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3351282979999999, 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200914 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472473+0000", 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472473+0000", 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472474+0000", 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.711e-06 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472483+0000", 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3319999999999992e-06 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472496+0000", 2024-09-13T14:42:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3682e-05 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472506+0000", 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8789999999999998e-06 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472566+0000", 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0161999999999998e-05 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472578+0000", 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1669e-05 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.767184+0000", 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2946060619999999 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.771503+0000", 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043188499999999999 2024-09-13T14:42:00.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.772054+0000", 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00055080299999999999 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.786326+0000", 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014272674000000001 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.807219+0000", 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020892085000000001 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.807273+0000", 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4376000000000003e-05 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.807363+0000", 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0026999999999997e-05 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.807601+0000", 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023798600000000001 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200919 7.e 7:7e04b78c:::benchmark_data_smithi003_122851_object89801:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:30.472524+0000", 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 390.09425963699999, 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3354817940000001, 2024-09-13T14:42:00.638 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200919 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472524+0000", 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472524+0000", 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472524+0000", 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5199999999999997e-07 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472527+0000", 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8159999999999998e-06 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472535+0000", 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0900000000000005e-06 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472542+0000", 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3600000000000001e-06 2024-09-13T14:42:00.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472624+0000", 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2052999999999995e-05 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472634+0000", 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9890000000000004e-06 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.786369+0000", 2024-09-13T14:42:00.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.313735573 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.786530+0000", 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000160414 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.786694+0000", 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000164065 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.797002+0000", 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010307818 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.807615+0000", 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01061395 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.807765+0000", 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014977799999999999 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.807840+0000", 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4333999999999997e-05 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.808006+0000", 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000166002 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200931 7.e 7:7c500fda:::benchmark_data_smithi003_122851_object89813:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:30.472553+0000", 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 390.09423034600002, 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3363158239999999, 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200931 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472553+0000", 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472553+0000", 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472554+0000", 2024-09-13T14:42:00.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1699999999999998e-07 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472556+0000", 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5600000000000001e-06 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472564+0000", 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9489999999999996e-06 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472570+0000", 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9120000000000003e-06 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472883+0000", 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031264200000000002 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.472892+0000", 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6670000000000008e-06 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.791604+0000", 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.318711741 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.791754+0000", 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014947100000000001 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.791926+0000", 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000172518 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.797053+0000", 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051274299999999997 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.807626+0000", 2024-09-13T14:42:00.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010572056999999999 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.808671+0000", 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010453109999999999 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.808735+0000", 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4548000000000006e-05 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:31.808869+0000", 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013350100000000001 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:214060 7.4 7:239d34de:::benchmark_data_smithi003_122851_object102942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:51.115344+0000", 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 369.45143978099998, 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.036788724, 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.653 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 214060 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.115344+0000", 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.115344+0000", 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.115345+0000", 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5960000000000001e-06 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.115351+0000", 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8690000000000001e-06 2024-09-13T14:42:00.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.115364+0000", 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3230999999999999e-05 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.115373+0000", 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8780000000000004e-06 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.115749+0000", 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037627400000000001 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.115759+0000", 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8619999999999998e-06 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.142415+0000", 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0266551829999999 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.142543+0000", 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012844899999999999 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.142637+0000", 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4415999999999999e-05 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.151909+0000", 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092720360000000009 2024-09-13T14:42:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.151964+0000", 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4691000000000002e-05 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152034+0000", 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9727999999999994e-05 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152088+0000", 2024-09-13T14:42:00.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4088999999999997e-05 2024-09-13T14:42:00.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152132+0000", 2024-09-13T14:42:00.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4422000000000003e-05 2024-09-13T14:42:00.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:214063 7.4 7:2f39cb8d:::benchmark_data_smithi003_122851_object102945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:51.122548+0000", 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 369.44423588400002, 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0298209149999999, 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 214063 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122548+0000", 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122548+0000", 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122550+0000", 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0660000000000002e-06 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122556+0000", 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2430000000000002e-06 2024-09-13T14:42:00.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122575+0000", 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9276999999999999e-05 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122583+0000", 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0709999999999995e-06 2024-09-13T14:42:00.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122760+0000", 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000176866 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122771+0000", 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0668e-05 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.147530+0000", 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0247595169999999 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.147642+0000", 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112043 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.147719+0000", 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7106000000000003e-05 2024-09-13T14:42:00.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152223+0000", 2024-09-13T14:42:00.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004503557 2024-09-13T14:42:00.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152280+0000", 2024-09-13T14:42:00.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6613999999999997e-05 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152291+0000", 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1800000000000001e-05 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152335+0000", 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3948e-05 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152368+0000", 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3139000000000002e-05 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:214065 7.4 7:2322a5d1:::benchmark_data_smithi003_122851_object102947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:51.122604+0000", 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 369.44417989599998, 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.038983153, 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.669 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 214065 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122604+0000", 2024-09-13T14:42:00.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122604+0000", 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122604+0000", 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9100000000000004e-07 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122607+0000", 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5170000000000002e-06 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122613+0000", 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.331e-06 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122618+0000", 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8110000000000002e-06 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122792+0000", 2024-09-13T14:42:00.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017461500000000001 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.122799+0000", 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3010000000000003e-06 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152412+0000", 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.029613012 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152493+0000", 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1830999999999996e-05 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.152566+0000", 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2908000000000001e-05 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.161423+0000", 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0088567439999999997 2024-09-13T14:42:00.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.161475+0000", 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1734999999999998e-05 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.161487+0000", 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2488e-05 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.161538+0000", 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0760000000000002e-05 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.161587+0000", 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8609000000000002e-05 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:214075 7.4 7:2a21e3ce:::benchmark_data_smithi003_122851_object102957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T14:42:00.677 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:51.135265+0000", 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 369.43151824099999, 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.035692396, 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 214075 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.135265+0000", 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.135265+0000", 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.135266+0000", 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.091e-06 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.135272+0000", 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2379999999999997e-06 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.135283+0000", 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1365e-05 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.135289+0000", 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5910000000000001e-06 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.139971+0000", 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046817380000000004 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:51.139982+0000", 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0302999999999999e-05 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.156750+0000", 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.016768949 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.156890+0000", 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013928700000000001 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.157021+0000", 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131373 2024-09-13T14:42:00.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.161614+0000", 2024-09-13T14:42:00.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045929220000000002 2024-09-13T14:42:00.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.166216+0000", 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046018539999999998 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.170821+0000", 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046052879999999999 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.170906+0000", 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4473000000000003e-05 2024-09-13T14:42:00.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:52.170958+0000", 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1924e-05 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:53919 8.c 8:3c8894e3:::benchmark_data_smithi003_139172_object53918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:36:46.522566+0000", 2024-09-13T14:42:00.685 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 314.04421780500002, 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.030694461, 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 53919 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.522566+0000", 2024-09-13T14:42:00.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.522566+0000", 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.522566+0000", 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6000000000000003e-07 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.523754+0000", 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001187848 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.523773+0000", 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8754999999999999e-05 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.523781+0000", 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999999e-06 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.523908+0000", 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012730799999999999 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.524036+0000", 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012819099999999999 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.524226+0000", 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018940800000000001 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549262+0000", 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025035999 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549273+0000", 2024-09-13T14:42:00.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0954e-05 2024-09-13T14:42:00.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553061+0000", 2024-09-13T14:42:00.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.003788667 2024-09-13T14:42:00.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553260+0000", 2024-09-13T14:42:00.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019860099999999999 2024-09-13T14:42:00.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.692 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:00.694 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:42:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:42:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:00.694 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:53924 8.c 8:3b907ba2:::benchmark_data_smithi003_139172_object53923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:42:00.695 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:36:46.526036+0000", 2024-09-13T14:42:00.695 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 314.04074722500002, 2024-09-13T14:42:00.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0276176050000001, 2024-09-13T14:42:00.695 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.695 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.695 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:42:00.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:00.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 53924 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526036+0000", 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526036+0000", 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526038+0000", 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.387e-06 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526061+0000", 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.304e-05 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526077+0000", 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6300999999999999e-05 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526084+0000", 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5109999999999997e-06 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527077+0000", 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0009923480000000001 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527347+0000", 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000270661 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527602+0000", 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025452299999999999 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549305+0000", 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021703324 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549314+0000", 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2439999999999993e-06 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553519+0000", 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.004205867 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553654+0000", 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000134399 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:53925 8.c 8:32910906:::benchmark_data_smithi003_139172_object53924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:42:00.702 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:36:46.526979+0000", 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 314.039804683, 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0417181719999999, 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 53925 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526979+0000", 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526979+0000", 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526980+0000", 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.187e-06 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527001+0000", 2024-09-13T14:42:00.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0908e-05 2024-09-13T14:42:00.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527013+0000", 2024-09-13T14:42:00.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2203e-05 2024-09-13T14:42:00.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527020+0000", 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1119999999999998e-06 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.541776+0000", 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014755819999999999 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.541947+0000", 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017083699999999999 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.542127+0000", 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000180003 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549346+0000", 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072187060000000001 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549354+0000", 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4910000000000005e-06 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553698+0000", 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0043439009999999 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.554556+0000", 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00085828399999999996 2024-09-13T14:42:00.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.568579+0000", 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014023066000000001 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.568697+0000", 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117654 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85389 8.c 8:32251d57:::benchmark_data_smithi003_139172_object85388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:07.765151+0000", 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 292.80163284600002, 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1862126320000002, 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:00.710 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85389 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765151+0000", 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765151+0000", 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765152+0000", 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.486e-06 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765419+0000", 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000266588 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765435+0000", 2024-09-13T14:42:00.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6041000000000001e-05 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765443+0000", 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3010000000000007e-06 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765544+0000", 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101245 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765557+0000", 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3161e-05 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.739019+0000", 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97346153300000005 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.739031+0000", 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2259000000000001e-05 2024-09-13T14:42:00.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.941457+0000", 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.202425351 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.941615+0000", 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015840799999999999 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.941794+0000", 2024-09-13T14:42:00.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178684 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.949747+0000", 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079533920000000001 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.949769+0000", 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2130999999999999e-05 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.949818+0000", 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9157999999999999e-05 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.949901+0000", 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2967999999999996e-05 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.951363+0000", 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014619259999999999 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.717 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85397 8.c 8:37504a20:::benchmark_data_smithi003_139172_object85396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:07.783684+0000", 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 292.78309932100001, 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1708010880000002, 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:00.718 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85397 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783684+0000", 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783684+0000", 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783685+0000", 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.299e-06 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783709+0000", 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3929000000000001e-05 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783724+0000", 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4225e-05 2024-09-13T14:42:00.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783731+0000", 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2570000000000001e-06 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.784233+0000", 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050265399999999995 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.784242+0000", 2024-09-13T14:42:00.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9490000000000002e-06 2024-09-13T14:42:00.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.739057+0000", 2024-09-13T14:42:00.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95481418100000004 2024-09-13T14:42:00.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.739064+0000", 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7950000000000008e-06 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.946256+0000", 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2071917219999999 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.946424+0000", 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016787 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.946771+0000", 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000346588 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.951533+0000", 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047621060000000003 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.951546+0000", 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3358e-05 2024-09-13T14:42:00.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.952975+0000", 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014288339999999999 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.953033+0000", 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7717999999999999e-05 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.954485+0000", 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001452603 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:156289 8.c 8:3a176f0a:::benchmark_data_smithi003_139172_object31496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:38:28.608616+0000", 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 211.95816744199999, 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0276335910000001, 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:00.726 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 156289 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608616+0000", 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608616+0000", 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608617+0000", 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4079999999999999e-06 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608623+0000", 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8379999999999998e-06 2024-09-13T14:42:00.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608640+0000", 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7215e-05 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608648+0000", 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8280000000000003e-06 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608712+0000", 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4113000000000001e-05 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608724+0000", 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1267000000000001e-05 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.623532+0000", 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0148086999999999 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.623608+0000", 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5431000000000003e-05 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.623726+0000", 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118579 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636094+0000", 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012368048 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636109+0000", 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4365999999999999e-05 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:00.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636145+0000", 2024-09-13T14:42:00.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6273000000000003e-05 2024-09-13T14:42:00.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:00.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636202+0000", 2024-09-13T14:42:00.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7065999999999999e-05 2024-09-13T14:42:00.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:00.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:00.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:00.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636250+0000", 2024-09-13T14:42:00.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7459000000000001e-05 2024-09-13T14:42:00.734 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.734 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.734 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.734 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:00.734 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:00.734 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:42:00.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:00.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 15 23549 23534 183.84 155.438 0.00581077 0.00543363 2024-09-13T14:42:00.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:01.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:01.054 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:01.156 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:01.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:00 smithi003 ceph-mon[104007]: pgmap v518: 89 pgs: 89 active+clean; 1.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-13T14:42:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:00 smithi003 ceph-mon[107278]: pgmap v518: 89 pgs: 89 active+clean; 1.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-13T14:42:01.192 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:01.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:01.328 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:01.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:01.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:00 smithi019 ceph-mon[91296]: pgmap v518: 89 pgs: 89 active+clean; 1.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-13T14:42:01.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:01.555 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:01.592 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:01.694 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:01.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:01.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:01.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 25794 25778 178.996 140.25 0.00557107 0.00558128 2024-09-13T14:42:01.865 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:01.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:01.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:42:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:02.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:02.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:02.143 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:02.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:02.395 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:02.418 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:02.519 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:02.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:02.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:02.683 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:02.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:02.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:02.856 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 27517 27501 171.864 107.688 0.00924995 0.00581255 2024-09-13T14:42:02.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:02.947 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:03.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:03.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:03.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:03.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:03.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:03.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:02 smithi019 ceph-mon[91296]: pgmap v519: 89 pgs: 89 active+clean; 1.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:42:03.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:02 smithi003 ceph-mon[107278]: pgmap v519: 89 pgs: 89 active+clean; 1.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:42:03.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:02 smithi003 ceph-mon[104007]: pgmap v519: 89 pgs: 89 active+clean; 1.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:42:03.448 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:03.482 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:03.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:03.623 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:03.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:03.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:03.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 29450 29434 167.222 120.812 0.00859413 0.0059741 2024-09-13T14:42:03.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:03.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:04.004 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:04.040 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:04.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:04.164 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:04.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:04.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:04.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:42:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:42:04] "GET /metrics HTTP/1.1" 200 36461 "" "Prometheus/2.51.0" 2024-09-13T14:42:04.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:04.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:04.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:04.676 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:04.818 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:04.840 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:04.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 31249 31233 162.656 112.438 0.00875452 0.00611444 2024-09-13T14:42:04.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:04.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:04.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:05.084 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:05.120 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:05.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:05.251 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:05.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:05.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:04 smithi019 ceph-mon[91296]: pgmap v520: 89 pgs: 89 active+clean; 1.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-13T14:42:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:04 smithi003 ceph-mon[107278]: pgmap v520: 89 pgs: 89 active+clean; 1.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-13T14:42:05.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:04 smithi003 ceph-mon[104007]: pgmap v520: 89 pgs: 89 active+clean; 1.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-13T14:42:05.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:05.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:05.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:05.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:05.770 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.808 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:05.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:05.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 33401 33385 160.489 134.5 0.00246656 0.00622283 2024-09-13T14:42:05.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:05.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:06.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.078 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:06.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:06.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:06.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:06.335 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:06.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:06.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:06.580 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:06.615 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:06.717 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:06.752 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:06.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:06.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 35177 35161 156.954 111 0.0118663 0.00636592 2024-09-13T14:42:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:06.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:06.997 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:07.030 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:07.132 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:07.165 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:07.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:06 smithi003 ceph-mon[104007]: pgmap v521: 89 pgs: 89 active+clean; 1.8 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-13T14:42:07.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:07 smithi003 ceph-mon[107278]: pgmap v521: 89 pgs: 89 active+clean; 1.8 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-13T14:42:07.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:07.300 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:07.402 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:07.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:07 smithi019 ceph-mon[91296]: pgmap v521: 89 pgs: 89 active+clean; 1.8 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-13T14:42:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:07.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:07.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:07.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:07.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:07.856 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 36956 36940 153.902 111.188 0.00543167 0.00649117 2024-09-13T14:42:07.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:07.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:07.984 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:08.085 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:08.107 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:08.209 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:08.247 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:08.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:08.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:08.379 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:08.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:08.518 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:08.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:08.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:08.763 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:08.798 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:08.856 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 38436 38420 150.064 92.5 0.00374871 0.00665861 2024-09-13T14:42:08.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:08.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:09.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:09.080 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:09.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:09.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:09.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:09.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:09.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:09.401 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:09 smithi003 ceph-mon[104007]: pgmap v522: 89 pgs: 89 active+clean; 1.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-09-13T14:42:09.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:09 smithi003 ceph-mon[107278]: pgmap v522: 89 pgs: 89 active+clean; 1.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-09-13T14:42:09.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:09 smithi019 ceph-mon[91296]: pgmap v522: 89 pgs: 89 active+clean; 1.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-09-13T14:42:09.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:42:09.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:09.504 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:09.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:09.606 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:09.639 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:09.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:09.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:09.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:09.856 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 40400 40384 148.457 122.75 0.00837782 0.0067302 2024-09-13T14:42:09.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:09.924 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:10.026 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:10.062 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:10.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:10.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:10.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:10.304 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:10.326 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:10.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:10.464 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:10.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:42:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:42:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:10.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:10.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:10.849 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:10.856 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 41896 41880 145.403 93.5 0.00669915 0.00687226 2024-09-13T14:42:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.887 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:10.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:10.988 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:11.021 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:11.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:11.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:11.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:11.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:11.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:11.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:11.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:11 smithi019 ceph-mon[91296]: pgmap v523: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-13T14:42:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:11 smithi003 ceph-mon[107278]: pgmap v523: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-13T14:42:11.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:11 smithi003 ceph-mon[104007]: pgmap v523: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-13T14:42:11.450 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:11.552 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:11.587 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:11.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:11.598 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:11.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:11.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:11.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:11.839 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:11.856 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 43813 43797 144.051 119.812 0.00775503 0.00693634 2024-09-13T14:42:11.857 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:11.959 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:11.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:11.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:12.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:12.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:12.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:12.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:12.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:12.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:12.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:12.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:12.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.727 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:12.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:12.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:12.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:42:12.858008+0000 min lat: 0.00126908 max lat: 0.258047 avg lat: 0.00701929 2024-09-13T14:42:12.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:42:12.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 45577 45561 142.353 110.25 0.00894356 0.00701929 2024-09-13T14:42:12.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:12.970 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:13.003 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:13.029 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 9 pg_num to 26 2024-09-13T14:42:13.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.139 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:13.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:13.161 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:42:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:13.205 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:42:13.205 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:13.237 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:42:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:13.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:13 smithi003 ceph-mon[104007]: pgmap v524: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T14:42:13.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:13 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1551931247' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:42:13.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:13.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:13 smithi003 ceph-mon[107278]: pgmap v524: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T14:42:13.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:13 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1551931247' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:42:13.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:13.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:13.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:13.280 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:42:13.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:13.316 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:42:13.316 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:13.351 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:42:13.351 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:13.381 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:42:13.381 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:13.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:13.413 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:42:13.413 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:13.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:13.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:13 smithi019 ceph-mon[91296]: pgmap v524: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T14:42:13.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:13 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1551931247' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:42:13.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:13.526 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:13.559 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:13.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:13.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:13.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:13.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:13.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 47386 47370 140.958 113.062 0.00615048 0.00708913 2024-09-13T14:42:13.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:13.971 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:14.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:14.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:14.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1551931247' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:42:14.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:14 smithi003 ceph-mon[107278]: osdmap e158: 8 total, 8 up, 8 in 2024-09-13T14:42:14.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2027655921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:14.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:14 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:42:14.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:42:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:42:14] "GET /metrics HTTP/1.1" 200 36466 "" "Prometheus/2.51.0" 2024-09-13T14:42:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1551931247' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:42:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:14 smithi003 ceph-mon[104007]: osdmap e158: 8 total, 8 up, 8 in 2024-09-13T14:42:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2027655921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:14.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:14 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:42:14.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:14.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:14.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:14.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:14.374 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:14.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:14.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1551931247' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:42:14.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:14 smithi019 ceph-mon[91296]: osdmap e158: 8 total, 8 up, 8 in 2024-09-13T14:42:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2027655921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:14.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:14 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:42:14.475 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:14.498 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:14.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:14.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:14.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:14.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:14.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:14.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 47764 47748 135.625 23.625 0.00365146 0.00710801 2024-09-13T14:42:14.876 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:14.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:15.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:15.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:15.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:15.246 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:15.275 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:15.377 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:15.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:15.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:15 smithi019 ceph-mon[91296]: pgmap v526: 89 pgs: 89 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-09-13T14:42:15.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:42:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:15 smithi019 ceph-mon[91296]: osdmap e159: 8 total, 8 up, 8 in 2024-09-13T14:42:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:15 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:42:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:15 smithi003 ceph-mon[104007]: pgmap v526: 89 pgs: 89 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-09-13T14:42:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:42:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:15 smithi003 ceph-mon[104007]: osdmap e159: 8 total, 8 up, 8 in 2024-09-13T14:42:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:15 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:42:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:15 smithi003 ceph-mon[107278]: pgmap v526: 89 pgs: 89 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-09-13T14:42:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:42:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:15 smithi003 ceph-mon[107278]: osdmap e159: 8 total, 8 up, 8 in 2024-09-13T14:42:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:15 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:42:15.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:15.544 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:15.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:15.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:15.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:15.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:15.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 48645 48629 132.122 55.0625 0.0030187 0.00756334 2024-09-13T14:42:15.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:15.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:16.067 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:16.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:16 smithi019 ceph-mon[91296]: osdmap e160: 8 total, 8 up, 8 in 2024-09-13T14:42:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:16 smithi019 ceph-mon[91296]: 9.2 scrub starts 2024-09-13T14:42:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:16 smithi019 ceph-mon[91296]: 9.4 scrub starts 2024-09-13T14:42:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:16 smithi019 ceph-mon[91296]: 9.5 scrub starts 2024-09-13T14:42:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:16 smithi019 ceph-mon[91296]: 9.2 scrub ok 2024-09-13T14:42:16.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:16.227 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:16.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:16.368 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:16.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:16.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[107278]: osdmap e160: 8 total, 8 up, 8 in 2024-09-13T14:42:16.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[107278]: 9.2 scrub starts 2024-09-13T14:42:16.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[107278]: 9.4 scrub starts 2024-09-13T14:42:16.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[107278]: 9.5 scrub starts 2024-09-13T14:42:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[107278]: 9.2 scrub ok 2024-09-13T14:42:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[104007]: osdmap e160: 8 total, 8 up, 8 in 2024-09-13T14:42:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[104007]: 9.2 scrub starts 2024-09-13T14:42:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[104007]: 9.4 scrub starts 2024-09-13T14:42:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[104007]: 9.5 scrub starts 2024-09-13T14:42:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:16 smithi003 ceph-mon[104007]: 9.2 scrub ok 2024-09-13T14:42:16.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:16.512 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:16.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:16.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:16.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:16.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:16.771 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.793 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:16.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:16.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 50093 50077 130.388 90.5 0.00425694 0.00766452 2024-09-13T14:42:16.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:16.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:17.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:17.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:17 smithi019 ceph-mon[91296]: pgmap v529: 99 pgs: 2 peering, 10 unknown, 87 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-13T14:42:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:17 smithi019 ceph-mon[91296]: 9.4 scrub ok 2024-09-13T14:42:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:17 smithi019 ceph-mon[91296]: 9.5 scrub ok 2024-09-13T14:42:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:17 smithi019 ceph-mon[91296]: 9.6 scrub starts 2024-09-13T14:42:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:17 smithi019 ceph-mon[91296]: 9.1 scrub starts 2024-09-13T14:42:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:17 smithi019 ceph-mon[91296]: 9.9 scrub starts 2024-09-13T14:42:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:17 smithi019 ceph-mon[91296]: 9.1 scrub ok 2024-09-13T14:42:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:17 smithi019 ceph-mon[91296]: 9.6 scrub ok 2024-09-13T14:42:17.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:17.304 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.346 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:17.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:17.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[104007]: pgmap v529: 99 pgs: 2 peering, 10 unknown, 87 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[104007]: 9.4 scrub ok 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[104007]: 9.5 scrub ok 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[104007]: 9.6 scrub starts 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[104007]: 9.1 scrub starts 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[104007]: 9.9 scrub starts 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[104007]: 9.1 scrub ok 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[104007]: 9.6 scrub ok 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[107278]: pgmap v529: 99 pgs: 2 peering, 10 unknown, 87 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[107278]: 9.4 scrub ok 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[107278]: 9.5 scrub ok 2024-09-13T14:42:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[107278]: 9.6 scrub starts 2024-09-13T14:42:17.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[107278]: 9.1 scrub starts 2024-09-13T14:42:17.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[107278]: 9.9 scrub starts 2024-09-13T14:42:17.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[107278]: 9.1 scrub ok 2024-09-13T14:42:17.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:17 smithi003 ceph-mon[107278]: 9.6 scrub ok 2024-09-13T14:42:17.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:17.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:17.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:17.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:17.729 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:17.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:17.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 51888 51872 129.659 112.188 0.00643248 0.00770746 2024-09-13T14:42:17.866 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.907 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:17.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:18.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.053 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:18.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:18.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:18.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:18 smithi019 ceph-mon[91296]: 9.14 scrub starts 2024-09-13T14:42:18.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:18 smithi019 ceph-mon[91296]: 9.14 scrub ok 2024-09-13T14:42:18.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:18 smithi019 ceph-mon[91296]: 9.9 scrub ok 2024-09-13T14:42:18.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:18 smithi019 ceph-mon[91296]: 9.11 scrub starts 2024-09-13T14:42:18.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:18 smithi019 ceph-mon[91296]: 9.15 scrub starts 2024-09-13T14:42:18.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:18 smithi019 ceph-mon[91296]: 9.11 scrub ok 2024-09-13T14:42:18.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[107278]: 9.14 scrub starts 2024-09-13T14:42:18.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[107278]: 9.14 scrub ok 2024-09-13T14:42:18.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[107278]: 9.9 scrub ok 2024-09-13T14:42:18.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[107278]: 9.11 scrub starts 2024-09-13T14:42:18.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[107278]: 9.15 scrub starts 2024-09-13T14:42:18.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[107278]: 9.11 scrub ok 2024-09-13T14:42:18.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[104007]: 9.14 scrub starts 2024-09-13T14:42:18.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[104007]: 9.14 scrub ok 2024-09-13T14:42:18.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[104007]: 9.9 scrub ok 2024-09-13T14:42:18.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[104007]: 9.11 scrub starts 2024-09-13T14:42:18.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[104007]: 9.15 scrub starts 2024-09-13T14:42:18.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:18 smithi003 ceph-mon[104007]: 9.11 scrub ok 2024-09-13T14:42:18.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:18.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:18.331 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:18.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:18.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:18.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:18.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:18.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:18.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:18.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:18.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 53773 53757 129.203 117.812 0.00748002 0.00773512 2024-09-13T14:42:18.860 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.910 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:18.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:19.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.053 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:19.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:19.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:19.176 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:19 smithi019 ceph-mon[91296]: 9.10 scrub starts 2024-09-13T14:42:19.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:19 smithi019 ceph-mon[91296]: pgmap v530: 99 pgs: 2 peering, 7 unknown, 90 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-13T14:42:19.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:19 smithi019 ceph-mon[91296]: 9.10 scrub ok 2024-09-13T14:42:19.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:19 smithi019 ceph-mon[91296]: 9.15 scrub ok 2024-09-13T14:42:19.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:19 smithi019 ceph-mon[91296]: 9.12 scrub starts 2024-09-13T14:42:19.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:19 smithi019 ceph-mon[91296]: 9.19 scrub starts 2024-09-13T14:42:19.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:19 smithi019 ceph-mon[91296]: 9.12 scrub ok 2024-09-13T14:42:19.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:19.295 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:19.334 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[107278]: 9.10 scrub starts 2024-09-13T14:42:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[107278]: pgmap v530: 99 pgs: 2 peering, 7 unknown, 90 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-13T14:42:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[107278]: 9.10 scrub ok 2024-09-13T14:42:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[107278]: 9.15 scrub ok 2024-09-13T14:42:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[107278]: 9.12 scrub starts 2024-09-13T14:42:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[107278]: 9.19 scrub starts 2024-09-13T14:42:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[107278]: 9.12 scrub ok 2024-09-13T14:42:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[104007]: 9.10 scrub starts 2024-09-13T14:42:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[104007]: pgmap v530: 99 pgs: 2 peering, 7 unknown, 90 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-13T14:42:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[104007]: 9.10 scrub ok 2024-09-13T14:42:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[104007]: 9.15 scrub ok 2024-09-13T14:42:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[104007]: 9.12 scrub starts 2024-09-13T14:42:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[104007]: 9.19 scrub starts 2024-09-13T14:42:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:19 smithi003 ceph-mon[104007]: 9.12 scrub ok 2024-09-13T14:42:19.437 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:19.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:19.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:19.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:19.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:19.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:19.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 55712 55696 128.906 121.188 0.0026888 0.00775202 2024-09-13T14:42:19.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:19.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:20.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.059 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:20.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:20.105 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:42:20.106 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:42:20.106 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-13T14:42:20.106 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:42:20.106 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:42:20.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-13T14:42:20.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:20.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:20 smithi019 ceph-mon[91296]: 9.0 scrub starts 2024-09-13T14:42:20.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:20 smithi019 ceph-mon[91296]: 9.0 scrub ok 2024-09-13T14:42:20.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:20 smithi019 ceph-mon[91296]: 9.19 scrub ok 2024-09-13T14:42:20.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:20 smithi019 ceph-mon[91296]: 9.7 scrub starts 2024-09-13T14:42:20.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:20 smithi019 ceph-mon[91296]: 9.8 scrub starts 2024-09-13T14:42:20.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:20 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:42:20.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:20 smithi019 ceph-mon[91296]: 9.8 scrub ok 2024-09-13T14:42:20.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:20 smithi019 ceph-mon[91296]: 9.7 scrub ok 2024-09-13T14:42:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:20.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:20.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[107278]: 9.0 scrub starts 2024-09-13T14:42:20.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[107278]: 9.0 scrub ok 2024-09-13T14:42:20.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[107278]: 9.19 scrub ok 2024-09-13T14:42:20.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[107278]: 9.7 scrub starts 2024-09-13T14:42:20.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[107278]: 9.8 scrub starts 2024-09-13T14:42:20.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:42:20.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[107278]: 9.8 scrub ok 2024-09-13T14:42:20.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[107278]: 9.7 scrub ok 2024-09-13T14:42:20.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[104007]: 9.0 scrub starts 2024-09-13T14:42:20.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[104007]: 9.0 scrub ok 2024-09-13T14:42:20.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[104007]: 9.19 scrub ok 2024-09-13T14:42:20.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[104007]: 9.7 scrub starts 2024-09-13T14:42:20.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[104007]: 9.8 scrub starts 2024-09-13T14:42:20.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:42:20.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[104007]: 9.8 scrub ok 2024-09-13T14:42:20.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:20 smithi003 ceph-mon[104007]: 9.7 scrub ok 2024-09-13T14:42:20.304 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:20.330 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:20.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:42:20.432 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:20.457 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:20.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:20.680 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:42:20.680 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:42:20.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:42:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:42:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:20.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:20.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:20.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:20.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 56585 56569 126.251 54.5625 0.00331443 0.00774152 2024-09-13T14:42:20.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:21.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:21.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:21.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:21.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:21.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:21.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: pgmap v531: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 20 KiB/s, 0 objects/s recovering 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: osdmap e161: 8 total, 8 up, 8 in 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: 9.18 scrub starts 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: osdmap e162: 8 total, 8 up, 8 in 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: 9.18 scrub ok 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/707730504' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:21 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: pgmap v531: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 20 KiB/s, 0 objects/s recovering 2024-09-13T14:42:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:42:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: osdmap e161: 8 total, 8 up, 8 in 2024-09-13T14:42:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: 9.18 scrub starts 2024-09-13T14:42:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: osdmap e162: 8 total, 8 up, 8 in 2024-09-13T14:42:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: 9.18 scrub ok 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/707730504' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: pgmap v531: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 20 KiB/s, 0 objects/s recovering 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: osdmap e161: 8 total, 8 up, 8 in 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: 9.18 scrub starts 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: osdmap e162: 8 total, 8 up, 8 in 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: 9.18 scrub ok 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/707730504' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:21 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:21.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:21.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:21.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:42:21.492 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:42:21.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:21.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:21.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:21.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:21.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:21.858 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 56585 56569 121.898 0 - 0.00774152 2024-09-13T14:42:21.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:21.984 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:22.085 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:22.112 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:22.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:22.238 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:22.340 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:22.379 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:22.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:22 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:42:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:22 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:42:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:22 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:22 smithi019 ceph-mon[91296]: osdmap e163: 8 total, 8 up, 8 in 2024-09-13T14:42:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:22 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T14:42:22.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:42:22.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:42:22.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:22.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[104007]: osdmap e163: 8 total, 8 up, 8 in 2024-09-13T14:42:22.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T14:42:22.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:42:22.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:42:22.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:22.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[107278]: osdmap e163: 8 total, 8 up, 8 in 2024-09-13T14:42:22.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:22 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T14:42:22.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:22.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:22.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:22.657 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:22.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:22.788 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:22.859 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 57505 57489 119.751 28.75 0.00252888 0.00830968 2024-09-13T14:42:22.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:22.914 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:23.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:23.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:23.142 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:23.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:23.289 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:23.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:23.393 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:23.415 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:23.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:23 smithi019 ceph-mon[91296]: pgmap v535: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:42:23.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:23 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T14:42:23.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:23 smithi019 ceph-mon[91296]: osdmap e164: 8 total, 8 up, 8 in 2024-09-13T14:42:23.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:23 smithi003 ceph-mon[107278]: pgmap v535: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:42:23.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:23 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T14:42:23.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:23 smithi003 ceph-mon[107278]: osdmap e164: 8 total, 8 up, 8 in 2024-09-13T14:42:23.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:23 smithi003 ceph-mon[104007]: pgmap v535: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:42:23.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:23 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T14:42:23.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:23 smithi003 ceph-mon[104007]: osdmap e164: 8 total, 8 up, 8 in 2024-09-13T14:42:23.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:23.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:23.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:23.783 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:23.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:23.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:23.859 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 58119 58103 117.126 38.375 0.00646311 0.00853231 2024-09-13T14:42:23.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.959 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:23.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:24.060 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:24.092 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:24.097 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:42:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:42:24] "GET /metrics HTTP/1.1" 200 36466 "" "Prometheus/2.51.0" 2024-09-13T14:42:24.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:24.228 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:24.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:24.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:24.479 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:24.516 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:24.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:24.618 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:24.649 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:24.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:24.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:24.859 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 59902 59886 116.948 111.438 0.0029469 0.00850585 2024-09-13T14:42:24.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:24.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:24.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:24 smithi019 ceph-mon[91296]: 9.17 scrub starts 2024-09-13T14:42:24.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:24 smithi019 ceph-mon[91296]: osdmap e165: 8 total, 8 up, 8 in 2024-09-13T14:42:24.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:24 smithi019 ceph-mon[91296]: pgmap v538: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail 2024-09-13T14:42:24.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-13T14:42:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:24 smithi003 ceph-mon[107278]: 9.17 scrub starts 2024-09-13T14:42:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:24 smithi003 ceph-mon[107278]: osdmap e165: 8 total, 8 up, 8 in 2024-09-13T14:42:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:24 smithi003 ceph-mon[107278]: pgmap v538: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail 2024-09-13T14:42:24.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-13T14:42:24.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:24 smithi003 ceph-mon[104007]: 9.17 scrub starts 2024-09-13T14:42:24.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:24 smithi003 ceph-mon[104007]: osdmap e165: 8 total, 8 up, 8 in 2024-09-13T14:42:24.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:24 smithi003 ceph-mon[104007]: pgmap v538: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail 2024-09-13T14:42:24.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-13T14:42:25.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:25.034 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:25.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:25.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:25.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:25.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:25.397 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:25.419 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:25.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:25.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:25.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:25.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:25.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:25.770 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:25.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-13T14:42:25.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:25 smithi003 ceph-mon[107278]: osdmap e166: 8 total, 8 up, 8 in 2024-09-13T14:42:25.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:25 smithi003 ceph-mon[107278]: osdmap e167: 8 total, 8 up, 8 in 2024-09-13T14:42:25.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-13T14:42:25.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:25 smithi003 ceph-mon[104007]: osdmap e166: 8 total, 8 up, 8 in 2024-09-13T14:42:25.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:25 smithi003 ceph-mon[104007]: osdmap e167: 8 total, 8 up, 8 in 2024-09-13T14:42:25.859 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 59931 59915 113.459 1.8125 0.00178541 0.00850291 2024-09-13T14:42:25.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:25.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:25 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-13T14:42:25.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:25 smithi019 ceph-mon[91296]: osdmap e166: 8 total, 8 up, 8 in 2024-09-13T14:42:25.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:25 smithi019 ceph-mon[91296]: osdmap e167: 8 total, 8 up, 8 in 2024-09-13T14:42:26.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:26.042 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:26.144 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:26.165 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:26.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:26.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:26.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:26.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:26.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:26.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:26.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:26.642 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:26.666 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:26.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:42:26.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:26.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:26.829 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[104007]: 9.16 scrub starts 2024-09-13T14:42:26.829 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[104007]: 9.16 scrub ok 2024-09-13T14:42:26.829 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[104007]: pgmap v541: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 5167/173532 objects misplaced (2.978%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:42:26.829 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-13T14:42:26.829 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-13T14:42:26.829 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[104007]: osdmap e168: 8 total, 8 up, 8 in 2024-09-13T14:42:26.829 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[107278]: 9.16 scrub starts 2024-09-13T14:42:26.830 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[107278]: 9.16 scrub ok 2024-09-13T14:42:26.830 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[107278]: pgmap v541: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 5167/173532 objects misplaced (2.978%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:42:26.830 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-13T14:42:26.830 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-13T14:42:26.830 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:26 smithi003 ceph-mon[107278]: osdmap e168: 8 total, 8 up, 8 in 2024-09-13T14:42:26.859 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 60197 60181 110.611 16.625 0.00484033 0.00902433 2024-09-13T14:42:26.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:26 smithi019 ceph-mon[91296]: 9.16 scrub starts 2024-09-13T14:42:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:26 smithi019 ceph-mon[91296]: 9.16 scrub ok 2024-09-13T14:42:26.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:26 smithi019 ceph-mon[91296]: pgmap v541: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 5167/173532 objects misplaced (2.978%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:42:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-13T14:42:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:26 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-13T14:42:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:26 smithi019 ceph-mon[91296]: osdmap e168: 8 total, 8 up, 8 in 2024-09-13T14:42:26.944 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:26.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:27.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.072 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:27.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:27.173 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.197 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:27.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:27.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:27.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:27.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:27.446 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:27.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.577 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:27.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:27.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:27.803 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:27.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:27.859 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 60488 60472 107.971 18.1875 0.00296681 0.00900258 2024-09-13T14:42:27.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:27.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:27 smithi019 ceph-mon[91296]: 9.17 scrub starts 2024-09-13T14:42:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.6", "id": [5, 6]} : dispatch 2024-09-13T14:42:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.d", "id": [5, 4]} : dispatch 2024-09-13T14:42:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]} : dispatch 2024-09-13T14:42:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.6", "id": [5, 6]}]': finished 2024-09-13T14:42:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.d", "id": [5, 4]}]': finished 2024-09-13T14:42:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]': finished 2024-09-13T14:42:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:27 smithi019 ceph-mon[91296]: osdmap e169: 8 total, 8 up, 8 in 2024-09-13T14:42:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:42:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[107278]: 9.17 scrub starts 2024-09-13T14:42:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.6", "id": [5, 6]} : dispatch 2024-09-13T14:42:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.d", "id": [5, 4]} : dispatch 2024-09-13T14:42:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]} : dispatch 2024-09-13T14:42:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.6", "id": [5, 6]}]': finished 2024-09-13T14:42:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.d", "id": [5, 4]}]': finished 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]': finished 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[107278]: osdmap e169: 8 total, 8 up, 8 in 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[104007]: 9.17 scrub starts 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.6", "id": [5, 6]} : dispatch 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.d", "id": [5, 4]} : dispatch 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]} : dispatch 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.6", "id": [5, 6]}]': finished 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.d", "id": [5, 4]}]': finished 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.10", "id": [7, 2]}]': finished 2024-09-13T14:42:27.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[104007]: osdmap e169: 8 total, 8 up, 8 in 2024-09-13T14:42:27.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:42:27.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:28.053 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:28.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:28.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:28.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:28.303 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:28.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:28.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:28.552 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:28.581 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:28.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:28.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:28.814 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:28.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:28.859 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 60589 60573 105.147 6.3125 0.00292844 0.00946881 2024-09-13T14:42:28.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:28.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:29.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:29.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:28 smithi019 ceph-mon[91296]: pgmap v544: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 8.7 MiB/s wr, 138 op/s; 5167/175758 objects misplaced (2.940%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:42:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:42:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:28 smithi019 ceph-mon[91296]: osdmap e170: 8 total, 8 up, 8 in 2024-09-13T14:42:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[107278]: pgmap v544: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 8.7 MiB/s wr, 138 op/s; 5167/175758 objects misplaced (2.940%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:42:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:42:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[107278]: osdmap e170: 8 total, 8 up, 8 in 2024-09-13T14:42:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[104007]: pgmap v544: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 8.7 MiB/s wr, 138 op/s; 5167/175758 objects misplaced (2.940%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:42:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:42:29.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:28 smithi003 ceph-mon[104007]: osdmap e170: 8 total, 8 up, 8 in 2024-09-13T14:42:29.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:29.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:29.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:29.440 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:29.463 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:29.565 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:29.590 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:29.692 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:29.720 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:29.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:29.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:29.861 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 60859 60843 102.761 16.875 0.00335335 0.00972666 2024-09-13T14:42:29.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:30.000 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:30.102 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:30.140 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:30.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:30.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:30.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:30.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:30.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:30.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:30.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:30.616 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:42:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:42:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:30.659 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:30.696 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:30.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:30.859 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 62319 62303 102.458 91.25 0.0149781 0.00975487 2024-09-13T14:42:30.868 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:30 smithi003 ceph-mon[107278]: osdmap e171: 8 total, 8 up, 8 in 2024-09-13T14:42:30.868 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:30 smithi003 ceph-mon[107278]: pgmap v547: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 94 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 14 MiB/s wr, 218 op/s; 3700/181614 objects degraded (2.037%); 3454/181614 objects misplaced (1.902%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:42:30.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:30 smithi003 ceph-mon[104007]: osdmap e171: 8 total, 8 up, 8 in 2024-09-13T14:42:30.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:30 smithi003 ceph-mon[104007]: pgmap v547: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 94 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 14 MiB/s wr, 218 op/s; 3700/181614 objects degraded (2.037%); 3454/181614 objects misplaced (1.902%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:42:30.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:30 smithi019 ceph-mon[91296]: osdmap e171: 8 total, 8 up, 8 in 2024-09-13T14:42:30.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:30 smithi019 ceph-mon[91296]: pgmap v547: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 94 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 14 MiB/s wr, 218 op/s; 3700/181614 objects degraded (2.037%); 3454/181614 objects misplaced (1.902%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:42:30.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:30.969 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:31.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:31.108 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:31.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:31.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:31.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:31.486 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:31.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:31.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:31.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:31.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:31.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:31.799 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:31.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:31.860 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 63830 63814 102.252 94.4375 0.00939995 0.00977217 2024-09-13T14:42:31.901 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:31.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:31 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 3700/181614 objects degraded (2.037%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:31.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:31 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 3700/181614 objects degraded (2.037%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:31.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:31 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 3700/181614 objects degraded (2.037%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:31.939 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:31.953 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:42:31.953 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:42:31.953 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:42:32.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:32.084 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:32.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:32.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:32.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:32.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:32.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:32.490 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:32.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:32.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:32.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:32.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:32.860 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:42:32.859956+0000 min lat: 0.00126908 max lat: 2.1988 avg lat: 0.00977877 2024-09-13T14:42:32.860 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:42:32.860 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 65414 65398 102.17 99 0.00363658 0.00977877 2024-09-13T14:42:32.877 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:32.899 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:32.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:32 smithi019 ceph-mon[91296]: pgmap v548: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 94 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 11 MiB/s wr, 173 op/s; 3700/181614 objects degraded (2.037%); 3454/181614 objects misplaced (1.902%); 6.0 MiB/s, 95 objects/s recovering 2024-09-13T14:42:32.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:32 smithi003 ceph-mon[107278]: pgmap v548: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 94 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 11 MiB/s wr, 173 op/s; 3700/181614 objects degraded (2.037%); 3454/181614 objects misplaced (1.902%); 6.0 MiB/s, 95 objects/s recovering 2024-09-13T14:42:32.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:32 smithi003 ceph-mon[104007]: pgmap v548: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 94 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 11 MiB/s wr, 173 op/s; 3700/181614 objects degraded (2.037%); 3454/181614 objects misplaced (1.902%); 6.0 MiB/s, 95 objects/s recovering 2024-09-13T14:42:33.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:33.045 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:33.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:33.184 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:33.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:33.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:33.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:33.458 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:33.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:33.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:33.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:33.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:33.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:33.861 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 66738 66722 101.694 82.75 0.0125141 0.009827 2024-09-13T14:42:33.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:33.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:34.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:34.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:34.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:34.188 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:34.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:34.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:34.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:34.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:34.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:42:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:42:34] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-09-13T14:42:34.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:34.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:34.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:34.690 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:34.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:34.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:34.861 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 68143 68127 101.364 87.8125 0.0196754 0.00986021 2024-09-13T14:42:34.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:34 smithi019 ceph-mon[91296]: pgmap v549: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 94 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 8.8 MiB/s wr, 141 op/s; 3700/183774 objects degraded (2.013%); 3454/183774 objects misplaced (1.879%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:42:34.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:34.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:34 smithi003 ceph-mon[107278]: pgmap v549: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 94 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 8.8 MiB/s wr, 141 op/s; 3700/183774 objects degraded (2.013%); 3454/183774 objects misplaced (1.879%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:42:34.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:34 smithi003 ceph-mon[104007]: pgmap v549: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 94 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 8.8 MiB/s wr, 141 op/s; 3700/183774 objects degraded (2.013%); 3454/183774 objects misplaced (1.879%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:42:34.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:35.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:35.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:35.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:35.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:35.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:35.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:35.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:35.522 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:35.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:35.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:35.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:35.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:35.861 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 69511 69495 100.994 85.5 0.00618921 0.00989428 2024-09-13T14:42:35.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:35.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:35.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10994/198993 objects degraded (5.525%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:35.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10994/198993 objects degraded (5.525%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:35.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10994/198993 objects degraded (5.525%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:36.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:36.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:36.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:36.209 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:36.311 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:36.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:36.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:36.478 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:36.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:36.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:36.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:36.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:36.861 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 70988 70972 100.797 92.3125 0.0101856 0.00991622 2024-09-13T14:42:36.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:36.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:36.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:36 smithi003 ceph-mon[107278]: pgmap v550: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 46 MiB/s wr, 740 op/s; 10994/198993 objects degraded (5.525%); 4168/198993 objects misplaced (2.095%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:42:36.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:36 smithi003 ceph-mon[104007]: pgmap v550: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 46 MiB/s wr, 740 op/s; 10994/198993 objects degraded (5.525%); 4168/198993 objects misplaced (2.095%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:42:37.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:37.055 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:37.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:37.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:36 smithi019 ceph-mon[91296]: pgmap v550: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 46 MiB/s wr, 740 op/s; 10994/198993 objects degraded (5.525%); 4168/198993 objects misplaced (2.095%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:42:37.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:37.300 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:37.334 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:37.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:37.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:37.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:37.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:37.727 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:42:37.727 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:42:37.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:38.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:38.053 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 72368 72352 100.473 86.25 0.00507485 0.00994775 2024-09-13T14:42:38.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:38.189 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:38.291 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:38.326 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:38.424 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:42:38.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:38.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:38.461 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:42:38.461 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:38.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:38.495 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:42:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:38.530 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:42:38.531 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:38.565 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:42:38.565 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:38.587 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:38.600 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:42:38.600 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:38.617 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:38.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:38.631 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:42:38.631 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:38.664 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:42:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:38.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:38.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.756 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:38.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:38.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:38.861 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 73758 73742 100.178 86.875 0.00938951 0.0099765 2024-09-13T14:42:38.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:39.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:39.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:39.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:39.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:39.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:39.307 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:38 smithi003 ceph-mon[104007]: pgmap v551: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 50 MiB/s wr, 801 op/s; 10994/202905 objects degraded (5.418%); 4168/202905 objects misplaced (2.054%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:42:39.307 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:38 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:39.307 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1342533213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:39.310 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:38 smithi003 ceph-mon[107278]: pgmap v551: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 50 MiB/s wr, 801 op/s; 10994/202905 objects degraded (5.418%); 4168/202905 objects misplaced (2.054%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:42:39.310 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:38 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:39.310 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1342533213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:39.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:39.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:39.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:38 smithi019 ceph-mon[91296]: pgmap v551: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 50 MiB/s wr, 801 op/s; 10994/202905 objects degraded (5.418%); 4168/202905 objects misplaced (2.054%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:42:39.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:38 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:39.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1342533213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:39.461 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:39.563 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:39.599 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:39.701 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:39.737 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:39.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:39.861 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 75269 75253 100.055 94.4375 0.0177285 0.00998996 2024-09-13T14:42:39.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:39.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:40.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:40.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:40.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:40.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:40.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:40.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:40.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:40.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:40.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:40.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:40.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:40.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:42:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:42:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:40.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:40.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:40.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:40.861 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 76770 76754 99.9251 93.8125 0.0121456 0.0100024 2024-09-13T14:42:40.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:40.999 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:41.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:42:41.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:41.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:40 smithi003 ceph-mon[107278]: pgmap v552: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 10994/220107 objects degraded (4.995%); 3630/220107 objects misplaced (1.649%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:41.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:40 smithi003 ceph-mon[104007]: pgmap v552: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 10994/220107 objects degraded (4.995%); 3630/220107 objects misplaced (1.649%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:41.268 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:41.306 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:41.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:41.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:40 smithi019 ceph-mon[91296]: pgmap v552: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 10994/220107 objects degraded (4.995%); 3630/220107 objects misplaced (1.649%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:41.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.449 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:41.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:41.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:41.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:41.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:41.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:41.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:41.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:41.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:41.862 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 78177 78161 99.6801 87.9375 0.0139819 0.010027 2024-09-13T14:42:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:41.963 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:42.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:42.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:42.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:42.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:42.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:42.430 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:42.489 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:42.532 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:42.564 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:42.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:42.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:42.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:42.862 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 79440 79424 99.265 78.9375 0.0223066 0.0100689 2024-09-13T14:42:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:42.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:42.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:43.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:43.123 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:43.168 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:43.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:43.314 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:43.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[107278]: pgmap v553: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 10994/220107 objects degraded (4.995%); 3630/220107 objects misplaced (1.649%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:42:43.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:43.356 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:43.357 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3314099414' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:43.357 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:43.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[104007]: pgmap v553: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 10994/220107 objects degraded (4.995%); 3630/220107 objects misplaced (1.649%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:42:43.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:43.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:43.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3314099414' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:43.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:42 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:43.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:42 smithi019 ceph-mon[91296]: pgmap v553: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 10994/220107 objects degraded (4.995%); 3630/220107 objects misplaced (1.649%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:42:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:42 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3314099414' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:42 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:43.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:43.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:43.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:43.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:42:43.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:43.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:43.790 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:42:43.855 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:43.862 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 80762 80746 98.9386 82.625 0.0196709 0.0101006 2024-09-13T14:42:43.884 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:43.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:44.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:44.114 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:42:44.114 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:42:44.115 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T14:42:44.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-13T14:42:44.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:44.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:44.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:44.294 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:44 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:44.294 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:44 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:42:44.294 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:44 smithi003 ceph-mon[107278]: osdmap e172: 8 total, 8 up, 8 in 2024-09-13T14:42:44.295 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:42:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:42:44] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-09-13T14:42:44.295 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:44 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:44.295 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:44 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:42:44.295 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:44 smithi003 ceph-mon[104007]: osdmap e172: 8 total, 8 up, 8 in 2024-09-13T14:42:44.311 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:44.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:44.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:44 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:44.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:44 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:42:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:44 smithi019 ceph-mon[91296]: osdmap e172: 8 total, 8 up, 8 in 2024-09-13T14:42:44.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.455 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:44.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:44.556 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:44.597 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:44.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:44.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:44.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:44.862 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 82173 82157 98.7318 88.1875 0.0102586 0.010123 2024-09-13T14:42:44.888 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:44.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:45.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:45.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:45.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:45.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:45.313 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:45.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:45 smithi019 ceph-mon[91296]: pgmap v554: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 10994/224088 objects degraded (4.906%); 3630/224088 objects misplaced (1.620%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:42:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10994/224088 objects degraded (4.906%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:45.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:45 smithi003 ceph-mon[104007]: pgmap v554: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 10994/224088 objects degraded (4.906%); 3630/224088 objects misplaced (1.620%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:42:45.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10994/224088 objects degraded (4.906%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:45 smithi003 ceph-mon[107278]: pgmap v554: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 10994/224088 objects degraded (4.906%); 3630/224088 objects misplaced (1.620%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:42:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10994/224088 objects degraded (4.906%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:45.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:45.553 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:45.589 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:45.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:45.731 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:45.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:45.862 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 83559 83543 98.5032 86.625 0.0164404 0.0101469 2024-09-13T14:42:45.868 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:45.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:46.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:46.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:191965 7.b 7:d6636894:::benchmark_data_smithi003_122851_object80847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:16.954453+0000", 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 449.17907301000002, 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.045611316, 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 191965 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.954453+0000", 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.954453+0000", 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.954455+0000", 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8139999999999999e-06 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.954464+0000", 2024-09-13T14:42:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4849999999999992e-06 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.954541+0000", 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7343999999999997e-05 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.954567+0000", 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.616e-05 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.970607+0000", 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016040201 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.970640+0000", 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2223e-05 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.982073+0000", 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.011433719 2024-09-13T14:42:46.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.982160+0000", 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6459999999999996e-05 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.982692+0000", 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053256500000000003 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.999844+0000", 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017151381 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.999903+0000", 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9540000000000003e-05 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.999966+0000", 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2947000000000006e-05 2024-09-13T14:42:46.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.000018+0000", 2024-09-13T14:42:46.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1811000000000003e-05 2024-09-13T14:42:46.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.000065+0000", 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6665999999999997e-05 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:191978 7.b 7:dce5a533:::benchmark_data_smithi003_122851_object80860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:16.975456+0000", 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 449.15807032200001, 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.033550178, 2024-09-13T14:42:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 191978 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.975456+0000", 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.975456+0000", 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.975458+0000", 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9080000000000002e-06 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.975463+0000", 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4020000000000001e-06 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.975480+0000", 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6194000000000001e-05 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.975487+0000", 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.17e-06 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.975553+0000", 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6215000000000002e-05 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:16.975563+0000", 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.023e-05 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.996019+0000", 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0204560549999999 2024-09-13T14:42:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.996149+0000", 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000129331 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.996283+0000", 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000134816 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.999950+0000", 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036666509999999999 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.000109+0000", 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000158598 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.008914+0000", 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0088055589999999993 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.008963+0000", 2024-09-13T14:42:46.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8572000000000002e-05 2024-09-13T14:42:46.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.009006+0000", 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3476999999999997e-05 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:192006 7.b 7:dc86c8c4:::benchmark_data_smithi003_122851_object80888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:17.011824+0000", 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 449.12170235999997, 2024-09-13T14:42:46.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0005264140000001, 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 192006 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.011824+0000", 2024-09-13T14:42:46.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.011824+0000", 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.011826+0000", 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1210000000000001e-06 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.011834+0000", 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6279999999999996e-06 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.011871+0000", 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6831e-05 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.011880+0000", 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.149e-06 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.012066+0000", 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000186047 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:17.012094+0000", 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8025999999999999e-05 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.000200+0000", 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98810636200000002 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.000292+0000", 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1372000000000005e-05 2024-09-13T14:42:46.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.000362+0000", 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0438999999999994e-05 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.012194+0000", 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011831611000000001 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.012202+0000", 2024-09-13T14:42:46.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4859999999999997e-06 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.012240+0000", 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7475999999999998e-05 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.012298+0000", 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8789000000000002e-05 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:18.012351+0000", 2024-09-13T14:42:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2077000000000003e-05 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:53936 8.7 8:e2b0e6a1:::benchmark_data_smithi003_139172_object53935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:36:46.539341+0000", 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 359.59418587699997, 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0088115660000001, 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.163 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 53936 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.539341+0000", 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.539341+0000", 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.539343+0000", 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8909999999999999e-06 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.539375+0000", 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1690999999999999e-05 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.539388+0000", 2024-09-13T14:42:46.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2499e-05 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.539398+0000", 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8640000000000008e-06 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.541134+0000", 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017368850000000001 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-13T14:42:46.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.541142+0000", 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2819999999999999e-06 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.546169+0000", 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005027541 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.546178+0000", 2024-09-13T14:42:46.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3310000000000004e-06 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.541175+0000", 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99499710799999996 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.541341+0000", 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016671399999999999 2024-09-13T14:42:46.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 6", 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.541488+0000", 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014655500000000001 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.545362+0000", 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038742889999999999 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.547955+0000", 2024-09-13T14:42:46.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025923999999999999 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.548027+0000", 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1989000000000005e-05 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.548152+0000", 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012552700000000001 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.170 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85394 8.7 8:e0051f26:::benchmark_data_smithi003_139172_object85393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:07.777586+0000", 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 338.35594038199997, 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1836367669999999, 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.171 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85394 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.777586+0000", 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.777586+0000", 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.777587+0000", 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7899999999999998e-07 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.777609+0000", 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1974e-05 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.777617+0000", 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4030000000000007e-06 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.777623+0000", 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0510000000000002e-06 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.777892+0000", 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026873799999999997 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.777902+0000", 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0566e-05 2024-09-13T14:42:46.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.744504+0000", 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96660128199999995 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.744512+0000", 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.755e-06 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.946435+0000", 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2019229419999999 2024-09-13T14:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.946568+0000", 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013228299999999999 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.946744+0000", 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017588900000000001 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.950932+0000", 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041885150000000003 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.950991+0000", 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8854000000000003e-05 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.958683+0000", 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076923820000000002 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.958738+0000", 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4805000000000001e-05 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.961223+0000", 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024847490000000001 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85362 8.f 8:ffe7732d:::benchmark_data_smithi003_139172_object85361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e138)", 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:08.740068+0000", 2024-09-13T14:42:46.178 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 337.39345809899999, 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.21853323, 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85362 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740068+0000", 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740068+0000", 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740070+0000", 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9690000000000001e-06 2024-09-13T14:42:46.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740133+0000", 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2700999999999995e-05 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740146+0000", 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3030999999999999e-05 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740156+0000", 2024-09-13T14:42:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0059e-05 2024-09-13T14:42:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740239+0000", 2024-09-13T14:42:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2614000000000006e-05 2024-09-13T14:42:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-13T14:42:46.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740248+0000", 2024-09-13T14:42:46.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8540000000000003e-06 2024-09-13T14:42:46.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.744468+0000", 2024-09-13T14:42:46.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004220629 2024-09-13T14:42:46.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.744477+0000", 2024-09-13T14:42:46.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1579999999999994e-06 2024-09-13T14:42:46.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:46.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:46.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.956913+0000", 2024-09-13T14:42:46.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.212435921 2024-09-13T14:42:46.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.958602+0000", 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016882939999999999 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85366 8.b 8:dedc80ff:::benchmark_data_smithi003_139172_object85365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e138)", 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:08.740170+0000", 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 337.393356074, 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.183052099, 2024-09-13T14:42:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85366 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740170+0000", 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740170+0000", 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740171+0000", 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.75e-07 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740229+0000", 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8013000000000002e-05 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740238+0000", 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9600000000000006e-06 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740245+0000", 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8560000000000001e-06 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740299+0000", 2024-09-13T14:42:46.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3773000000000002e-05 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740309+0000", 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0413e-05 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.921671+0000", 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.181362257 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.923222+0000", 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015511520000000001 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85373 8.0 8:0acbbb63:::benchmark_data_smithi003_139172_object85372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e138)", 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:08.740265+0000", 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 337.39326191599997, 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.215601215, 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:42:46.192 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85373 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740265+0000", 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740265+0000", 2024-09-13T14:42:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740265+0000", 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4099999999999999e-07 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740315+0000", 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9518000000000003e-05 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740324+0000", 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8859999999999999e-06 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740331+0000", 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.404e-06 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.741803+0000", 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001472587 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.741812+0000", 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5969999999999992e-06 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.953475+0000", 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2116628970000001 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.955866+0000", 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023907849999999999 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85375 8.0 8:09b74f24:::benchmark_data_smithi003_139172_object85374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e138)", 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:08.740342+0000", 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 337.39318469199998, 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2163280000000001, 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85375 2024-09-13T14:42:46.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740342+0000", 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740342+0000", 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740342+0000", 2024-09-13T14:42:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5599999999999995e-07 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740401+0000", 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8548000000000003e-05 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740410+0000", 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4769999999999992e-06 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740417+0000", 2024-09-13T14:42:46.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1999999999999999e-06 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.741837+0000", 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001420312 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.741844+0000", 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5279999999999998e-06 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.956575+0000", 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2147303 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.956670+0000", 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5079000000000002e-05 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85376 8.0 8:030311ff:::benchmark_data_smithi003_139172_object85375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e138)", 2024-09-13T14:42:46.201 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:08.740426+0000", 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 337.39310000699999, 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.244452935, 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85376 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740426+0000", 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740426+0000", 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740427+0000", 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6499999999999999e-07 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740476+0000", 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9119000000000002e-05 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740484+0000", 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0469999999999994e-06 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740489+0000", 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.378e-06 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.741879+0000", 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013892030000000001 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.741886+0000", 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0419999999999997e-06 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.956698+0000", 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.214812206 2024-09-13T14:42:46.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.957327+0000", 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00062928400000000003 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.957489+0000", 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016228400000000001 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.967646+0000", 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010156926 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.972368+0000", 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047212790000000001 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.983456+0000", 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01108871 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.983615+0000", 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015835299999999999 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.984879+0000", 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012646389999999999 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85377 8.b 8:d33fa6e3:::benchmark_data_smithi003_139172_object85376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e138)", 2024-09-13T14:42:46.208 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:08.740499+0000", 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 337.39302789700002, 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1842192730000001, 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85377 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740499+0000", 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740499+0000", 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740499+0000", 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9900000000000001e-07 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740546+0000", 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7382999999999997e-05 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740555+0000", 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4479999999999994e-06 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.740561+0000", 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2020000000000001e-06 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.745002+0000", 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044410550000000002 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.745011+0000", 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8359999999999998e-06 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.924594+0000", 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1795833170000001 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.924718+0000", 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123533 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85379 8.0 8:09501ee7:::benchmark_data_smithi003_139172_object85378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e138)", 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:08.742552+0000", 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 337.39097411799997, 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2441934889999999, 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.213 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85379 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742552+0000", 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742552+0000", 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742553+0000", 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0899999999999999e-06 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742609+0000", 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5352000000000001e-05 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742685+0000", 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6435999999999998e-05 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742694+0000", 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7930000000000003e-06 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.743480+0000", 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00078619499999999995 2024-09-13T14:42:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.743490+0000", 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9450000000000005e-06 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.960176+0000", 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.216685894 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.960289+0000", 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112971 2024-09-13T14:42:46.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.960432+0000", 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014255499999999999 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.967653+0000", 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072216190000000003 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.980049+0000", 2024-09-13T14:42:46.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012395586 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.984930+0000", 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048810119999999997 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.984992+0000", 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2398999999999997e-05 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.986746+0000", 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017536419999999999 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85386 8.0 8:0ed0ed4b:::benchmark_data_smithi003_139172_object85385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e138)", 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:08.742729+0000", 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 337.3907974, 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.246589618, 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.220 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85386 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742729+0000", 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742729+0000", 2024-09-13T14:42:46.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742730+0000", 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6199999999999997e-07 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742785+0000", 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4849000000000003e-05 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742795+0000", 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0424e-05 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742802+0000", 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6440000000000003e-06 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.743515+0000", 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000712987 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.743522+0000", 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5100000000000001e-06 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.963466+0000", 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2199438819999999 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.963607+0000", 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014092999999999999 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.963766+0000", 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015862800000000001 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.967659+0000", 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003893311 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.983413+0000", 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01575387 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.986800+0000", 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00338763 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.987753+0000", 2024-09-13T14:42:46.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00095221300000000004 2024-09-13T14:42:46.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.989319+0000", 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001566078 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85407 8.f 8:f6c9adfe:::benchmark_data_smithi003_139172_object85406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e138)", 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:08.742812+0000", 2024-09-13T14:42:46.228 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 337.39071472099999, 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.231031164, 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85407 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742812+0000", 2024-09-13T14:42:46.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742812+0000", 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742812+0000", 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7899999999999998e-07 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742869+0000", 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6589000000000003e-05 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742879+0000", 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0002e-05 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.742886+0000", 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2049999999999996e-06 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.744538+0000", 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001651923 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.744546+0000", 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6820000000000003e-06 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.959868+0000", 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2153218429999999 2024-09-13T14:42:46.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.960591+0000", 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00072322599999999999 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.960728+0000", 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013728300000000001 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.972823+0000", 2024-09-13T14:42:46.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012094631999999999 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.972836+0000", 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2850999999999999e-05 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.972890+0000", 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4527000000000002e-05 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.972961+0000", 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0807000000000002e-05 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.973843+0000", 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00088201499999999999 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:225505 8.7 8:e4e317bb:::benchmark_data_smithi003_139172_object100712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:39:50.728037+0000", 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 175.40548944700001, 2024-09-13T14:42:46.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.050753273, 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 225505 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.728037+0000", 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.728037+0000", 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.728039+0000", 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.575e-06 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.728043+0000", 2024-09-13T14:42:46.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6450000000000004e-06 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.728052+0000", 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2359999999999993e-06 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.728059+0000", 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2650000000000002e-06 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.743570+0000", 2024-09-13T14:42:46.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015511155 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.743579+0000", 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9679999999999995e-06 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.771604+0000", 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0280254049999999 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.771698+0000", 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4185999999999996e-05 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.771773+0000", 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4482999999999998e-05 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.778667+0000", 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068941610000000002 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.778705+0000", 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8235000000000002e-05 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.778716+0000", 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.045e-05 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.778747+0000", 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0830000000000001e-05 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.778790+0000", 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3679e-05 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:225539 8.7 8:e342870d:::benchmark_data_smithi003_139172_object100746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:39:50.774397+0000", 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 175.35912996499999, 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0129596670000001, 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.242 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 225539 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.774397+0000", 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.774397+0000", 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.774398+0000", 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.61e-06 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.774405+0000", 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4479999999999999e-06 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.774421+0000", 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5999999999999999e-05 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.774428+0000", 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3499999999999999e-06 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.774513+0000", 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5290000000000005e-05 2024-09-13T14:42:46.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.774523+0000", 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0210000000000001e-05 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.775337+0000", 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0008131629999999 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.775414+0000", 2024-09-13T14:42:46.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7441999999999995e-05 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.775494+0000", 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0098000000000005e-05 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.778899+0000", 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034053830000000001 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787200+0000", 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0083000810000000008 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787255+0000", 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5087000000000003e-05 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787314+0000", 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9191000000000003e-05 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787356+0000", 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2314000000000002e-05 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:225549 8.7 8:e19bc36e:::benchmark_data_smithi003_139172_object100756:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:39:50.785847+0000", 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 175.347679833, 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0016850770000001, 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 225549 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.785847+0000", 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.785847+0000", 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.785848+0000", 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4729999999999999e-06 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.785860+0000", 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2061e-05 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.785874+0000", 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4027e-05 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.785882+0000", 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0509999999999997e-06 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.785946+0000", 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4150000000000001e-05 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.785956+0000", 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3100000000000006e-06 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.778952+0000", 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.992996659 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.779073+0000", 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012058 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.779189+0000", 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116347 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787413+0000", 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082239050000000001 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787429+0000", 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5967000000000001e-05 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787463+0000", 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4159000000000003e-05 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787493+0000", 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0005000000000002e-05 2024-09-13T14:42:46.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787532+0000", 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8383000000000002e-05 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:225555 8.7 8:e34d3d14:::benchmark_data_smithi003_139172_object100762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:39:50.791982+0000", 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 175.341544183, 2024-09-13T14:42:46.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99573471000000002, 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 225555 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.791982+0000", 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.791982+0000", 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.791983+0000", 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.051e-06 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.791987+0000", 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1139999999999999e-06 2024-09-13T14:42:46.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.791997+0000", 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2660000000000007e-06 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792003+0000", 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0009999999999998e-06 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792071+0000", 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8019e-05 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792082+0000", 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1491000000000001e-05 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.783153+0000", 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.991071221 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.783267+0000", 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011341300000000001 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.783386+0000", 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011930300000000001 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787588+0000", 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042017770000000003 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787600+0000", 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1931e-05 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787635+0000", 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5475000000000002e-05 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787682+0000", 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6748e-05 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787717+0000", 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4900000000000001e-05 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:225556 8.7 8:e44500dc:::benchmark_data_smithi003_139172_object100763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:39:50.792012+0000", 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 175.34151414900001, 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0195049199999999, 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.263 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 225556 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792012+0000", 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792012+0000", 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792013+0000", 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6899999999999998e-07 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792014+0000", 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.497e-06 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792018+0000", 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6610000000000002e-06 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792021+0000", 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.88e-06 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792103+0000", 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2399999999999997e-05 2024-09-13T14:42:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.792109+0000", 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.268e-06 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787750+0000", 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99564062099999995 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787847+0000", 2024-09-13T14:42:46.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6946000000000003e-05 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.787965+0000", 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118068 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.795794+0000", 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078287180000000001 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.803609+0000", 2024-09-13T14:42:46.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078157260000000003 2024-09-13T14:42:46.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.811394+0000", 2024-09-13T14:42:46.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077842950000000001 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.811460+0000", 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6360999999999997e-05 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.811517+0000", 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.711e-05 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:225563 8.7 8:e50eed16:::benchmark_data_smithi003_139172_object100770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:39:50.799946+0000", 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 175.333580485, 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0117036859999999, 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 225563 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.271 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.799946+0000", 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.799946+0000", 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:42:46.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.799947+0000", 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.268e-06 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.799951+0000", 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1330000000000001e-06 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.799963+0000", 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1819e-05 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.799970+0000", 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7349999999999999e-06 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.800036+0000", 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6335999999999996e-05 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:50.800045+0000", 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9360000000000005e-06 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.792310+0000", 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99226454100000006 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.792416+0000", 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106147 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.792523+0000", 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010675099999999999 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.803649+0000", 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011126504000000001 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.803662+0000", 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2568999999999999e-05 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.811563+0000", 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079017080000000003 2024-09-13T14:42:46.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.811611+0000", 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7786e-05 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:39:51.811650+0000", 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8452999999999999e-05 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:42:46.277 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:42:46.287 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:46.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:46.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:46.521 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:46.521 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:46.563 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:46.594 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:46.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:42:46.695 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:46.730 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:46.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:46.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:46.862 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 84858 84842 98.1817 81.1875 0.00690514 0.0101804 2024-09-13T14:42:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:46.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:46.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:47.016 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:47.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:47.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:47.258 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:47.286 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:47 smithi003 ceph-mon[107278]: pgmap v556: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 10994/241263 objects degraded (4.557%); 3038/241263 objects misplaced (1.259%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:42:47.287 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:47 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:47.287 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:47 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:47.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:47 smithi003 ceph-mon[104007]: pgmap v556: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 10994/241263 objects degraded (4.557%); 3038/241263 objects misplaced (1.259%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:42:47.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:47 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:47.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:47 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:47.289 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:47.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:47 smithi019 ceph-mon[91296]: pgmap v556: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 10994/241263 objects degraded (4.557%); 3038/241263 objects misplaced (1.259%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:42:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:47 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:47 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:47.543 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.594 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:47.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:47.695 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:47.735 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:47.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:47.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 86045 86029 97.7454 74.1875 0.00808809 0.0102241 2024-09-13T14:42:47.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:47.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:48.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:48.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.156 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:48.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:48.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:48.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:48.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:48 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:42:48.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:48 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:42:48.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:48 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:48.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:48 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:42:48.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:48 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:42:48.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:48 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:48.393 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:48.429 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:48.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:48 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:42:48.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:48 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:42:48.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:48 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:48.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:48.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:48.673 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.715 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:48.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:48.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:48.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:48.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:48.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 87404 87388 97.5165 84.9375 0.00841649 0.01025 2024-09-13T14:42:48.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:48.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.994 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:48.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:49.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:49.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:49.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:49.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:49 smithi003 ceph-mon[104007]: pgmap v557: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 10994/245073 objects degraded (4.486%); 3038/245073 objects misplaced (1.240%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:42:49.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:49 smithi003 ceph-mon[107278]: pgmap v557: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 10994/245073 objects degraded (4.486%); 3038/245073 objects misplaced (1.240%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:42:49.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:49.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:49.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:49 smithi019 ceph-mon[91296]: pgmap v557: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s; 10994/245073 objects degraded (4.486%); 3038/245073 objects misplaced (1.240%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:42:49.508 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:49.544 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:49.646 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:49.688 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:49.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:49.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:49.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 88786 88770 97.3209 86.375 0.0147257 0.0102705 2024-09-13T14:42:49.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:49.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:50.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:50.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:50.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:42:50.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:50.265 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:50.318 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10994/261255 objects degraded (4.208%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:50.318 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10994/261255 objects degraded (4.208%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:50.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:50.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10994/261255 objects degraded (4.208%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:50.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:42:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:42:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:50.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:50.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:50.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:50.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:50.692 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:50.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:50.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:50.831 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:50.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 90075 90059 97.0318 80.5625 0.00490756 0.0103011 2024-09-13T14:42:50.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.970 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:50.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:51.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:51.110 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:51.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:51.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:51.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:51 smithi003 ceph-mon[104007]: pgmap v558: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 10994/261255 objects degraded (4.208%); 2397/261255 objects misplaced (0.917%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:51.279 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:51 smithi003 ceph-mon[107278]: pgmap v558: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 10994/261255 objects degraded (4.208%); 2397/261255 objects misplaced (0.917%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:51.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:51.393 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:51.406 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:51.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:51 smithi019 ceph-mon[91296]: pgmap v558: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 10994/261255 objects degraded (4.208%); 2397/261255 objects misplaced (0.917%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:51.456 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:42:51.457 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:51.493 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:42:51.493 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:51.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:51.536 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:42:51.536 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:51.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:51.578 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:42:51.578 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:51.617 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:42:51.617 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:51.642 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:51.658 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:42:51.658 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:51.674 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:51.691 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:42:51.691 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:51.735 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:42:51.736 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:51.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:51.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:51.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 91363 91347 96.7513 80.5 0.0153626 0.0103308 2024-09-13T14:42:51.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:51.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:52.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:52.104 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 297 (0%) 2024-09-13T14:42:52.104 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:42:52.104 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.43113 -> 6.43113 (expected baseline 5.69951) 2024-09-13T14:42:52.104 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 24 -> 24 pgs (0.646465 -> 0.646465 * mean) 2024-09-13T14:42:52.104 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 44 -> 44 pgs (1.18519 -> 1.18519 * mean) 2024-09-13T14:42:52.105 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:42:52.105 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-09-13T14:42:52.105 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 1 2024-09-13T14:42:52.105 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:42:52.105 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0275 2024-09-13T14:42:52.105 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.2753 2024-09-13T14:42:52.105 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T14:42:52.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:52.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:52.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:52.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:52.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:52 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1894925244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:52.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:52 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1894925244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:52.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:52 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1894925244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:52.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:52.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:52.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:52.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:52.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:52.802 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:52.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:42:52.863514+0000 min lat: 0.00126908 max lat: 2.1988 avg lat: 0.0103482 2024-09-13T14:42:52.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:42:52.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 92756 92740 96.5897 87.0625 0.0053015 0.0103482 2024-09-13T14:42:52.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:52.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:52.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:53.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.084 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:53.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:53.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:53.225 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:53.300 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T14:42:53.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-13T14:42:53.327 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:53.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:53 smithi003 ceph-mon[104007]: pgmap v559: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 10994/261255 objects degraded (4.208%); 2397/261255 objects misplaced (0.917%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:53.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:53 smithi003 ceph-mon[104007]: from='client.55729 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:42:53.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:53 smithi003 ceph-mon[107278]: pgmap v559: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 10994/261255 objects degraded (4.208%); 2397/261255 objects misplaced (0.917%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:53.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:53 smithi003 ceph-mon[107278]: from='client.55729 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:42:53.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:53 smithi019 ceph-mon[91296]: pgmap v559: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 10994/261255 objects degraded (4.208%); 2397/261255 objects misplaced (0.917%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:53 smithi019 ceph-mon[91296]: from='client.55729 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:42:53.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:53.504 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:53.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:53.744 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.771 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:53.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:53.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 93972 93956 96.252 76 0.00440763 0.0103843 2024-09-13T14:42:53.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:53.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:54.010 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:54.045 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:54.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:54.185 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:54.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:54.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:54.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:42:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:42:54] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:54.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:54.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:54.699 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.742 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:54.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:54.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:54.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 95243 95227 95.9807 79.4375 0.00752889 0.0104133 2024-09-13T14:42:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:54.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:54.980 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:55.019 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:55.122 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:55.162 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:55.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:55.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:55.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:55.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:55.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:55.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:55 smithi019 ceph-mon[91296]: pgmap v560: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 10994/264783 objects degraded (4.152%); 2397/264783 objects misplaced (0.905%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:55.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:55 smithi003 ceph-mon[107278]: pgmap v560: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 10994/264783 objects degraded (4.152%); 2397/264783 objects misplaced (0.905%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:55 smithi003 ceph-mon[104007]: pgmap v560: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 10994/264783 objects degraded (4.152%); 2397/264783 objects misplaced (0.905%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:42:55.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:55.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:55.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:55.801 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:55.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 96368 96352 95.5732 70.3125 0.00295348 0.0104403 2024-09-13T14:42:55.935 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:55.961 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:56.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:56.085 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:56.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:56.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:56.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:56.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:56.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:56.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10994/264783 objects degraded (4.152%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:56.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:56 smithi019 ceph-mon[91296]: osdmap e173: 8 total, 8 up, 8 in 2024-09-13T14:42:56.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10994/264783 objects degraded (4.152%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:56.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:56 smithi003 ceph-mon[107278]: osdmap e173: 8 total, 8 up, 8 in 2024-09-13T14:42:56.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10994/264783 objects degraded (4.152%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:42:56.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:56 smithi003 ceph-mon[104007]: osdmap e173: 8 total, 8 up, 8 in 2024-09-13T14:42:56.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:56.458 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:56.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:56.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:56.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:56.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:56.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:56.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 96787 96771 94.4891 26.1875 0.00592771 0.0105776 2024-09-13T14:42:56.868 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:56.970 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:57.110 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:57.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:57.252 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:57.291 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:57.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:57.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:57 smithi019 ceph-mon[91296]: pgmap v562: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 82 MiB/s wr, 1.32k op/s; 10994/280875 objects degraded (3.914%); 1632/280875 objects misplaced (0.581%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:42:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:57 smithi019 ceph-mon[91296]: osdmap e174: 8 total, 8 up, 8 in 2024-09-13T14:42:57.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:57 smithi003 ceph-mon[107278]: pgmap v562: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 82 MiB/s wr, 1.32k op/s; 10994/280875 objects degraded (3.914%); 1632/280875 objects misplaced (0.581%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:42:57.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:57 smithi003 ceph-mon[107278]: osdmap e174: 8 total, 8 up, 8 in 2024-09-13T14:42:57.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:57 smithi003 ceph-mon[104007]: pgmap v562: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 82 MiB/s wr, 1.32k op/s; 10994/280875 objects degraded (3.914%); 1632/280875 objects misplaced (0.581%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:42:57.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:57 smithi003 ceph-mon[104007]: osdmap e174: 8 total, 8 up, 8 in 2024-09-13T14:42:57.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:57.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:57.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:57.705 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:57.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:57.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:57.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 97944 97928 94.1479 72.3125 0.0225529 0.010615 2024-09-13T14:42:57.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:57.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:58.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:58.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:58.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:58.271 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:58.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:58.415 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:58.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:58.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:58.651 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:58.691 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:58.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:58.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:58.863 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 99150 99134 93.8633 75.375 0.00538173 0.0106487 2024-09-13T14:42:58.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:58.973 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:59.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:59.119 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:58 smithi019 ceph-mon[91296]: pgmap v564: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s; 10994/284103 objects degraded (3.870%); 1632/284103 objects misplaced (0.574%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:42:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:42:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:58 smithi003 ceph-mon[107278]: pgmap v564: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s; 10994/284103 objects degraded (3.870%); 1632/284103 objects misplaced (0.574%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:42:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:42:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:58 smithi003 ceph-mon[104007]: pgmap v564: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s; 10994/284103 objects degraded (3.870%); 1632/284103 objects misplaced (0.574%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:42:59.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:42:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:42:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:59.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:59.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:59.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:59.394 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:59.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:59.530 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:59.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:59.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:59.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:59.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:59.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:42:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:59.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 100379 100363 93.6088 76.8125 0.00764321 0.0106783 2024-09-13T14:42:59.911 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:59.946 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:42:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:00.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:00.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:00.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:00.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:00.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:00.370 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:00.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:00.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:00.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:00.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:00.638 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:43:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:43:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:00.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:00.778 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:00.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:00.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 101518 101502 93.279 71.1875 0.0158502 0.0107157 2024-09-13T14:43:00.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:00.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:01.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:43:01.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:01.033 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:00 smithi003 ceph-mon[107278]: pgmap v565: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 5733/296448 objects degraded (1.934%); 5796/296448 objects misplaced (1.955%); 7.9 MiB/s, 125 objects/s recovering 2024-09-13T14:43:01.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5733/296448 objects degraded (1.934%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:01.034 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:00 smithi003 ceph-mon[104007]: pgmap v565: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 5733/296448 objects degraded (1.934%); 5796/296448 objects misplaced (1.955%); 7.9 MiB/s, 125 objects/s recovering 2024-09-13T14:43:01.034 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5733/296448 objects degraded (1.934%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.053 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:01.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:01.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:01.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:00 smithi019 ceph-mon[91296]: pgmap v565: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 5733/296448 objects degraded (1.934%); 5796/296448 objects misplaced (1.955%); 7.9 MiB/s, 125 objects/s recovering 2024-09-13T14:43:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5733/296448 objects degraded (1.934%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:01.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:01.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:01.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:01.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:01.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:01.563 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:01.604 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:01.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:01.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:01.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 102695 102679 92.9932 73.5625 0.00757048 0.0107481 2024-09-13T14:43:01.884 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:01.986 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:02.009 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:02.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:02.150 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:02.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:02.431 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:02.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.577 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:02.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:02.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:02.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:02.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:02.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 103692 103676 92.5548 62.3125 0.00383001 0.0107975 2024-09-13T14:43:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:02 smithi003 ceph-mon[107278]: pgmap v566: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 82 MiB/s wr, 1.32k op/s; 5733/296448 objects degraded (1.934%); 5796/296448 objects misplaced (1.955%); 7.9 MiB/s, 125 objects/s recovering 2024-09-13T14:43:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:02 smithi003 ceph-mon[104007]: pgmap v566: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 82 MiB/s wr, 1.32k op/s; 5733/296448 objects degraded (1.934%); 5796/296448 objects misplaced (1.955%); 7.9 MiB/s, 125 objects/s recovering 2024-09-13T14:43:02.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:02.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:03.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:03.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:02 smithi019 ceph-mon[91296]: pgmap v566: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 82 MiB/s wr, 1.32k op/s; 5733/296448 objects degraded (1.934%); 5796/296448 objects misplaced (1.955%); 7.9 MiB/s, 125 objects/s recovering 2024-09-13T14:43:03.224 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:03.257 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:03.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:03.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:03.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:03.476 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:03.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:03.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:03.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:03.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:03.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:03.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:03.827 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:03.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 104750 104734 92.1825 66.125 0.00962372 0.0108432 2024-09-13T14:43:03.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:03.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:04.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:04.111 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:04.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:04.251 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:04.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:04.353 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:04.398 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:04.422 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:43:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:43:04] "GET /metrics HTTP/1.1" 200 36494 "" "Prometheus/2.51.0" 2024-09-13T14:43:04.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:04.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:04.649 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:04.686 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:04.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:04.788 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:04.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:04.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:04.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 105909 105893 91.9082 72.4375 0.0114723 0.0108751 2024-09-13T14:43:04.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:04.978 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:05.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:05.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:04 smithi003 ceph-mon[107278]: pgmap v567: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s; 5733/299796 objects degraded (1.912%); 5796/299796 objects misplaced (1.933%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:43:05.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:04 smithi003 ceph-mon[104007]: pgmap v567: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s; 5733/299796 objects degraded (1.912%); 5796/299796 objects misplaced (1.933%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:43:05.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:05.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:04 smithi019 ceph-mon[91296]: pgmap v567: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s; 5733/299796 objects degraded (1.912%); 5796/299796 objects misplaced (1.933%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:43:05.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:05.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:05.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:05.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:05.402 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:05.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:05.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:05.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:05.688 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:05.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:05.801 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:43:05.830 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:05.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 107025 107009 91.6046 69.75 0.00838769 0.010912 2024-09-13T14:43:05.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:05.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:06.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:06.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:06.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:06.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1401210605' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:05 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5733/299796 objects degraded (1.912%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:06.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1401210605' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:06.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:05 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:06.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5733/299796 objects degraded (1.912%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:06.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1401210605' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:06.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:05 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:06.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5733/299796 objects degraded (1.912%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:06.219 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:06.260 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:06.362 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:06.404 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:06.506 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:06.545 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:06.647 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.687 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:06.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:06.789 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:06.830 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:06.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 108183 108167 91.3446 72.375 0.010799 0.0109419 2024-09-13T14:43:06.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:06.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:06.986 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:07.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:07.081 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:06 smithi003 ceph-mon[107278]: pgmap v568: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 5733/313509 objects degraded (1.829%); 4649/313509 objects misplaced (1.483%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:43:07.081 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:06 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:07.081 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:06 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:07.082 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:06 smithi003 ceph-mon[107278]: osdmap e175: 8 total, 8 up, 8 in 2024-09-13T14:43:07.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:06 smithi003 ceph-mon[104007]: pgmap v568: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 5733/313509 objects degraded (1.829%); 4649/313509 objects misplaced (1.483%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:43:07.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:06 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:07.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:06 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:07.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:06 smithi003 ceph-mon[104007]: osdmap e175: 8 total, 8 up, 8 in 2024-09-13T14:43:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:07.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:06 smithi019 ceph-mon[91296]: pgmap v568: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 5733/313509 objects degraded (1.829%); 4649/313509 objects misplaced (1.483%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:43:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:06 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:06 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:06 smithi019 ceph-mon[91296]: osdmap e175: 8 total, 8 up, 8 in 2024-09-13T14:43:07.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:07.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:07.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:07.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.395 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:07.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:07.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:07.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:07.641 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.683 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:07.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:07.700 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:07.700 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:07.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:07.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:07.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 109278 109262 91.0391 68.4375 0.0138636 0.0109791 2024-09-13T14:43:07.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:07.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:07.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:08.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:08.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:08.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:08.251 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:08.353 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:08.389 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:08.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:08.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:08.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:08.632 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.674 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:08.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:08.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:08.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:43:08.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout:moved 51 / 297 (17.1717%) 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.43113 -> 6.52759 (expected baseline 5.69951) 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 24 -> 33 pgs (0.646465 -> 0.888889 * mean) 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 44 -> 44 pgs (1.18519 -> 1.18519 * mean) 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 3 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:43:08.857 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0328 2024-09-13T14:43:08.858 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0361 2024-09-13T14:43:08.858 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 weight 1.0000 -> 0.6510 2024-09-13T14:43:08.858 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 weight 1.0000 -> 0.7067 2024-09-13T14:43:08.858 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 weight 1.0000 -> 0.7445 2024-09-13T14:43:08.858 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 weight 1.0000 -> 0.8825 2024-09-13T14:43:08.858 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T14:43:08.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 110357 110341 90.7285 67.4375 0.0103477 0.0110164 2024-09-13T14:43:08.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.995 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:08.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:09.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:09.134 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:09.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:08 smithi019 ceph-mon[91296]: pgmap v570: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 5733/316533 objects degraded (1.811%); 4649/316533 objects misplaced (1.469%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:43:09.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:08 smithi003 ceph-mon[104007]: pgmap v570: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 5733/316533 objects degraded (1.811%); 4649/316533 objects misplaced (1.469%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:43:09.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:43:08 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:43:08.852+0000 7f0f7ea9d640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-13T14:43:09.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:08 smithi003 ceph-mon[107278]: pgmap v570: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 5733/316533 objects degraded (1.811%); 4649/316533 objects misplaced (1.469%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:43:09.236 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:09.277 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:09.378 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:09.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:09.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.563 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:09.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:09.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:09.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:43:09.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:09.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:09.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:09.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 111463 111447 90.4479 69.125 0.0166363 0.0110511 2024-09-13T14:43:09.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:09 smithi003 ceph-mon[107278]: from='client.35919 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:43:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:09 smithi003 ceph-mon[104007]: from='client.35919 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:43:09.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:10.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:10.082 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:09 smithi019 ceph-mon[91296]: from='client.35919 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:43:10.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:10.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:10.182 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T14:43:10.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-09-13T14:43:10.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:10.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:10.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.430 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:10.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:10.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:43:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:43:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:10.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:10.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:10.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:10.725 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:10.725 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:10.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:10.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:10.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 112524 112508 90.1384 66.3125 0.00635036 0.0110886 2024-09-13T14:43:10.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:10 smithi003 ceph-mon[107278]: pgmap v571: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 5733/330213 objects degraded (1.736%); 3424/330213 objects misplaced (1.037%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:43:10.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5733/330213 objects degraded (1.736%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:10.879 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:10 smithi003 ceph-mon[104007]: pgmap v571: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 5733/330213 objects degraded (1.736%); 3424/330213 objects misplaced (1.037%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:43:10.879 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5733/330213 objects degraded (1.736%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:10.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:11.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:11.103 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:11.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:10 smithi019 ceph-mon[91296]: pgmap v571: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 5733/330213 objects degraded (1.736%); 3424/330213 objects misplaced (1.037%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:43:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5733/330213 objects degraded (1.736%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:11.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.289 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:11.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:11.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:11.425 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:11.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:11.566 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:11.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:11.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:11.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:11.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:11.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:11.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 113574 113558 89.8281 65.625 0.0058186 0.011127 2024-09-13T14:43:11.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.982 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:11.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:12.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:12.123 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:12.225 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:12.269 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:12.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:12.410 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:12.512 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:12.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:12.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:12.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:12.657 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:12.692 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:12.711 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:43:12.712 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:12.742 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:43:12.742 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:12.776 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:43:12.776 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:12.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:12.817 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:43:12.818 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:12.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:12.848 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:43:12.849 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:12.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:43:12.865258+0000 min lat: 0.00126908 max lat: 2.1988 avg lat: 0.0111691 2024-09-13T14:43:12.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:43:12.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 114586 114570 89.4958 63.25 0.0103646 0.0111691 2024-09-13T14:43:12.882 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:43:12.883 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:12.914 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:43:12.914 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:12.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:12.945 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:43:12.946 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:12.965 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:12 smithi003 ceph-mon[104007]: pgmap v572: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 5733/330213 objects degraded (1.736%); 3424/330213 objects misplaced (1.037%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:43:12.965 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:12 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/129573699' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:12.965 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:12.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:12 smithi003 ceph-mon[107278]: pgmap v572: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 5733/330213 objects degraded (1.736%); 3424/330213 objects misplaced (1.037%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:43:12.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:12 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/129573699' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:12.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:12.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:13.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:13.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:13.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:12 smithi019 ceph-mon[91296]: pgmap v572: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 5733/330213 objects degraded (1.736%); 3424/330213 objects misplaced (1.037%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:43:13.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:12 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/129573699' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:13.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:13.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:13.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:13.359 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.398 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:13.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:13.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:13.541 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:13.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:13.690 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:13.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:13.832 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:13.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 115676 115660 89.2319 68.125 0.00979178 0.0112009 2024-09-13T14:43:13.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:13.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:14.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.110 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:14.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:14.211 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.246 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:14.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:14.320 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:43:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:43:14] "GET /metrics HTTP/1.1" 200 36496 "" "Prometheus/2.51.0" 2024-09-13T14:43:14.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:14.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:14.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:14.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:14.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:14.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:14.809 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:14.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 116786 116770 88.9897 69.375 0.0156531 0.0112328 2024-09-13T14:43:14.911 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:14.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:15.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:15.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:14 smithi019 ceph-mon[91296]: pgmap v573: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 5733/333219 objects degraded (1.720%); 3424/333219 objects misplaced (1.028%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:43:15.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:14 smithi003 ceph-mon[107278]: pgmap v573: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 5733/333219 objects degraded (1.720%); 3424/333219 objects misplaced (1.028%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:43:15.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:14 smithi003 ceph-mon[104007]: pgmap v573: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 5733/333219 objects degraded (1.720%); 3424/333219 objects misplaced (1.028%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:43:15.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:15.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:15.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:15.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:15.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.514 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:15.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:15.576 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:43:15.615 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:15.658 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:15.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:15.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:15.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:15.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 117908 117892 88.7623 70.125 0.00350761 0.0112491 2024-09-13T14:43:15.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:15.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:16.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:16.055 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:16.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:43:16.124 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2054164741' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5733/333219 objects degraded (1.720%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[107278]: osdmap e176: 8 total, 8 up, 8 in 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2054164741' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5733/333219 objects degraded (1.720%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:16.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:15 smithi003 ceph-mon[104007]: osdmap e176: 8 total, 8 up, 8 in 2024-09-13T14:43:16.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2054164741' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:15 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:15 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5733/333219 objects degraded (1.720%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:15 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:15 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:15 smithi019 ceph-mon[91296]: osdmap e176: 8 total, 8 up, 8 in 2024-09-13T14:43:16.186 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:16.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:16.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:16.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:16.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:16.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.569 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:16.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:16.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:16.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:16.805 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:16.846 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:16.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:16.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 118573 118557 88.2004 41.5625 0.0151625 0.0113331 2024-09-13T14:43:16.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:16.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:17.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:17.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:16 smithi019 ceph-mon[91296]: pgmap v575: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 5733/346134 objects degraded (1.656%); 2056/346134 objects misplaced (0.594%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:43:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:16 smithi019 ceph-mon[91296]: osdmap e177: 8 total, 8 up, 8 in 2024-09-13T14:43:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:16 smithi003 ceph-mon[107278]: pgmap v575: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 5733/346134 objects degraded (1.656%); 2056/346134 objects misplaced (0.594%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:43:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:16 smithi003 ceph-mon[107278]: osdmap e177: 8 total, 8 up, 8 in 2024-09-13T14:43:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:16 smithi003 ceph-mon[104007]: pgmap v575: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 5733/346134 objects degraded (1.656%); 2056/346134 objects misplaced (0.594%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:43:17.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:16 smithi003 ceph-mon[104007]: osdmap e177: 8 total, 8 up, 8 in 2024-09-13T14:43:17.219 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:17.265 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:17.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.407 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:17.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:17.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:17.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:17.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:17.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:17.782 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:17.826 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:17.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 119719 119703 88.0053 71.625 0.0147275 0.0113582 2024-09-13T14:43:17.921 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:43:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:43:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:43:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:43:17.921 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:43:17.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:18.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:18.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:18.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:18.249 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:18.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:18.389 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:18.491 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:18.533 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:18.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:18.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:43:18.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:18.775 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:18.810 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:18.815 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:18.849 INFO:teuthology.orchestra.run.smithi003.stdout:moved 49 / 297 (16.4983%) 2024-09-13T14:43:18.849 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:43:18.849 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.43113 -> 6.54671 (expected baseline 5.69951) 2024-09-13T14:43:18.849 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 24 -> 33 pgs (0.646465 -> 0.888889 * mean) 2024-09-13T14:43:18.849 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 44 -> 44 pgs (1.18519 -> 1.18519 * mean) 2024-09-13T14:43:18.849 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:43:18.849 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-09-13T14:43:18.849 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 3 2024-09-13T14:43:18.849 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:43:18.850 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0359 2024-09-13T14:43:18.850 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0395 2024-09-13T14:43:18.850 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 weight 1.0000 -> 0.6558 2024-09-13T14:43:18.850 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 weight 1.0000 -> 0.7064 2024-09-13T14:43:18.850 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 weight 1.0000 -> 0.7437 2024-09-13T14:43:18.850 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 weight 1.0000 -> 0.8936 2024-09-13T14:43:18.868 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 120809 120793 87.772 68.125 0.0279309 0.0113869 2024-09-13T14:43:18.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:18.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:19.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:19.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:18 smithi019 ceph-mon[91296]: pgmap v577: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 782 op/s; 5733/348987 objects degraded (1.643%); 2056/348987 objects misplaced (0.589%); 5.3 MiB/s, 85 objects/s recovering 2024-09-13T14:43:19.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'48739','3':'58561','5':'42976','7':'46297'}"} : dispatch 2024-09-13T14:43:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:18 smithi003 ceph-mon[107278]: pgmap v577: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 782 op/s; 5733/348987 objects degraded (1.643%); 2056/348987 objects misplaced (0.589%); 5.3 MiB/s, 85 objects/s recovering 2024-09-13T14:43:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'48739','3':'58561','5':'42976','7':'46297'}"} : dispatch 2024-09-13T14:43:19.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:18 smithi003 ceph-mon[104007]: pgmap v577: 99 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 782 op/s; 5733/348987 objects degraded (1.643%); 2056/348987 objects misplaced (0.589%); 5.3 MiB/s, 85 objects/s recovering 2024-09-13T14:43:19.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'48739','3':'58561','5':'42976','7':'46297'}"} : dispatch 2024-09-13T14:43:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:19.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:19.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:43:19.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:19.439 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:19.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:19.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:19.557 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T14:43:19.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-13T14:43:19.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:19.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:19.567 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:19.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:19.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:19.817 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:19.869 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 120826 120810 86.7746 1.0625 0.0037059 0.0113883 2024-09-13T14:43:19.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:19.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:20.043 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.065 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:20.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:20.069 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:19 smithi003 ceph-mon[107278]: from='client.35931 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:43:20.070 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:19 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'48739','3':'58561','5':'42976','7':'46297'}"}]': finished 2024-09-13T14:43:20.070 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:19 smithi003 ceph-mon[107278]: osdmap e178: 8 total, 8 up, 8 in 2024-09-13T14:43:20.070 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:19 smithi003 ceph-mon[107278]: osdmap e179: 8 total, 8 up, 8 in 2024-09-13T14:43:20.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:19 smithi003 ceph-mon[104007]: from='client.35931 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:43:20.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:19 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'48739','3':'58561','5':'42976','7':'46297'}"}]': finished 2024-09-13T14:43:20.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:19 smithi003 ceph-mon[104007]: osdmap e178: 8 total, 8 up, 8 in 2024-09-13T14:43:20.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:19 smithi003 ceph-mon[104007]: osdmap e179: 8 total, 8 up, 8 in 2024-09-13T14:43:20.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:20.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:19 smithi019 ceph-mon[91296]: from='client.35931 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:43:20.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:19 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'48739','3':'58561','5':'42976','7':'46297'}"}]': finished 2024-09-13T14:43:20.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:19 smithi019 ceph-mon[91296]: osdmap e178: 8 total, 8 up, 8 in 2024-09-13T14:43:20.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:19 smithi019 ceph-mon[91296]: osdmap e179: 8 total, 8 up, 8 in 2024-09-13T14:43:20.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:20.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:20.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:20.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:20.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:20.673 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:20.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:43:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:43:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:20.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:20.869 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 120826 120810 85.7886 0 - 0.0113883 2024-09-13T14:43:20.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:20 smithi003 ceph-mon[107278]: pgmap v579: 99 pgs: 2 unknown, 17 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s; 5733/361356 objects degraded (1.587%); 1002/361356 objects misplaced (0.277%); 13 MiB/s, 201 objects/s recovering 2024-09-13T14:43:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:20 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5733/361356 objects degraded (1.587%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:20.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:20 smithi003 ceph-mon[107278]: osdmap e180: 8 total, 8 up, 8 in 2024-09-13T14:43:20.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:20 smithi003 ceph-mon[104007]: pgmap v579: 99 pgs: 2 unknown, 17 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s; 5733/361356 objects degraded (1.587%); 1002/361356 objects misplaced (0.277%); 13 MiB/s, 201 objects/s recovering 2024-09-13T14:43:20.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:20 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:20.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5733/361356 objects degraded (1.587%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:20.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:20 smithi003 ceph-mon[104007]: osdmap e180: 8 total, 8 up, 8 in 2024-09-13T14:43:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:20.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:21.057 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:21.086 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:20 smithi019 ceph-mon[91296]: pgmap v579: 99 pgs: 2 unknown, 17 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s; 5733/361356 objects degraded (1.587%); 1002/361356 objects misplaced (0.277%); 13 MiB/s, 201 objects/s recovering 2024-09-13T14:43:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:20 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5733/361356 objects degraded (1.587%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:20 smithi019 ceph-mon[91296]: osdmap e180: 8 total, 8 up, 8 in 2024-09-13T14:43:21.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:21.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:21.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:21.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:21.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:21.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:21.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:21.609 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:21.711 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:21.733 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:21.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:21.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:21.868 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 120913 120897 84.8858 2.71875 0.00314676 0.0115935 2024-09-13T14:43:21.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:21.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:22.085 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:22.108 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:22.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:21 smithi019 ceph-mon[91296]: 9.d starting backfill to osd.6 from (0'0,0'0] MAX to 177'7581 2024-09-13T14:43:22.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:21 smithi003 ceph-mon[107278]: 9.d starting backfill to osd.6 from (0'0,0'0] MAX to 177'7581 2024-09-13T14:43:22.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:21 smithi003 ceph-mon[104007]: 9.d starting backfill to osd.6 from (0'0,0'0] MAX to 177'7581 2024-09-13T14:43:22.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:22.233 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:22.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:22.334 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:22.356 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:22.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:22.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:22.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:22.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:22.708 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:22.730 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:22.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:22.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:22.854 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:22.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:22.868 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 90 16 120913 120897 83.9427 0 - 0.0115935 2024-09-13T14:43:22.956 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:22.978 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:23.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:23.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:22 smithi019 ceph-mon[91296]: pgmap v582: 99 pgs: 2 unknown, 17 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 5733/361356 objects degraded (1.587%); 1002/361356 objects misplaced (0.277%); 6.6 MiB/s, 104 objects/s recovering 2024-09-13T14:43:23.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:22 smithi003 ceph-mon[107278]: pgmap v582: 99 pgs: 2 unknown, 17 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 5733/361356 objects degraded (1.587%); 1002/361356 objects misplaced (0.277%); 6.6 MiB/s, 104 objects/s recovering 2024-09-13T14:43:23.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:22 smithi003 ceph-mon[104007]: pgmap v582: 99 pgs: 2 unknown, 17 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 5733/361356 objects degraded (1.587%); 1002/361356 objects misplaced (0.277%); 6.6 MiB/s, 104 objects/s recovering 2024-09-13T14:43:23.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:23.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:23.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:23.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:23.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:23.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:23.475 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:23.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.600 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:23.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:23.701 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:23.723 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:23.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:23.847 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:23.869 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 91 16 120913 120897 83.0203 0 - 0.0115935 2024-09-13T14:43:23.949 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:23.971 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:24.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:24.096 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:24.198 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:24.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:24.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:43:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:43:24] "GET /metrics HTTP/1.1" 200 36496 "" "Prometheus/2.51.0" 2024-09-13T14:43:24.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:24.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:24.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:24.594 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:24.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:24.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:24.821 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:24.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:24.869 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 92 16 120913 120897 82.118 0 - 0.0115935 2024-09-13T14:43:24.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:24.958 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:24.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:25.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:25.100 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:25.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:24 smithi019 ceph-mon[91296]: pgmap v583: 99 pgs: 2 unknown, 17 peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 45 MiB/s wr, 716 op/s; 9425/361896 objects degraded (2.604%); 1002/361896 objects misplaced (0.277%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:43:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:24 smithi003 ceph-mon[107278]: pgmap v583: 99 pgs: 2 unknown, 17 peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 45 MiB/s wr, 716 op/s; 9425/361896 objects degraded (2.604%); 1002/361896 objects misplaced (0.277%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:43:25.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:24 smithi003 ceph-mon[104007]: pgmap v583: 99 pgs: 2 unknown, 17 peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 45 MiB/s wr, 716 op/s; 9425/361896 objects degraded (2.604%); 1002/361896 objects misplaced (0.277%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:43:25.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:25.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:25.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:25.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:25.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:25.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:25.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:25.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:25.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:25.853 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:25.869 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 93 16 120913 120897 81.2351 0 - 0.0115935 2024-09-13T14:43:25.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:25.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:26.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:26.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:26.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9425/361896 objects degraded (2.604%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:26.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9425/361896 objects degraded (2.604%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9425/361896 objects degraded (2.604%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:26.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:26.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:26.334 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:26.356 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:26.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:26.480 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:26.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:26.584 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:43:26.584 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:43:26.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:43:26.585 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:43:26.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.585 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200914 7.e 7.10f306ae (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T14:43:26.585 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:29.556714+0000", 2024-09-13T14:43:26.585 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 477.02528566400002, 2024-09-13T14:43:26.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.912808116, 2024-09-13T14:43:26.586 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.586 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.586 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:43:26.586 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:43:26.586 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:43:26.587 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:43:26.587 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200914 2024-09-13T14:43:26.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.587 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556714+0000", 2024-09-13T14:43:26.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556714+0000", 2024-09-13T14:43:26.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556716+0000", 2024-09-13T14:43:26.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.59e-06 2024-09-13T14:43:26.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556721+0000", 2024-09-13T14:43:26.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8219999999999997e-06 2024-09-13T14:43:26.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556732+0000", 2024-09-13T14:43:26.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1268e-05 2024-09-13T14:43:26.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556740+0000", 2024-09-13T14:43:26.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7619999999999995e-06 2024-09-13T14:43:26.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.557265+0000", 2024-09-13T14:43:26.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052527599999999998 2024-09-13T14:43:26.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:43:26.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.557276+0000", 2024-09-13T14:43:26.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1069e-05 2024-09-13T14:43:26.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469510+0000", 2024-09-13T14:43:26.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91223351799999997 2024-09-13T14:43:26.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469522+0000", 2024-09-13T14:43:26.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2811e-05 2024-09-13T14:43:26.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.600 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.601 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.601 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200917 7.8 7.f1acfa8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T14:43:26.601 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:29.556886+0000", 2024-09-13T14:43:26.602 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 477.02511425799997, 2024-09-13T14:43:26.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91263592800000004, 2024-09-13T14:43:26.602 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.602 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.603 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:43:26.603 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:43:26.603 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:43:26.603 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:43:26.603 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200917 2024-09-13T14:43:26.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.604 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556886+0000", 2024-09-13T14:43:26.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556886+0000", 2024-09-13T14:43:26.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556886+0000", 2024-09-13T14:43:26.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.75e-07 2024-09-13T14:43:26.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556891+0000", 2024-09-13T14:43:26.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3100000000000002e-06 2024-09-13T14:43:26.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556900+0000", 2024-09-13T14:43:26.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5680000000000005e-06 2024-09-13T14:43:26.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556907+0000", 2024-09-13T14:43:26.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9e-06 2024-09-13T14:43:26.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.557962+0000", 2024-09-13T14:43:26.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010545660000000001 2024-09-13T14:43:26.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:43:26.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.557971+0000", 2024-09-13T14:43:26.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8580000000000006e-06 2024-09-13T14:43:26.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469510+0000", 2024-09-13T14:43:26.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91153902899999995 2024-09-13T14:43:26.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469522+0000", 2024-09-13T14:43:26.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2021999999999999e-05 2024-09-13T14:43:26.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.613 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.614 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200918 7.8 7.201f1348 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T14:43:26.614 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:29.556919+0000", 2024-09-13T14:43:26.614 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 477.02508063200003, 2024-09-13T14:43:26.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91263936999999995, 2024-09-13T14:43:26.614 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.614 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.614 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:43:26.615 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:43:26.615 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:43:26.615 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:43:26.615 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200918 2024-09-13T14:43:26.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.615 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556919+0000", 2024-09-13T14:43:26.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556919+0000", 2024-09-13T14:43:26.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556920+0000", 2024-09-13T14:43:26.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0999999999999999e-07 2024-09-13T14:43:26.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556922+0000", 2024-09-13T14:43:26.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3769999999999999e-06 2024-09-13T14:43:26.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556930+0000", 2024-09-13T14:43:26.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6920000000000002e-06 2024-09-13T14:43:26.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556935+0000", 2024-09-13T14:43:26.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.327e-06 2024-09-13T14:43:26.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.557996+0000", 2024-09-13T14:43:26.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010604589999999999 2024-09-13T14:43:26.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:43:26.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.558003+0000", 2024-09-13T14:43:26.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5259999999999996e-06 2024-09-13T14:43:26.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469549+0000", 2024-09-13T14:43:26.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91154547500000005 2024-09-13T14:43:26.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469559+0000", 2024-09-13T14:43:26.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0003999999999999e-05 2024-09-13T14:43:26.624 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.624 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.624 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.624 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200919 7.e 7.31ed207e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T14:43:26.625 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:29.556943+0000", 2024-09-13T14:43:26.625 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 477.02505703200001, 2024-09-13T14:43:26.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91261713899999997, 2024-09-13T14:43:26.625 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.625 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.625 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:43:26.626 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:43:26.626 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:43:26.626 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:43:26.626 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200919 2024-09-13T14:43:26.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.626 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556943+0000", 2024-09-13T14:43:26.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556943+0000", 2024-09-13T14:43:26.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556943+0000", 2024-09-13T14:43:26.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6400000000000003e-07 2024-09-13T14:43:26.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556947+0000", 2024-09-13T14:43:26.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2009999999999999e-06 2024-09-13T14:43:26.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556954+0000", 2024-09-13T14:43:26.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0489999999999998e-06 2024-09-13T14:43:26.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.556960+0000", 2024-09-13T14:43:26.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2659999999999998e-06 2024-09-13T14:43:26.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.557306+0000", 2024-09-13T14:43:26.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000346316 2024-09-13T14:43:26.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:43:26.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.557314+0000", 2024-09-13T14:43:26.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7859999999999997e-06 2024-09-13T14:43:26.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469551+0000", 2024-09-13T14:43:26.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91223698099999995 2024-09-13T14:43:26.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469560+0000", 2024-09-13T14:43:26.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0759999999999992e-06 2024-09-13T14:43:26.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.635 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.635 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200931 7.e 7.5bf00a3e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T14:43:26.636 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:29.579721+0000", 2024-09-13T14:43:26.636 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 477.00227855100002, 2024-09-13T14:43:26.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88987400299999997, 2024-09-13T14:43:26.636 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.636 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:43:26.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:43:26.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:43:26.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:43:26.637 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200931 2024-09-13T14:43:26.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.637 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.579721+0000", 2024-09-13T14:43:26.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.579721+0000", 2024-09-13T14:43:26.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.579723+0000", 2024-09-13T14:43:26.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9130000000000001e-06 2024-09-13T14:43:26.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.579731+0000", 2024-09-13T14:43:26.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6580000000000002e-06 2024-09-13T14:43:26.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.579754+0000", 2024-09-13T14:43:26.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.34e-05 2024-09-13T14:43:26.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.579763+0000", 2024-09-13T14:43:26.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1840000000000005e-06 2024-09-13T14:43:26.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.579837+0000", 2024-09-13T14:43:26.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3904000000000003e-05 2024-09-13T14:43:26.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:43:26.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.579848+0000", 2024-09-13T14:43:26.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0365e-05 2024-09-13T14:43:26.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469585+0000", 2024-09-13T14:43:26.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88973760800000001 2024-09-13T14:43:26.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469595+0000", 2024-09-13T14:43:26.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9709999999999999e-06 2024-09-13T14:43:26.645 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.646 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200932 7.8 7.1c6964a8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T14:43:26.646 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:29.580131+0000", 2024-09-13T14:43:26.646 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 477.00186935599999, 2024-09-13T14:43:26.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88945857500000003, 2024-09-13T14:43:26.647 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.647 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:43:26.647 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200932 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.580131+0000", 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.580131+0000", 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.580131+0000", 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2099999999999995e-07 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.580137+0000", 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5149999999999997e-06 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.580151+0000", 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3859999999999999e-05 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.580158+0000", 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6769999999999995e-06 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.580218+0000", 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9827999999999997e-05 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:43:26.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.580227+0000", 2024-09-13T14:43:26.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3019999999999995e-06 2024-09-13T14:43:26.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469580+0000", 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88935337999999997 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469589+0000", 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1919999999999994e-06 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.44414.0:200936 7.8 7.547dfe38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T14:43:26.653 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:35:29.583070+0000", 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 476.99892944300001, 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88655183299999996, 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.44414", 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3592298043", 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 200936 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.583070+0000", 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.583070+0000", 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.583072+0000", 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1939999999999999e-06 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.583077+0000", 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4550000000000003e-06 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.583094+0000", 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7141000000000001e-05 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.583102+0000", 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6729999999999992e-06 2024-09-13T14:43:26.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.583161+0000", 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9494000000000003e-05 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:29.583170+0000", 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.799e-06 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469612+0000", 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88644182800000004 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:35:30.469622+0000", 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0249e-05 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:53934 8.f 8:fe685b29:::benchmark_data_smithi003_139172_object53933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:36:46.534486+0000", 2024-09-13T14:43:26.658 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 400.04751340899998, 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0144860449999999, 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 53934 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.534486+0000", 2024-09-13T14:43:26.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.534486+0000", 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.534488+0000", 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5740000000000001e-06 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.534510+0000", 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1699999999999999e-05 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.534519+0000", 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6460000000000005e-06 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.534528+0000", 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1610000000000003e-06 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.534597+0000", 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9445999999999999e-05 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.534604+0000", 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4259999999999997e-06 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.537030+0000", 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024260480000000001 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.537040+0000", 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4269999999999997e-06 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.539720+0000", 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.002679855 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.541268+0000", 2024-09-13T14:43:26.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015478930000000001 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1", 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.541670+0000", 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00040273199999999999 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.547280+0000", 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056100000000000004 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:26.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.548799+0000", 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015182800000000001 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.548880+0000", 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1383000000000003e-05 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.548973+0000", 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2473999999999996e-05 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.665 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:53958 8.f 8:fbfb2c94:::benchmark_data_smithi003_139172_object53957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:36:46.566756+0000", 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 400.015244322, 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98601073400000006, 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 53958 2024-09-13T14:43:26.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.566756+0000", 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.566756+0000", 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.566757+0000", 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.192e-06 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.566787+0000", 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0729999999999999e-05 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.566800+0000", 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2232000000000001e-05 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.566806+0000", 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2650000000000002e-06 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.566886+0000", 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0499000000000003e-05 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.566898+0000", 2024-09-13T14:43:26.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1535999999999999e-05 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.544461+0000", 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97756331100000005 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.544620+0000", 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015901700000000001 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1", 2024-09-13T14:43:26.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.544940+0000", 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031945500000000001 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.547388+0000", 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024482319999999998 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.552599+0000", 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052113560000000003 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.552671+0000", 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1124000000000005e-05 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.552766+0000", 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5785000000000005e-05 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.672 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35454.0:243962 8.a e147/130 8:53f6dab3:::benchmark_data_smithi003_139172_object119169:head v 147'15542, mlcod=147'15539)", 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:40:12.664674+0000", 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 193.917326293, 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.096324710999999993, 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.664674+0000", 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.664674+0000", 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.664674+0000", 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5100000000000001e-07 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.664677+0000", 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0879999999999999e-06 2024-09-13T14:43:26.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.664683+0000", 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7339999999999996e-06 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.664687+0000", 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3279999999999999e-06 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.668568+0000", 2024-09-13T14:43:26.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038809780000000002 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.668587+0000", 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8598999999999999e-05 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.674864+0000", 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062769150000000001 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.760998+0000", 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.086134717999999999 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35454.0:243974 8.d e147/130 8:b104d5fd:::benchmark_data_smithi003_139172_object119181:head v 147'15106, mlcod=147'15104)", 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:40:12.683578+0000", 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 193.898421439, 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.092275486000000004, 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.677 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.683578+0000", 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.683578+0000", 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.683580+0000", 2024-09-13T14:43:26.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2780000000000001e-06 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.683592+0000", 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2561999999999999e-05 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.760954+0000", 2024-09-13T14:43:26.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.077361899999999997 2024-09-13T14:43:26.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.760970+0000", 2024-09-13T14:43:26.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5707000000000001e-05 2024-09-13T14:43:26.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.761161+0000", 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019122900000000001 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.761181+0000", 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9760999999999999e-05 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.775755+0000", 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01457368 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.775854+0000", 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9369000000000004e-05 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35454.0:243979 8.e e147/130 8:703f5d12:::benchmark_data_smithi003_139172_object119186:head v 147'14983, mlcod=147'14981)", 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:40:12.690774+0000", 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 193.891226281, 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.097958542999999995, 2024-09-13T14:43:26.682 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.690774+0000", 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.690774+0000", 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.690775+0000", 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1880000000000001e-06 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.690784+0000", 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1239999999999994e-06 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.760958+0000", 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070173846999999998 2024-09-13T14:43:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.760970+0000", 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2473999999999999e-05 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.761122+0000", 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000151526 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.761141+0000", 2024-09-13T14:43:26.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.969e-05 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.787779+0000", 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026637922000000001 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:40:12.788732+0000", 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00095277199999999995 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.686 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:36986 9.0 e157/152 9:0f008b62:::benchmark_data_smithi003_156985_object36985:head v 157'2382, mlcod=157'2380)", 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:07.872995+0000", 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 78.709005081000001, 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16962896399999999, 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.872995+0000", 2024-09-13T14:43:26.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.872995+0000", 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.872996+0000", 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7000000000000003e-07 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873018+0000", 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2504e-05 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873029+0000", 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1243000000000001e-05 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873036+0000", 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.173e-06 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873089+0000", 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3143999999999998e-05 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873108+0000", 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9573e-05 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042505+0000", 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16939712600000001 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042624+0000", 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118331 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:36988 9.6 e157/152 9:669fab83:::benchmark_data_smithi003_156985_object36987:head v 157'2342, mlcod=157'2340)", 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:07.874028+0000", 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 78.707971618000002, 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17089444100000001, 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874028+0000", 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874028+0000", 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874029+0000", 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8999999999999995e-07 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874080+0000", 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1335999999999998e-05 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874091+0000", 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0913e-05 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874098+0000", 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5220000000000004e-06 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042652+0000", 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16855393799999999 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042683+0000", 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1177999999999999e-05 2024-09-13T14:43:26.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.044809+0000", 2024-09-13T14:43:26.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021263950000000001 2024-09-13T14:43:26.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.044923+0000", 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011336900000000001 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:36987 9.0 e157/152 9:0dc893eb:::benchmark_data_smithi003_156985_object36986:head v 157'2383, mlcod=157'2380)", 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:07.874717+0000", 2024-09-13T14:43:26.696 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 78.707282848999995, 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17119854900000001, 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874717+0000", 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874717+0000", 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874718+0000", 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.99e-07 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874745+0000", 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7538999999999998e-05 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042433+0000", 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16768813399999999 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042462+0000", 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8983e-05 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042710+0000", 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024720500000000002 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042728+0000", 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8365000000000001e-05 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.045743+0000", 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003014961 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.045916+0000", 2024-09-13T14:43:26.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017266300000000001 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41012 9.6 e157/152 9:63d12f41:::benchmark_data_smithi003_156985_object41011:head v 157'2584, mlcod=157'2582)", 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.198172+0000", 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.383827800000006, 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.229939433, 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198172+0000", 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198172+0000", 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198173+0000", 2024-09-13T14:43:26.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1100000000000005e-07 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198285+0000", 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111656 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198296+0000", 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1303e-05 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198302+0000", 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5740000000000001e-06 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198371+0000", 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8409000000000002e-05 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198389+0000", 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8077e-05 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.428071+0000", 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22968186900000001 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.428112+0000", 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0734e-05 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41014 9.0 e157/152 9:057a3b88:::benchmark_data_smithi003_156985_object41013:head v 157'2642, mlcod=157'2639)", 2024-09-13T14:43:26.705 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.200366+0000", 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.381633601999994, 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.225644175, 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200366+0000", 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200366+0000", 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200367+0000", 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7000000000000003e-07 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200409+0000", 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1468999999999999e-05 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200420+0000", 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1521e-05 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200427+0000", 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5740000000000001e-06 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200485+0000", 2024-09-13T14:43:26.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8315e-05 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200505+0000", 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0066999999999999e-05 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425972+0000", 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.225466323 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.426010+0000", 2024-09-13T14:43:26.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8936000000000001e-05 2024-09-13T14:43:26.710 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.710 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.710 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.710 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41016 9.6 e157/152 9:63031843:::benchmark_data_smithi003_156985_object41015:head v 157'2585, mlcod=157'2582)", 2024-09-13T14:43:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:26.711 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.201276+0000", 2024-09-13T14:43:26.711 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.380723576999998, 2024-09-13T14:43:26.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22863520700000001, 2024-09-13T14:43:26.711 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.711 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201276+0000", 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201276+0000", 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201277+0000", 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3099999999999996e-07 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201315+0000", 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7508000000000001e-05 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201338+0000", 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3682000000000001e-05 2024-09-13T14:43:26.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201347+0000", 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3100000000000001e-06 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201424+0000", 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6872000000000005e-05 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201442+0000", 2024-09-13T14:43:26.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8805e-05 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.429846+0000", 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.228403949 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.429912+0000", 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5149999999999998e-05 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41023 9.2 e157/152 9:40cd60d0:::benchmark_data_smithi003_156985_object41022:head v 157'2589, mlcod=157'2586)", 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.203360+0000", 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.378640017999999, 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22830435900000001, 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.203360+0000", 2024-09-13T14:43:26.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.203360+0000", 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.203361+0000", 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5799999999999998e-07 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.203403+0000", 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2021000000000003e-05 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425885+0000", 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.222481715 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425897+0000", 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2736e-05 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425964+0000", 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6964999999999993e-05 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425986+0000", 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1676999999999999e-05 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.431557+0000", 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055714179999999999 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.431664+0000", 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106869 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41025 9.6 e157/152 9:668eb8d4:::benchmark_data_smithi003_156985_object41024:head v 157'2586, mlcod=157'2582)", 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.204204+0000", 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.377795890000002, 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.225741672, 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:43:26.720 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.204204+0000", 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.204204+0000", 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.204205+0000", 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.133e-06 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.204253+0000", 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8340999999999997e-05 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425909+0000", 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22165542399999999 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425919+0000", 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0074999999999999e-05 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.427060+0000", 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00114062 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.427080+0000", 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0298000000000001e-05 2024-09-13T14:43:26.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.429916+0000", 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002835859 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.429946+0000", 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9921999999999999e-05 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.724 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:43:26.725 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:43:26.725 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:43:26.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:26.835 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:26.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:26.869 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 94 15 120913 120898 80.3716 0.0125 7.45145 0.0116551 2024-09-13T14:43:26.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:26.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:27.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:27.087 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:27.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:26 smithi019 ceph-mon[91296]: pgmap v584: 99 pgs: 4 peering, 11 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 31 MiB/s wr, 502 op/s; 62799/362697 objects degraded (17.314%); 7589/362697 objects misplaced (2.092%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:43:27.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:26 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-09-13T14:43:27.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:26 smithi003 ceph-mon[107278]: pgmap v584: 99 pgs: 4 peering, 11 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 31 MiB/s wr, 502 op/s; 62799/362697 objects degraded (17.314%); 7589/362697 objects misplaced (2.092%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:43:27.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:26 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-09-13T14:43:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:26 smithi003 ceph-mon[104007]: pgmap v584: 99 pgs: 4 peering, 11 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 31 MiB/s wr, 502 op/s; 62799/362697 objects degraded (17.314%); 7589/362697 objects misplaced (2.092%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:43:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:26 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-09-13T14:43:27.189 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:27.211 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:27.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:27.335 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:27.437 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:27.462 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:27.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:27.564 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:27.589 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:27.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:27.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:27.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:27.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:27.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:27.869 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 95 15 120913 120898 79.5256 0 - 0.0116551 2024-09-13T14:43:27.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:27.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:28.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:28.086 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:28.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:28.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:28.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:28.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:28.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:28.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:28.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:28.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:28.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:28.700 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:28.723 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:28.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:28.847 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:28.869 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 96 15 120913 120898 78.6973 0 - 0.0116551 2024-09-13T14:43:28.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:28.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:29.046 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:43:29.046 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:43:29.046 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:43:29.046 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:43:29.046 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:43:29.046 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:43:29.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:29.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:29.047 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:43:29.047 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:43:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:29.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:29.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:28 smithi019 ceph-mon[91296]: pgmap v585: 99 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 activating+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 3.6 MiB/s wr, 57 op/s; 75916/362730 objects degraded (20.929%); 8097/362730 objects misplaced (2.232%); 14 MiB/s, 218 objects/s recovering 2024-09-13T14:43:29.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:28 smithi019 ceph-mon[91296]: osdmap e181: 8 total, 8 up, 8 in 2024-09-13T14:43:29.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1804869350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:28 smithi003 ceph-mon[104007]: pgmap v585: 99 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 activating+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 3.6 MiB/s wr, 57 op/s; 75916/362730 objects degraded (20.929%); 8097/362730 objects misplaced (2.232%); 14 MiB/s, 218 objects/s recovering 2024-09-13T14:43:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:28 smithi003 ceph-mon[104007]: osdmap e181: 8 total, 8 up, 8 in 2024-09-13T14:43:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1804869350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:29.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:28 smithi003 ceph-mon[107278]: pgmap v585: 99 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 activating+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 3.6 MiB/s wr, 57 op/s; 75916/362730 objects degraded (20.929%); 8097/362730 objects misplaced (2.232%); 14 MiB/s, 218 objects/s recovering 2024-09-13T14:43:29.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:28 smithi003 ceph-mon[107278]: osdmap e181: 8 total, 8 up, 8 in 2024-09-13T14:43:29.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1804869350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:29.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:29.225 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:29.327 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:29.349 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:29.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:29.473 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:29.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:29.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:29.700 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:29.725 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:29.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:29.826 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:29.849 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:29.869 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 97 15 120913 120898 77.886 0 - 0.0116551 2024-09-13T14:43:29.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:29.982 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:30.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:30.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:30.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:30.212 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:29 smithi019 ceph-mon[91296]: osdmap e182: 8 total, 8 up, 8 in 2024-09-13T14:43:30.212 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:30.235 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:30.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:30.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:30.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:29 smithi003 ceph-mon[107278]: osdmap e182: 8 total, 8 up, 8 in 2024-09-13T14:43:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:29 smithi003 ceph-mon[104007]: osdmap e182: 8 total, 8 up, 8 in 2024-09-13T14:43:30.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:30.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:30.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:30.611 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:30.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:30.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:43:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:43:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:30.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:30.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:30.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:30.868 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:30.870 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 97.1407 2024-09-13T14:43:30.870 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 120913 2024-09-13T14:43:30.870 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T14:43:30.870 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T14:43:30.870 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 77.795 2024-09-13T14:43:30.870 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 49.3501 2024-09-13T14:43:30.870 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 229.562 2024-09-13T14:43:30.870 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:43:30.871 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1244 2024-09-13T14:43:30.871 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 789.613 2024-09-13T14:43:30.871 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3673 2024-09-13T14:43:30.871 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-09-13T14:43:30.871 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0128143 2024-09-13T14:43:30.871 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.118794 2024-09-13T14:43:30.871 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 11.0728 2024-09-13T14:43:30.871 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00126908 2024-09-13T14:43:30.883 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:43:30.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:31.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:31.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.149 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:31.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:31.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:31.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:31.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:30 smithi019 ceph-mon[91296]: pgmap v588: 99 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 activating+undersized+remapped, 1 creating+peering, 8 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 3.6 MiB/s wr, 57 op/s; 68062/362730 objects degraded (18.764%); 14871/362730 objects misplaced (4.100%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:43:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:30 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:43:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:30 smithi019 ceph-mon[91296]: osdmap e183: 8 total, 8 up, 8 in 2024-09-13T14:43:31.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 68062/362730 objects degraded (18.764%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:31.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:30 smithi003 ceph-mon[104007]: pgmap v588: 99 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 activating+undersized+remapped, 1 creating+peering, 8 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 3.6 MiB/s wr, 57 op/s; 68062/362730 objects degraded (18.764%); 14871/362730 objects misplaced (4.100%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:43:31.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:30 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:43:31.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:30 smithi003 ceph-mon[104007]: osdmap e183: 8 total, 8 up, 8 in 2024-09-13T14:43:31.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 68062/362730 objects degraded (18.764%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:30 smithi003 ceph-mon[107278]: pgmap v588: 99 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 activating+undersized+remapped, 1 creating+peering, 8 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 3.6 MiB/s wr, 57 op/s; 68062/362730 objects degraded (18.764%); 14871/362730 objects misplaced (4.100%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:43:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:30 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:43:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:30 smithi003 ceph-mon[107278]: osdmap e183: 8 total, 8 up, 8 in 2024-09-13T14:43:31.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 68062/362730 objects degraded (18.764%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:31.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:31.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:31.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:31.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:31.690 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:31.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:31.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:31.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:31.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:32.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:32.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:32.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:32.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:32.353 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:32.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:32.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:32.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:32.622 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:32.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:32.754 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:32.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:32.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:32.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:32.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:33.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:33.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:33.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:33.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:33.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:33.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:33.288 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:33.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:33.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:33.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:32 smithi019 ceph-mon[91296]: pgmap v590: 99 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 1 creating+peering, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 256 KiB/s wr, 10 op/s; 60578/362652 objects degraded (16.704%); 22298/362652 objects misplaced (6.149%); 6.2 MiB/s, 98 objects/s recovering 2024-09-13T14:43:33.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:32 smithi003 ceph-mon[107278]: pgmap v590: 99 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 1 creating+peering, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 256 KiB/s wr, 10 op/s; 60578/362652 objects degraded (16.704%); 22298/362652 objects misplaced (6.149%); 6.2 MiB/s, 98 objects/s recovering 2024-09-13T14:43:33.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:32 smithi003 ceph-mon[104007]: pgmap v590: 99 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 1 creating+peering, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 256 KiB/s wr, 10 op/s; 60578/362652 objects degraded (16.704%); 22298/362652 objects misplaced (6.149%); 6.2 MiB/s, 98 objects/s recovering 2024-09-13T14:43:33.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:33.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:33.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:33.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:33.816 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:33.848 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:33.950 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:33.982 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:34.084 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.113 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:34.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:34.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:43:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:43:34] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-09-13T14:43:34.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:34.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:34.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:34.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:34.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:34.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:34.763 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:34.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:34.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:34.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:35.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:35.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:35.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:35.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:35.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:35.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:35.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:34 smithi019 ceph-mon[91296]: pgmap v591: 99 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 1 creating+peering, 9 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 139 KiB/s wr, 19 op/s; 56886/362448 objects degraded (15.695%); 25637/362448 objects misplaced (7.073%) 2024-09-13T14:43:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:34 smithi003 ceph-mon[107278]: pgmap v591: 99 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 1 creating+peering, 9 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 139 KiB/s wr, 19 op/s; 56886/362448 objects degraded (15.695%); 25637/362448 objects misplaced (7.073%) 2024-09-13T14:43:35.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:34 smithi003 ceph-mon[104007]: pgmap v591: 99 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 1 creating+peering, 9 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 139 KiB/s wr, 19 op/s; 56886/362448 objects degraded (15.695%); 25637/362448 objects misplaced (7.073%) 2024-09-13T14:43:35.462 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.493 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:35.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:35.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:35.624 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:35.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:35.760 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:35.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:35.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:36.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:36.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:36.176 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:36.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:36.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:36.333 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 56886/362448 objects degraded (15.695%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:36.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 56886/362448 objects degraded (15.695%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:36.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:36.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 56886/362448 objects degraded (15.695%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:36.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:36.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:36.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:36.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:36.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:36.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:36.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:36.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:37.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:37.039 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:37.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:37.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:37.166 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:37.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:36 smithi003 ceph-mon[107278]: pgmap v592: 99 pgs: 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 132 B/s rd, 108 KiB/s wr, 159 op/s; 56228/359127 objects degraded (15.657%); 25065/359127 objects misplaced (6.979%); 15 MiB/s, 236 objects/s recovering 2024-09-13T14:43:37.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:36 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:43:37.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:36 smithi003 ceph-mon[104007]: pgmap v592: 99 pgs: 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 132 B/s rd, 108 KiB/s wr, 159 op/s; 56228/359127 objects degraded (15.657%); 25065/359127 objects misplaced (6.979%); 15 MiB/s, 236 objects/s recovering 2024-09-13T14:43:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:36 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:43:37.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:37.298 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:37.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:37.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:36 smithi019 ceph-mon[91296]: pgmap v592: 99 pgs: 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 132 B/s rd, 108 KiB/s wr, 159 op/s; 56228/359127 objects degraded (15.657%); 25065/359127 objects misplaced (6.979%); 15 MiB/s, 236 objects/s recovering 2024-09-13T14:43:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:36 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:43:37.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:37.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:37.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:37.676 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:37.707 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:37.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:37.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:37.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:37.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:38.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:38.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:38.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:38.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:38.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:38.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:38.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:38.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:38.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:38.803 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:38.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:38.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.938 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:38.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:39.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:39.067 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:39.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:39.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:39.309 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:39.332 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:38 smithi019 ceph-mon[91296]: pgmap v593: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 118 B/s rd, 96 KiB/s wr, 229 op/s; 56238/356850 objects degraded (15.760%); 24676/356850 objects misplaced (6.915%); 22 MiB/s, 355 objects/s recovering 2024-09-13T14:43:39.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:38 smithi003 ceph-mon[107278]: pgmap v593: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 118 B/s rd, 96 KiB/s wr, 229 op/s; 56238/356850 objects degraded (15.760%); 24676/356850 objects misplaced (6.915%); 22 MiB/s, 355 objects/s recovering 2024-09-13T14:43:39.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:38 smithi003 ceph-mon[104007]: pgmap v593: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 118 B/s rd, 96 KiB/s wr, 229 op/s; 56238/356850 objects degraded (15.760%); 24676/356850 objects misplaced (6.915%); 22 MiB/s, 355 objects/s recovering 2024-09-13T14:43:39.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:39.463 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:39.565 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:39.590 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:39.692 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:39.722 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:39.824 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:39.858 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:39.960 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:39.992 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:43:40.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:40.140 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:40.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:40.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:40.303 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:40.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:40.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:40.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:40.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:40.588 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:40.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:43:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:43:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:40.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:40.717 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:40.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:40.847 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:40.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:40.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:41.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:41.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:41.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:41.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:41.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:40 smithi003 ceph-mon[104007]: pgmap v594: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 83 KiB/s wr, 406 op/s; 55178/350607 objects degraded (15.738%); 24201/350607 objects misplaced (6.903%); 24 MiB/s, 384 objects/s recovering 2024-09-13T14:43:41.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 55178/350607 objects degraded (15.738%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:41.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:40 smithi003 ceph-mon[107278]: pgmap v594: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 83 KiB/s wr, 406 op/s; 55178/350607 objects degraded (15.738%); 24201/350607 objects misplaced (6.903%); 24 MiB/s, 384 objects/s recovering 2024-09-13T14:43:41.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 55178/350607 objects degraded (15.738%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:41.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:41.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:40 smithi019 ceph-mon[91296]: pgmap v594: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 83 KiB/s wr, 406 op/s; 55178/350607 objects degraded (15.738%); 24201/350607 objects misplaced (6.903%); 24 MiB/s, 384 objects/s recovering 2024-09-13T14:43:41.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 55178/350607 objects degraded (15.738%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:41.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:41.536 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:41.564 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:41.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:41.696 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:41.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:41.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:41.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:43:41.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:42.002 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:43:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:42.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:42.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:42.142 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:42.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:41 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3173892934' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:42.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:41 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:42.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:41 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3173892934' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:42.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:41 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:42.245 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:42.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:42.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.413 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:42.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:42.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:41 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3173892934' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:41 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:42.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:42.546 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:42.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:42.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:42.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:42.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:42.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:42.979 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:43.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:43.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:43.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:43.250 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:43.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:43.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:43.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:42 smithi003 ceph-mon[104007]: pgmap v595: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 87 B/s rd, 71 KiB/s wr, 396 op/s; 55281/348927 objects degraded (15.843%); 24025/348927 objects misplaced (6.885%); 21 MiB/s, 336 objects/s recovering 2024-09-13T14:43:43.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:43.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:43.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[104007]: osdmap e184: 8 total, 8 up, 8 in 2024-09-13T14:43:43.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:43:43.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:43.386 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:42 smithi003 ceph-mon[107278]: pgmap v595: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 87 B/s rd, 71 KiB/s wr, 396 op/s; 55281/348927 objects degraded (15.843%); 24025/348927 objects misplaced (6.885%); 21 MiB/s, 336 objects/s recovering 2024-09-13T14:43:43.386 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:43.387 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:43.387 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[107278]: osdmap e184: 8 total, 8 up, 8 in 2024-09-13T14:43:43.387 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:43:43.387 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:43.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:42 smithi019 ceph-mon[91296]: pgmap v595: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 87 B/s rd, 71 KiB/s wr, 396 op/s; 55281/348927 objects degraded (15.843%); 24025/348927 objects misplaced (6.885%); 21 MiB/s, 336 objects/s recovering 2024-09-13T14:43:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:42 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:42 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:42 smithi019 ceph-mon[91296]: osdmap e184: 8 total, 8 up, 8 in 2024-09-13T14:43:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:43:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:43.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:43.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:43.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:43.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:43.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:43.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:43.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:43.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:44.020 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 297 (0%) 2024-09-13T14:43:44.020 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:43:44.021 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.54671 -> 6.54671 (expected baseline 5.69951) 2024-09-13T14:43:44.021 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.5 with 27 -> 27 pgs (0.727273 -> 0.727273 * mean) 2024-09-13T14:43:44.021 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 47 -> 47 pgs (1.26599 -> 1.26599 * mean) 2024-09-13T14:43:44.021 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:43:44.021 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-09-13T14:43:44.022 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 3 2024-09-13T14:43:44.022 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:43:44.022 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0371 2024-09-13T14:43:44.022 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.3713 2024-09-13T14:43:44.023 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T14:43:44.076 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:44.105 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:44.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:44.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:44.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:44.357 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:44.385 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:43:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:43:44] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:43:44.391 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:44.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:44.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:44.631 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:44.662 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:44.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:44.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:44.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.794 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:44.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:44.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:44.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:45.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:45.181 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:45.205 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:45.307 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.338 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:45.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:45.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:45 smithi003 ceph-mon[107278]: pgmap v597: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 477 op/s; 55281/348123 objects degraded (15.880%); 23783/348123 objects misplaced (6.832%); 23 MiB/s, 375 objects/s recovering 2024-09-13T14:43:45.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:45 smithi003 ceph-mon[107278]: from='client.45959 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:43:45.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:45 smithi003 ceph-mon[104007]: pgmap v597: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 477 op/s; 55281/348123 objects degraded (15.880%); 23783/348123 objects misplaced (6.832%); 23 MiB/s, 375 objects/s recovering 2024-09-13T14:43:45.371 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:45 smithi003 ceph-mon[104007]: from='client.45959 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:43:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:45 smithi019 ceph-mon[91296]: pgmap v597: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 477 op/s; 55281/348123 objects degraded (15.880%); 23783/348123 objects misplaced (6.832%); 23 MiB/s, 375 objects/s recovering 2024-09-13T14:43:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:45 smithi019 ceph-mon[91296]: from='client.45959 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:43:45.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:45.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:45.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:45.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:45.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:45.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:45.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:45.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:45.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:46.023 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:46.046 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T14:43:46.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-13T14:43:46.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:46.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:46.277 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:46.410 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:46.424 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 55281/348123 objects degraded (15.880%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:46.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 55281/348123 objects degraded (15.880%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:46.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 55281/348123 objects degraded (15.880%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:46.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:46.539 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:46.545 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:46.567 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:46.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:46.704 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:46.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:43:46.806 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:46.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:46.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:46.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:47.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.107 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:47.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:47.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:47.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.385 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:47.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:47 smithi019 ceph-mon[91296]: pgmap v598: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 568 op/s; 54240/342069 objects degraded (15.856%); 23313/342069 objects misplaced (6.815%); 14 MiB/s, 221 objects/s recovering 2024-09-13T14:43:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:47 smithi003 ceph-mon[107278]: pgmap v598: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 568 op/s; 54240/342069 objects degraded (15.856%); 23313/342069 objects misplaced (6.815%); 14 MiB/s, 221 objects/s recovering 2024-09-13T14:43:47.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:47 smithi003 ceph-mon[104007]: pgmap v598: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 7.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 568 op/s; 54240/342069 objects degraded (15.856%); 23313/342069 objects misplaced (6.815%); 14 MiB/s, 221 objects/s recovering 2024-09-13T14:43:47.486 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:47.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:47.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:47.653 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:47.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:47.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:47.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:47.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:48.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:48.188 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.223 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:48.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:48.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:48.372 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:48 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:43:48.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:48 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:43:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:48 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:43:48.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:48.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:48.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:48.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:48.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:48.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.793 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:48.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:48.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:48.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:49.034 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:49 smithi019 ceph-mon[91296]: pgmap v599: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 54364/339597 objects degraded (16.008%); 22843/339597 objects misplaced (6.727%); 14 MiB/s, 218 objects/s recovering 2024-09-13T14:43:49.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:49.076 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:49.178 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:49.213 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:49.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:49.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:49.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:49 smithi003 ceph-mon[107278]: pgmap v599: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 54364/339597 objects degraded (16.008%); 22843/339597 objects misplaced (6.727%); 14 MiB/s, 218 objects/s recovering 2024-09-13T14:43:49.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:49 smithi003 ceph-mon[104007]: pgmap v599: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 54364/339597 objects degraded (16.008%); 22843/339597 objects misplaced (6.727%); 14 MiB/s, 218 objects/s recovering 2024-09-13T14:43:49.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:49.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:49.578 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:49.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:49.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.756 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:49.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:49.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:49.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:49.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:50.024 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:50.126 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:50.157 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:50.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:50.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:50.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:50.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:50.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:50.588 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:50.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:43:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:43:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:50.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:50.717 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:50.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:50.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:50.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:50.882 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:50.882 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:50.959 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:50.999 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:51.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:51.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:51.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:51 smithi003 ceph-mon[107278]: pgmap v600: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 53276/333351 objects degraded (15.982%); 22355/333351 objects misplaced (6.706%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:43:51.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 53276/333351 objects degraded (15.982%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:51.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:51 smithi003 ceph-mon[104007]: pgmap v600: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 53276/333351 objects degraded (15.982%); 22355/333351 objects misplaced (6.706%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:43:51.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 53276/333351 objects degraded (15.982%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:51.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:51.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:51.378 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:51.416 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:51.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:51 smithi019 ceph-mon[91296]: pgmap v600: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 53276/333351 objects degraded (15.982%); 22355/333351 objects misplaced (6.706%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:43:51.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 53276/333351 objects degraded (15.982%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:51.518 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.546 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:51.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:51.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:43:51.646 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:51.677 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:51.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:51.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:51.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:51.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:51.981 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:51.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:52.017 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:52.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:52.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:52.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:52.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:52.282 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:52.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:52.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:52.531 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:52.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:52.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:52.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:52.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:52.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:52.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:53.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:53.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:53.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:53.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:53.368 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:53 smithi019 ceph-mon[91296]: pgmap v601: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 577 op/s; 53400/331599 objects degraded (16.104%); 22154/331599 objects misplaced (6.681%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:43:53.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:53.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:53 smithi003 ceph-mon[104007]: pgmap v601: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 577 op/s; 53400/331599 objects degraded (16.104%); 22154/331599 objects misplaced (6.681%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:43:53.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:53 smithi003 ceph-mon[107278]: pgmap v601: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 577 op/s; 53400/331599 objects degraded (16.104%); 22154/331599 objects misplaced (6.681%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:43:53.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:53.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:53.630 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:53.669 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:53.771 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.805 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:53.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:53.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:53.946 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:54.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:54.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:54.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:54.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:54.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:54.356 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:43:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:43:54] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:43:54.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:54.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:54.599 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:54.626 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:54.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:54.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:54.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:54.899 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:55.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:55.031 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:55.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:55.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:54 smithi019 ceph-mon[91296]: pgmap v602: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 521 op/s; 53400/330750 objects degraded (16.145%); 21882/330750 objects misplaced (6.616%); 13 MiB/s, 213 objects/s recovering 2024-09-13T14:43:55.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:43:55.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:43:55.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:54 smithi003 ceph-mon[107278]: pgmap v602: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 521 op/s; 53400/330750 objects degraded (16.145%); 21882/330750 objects misplaced (6.616%); 13 MiB/s, 213 objects/s recovering 2024-09-13T14:43:55.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:43:55.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:43:55.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:54 smithi003 ceph-mon[104007]: pgmap v602: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 521 op/s; 53400/330750 objects degraded (16.145%); 21882/330750 objects misplaced (6.616%); 13 MiB/s, 213 objects/s recovering 2024-09-13T14:43:55.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:43:55.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:43:55.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:55.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:55.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:55.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.436 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:55.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:55.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:55.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:55.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:55.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:55.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:55.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:55.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:55.978 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:56.010 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 53400/330750 objects degraded (16.145%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:56.010 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 53400/330750 objects degraded (16.145%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:56.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:56.112 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:56.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 53400/330750 objects degraded (16.145%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:43:56.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:56.249 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:56.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:56.379 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:56.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:56.521 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:56.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:56.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:56.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:56 smithi003 ceph-mon[104007]: pgmap v603: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 663 op/s; 52291/324219 objects degraded (16.128%); 21392/324219 objects misplaced (6.598%); 16 MiB/s, 253 objects/s recovering 2024-09-13T14:43:56.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:56 smithi003 ceph-mon[107278]: pgmap v603: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 663 op/s; 52291/324219 objects degraded (16.128%); 21392/324219 objects misplaced (6.598%); 16 MiB/s, 253 objects/s recovering 2024-09-13T14:43:56.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:56.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:56.913 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:56.948 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:56.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:57.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:57.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:57.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:57.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:56 smithi019 ceph-mon[91296]: pgmap v603: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 663 op/s; 52291/324219 objects degraded (16.128%); 21392/324219 objects misplaced (6.598%); 16 MiB/s, 253 objects/s recovering 2024-09-13T14:43:57.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:57.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:57.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:57.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:57.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:57.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:57.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:57.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:57.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:57.797 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:57.899 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:58.033 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.065 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:58.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:58.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:58.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:58.348 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:58.450 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:58.478 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:58.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:58.711 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:58.734 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:58.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:58.859 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:58.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:58.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:59.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:59.124 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:59.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:58 smithi019 ceph-mon[91296]: pgmap v604: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 563 op/s; 52362/321774 objects degraded (16.273%); 20920/321774 objects misplaced (6.501%); 14 MiB/s, 223 objects/s recovering 2024-09-13T14:43:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:58 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/176570247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:58 smithi003 ceph-mon[107278]: pgmap v604: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 563 op/s; 52362/321774 objects degraded (16.273%); 20920/321774 objects misplaced (6.501%); 14 MiB/s, 223 objects/s recovering 2024-09-13T14:43:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:58 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/176570247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:58 smithi003 ceph-mon[104007]: pgmap v604: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 563 op/s; 52362/321774 objects degraded (16.273%); 20920/321774 objects misplaced (6.501%); 14 MiB/s, 223 objects/s recovering 2024-09-13T14:43:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:58 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/176570247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:59.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:59.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:59.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:59.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:59.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:59.551 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:59.653 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:59.690 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:59.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:59.712 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 297 (0%) 2024-09-13T14:43:59.712 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:43:59.712 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.54671 -> 6.54671 (expected baseline 5.69951) 2024-09-13T14:43:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.5 with 27 -> 27 pgs (0.727273 -> 0.727273 * mean) 2024-09-13T14:43:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 47 -> 47 pgs (1.26599 -> 1.26599 * mean) 2024-09-13T14:43:59.713 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:43:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-09-13T14:43:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 3 2024-09-13T14:43:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:43:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0359 2024-09-13T14:43:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.3586 2024-09-13T14:43:59.713 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T14:43:59.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:43:59.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:59.955 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:43:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:00.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:00.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:00.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:43:59 smithi019 ceph-mon[91296]: from='client.45971 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:00.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:43:59 smithi003 ceph-mon[104007]: from='client.45971 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:00.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:43:59 smithi003 ceph-mon[107278]: from='client.45971 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:00.215 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:00.239 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:00.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:00.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:00.367 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:00.469 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:00.501 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:00.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:00.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:00.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:44:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:44:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:00.746 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:00.778 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:00.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:00.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:00.912 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:00.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:01.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:01.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:01.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:00 smithi003 ceph-mon[107278]: pgmap v605: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 51298/315675 objects degraded (16.250%); 20375/315675 objects misplaced (6.454%); 16 MiB/s, 261 objects/s recovering 2024-09-13T14:44:01.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 51298/315675 objects degraded (16.250%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:01.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:00 smithi003 ceph-mon[104007]: pgmap v605: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 51298/315675 objects degraded (16.250%); 20375/315675 objects misplaced (6.454%); 16 MiB/s, 261 objects/s recovering 2024-09-13T14:44:01.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 51298/315675 objects degraded (16.250%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:01.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:01.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:00 smithi019 ceph-mon[91296]: pgmap v605: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 51298/315675 objects degraded (16.250%); 20375/315675 objects misplaced (6.454%); 16 MiB/s, 261 objects/s recovering 2024-09-13T14:44:01.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 51298/315675 objects degraded (16.250%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:01.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:01.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:01.329 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:01.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:01.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:01.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:01.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:01.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:01.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:01.860 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:01.890 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:01.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:02.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:02.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:44:02.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:44:02.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:44:02.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:44:02.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:44:02.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:44:02.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:44:02.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:44:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:02.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:02.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:02.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:02.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:02.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:02.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:02.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:02.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.617 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:02.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:02.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:02.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:02.864 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:02.897 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:02.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:03.032 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:03.134 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:03.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:02 smithi003 ceph-mon[104007]: pgmap v606: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 534 op/s; 51357/314118 objects degraded (16.350%); 20219/314118 objects misplaced (6.437%); 13 MiB/s, 204 objects/s recovering 2024-09-13T14:44:03.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:02 smithi003 ceph-mon[107278]: pgmap v606: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 534 op/s; 51357/314118 objects degraded (16.350%); 20219/314118 objects misplaced (6.437%); 13 MiB/s, 204 objects/s recovering 2024-09-13T14:44:03.164 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:02 smithi019 ceph-mon[91296]: pgmap v606: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 534 op/s; 51357/314118 objects degraded (16.350%); 20219/314118 objects misplaced (6.437%); 13 MiB/s, 204 objects/s recovering 2024-09-13T14:44:03.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:03.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:03.411 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:03.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.440 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:03.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:03.496 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:03.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:03.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:03.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:03.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:03.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:03.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:03.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:03.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:04.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:04.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:04.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:04.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:44:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:44:04] "GET /metrics HTTP/1.1" 200 36558 "" "Prometheus/2.51.0" 2024-09-13T14:44:04.261 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.294 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:04.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:04.395 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:04.418 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:04.520 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:04.556 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:04.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:04.692 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:04.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:04.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:04.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:04.976 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:05.076 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:04 smithi003 ceph-mon[104007]: pgmap v607: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 505 op/s; 51357/313374 objects degraded (16.388%); 19875/313374 objects misplaced (6.342%); 14 MiB/s, 226 objects/s recovering 2024-09-13T14:44:05.076 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:04 smithi003 ceph-mon[107278]: pgmap v607: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 505 op/s; 51357/313374 objects degraded (16.388%); 19875/313374 objects misplaced (6.342%); 14 MiB/s, 226 objects/s recovering 2024-09-13T14:44:05.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:05.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:05.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:04 smithi019 ceph-mon[91296]: pgmap v607: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 505 op/s; 51357/313374 objects degraded (16.388%); 19875/313374 objects misplaced (6.342%); 14 MiB/s, 226 objects/s recovering 2024-09-13T14:44:05.231 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:05.263 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:05.365 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:05.395 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:05.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:05.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:05.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:05.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:05.764 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:44:05.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:05.810 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:05.911 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:05.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 51357/313374 objects degraded (16.388%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:05.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2088869048' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 51357/313374 objects degraded (16.388%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:05.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2088869048' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:05.942 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:05.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:44:06.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:06.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:06.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 51357/313374 objects degraded (16.388%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2088869048' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:06.188 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:06.215 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:06.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:06.344 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:06.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:06.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:06.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:06.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:06.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:06.708 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.746 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:06.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:06.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.878 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:06.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:06.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:07.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:07.032 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:44:07.033 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:44:07.033 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T14:44:07.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-13T14:44:07.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:06 smithi003 ceph-mon[104007]: pgmap v608: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 50148/306816 objects degraded (16.345%); 19365/306816 objects misplaced (6.312%); 17 MiB/s, 275 objects/s recovering 2024-09-13T14:44:07.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:06 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:07.049 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:06 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2088869048' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:07.049 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:06 smithi003 ceph-mon[104007]: osdmap e185: 8 total, 8 up, 8 in 2024-09-13T14:44:07.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:06 smithi003 ceph-mon[107278]: pgmap v608: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 50148/306816 objects degraded (16.345%); 19365/306816 objects misplaced (6.312%); 17 MiB/s, 275 objects/s recovering 2024-09-13T14:44:07.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:06 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:07.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:06 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2088869048' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:07.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:06 smithi003 ceph-mon[107278]: osdmap e185: 8 total, 8 up, 8 in 2024-09-13T14:44:07.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:07.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:07.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:06 smithi019 ceph-mon[91296]: pgmap v608: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 50148/306816 objects degraded (16.345%); 19365/306816 objects misplaced (6.312%); 17 MiB/s, 275 objects/s recovering 2024-09-13T14:44:07.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:06 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:07.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:06 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2088869048' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:07.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:06 smithi019 ceph-mon[91296]: osdmap e185: 8 total, 8 up, 8 in 2024-09-13T14:44:07.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:07.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:07.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:07.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:07.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:07.571 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:07.604 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:07.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:07.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:07.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:07.880 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:07.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:07.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:08.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:08.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:08.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:08.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:08.282 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:44:08.282 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:44:08.283 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:44:08.283 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:44:08.283 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:44:08.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:08.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:08.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:08.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:08.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:08.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:08.612 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:08.714 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:08.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:08.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:08.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:08.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:09.004 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:09.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:09.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:08 smithi003 ceph-mon[104007]: pgmap v610: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 50235/304281 objects degraded (16.509%); 18826/304281 objects misplaced (6.187%); 16 MiB/s, 255 objects/s recovering 2024-09-13T14:44:09.126 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:08 smithi003 ceph-mon[107278]: pgmap v610: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 50235/304281 objects degraded (16.509%); 18826/304281 objects misplaced (6.187%); 16 MiB/s, 255 objects/s recovering 2024-09-13T14:44:09.135 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:09.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:08 smithi019 ceph-mon[91296]: pgmap v610: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 50235/304281 objects degraded (16.509%); 18826/304281 objects misplaced (6.187%); 16 MiB/s, 255 objects/s recovering 2024-09-13T14:44:09.237 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:09.271 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:09.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:09.407 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:09.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:09.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:09.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:09.694 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:09.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:09.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:44:09.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:09.860 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:09.862 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:09.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:09.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:44:09.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:10.086 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:10.111 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:10.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:10.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:10.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:10.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:10.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:10.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:10.674 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:44:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:44:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:10.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:10.810 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:10.912 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.008 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:11.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:11.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:11.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:10 smithi019 ceph-mon[91296]: pgmap v611: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 588 op/s; 49147/298017 objects degraded (16.491%); 18291/298017 objects misplaced (6.138%); 16 MiB/s, 256 objects/s recovering 2024-09-13T14:44:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 49147/298017 objects degraded (16.491%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:11.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:10 smithi003 ceph-mon[104007]: pgmap v611: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 588 op/s; 49147/298017 objects degraded (16.491%); 18291/298017 objects misplaced (6.138%); 16 MiB/s, 256 objects/s recovering 2024-09-13T14:44:11.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 49147/298017 objects degraded (16.491%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:10 smithi003 ceph-mon[107278]: pgmap v611: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 588 op/s; 49147/298017 objects degraded (16.491%); 18291/298017 objects misplaced (6.138%); 16 MiB/s, 256 objects/s recovering 2024-09-13T14:44:11.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 49147/298017 objects degraded (16.491%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:11.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:11.398 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:11.435 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:11.536 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:11.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:11.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:11.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:11.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:11.852 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:11.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:11.988 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:12.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.120 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:12.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:12.221 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:12.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:12.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:12.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:12.535 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:12.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:12.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:12.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:12.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:12.917 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:12.953 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:12.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:13.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:13.084 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:12 smithi019 ceph-mon[91296]: pgmap v612: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 49238/296424 objects degraded (16.611%); 18117/296424 objects misplaced (6.112%); 16 MiB/s, 254 objects/s recovering 2024-09-13T14:44:13.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:13.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:12 smithi003 ceph-mon[107278]: pgmap v612: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 49238/296424 objects degraded (16.611%); 18117/296424 objects misplaced (6.112%); 16 MiB/s, 254 objects/s recovering 2024-09-13T14:44:13.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:12 smithi003 ceph-mon[104007]: pgmap v612: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 49238/296424 objects degraded (16.611%); 18117/296424 objects misplaced (6.112%); 16 MiB/s, 254 objects/s recovering 2024-09-13T14:44:13.198 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:13.223 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:13.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:13.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:13.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:13.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:13.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:13.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:13.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:13.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:13.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:13.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:13.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:13.958 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:14.060 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:14.092 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:14.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:14.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:14.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:14.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:14.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:14.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:44:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:44:14] "GET /metrics HTTP/1.1" 200 36561 "" "Prometheus/2.51.0" 2024-09-13T14:44:14.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:14.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:14.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:14.670 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:14.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:14.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:14.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:14 smithi003 ceph-mon[104007]: pgmap v613: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 49238/295500 objects degraded (16.663%); 17740/295500 objects misplaced (6.003%); 16 MiB/s, 258 objects/s recovering 2024-09-13T14:44:14.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:14 smithi003 ceph-mon[107278]: pgmap v613: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 49238/295500 objects degraded (16.663%); 17740/295500 objects misplaced (6.003%); 16 MiB/s, 258 objects/s recovering 2024-09-13T14:44:14.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:14.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:15.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:15.092 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:14 smithi019 ceph-mon[91296]: pgmap v613: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 49238/295500 objects degraded (16.663%); 17740/295500 objects misplaced (6.003%); 16 MiB/s, 258 objects/s recovering 2024-09-13T14:44:15.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:15.231 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:15.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:15.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:15.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.514 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:15.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:15.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:15.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:15.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:15.781 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:15.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:15.918 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:16.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:16.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:16.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:16.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:15 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 49238/295500 objects degraded (16.663%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:16.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:15 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 49238/295500 objects degraded (16.663%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:16.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:15 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 49238/295500 objects degraded (16.663%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:16.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:16.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:16.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:16.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:16.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:16.582 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:16.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:16.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:16.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:16.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:16.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:17.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:17.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:17.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:16 smithi019 ceph-mon[91296]: pgmap v614: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 599 op/s; 48055/288822 objects degraded (16.638%); 17213/288822 objects misplaced (5.960%); 16 MiB/s, 257 objects/s recovering 2024-09-13T14:44:17.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:16 smithi003 ceph-mon[104007]: pgmap v614: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 599 op/s; 48055/288822 objects degraded (16.638%); 17213/288822 objects misplaced (5.960%); 16 MiB/s, 257 objects/s recovering 2024-09-13T14:44:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:16 smithi003 ceph-mon[107278]: pgmap v614: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 599 op/s; 48055/288822 objects degraded (16.638%); 17213/288822 objects misplaced (5.960%); 16 MiB/s, 257 objects/s recovering 2024-09-13T14:44:17.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:17.286 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.319 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:17.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:17.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:17.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:17.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:17.568 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:17.600 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:17.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:17.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:17.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:17.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:17.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:18.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:18.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:18.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:18.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:18.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:18.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:18.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:18.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:18.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:18.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:18.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:18.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:18.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:18.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:19.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:19.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:19.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:19.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:18 smithi019 ceph-mon[91296]: pgmap v615: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 48184/286110 objects degraded (16.841%); 16492/286110 objects misplaced (5.764%); 17 MiB/s, 267 objects/s recovering 2024-09-13T14:44:19.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:18 smithi003 ceph-mon[104007]: pgmap v615: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 48184/286110 objects degraded (16.841%); 16492/286110 objects misplaced (5.764%); 17 MiB/s, 267 objects/s recovering 2024-09-13T14:44:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:18 smithi003 ceph-mon[107278]: pgmap v615: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 48184/286110 objects degraded (16.841%); 16492/286110 objects misplaced (5.764%); 17 MiB/s, 267 objects/s recovering 2024-09-13T14:44:19.282 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:19.313 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:19.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:19.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:19.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:19.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:19.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:19.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:19.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:19.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:19.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:19.970 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:19.999 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:20.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:20.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:20.249 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.280 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:20.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:20.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:20.412 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:20.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:20.544 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:20.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:20.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:44:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:44:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:20.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:20.789 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:20.819 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:20.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:21.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:20 smithi003 ceph-mon[107278]: pgmap v616: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 693 op/s; 47014/279312 objects degraded (16.832%); 15875/279312 objects misplaced (5.684%); 19 MiB/s, 300 objects/s recovering 2024-09-13T14:44:21.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 47014/279312 objects degraded (16.832%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:21.032 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:20 smithi003 ceph-mon[107278]: osdmap e186: 8 total, 8 up, 8 in 2024-09-13T14:44:21.032 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:20 smithi003 ceph-mon[104007]: pgmap v616: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 693 op/s; 47014/279312 objects degraded (16.832%); 15875/279312 objects misplaced (5.684%); 19 MiB/s, 300 objects/s recovering 2024-09-13T14:44:21.032 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 47014/279312 objects degraded (16.832%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:21.032 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:20 smithi003 ceph-mon[104007]: osdmap e186: 8 total, 8 up, 8 in 2024-09-13T14:44:21.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:21.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:44:21.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:44:21.092 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:44:21.092 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:44:21.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:21.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:21.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:21.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:21.102 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:21.104 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:21.105 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:21.106 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:44:21.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:21.148 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:44:21.148 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:20 smithi019 ceph-mon[91296]: pgmap v616: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 693 op/s; 47014/279312 objects degraded (16.832%); 15875/279312 objects misplaced (5.684%); 19 MiB/s, 300 objects/s recovering 2024-09-13T14:44:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 47014/279312 objects degraded (16.832%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:20 smithi019 ceph-mon[91296]: osdmap e186: 8 total, 8 up, 8 in 2024-09-13T14:44:21.184 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:44:21.185 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:21.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:21.219 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:44:21.220 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:21.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:21.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:21.400 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:21.404 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 297 (0%) 2024-09-13T14:44:21.404 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:44:21.404 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.54671 -> 6.54671 (expected baseline 5.69951) 2024-09-13T14:44:21.404 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.5 with 27 -> 27 pgs (0.727273 -> 0.727273 * mean) 2024-09-13T14:44:21.404 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 47 -> 47 pgs (1.26599 -> 1.26599 * mean) 2024-09-13T14:44:21.404 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:44:21.405 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-09-13T14:44:21.405 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 1 2024-09-13T14:44:21.405 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:44:21.405 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0331 2024-09-13T14:44:21.405 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.3311 2024-09-13T14:44:21.405 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T14:44:21.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:21.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:21.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:21.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:21.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:21.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:21.912 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:21.939 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:22.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:22.074 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:22.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:22.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1837599204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:22.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:21 smithi003 ceph-mon[107278]: osdmap e187: 8 total, 8 up, 8 in 2024-09-13T14:44:22.153 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1837599204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:22.153 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:21 smithi003 ceph-mon[104007]: osdmap e187: 8 total, 8 up, 8 in 2024-09-13T14:44:22.177 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:22.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1837599204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:22.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:21 smithi019 ceph-mon[91296]: osdmap e187: 8 total, 8 up, 8 in 2024-09-13T14:44:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.214 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:22.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:22.315 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:22.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:22.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:22.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:22.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:22.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:22.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:22.863 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.893 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:22.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:22 smithi003 ceph-mon[107278]: from='client.45983 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:22 smithi003 ceph-mon[107278]: pgmap v619: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 47129/277647 objects degraded (16.974%); 15671/277647 objects misplaced (5.644%); 19 MiB/s, 303 objects/s recovering 2024-09-13T14:44:22.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:22 smithi003 ceph-mon[104007]: from='client.45983 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:22 smithi003 ceph-mon[104007]: pgmap v619: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 47129/277647 objects degraded (16.974%); 15671/277647 objects misplaced (5.644%); 19 MiB/s, 303 objects/s recovering 2024-09-13T14:44:22.994 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:23.022 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:23.124 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:23.161 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:22 smithi019 ceph-mon[91296]: from='client.45983 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:22 smithi019 ceph-mon[91296]: pgmap v619: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 47129/277647 objects degraded (16.974%); 15671/277647 objects misplaced (5.644%); 19 MiB/s, 303 objects/s recovering 2024-09-13T14:44:23.263 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:23.291 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:23.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:23.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:23.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:23.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:23.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:23.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:23.833 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T14:44:23.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-13T14:44:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:23.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:23.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:24.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:24.142 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:24.172 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:24.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:24.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:24.418 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:24.421 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:44:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:44:24] "GET /metrics HTTP/1.1" 200 36561 "" "Prometheus/2.51.0" 2024-09-13T14:44:24.451 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:24.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:24.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:24.720 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:24.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:24.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:44:24.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:24.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:25.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:25.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:24 smithi003 ceph-mon[104007]: pgmap v620: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 497 op/s; 47129/276864 objects degraded (17.022%); 15277/276864 objects misplaced (5.518%); 15 MiB/s, 245 objects/s recovering 2024-09-13T14:44:25.015 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:24 smithi003 ceph-mon[107278]: pgmap v620: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 497 op/s; 47129/276864 objects degraded (17.022%); 15277/276864 objects misplaced (5.518%); 15 MiB/s, 245 objects/s recovering 2024-09-13T14:44:25.104 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:25.131 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:25.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:24 smithi019 ceph-mon[91296]: pgmap v620: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 497 op/s; 47129/276864 objects degraded (17.022%); 15277/276864 objects misplaced (5.518%); 15 MiB/s, 245 objects/s recovering 2024-09-13T14:44:25.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:25.262 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:25.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:25.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:25.537 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:25.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:25.669 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:25.771 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.804 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:25.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:25.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:25.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 47129/276864 objects degraded (17.022%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:25.917 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 47129/276864 objects degraded (17.022%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:25.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:25.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:26.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:26.103 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:26.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 47129/276864 objects degraded (17.022%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:26.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:26.232 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:26.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:26.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:26.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:26.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:26.615 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:26.644 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:26.746 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:26.777 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:26.851 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:44:26.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:26 smithi003 ceph-mon[104007]: pgmap v621: 99 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 40272/269964 objects degraded (14.918%); 20285/269964 objects misplaced (7.514%); 18 MiB/s, 286 objects/s recovering 2024-09-13T14:44:26.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1511699774' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:26.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:26 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:26.878 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:26 smithi003 ceph-mon[107278]: pgmap v621: 99 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 40272/269964 objects degraded (14.918%); 20285/269964 objects misplaced (7.514%); 18 MiB/s, 286 objects/s recovering 2024-09-13T14:44:26.878 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1511699774' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:26.878 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:26 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:26.879 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.909 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:26.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:27.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:27.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:27.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:27.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:26 smithi019 ceph-mon[91296]: pgmap v621: 99 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 40272/269964 objects degraded (14.918%); 20285/269964 objects misplaced (7.514%); 18 MiB/s, 286 objects/s recovering 2024-09-13T14:44:27.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1511699774' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:27.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:26 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:27.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:27.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:27.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:27.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:27.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:27.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:27.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:27.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:27.765 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:27.794 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:27.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:27.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:27.835 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[107278]: osdmap e188: 8 total, 8 up, 8 in 2024-09-13T14:44:27.835 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:27.835 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:27.835 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:27.835 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:27.835 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:27.835 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[104007]: osdmap e188: 8 total, 8 up, 8 in 2024-09-13T14:44:27.835 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:27.896 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:27.927 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:28.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:28.060 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:28.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:28.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:28.162 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:27 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:27 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:27 smithi019 ceph-mon[91296]: osdmap e188: 8 total, 8 up, 8 in 2024-09-13T14:44:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:28.194 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:28.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:28.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:28.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:28.420 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:28.422 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:28.463 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:28.463 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:28.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:28.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:28.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:28.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:28.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:28.789 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:28.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:28.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:28.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:29.033 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:29.066 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:29.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:29.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:28 smithi019 ceph-mon[91296]: pgmap v623: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 559 op/s; 40360/267504 objects degraded (15.088%); 20099/267504 objects misplaced (7.514%); 13 MiB/s, 213 objects/s recovering 2024-09-13T14:44:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:44:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:44:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:28 smithi003 ceph-mon[104007]: pgmap v623: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 559 op/s; 40360/267504 objects degraded (15.088%); 20099/267504 objects misplaced (7.514%); 13 MiB/s, 213 objects/s recovering 2024-09-13T14:44:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:44:29.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:44:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:28 smithi003 ceph-mon[107278]: pgmap v623: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 559 op/s; 40360/267504 objects degraded (15.088%); 20099/267504 objects misplaced (7.514%); 13 MiB/s, 213 objects/s recovering 2024-09-13T14:44:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:44:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:44:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:29.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:29.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:29.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:29.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:29.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:29.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:29.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:29.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:29.739 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:29.769 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:29.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:29.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:30.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:30.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:44:30.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:30.183 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:30.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:30.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:30.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.460 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:30.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:30.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:30.592 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:30.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:44:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:44:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:30.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:30.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:30.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:30.867 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:30.910 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:44:30.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:31.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:31.112 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:31.149 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:31.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:30 smithi019 ceph-mon[91296]: pgmap v624: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 703 op/s; 39320/260667 objects degraded (15.084%); 19230/260667 objects misplaced (7.377%); 18 MiB/s, 294 objects/s recovering 2024-09-13T14:44:31.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 39320/260667 objects degraded (15.084%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:31.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:30 smithi003 ceph-mon[107278]: pgmap v624: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 703 op/s; 39320/260667 objects degraded (15.084%); 19230/260667 objects misplaced (7.377%); 18 MiB/s, 294 objects/s recovering 2024-09-13T14:44:31.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 39320/260667 objects degraded (15.084%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:31.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:30 smithi003 ceph-mon[104007]: pgmap v624: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 703 op/s; 39320/260667 objects degraded (15.084%); 19230/260667 objects misplaced (7.377%); 18 MiB/s, 294 objects/s recovering 2024-09-13T14:44:31.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 39320/260667 objects degraded (15.084%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:31.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.293 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:31.316 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:31.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:31.431 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:31.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:31.562 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:31.664 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.706 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:31.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:31.808 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:31.853 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:31.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:31.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:32.084 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:32.111 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:32.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:32.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:32.309 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:44:32.309 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:32.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:32.362 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:44:32.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:32.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:32.405 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:44:32.405 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:32.443 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:44:32.443 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:32.473 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:44:32.474 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:32.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:32.513 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:44:32.513 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:44:32.556 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:32.593 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:44:32.593 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:32.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.685 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:32.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:32.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:32.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:32.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:32.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:33.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:33.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:32 smithi003 ceph-mon[107278]: pgmap v625: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 39449/258759 objects degraded (15.245%); 19014/258759 objects misplaced (7.348%); 15 MiB/s, 245 objects/s recovering 2024-09-13T14:44:33.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1306883411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:33.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:32 smithi003 ceph-mon[104007]: pgmap v625: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 39449/258759 objects degraded (15.245%); 19014/258759 objects misplaced (7.348%); 15 MiB/s, 245 objects/s recovering 2024-09-13T14:44:33.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1306883411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:33.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:33.148 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:32 smithi019 ceph-mon[91296]: pgmap v625: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 39449/258759 objects degraded (15.245%); 19014/258759 objects misplaced (7.348%); 15 MiB/s, 245 objects/s recovering 2024-09-13T14:44:33.148 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1306883411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:33.185 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 297 (0%) 2024-09-13T14:44:33.185 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:44:33.185 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.54671 -> 6.54671 (expected baseline 5.69951) 2024-09-13T14:44:33.185 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.5 with 27 -> 27 pgs (0.727273 -> 0.727273 * mean) 2024-09-13T14:44:33.185 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 47 -> 47 pgs (1.26599 -> 1.26599 * mean) 2024-09-13T14:44:33.185 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:44:33.185 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-09-13T14:44:33.185 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 1 2024-09-13T14:44:33.186 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:44:33.186 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0319 2024-09-13T14:44:33.186 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.3186 2024-09-13T14:44:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T14:44:33.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:33.236 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:33.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:33.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:33.474 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:44:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:44:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:44:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.45869.0:158199 9.e e188/152 9:76c26fa2:::benchmark_data_smithi003_156985_object37283:head v 188'9797, mlcod=188'9795)", 2024-09-13T14:44:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:44:33.471232+0000", 2024-09-13T14:44:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.000895526, 2024-09-13T14:44:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000925025, 2024-09-13T14:44:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:44:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "started", 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:33.471232+0000", 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:33.471232+0000", 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:44:33.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:33.471231+0000", 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:33.471238+0000", 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3780000000000003e-06 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:33.471253+0000", 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4744e-05 2024-09-13T14:44:33.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:33.471259+0000", 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3090000000000001e-06 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:33.471327+0000", 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7322000000000004e-05 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:33.471351+0000", 2024-09-13T14:44:33.477 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4760999999999999e-05 2024-09-13T14:44:33.478 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:44:33.478 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:44:33.478 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:44:33.478 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:44:33.478 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-09-13T14:44:33.478 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 1 2024-09-13T14:44:33.478 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:44:33.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:33.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:33.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:33.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:33.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:33.746 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:33.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:33.798 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:33.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:33.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.932 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:33.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:34.033 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:34.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:34.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:34.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:44:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:44:34] "GET /metrics HTTP/1.1" 200 36558 "" "Prometheus/2.51.0" 2024-09-13T14:44:34.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:34.233 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-13T14:44:34.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-09-13T14:44:34.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:34.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:34.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:34.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:34.647 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.679 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:34.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:34.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:34.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:35.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:35.089 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:35.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:34 smithi019 ceph-mon[91296]: from='client.55792 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:35.177 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:34 smithi019 ceph-mon[91296]: pgmap v626: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 39449/257955 objects degraded (15.293%); 19014/257955 objects misplaced (7.371%); 13 MiB/s, 205 objects/s recovering 2024-09-13T14:44:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:34 smithi003 ceph-mon[107278]: from='client.55792 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:35.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:34 smithi003 ceph-mon[107278]: pgmap v626: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 39449/257955 objects degraded (15.293%); 19014/257955 objects misplaced (7.371%); 13 MiB/s, 205 objects/s recovering 2024-09-13T14:44:35.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:34 smithi003 ceph-mon[104007]: from='client.55792 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:34 smithi003 ceph-mon[104007]: pgmap v626: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 39449/257955 objects degraded (15.293%); 19014/257955 objects misplaced (7.371%); 13 MiB/s, 205 objects/s recovering 2024-09-13T14:44:35.191 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:35.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:44:35.227 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:35.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:35.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:35.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:44:35.523 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:35.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:35.626 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:35.640 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:35.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:35.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:35.740 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:35.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:35.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:35.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:35.961 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:36.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:36.073 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 39449/257955 objects degraded (15.293%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:36.089 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 39449/257955 objects degraded (15.293%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:36.090 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 39449/257955 objects degraded (15.293%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:36.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:36.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:36.276 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:36.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:36.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:36.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:36.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:36.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:36.710 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:36.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:36.847 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:36.948 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:36.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:37.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:37.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:37.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:36 smithi003 ceph-mon[107278]: pgmap v627: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 38487/250956 objects degraded (15.336%); 18107/250956 objects misplaced (7.215%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:44:37.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:36 smithi003 ceph-mon[104007]: pgmap v627: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 38487/250956 objects degraded (15.336%); 18107/250956 objects misplaced (7.215%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:44:37.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:36 smithi019 ceph-mon[91296]: pgmap v627: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 38487/250956 objects degraded (15.336%); 18107/250956 objects misplaced (7.215%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:44:37.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:37.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:37.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:37.513 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:44:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:44:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:44:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:44:37.513 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:44:37.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:37.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.766 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:37.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:37.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:44:37.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3074319613' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:37.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:37 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:37.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:38.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:38.053 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:38.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:38.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3074319613' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:38.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:37 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:38.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3074319613' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:38.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:37 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:38.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:38.302 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:38.433 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:38.460 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:38.565 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:38.595 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:38.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:38.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:38.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:38.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:38.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:39.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:39.139 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:39.161 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:39.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:39.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:38 smithi019 ceph-mon[91296]: pgmap v628: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 620 op/s; 38620/248163 objects degraded (15.562%); 17887/248163 objects misplaced (7.208%); 12 MiB/s, 190 objects/s recovering 2024-09-13T14:44:39.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:38 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:39.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:38 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:39.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:38 smithi019 ceph-mon[91296]: osdmap e189: 8 total, 8 up, 8 in 2024-09-13T14:44:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:38 smithi003 ceph-mon[104007]: pgmap v628: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 620 op/s; 38620/248163 objects degraded (15.562%); 17887/248163 objects misplaced (7.208%); 12 MiB/s, 190 objects/s recovering 2024-09-13T14:44:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:38 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:38 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:39.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:38 smithi003 ceph-mon[104007]: osdmap e189: 8 total, 8 up, 8 in 2024-09-13T14:44:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:38 smithi003 ceph-mon[107278]: pgmap v628: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 620 op/s; 38620/248163 objects degraded (15.562%); 17887/248163 objects misplaced (7.208%); 12 MiB/s, 190 objects/s recovering 2024-09-13T14:44:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:38 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:38 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:38 smithi003 ceph-mon[107278]: osdmap e189: 8 total, 8 up, 8 in 2024-09-13T14:44:39.263 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:39.292 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:39.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:39.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:39.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:39.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:39.674 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:39.706 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:39.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:39.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:39.970 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.009 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:40.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:40.110 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:40.141 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:40.243 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:40.275 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:40.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:40.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:40.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:40.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:40.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:40.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:44:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:44:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:40.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:40.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:40.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:40.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:44:40.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:40 smithi003 ceph-mon[107278]: pgmap v630: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 37660/241944 objects degraded (15.566%); 17021/241944 objects misplaced (7.035%); 13 MiB/s, 201 objects/s recovering 2024-09-13T14:44:40.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 37660/241944 objects degraded (15.566%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:40.928 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:40 smithi003 ceph-mon[104007]: pgmap v630: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 37660/241944 objects degraded (15.566%); 17021/241944 objects misplaced (7.035%); 13 MiB/s, 201 objects/s recovering 2024-09-13T14:44:40.928 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 37660/241944 objects degraded (15.566%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:40.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:41.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:41.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:41.168 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:40 smithi019 ceph-mon[91296]: pgmap v630: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 37660/241944 objects degraded (15.566%); 17021/241944 objects misplaced (7.035%); 13 MiB/s, 201 objects/s recovering 2024-09-13T14:44:41.168 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 37660/241944 objects degraded (15.566%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:41.183 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:41.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:41.312 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:41.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:44:41.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:41.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:41.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:44:41.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:44:41.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:44:41.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:44:41.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:44:41.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:44:41.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:41.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:41.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:41.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:41.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:41.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:41.481 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:44:41.482 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:44:41.482 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:41.482 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:41.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:41.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:41.708 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:41.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:41.789 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:41.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:41.872 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:41.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:42.037 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:42.139 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:42.172 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:42.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:41 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2868618726' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:42.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:41 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2868618726' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:42.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:41 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2868618726' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:42.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:42.305 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:42.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:42.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:42.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:42.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:42.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:42.712 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:42.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:42.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:42.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:42.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:43.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:43.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:43.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:42 smithi019 ceph-mon[91296]: pgmap v631: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 37749/240243 objects degraded (15.713%); 16842/240243 objects misplaced (7.010%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:44:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:43.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:42 smithi003 ceph-mon[104007]: pgmap v631: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 37749/240243 objects degraded (15.713%); 16842/240243 objects misplaced (7.010%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:44:43.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:43.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:42 smithi003 ceph-mon[107278]: pgmap v631: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 37749/240243 objects degraded (15.713%); 16842/240243 objects misplaced (7.010%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:44:43.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:43.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:43.266 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:43.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:43.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:43.402 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:43.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:43.538 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:43.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:43.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:43.812 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:43.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:43.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:43.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:44.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:44.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:44.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:44:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:44:44] "GET /metrics HTTP/1.1" 200 36559 "" "Prometheus/2.51.0" 2024-09-13T14:44:44.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:44.503 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:44.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:44.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:44.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:44.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:44.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:45.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:45.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:45.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:44 smithi019 ceph-mon[91296]: pgmap v632: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 37749/239487 objects degraded (15.762%); 16842/239487 objects misplaced (7.033%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:44:45.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:44 smithi003 ceph-mon[104007]: pgmap v632: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 37749/239487 objects degraded (15.762%); 16842/239487 objects misplaced (7.033%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:44:45.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:44 smithi003 ceph-mon[107278]: pgmap v632: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 37749/239487 objects degraded (15.762%); 16842/239487 objects misplaced (7.033%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:44:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:45.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:45.468 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:45.493 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:45.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:45.620 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:45.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:45.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:45.767 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:45.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:45.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:46.006 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:46.035 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:46.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:46.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:46.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:46.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 37749/239487 objects degraded (15.762%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:46.298 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 37749/239487 objects degraded (15.762%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:46.298 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 297 (0%) 2024-09-13T14:44:46.298 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.54671 -> 6.54671 (expected baseline 5.69951) 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.5 with 27 -> 27 pgs (0.727273 -> 0.727273 * mean) 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 47 -> 47 pgs (1.26599 -> 1.26599 * mean) 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 1 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0301 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.3007 2024-09-13T14:44:46.299 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T14:44:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:46.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:46.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:46.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 37749/239487 objects degraded (15.762%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:46.442 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:46.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:46.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:46.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:46.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:46.852 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:46.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:46.988 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:47.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:47.123 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:47.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:47.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:47.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:47.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:46 smithi019 ceph-mon[91296]: pgmap v633: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 36715/232839 objects degraded (15.768%); 15978/232839 objects misplaced (6.862%); 13 MiB/s, 203 objects/s recovering 2024-09-13T14:44:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:46 smithi019 ceph-mon[91296]: from='client.46013 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:47.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:46 smithi003 ceph-mon[104007]: pgmap v633: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 36715/232839 objects degraded (15.768%); 15978/232839 objects misplaced (6.862%); 13 MiB/s, 203 objects/s recovering 2024-09-13T14:44:47.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:46 smithi003 ceph-mon[104007]: from='client.46013 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:46 smithi003 ceph-mon[107278]: pgmap v633: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 36715/232839 objects degraded (15.768%); 15978/232839 objects misplaced (6.862%); 13 MiB/s, 203 objects/s recovering 2024-09-13T14:44:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:46 smithi003 ceph-mon[107278]: from='client.46013 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:47.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:47.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:47.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:47.671 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:47.773 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:47.811 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:47.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:47.913 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:47.952 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:48.054 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:48.080 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:48.182 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:48.204 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:48.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:48.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:48.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:48.458 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:48.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:48.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:48.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:48.731 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:48.833 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.872 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:48.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:48.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:49.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:49.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:49.147 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:49.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:49.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:48 smithi003 ceph-mon[104007]: pgmap v634: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 599 op/s; 36786/230175 objects degraded (15.982%); 15790/230175 objects misplaced (6.860%); 13 MiB/s, 206 objects/s recovering 2024-09-13T14:44:49.281 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:48 smithi003 ceph-mon[107278]: pgmap v634: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 599 op/s; 36786/230175 objects degraded (15.982%); 15790/230175 objects misplaced (6.860%); 13 MiB/s, 206 objects/s recovering 2024-09-13T14:44:49.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:49.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.420 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:49.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:48 smithi019 ceph-mon[91296]: pgmap v634: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 599 op/s; 36786/230175 objects degraded (15.982%); 15790/230175 objects misplaced (6.860%); 13 MiB/s, 206 objects/s recovering 2024-09-13T14:44:49.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:49.546 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:49.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:49.678 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:49.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:49.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:49.920 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.953 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:49.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:50.054 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.092 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:50.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:50.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:50.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:50.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:50.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:50.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:50.464 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:50.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:50.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:50.629 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:44:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:44:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:50.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:50.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:50.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:51.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:51.051 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:51.080 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T14:44:51.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-13T14:44:51.124 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:50 smithi019 ceph-mon[91296]: pgmap v635: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 699 op/s; 35868/223476 objects degraded (16.050%); 14960/223476 objects misplaced (6.694%); 16 MiB/s, 250 objects/s recovering 2024-09-13T14:44:51.124 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 35868/223476 objects degraded (16.050%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:51.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:51.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:51.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:51.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:51.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:51.413 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:51.413 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:51.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:50 smithi003 ceph-mon[107278]: pgmap v635: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 699 op/s; 35868/223476 objects degraded (16.050%); 14960/223476 objects misplaced (6.694%); 16 MiB/s, 250 objects/s recovering 2024-09-13T14:44:51.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 35868/223476 objects degraded (16.050%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:51.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:50 smithi003 ceph-mon[104007]: pgmap v635: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 699 op/s; 35868/223476 objects degraded (16.050%); 14960/223476 objects misplaced (6.694%); 16 MiB/s, 250 objects/s recovering 2024-09-13T14:44:51.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 35868/223476 objects degraded (16.050%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:51.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:51.487 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:51.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:51.658 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:51.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:51.789 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:51.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:51.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:51.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:52.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:52.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:52.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:44:52.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:52.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:52.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:52.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:52.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:52.500 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:52.531 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:52.557 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:52.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:52.672 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:52.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:52.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:52.805 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:52.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:52.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:52.939 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:53.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:53.082 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:53.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:53.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:53.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:53.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:52 smithi019 ceph-mon[91296]: pgmap v636: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 512 op/s; 35868/223476 objects degraded (16.050%); 14960/223476 objects misplaced (6.694%); 11 MiB/s, 169 objects/s recovering 2024-09-13T14:44:53.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:52 smithi003 ceph-mon[104007]: pgmap v636: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 512 op/s; 35868/223476 objects degraded (16.050%); 14960/223476 objects misplaced (6.694%); 11 MiB/s, 169 objects/s recovering 2024-09-13T14:44:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:52 smithi003 ceph-mon[107278]: pgmap v636: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 512 op/s; 35868/223476 objects degraded (16.050%); 14960/223476 objects misplaced (6.694%); 11 MiB/s, 169 objects/s recovering 2024-09-13T14:44:53.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:53.500 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:53.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:44:53.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:53.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:53.796 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:53.823 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:53.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:53.956 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:54.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:54.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:54.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:44:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:44:54] "GET /metrics HTTP/1.1" 200 36559 "" "Prometheus/2.51.0" 2024-09-13T14:44:54.209 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:54.239 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:54.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:54.373 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:54.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:54.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:54.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:54.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:54.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:54.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:54.913 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:54.944 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:55.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:55.053 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:54 smithi019 ceph-mon[91296]: pgmap v637: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 539 op/s; 35962/220827 objects degraded (16.285%); 14760/220827 objects misplaced (6.684%); 11 MiB/s, 170 objects/s recovering 2024-09-13T14:44:55.080 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:55.182 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:55.216 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:55.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:55.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:55.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:55.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:54 smithi003 ceph-mon[104007]: pgmap v637: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 539 op/s; 35962/220827 objects degraded (16.285%); 14760/220827 objects misplaced (6.684%); 11 MiB/s, 170 objects/s recovering 2024-09-13T14:44:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:54 smithi003 ceph-mon[107278]: pgmap v637: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 539 op/s; 35962/220827 objects degraded (16.285%); 14760/220827 objects misplaced (6.684%); 11 MiB/s, 170 objects/s recovering 2024-09-13T14:44:55.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:55.485 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:55.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:55.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:55.683 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:55.683 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:55.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:55.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:55.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:55.978 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:44:56.027 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:56.055 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:56.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:56.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:56.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2355264742' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:56.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 35962/220827 objects degraded (16.285%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:56.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2355264742' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:56.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 35962/220827 objects degraded (16.285%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:56.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:56.301 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:56.330 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:56.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2355264742' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 35962/220827 objects degraded (16.285%), 9 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:56.432 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:56.460 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:56.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:56.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:56.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:56.750 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:56.853 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:56.879 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:56.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:57.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:57.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:57.161 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:57.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:56 smithi003 ceph-mon[107278]: pgmap v638: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 34920/213786 objects degraded (16.334%); 13774/213786 objects misplaced (6.443%); 16 MiB/s, 254 objects/s recovering 2024-09-13T14:44:57.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:56 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:57.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2355264742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:57.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:56 smithi003 ceph-mon[107278]: osdmap e190: 8 total, 8 up, 8 in 2024-09-13T14:44:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:56 smithi003 ceph-mon[104007]: pgmap v638: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 34920/213786 objects degraded (16.334%); 13774/213786 objects misplaced (6.443%); 16 MiB/s, 254 objects/s recovering 2024-09-13T14:44:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:56 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2355264742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:56 smithi003 ceph-mon[104007]: osdmap e190: 8 total, 8 up, 8 in 2024-09-13T14:44:57.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:57.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:57.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:56 smithi019 ceph-mon[91296]: pgmap v638: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 34920/213786 objects degraded (16.334%); 13774/213786 objects misplaced (6.443%); 16 MiB/s, 254 objects/s recovering 2024-09-13T14:44:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:56 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2355264742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:56 smithi019 ceph-mon[91296]: osdmap e190: 8 total, 8 up, 8 in 2024-09-13T14:44:57.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:57.491 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:57.551 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:57.586 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:57.688 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:57.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:57.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:57.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:57.853 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 297 (0%) 2024-09-13T14:44:57.853 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:44:57.853 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.54671 -> 6.54671 (expected baseline 5.69951) 2024-09-13T14:44:57.853 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.5 with 27 -> 27 pgs (0.727273 -> 0.727273 * mean) 2024-09-13T14:44:57.853 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 47 -> 47 pgs (1.26599 -> 1.26599 * mean) 2024-09-13T14:44:57.853 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:44:57.853 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-09-13T14:44:57.853 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 0.05 2024-09-13T14:44:57.854 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:44:57.854 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0287 2024-09-13T14:44:57.854 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.2871 2024-09-13T14:44:57.854 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T14:44:57.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:57.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.993 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:57.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:58.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:58.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:58.177 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:44:58.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.271 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:58.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:58.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:58.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:58.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:58.566 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:58.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:58.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:58.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:58.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:58.892 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:58.919 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:58.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[107278]: pgmap v640: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 35030/211050 objects degraded (16.598%); 13578/211050 objects misplaced (6.434%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:44:58.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[107278]: from='client.46025 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[107278]: osdmap e191: 8 total, 8 up, 8 in 2024-09-13T14:44:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[104007]: pgmap v640: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 35030/211050 objects degraded (16.598%); 13578/211050 objects misplaced (6.434%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:44:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[104007]: from='client.46025 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:58 smithi003 ceph-mon[104007]: osdmap e191: 8 total, 8 up, 8 in 2024-09-13T14:44:59.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:59.029 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T14:44:59.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-13T14:44:59.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:59.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:59.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:58 smithi019 ceph-mon[91296]: pgmap v640: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 35030/211050 objects degraded (16.598%); 13578/211050 objects misplaced (6.434%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:44:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:44:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:58 smithi019 ceph-mon[91296]: from='client.46025 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:44:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:58 smithi019 ceph-mon[91296]: osdmap e191: 8 total, 8 up, 8 in 2024-09-13T14:44:59.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:59.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:44:59.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:59.325 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:59.426 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:59.453 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:59.555 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:59.582 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:59.597 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:59.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:59.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:44:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:59.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:59.871 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:44:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:59.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:00.006 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:00.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:00.139 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:00.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:00.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:44:59 smithi019 ceph-mon[91296]: osdmap e192: 8 total, 8 up, 8 in 2024-09-13T14:45:00.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:00.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:44:59 smithi003 ceph-mon[104007]: osdmap e192: 8 total, 8 up, 8 in 2024-09-13T14:45:00.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:44:59 smithi003 ceph-mon[107278]: osdmap e192: 8 total, 8 up, 8 in 2024-09-13T14:45:00.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:00.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:00.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:00.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:00.475 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:45:00.475 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:00.508 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:45:00.508 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:00.536 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:00.554 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:45:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:45:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:00.563 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:00.564 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:45:00.564 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:00.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:00.616 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:45:00.616 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:00.648 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:45:00.648 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:00.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:00.682 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:45:00.682 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:00.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:00.714 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:45:00.714 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:00.750 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:45:00.751 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:00.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:00.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:00.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:00.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:01.082 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:01.125 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:01.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:01.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:01.259 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:01 smithi019 ceph-mon[91296]: pgmap v643: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 32702/206649 objects degraded (15.825%); 14814/206649 objects misplaced (7.169%); 17 MiB/s, 279 objects/s recovering 2024-09-13T14:45:01.260 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:01 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1256265602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:01.260 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:01 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 32702/206649 objects degraded (15.825%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:01.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:01.365 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.426 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:01.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:01 smithi003 ceph-mon[107278]: pgmap v643: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 32702/206649 objects degraded (15.825%); 14814/206649 objects misplaced (7.169%); 17 MiB/s, 279 objects/s recovering 2024-09-13T14:45:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:01 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1256265602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:01 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 32702/206649 objects degraded (15.825%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:00 smithi003 ceph-mon[104007]: pgmap v643: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 32702/206649 objects degraded (15.825%); 14814/206649 objects misplaced (7.169%); 17 MiB/s, 279 objects/s recovering 2024-09-13T14:45:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:01 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1256265602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:01 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 32702/206649 objects degraded (15.825%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:01.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:01.564 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:01.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:01.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:01.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:01.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:01.945 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:45:01.945 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:45:01.945 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:45:01.945 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:45:01.945 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:45:01.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:01.993 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:02.095 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.133 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:02.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:02.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:02.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:02.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:02.401 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:02.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:02.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:02.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:45:02.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:02.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:02.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:02.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:02.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:02.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:03.019 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:03.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:03.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:03.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.244 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:03.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:03.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:03 smithi019 ceph-mon[91296]: pgmap v644: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 518 op/s; 31968/204450 objects degraded (15.636%); 14814/204450 objects misplaced (7.246%); 11 MiB/s, 171 objects/s recovering 2024-09-13T14:45:03.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.390 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:03.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:03.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:03 smithi003 ceph-mon[107278]: pgmap v644: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 518 op/s; 31968/204450 objects degraded (15.636%); 14814/204450 objects misplaced (7.246%); 11 MiB/s, 171 objects/s recovering 2024-09-13T14:45:03.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:03 smithi003 ceph-mon[104007]: pgmap v644: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 518 op/s; 31968/204450 objects degraded (15.636%); 14814/204450 objects misplaced (7.246%); 11 MiB/s, 171 objects/s recovering 2024-09-13T14:45:03.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:03.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:03.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:03.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:03.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:03.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:04.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:04.075 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:04.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:04.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:45:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:45:04] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:45:04.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:04.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.352 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:04.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:04.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:04.479 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:04.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:04.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:04.708 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:04.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:04.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:04.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:04.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:04.981 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:05.005 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:05.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:05.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:05.239 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:05.268 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:05.370 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:05.410 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:05 smithi019 ceph-mon[91296]: pgmap v645: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 510 op/s; 32040/202020 objects degraded (15.860%); 14629/202020 objects misplaced (7.241%); 9.3 MiB/s, 149 objects/s recovering 2024-09-13T14:45:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:05 smithi003 ceph-mon[107278]: pgmap v645: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 510 op/s; 32040/202020 objects degraded (15.860%); 14629/202020 objects misplaced (7.241%); 9.3 MiB/s, 149 objects/s recovering 2024-09-13T14:45:05.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:05 smithi003 ceph-mon[104007]: pgmap v645: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 510 op/s; 32040/202020 objects degraded (15.860%); 14629/202020 objects misplaced (7.241%); 9.3 MiB/s, 149 objects/s recovering 2024-09-13T14:45:05.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:05.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:05.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:05.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:05.821 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:05.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:05.953 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:05.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:06.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:06.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:06.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 32040/202020 objects degraded (15.860%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:06.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 32040/202020 objects degraded (15.860%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:06.202 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:06.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:06.262 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:06.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:06.391 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:06.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 32040/202020 objects degraded (15.860%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:06.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:06.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:06.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.650 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:06.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:06.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:06.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:06.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:07.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:07.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:07 smithi003 ceph-mon[104007]: pgmap v646: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 678 op/s; 30992/194751 objects degraded (15.914%); 14022/194751 objects misplaced (7.200%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:45:07.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:07 smithi003 ceph-mon[107278]: pgmap v646: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 678 op/s; 30992/194751 objects degraded (15.914%); 14022/194751 objects misplaced (7.200%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:45:07.188 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:07.217 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:07.324 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:07 smithi019 ceph-mon[91296]: pgmap v646: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 678 op/s; 30992/194751 objects degraded (15.914%); 14022/194751 objects misplaced (7.200%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:45:07.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:07.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:07.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.499 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:07.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:07.600 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:07.630 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:07.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.765 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:07.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout:moved 24 / 297 (8.08081%) 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout:avg 37.125 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.54671 -> 10.8563 (expected baseline 5.69951) 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.5 with 27 -> 22 pgs (0.727273 -> 0.592593 * mean) 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 47 -> 52 pgs (1.26599 -> 1.40067 * mean) 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 1 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T14:45:07.839 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0275 2024-09-13T14:45:07.840 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0302 2024-09-13T14:45:07.840 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 weight 0.6558 -> 0.4186 2024-09-13T14:45:07.840 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 weight 0.7064 -> 0.4729 2024-09-13T14:45:07.840 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 weight 0.7437 -> 0.5792 2024-09-13T14:45:07.840 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T14:45:07.866 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:07.895 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:07.997 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.025 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:08.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:08.126 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:08.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:08.170 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:45:07 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T14:45:07.837+0000 7f0f7ea9d640 -1 mgr.server reply reply (3) No such process no change 2024-09-13T14:45:08.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:08.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:08.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:08.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:08.464 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:08.566 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:08.602 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:08.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:08.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:08.779 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:45:08.779 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T14:45:08.779 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:85383 8.8 8:15106bc3:::benchmark_data_smithi003_139172_object85382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:37:07.756010+0000", 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 481.01669918800002, 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2226527759999999, 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 85383 2024-09-13T14:45:08.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756010+0000", 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756010+0000", 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756009+0000", 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756096+0000", 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7180000000000002e-05 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756109+0000", 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3271000000000001e-05 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.782 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756114+0000", 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2969999999999997e-06 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756149+0000", 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5515000000000001e-05 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756155+0000", 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.648e-06 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:08.743978+0000", 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98782341399999996 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:08.743993+0000", 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5058000000000001e-05 2024-09-13T14:45:08.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.954092+0000", 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.210098619 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.954256+0000", 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016445300000000001 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.785 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.954442+0000", 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000185778 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.970311+0000", 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015868554999999999 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.972232+0000", 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0019213940000000001 2024-09-13T14:45:08.786 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.976782+0000", 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0045501370000000001 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.976843+0000", 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1109000000000001e-05 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.978663+0000", 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0018192919999999999 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.787 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:85384 8.8 8:1b324b81:::benchmark_data_smithi003_139172_object85383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:37:07.756201+0000", 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 481.01650840399998, 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.22686502, 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:08.788 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 85384 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756201+0000", 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756201+0000", 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.789 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756199+0000", 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756275+0000", 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5346000000000007e-05 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756286+0000", 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1446e-05 2024-09-13T14:45:08.790 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756291+0000", 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3390000000000003e-06 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756339+0000", 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8519000000000001e-05 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.756349+0000", 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6700000000000006e-06 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:08.744033+0000", 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98768387999999996 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:08.744041+0000", 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3920000000000003e-06 2024-09-13T14:45:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.957455+0000", 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2134134759999999 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.957569+0000", 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000114716 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.793 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.957752+0000", 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00018263399999999999 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.970319+0000", 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012566631 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.972264+0000", 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001944994 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.794 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.981370+0000", 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0091062950000000004 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.981454+0000", 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4302999999999996e-05 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.983066+0000", 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001611456 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.795 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:85387 8.8 8:102f4d8e:::benchmark_data_smithi003_139172_object85386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:37:07.761659+0000", 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 481.011050033, 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.223439258, 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 85387 2024-09-13T14:45:08.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.761659+0000", 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.761659+0000", 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.797 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.761658+0000", 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.761703+0000", 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4780999999999997e-05 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.761714+0000", 2024-09-13T14:45:08.798 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1501000000000001e-05 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.761719+0000", 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7469999999999996e-06 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.761757+0000", 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7750000000000003e-05 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.799 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.761764+0000", 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9999999999999999e-06 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:08.744085+0000", 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98232111899999996 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.800 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:08.744092+0000", 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6279999999999996e-06 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.960400+0000", 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2163080550000001 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.960568+0000", 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016779300000000001 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.801 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.960721+0000", 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000152449 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.970325+0000", 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.009604329 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.974626+0000", 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0043014300000000002 2024-09-13T14:45:08.802 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.983285+0000", 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0086586630000000005 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.983436+0000", 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015058199999999999 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.985098+0000", 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0016625920000000001 2024-09-13T14:45:08.803 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:85402 8.8 8:16990cc2:::benchmark_data_smithi003_139172_object85401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:37:07.792273+0000", 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 480.98043627200002, 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1945746150000001, 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 85402 2024-09-13T14:45:08.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.792273+0000", 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.792273+0000", 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.805 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.792271+0000", 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.792323+0000", 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1363999999999998e-05 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.792334+0000", 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1328999999999999e-05 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.806 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.792339+0000", 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6110000000000004e-06 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.792390+0000", 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1795999999999997e-05 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:07.792400+0000", 2024-09-13T14:45:08.807 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5920000000000007e-06 2024-09-13T14:45:08.808 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.808 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.808 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.808 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:08.744139+0000", 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95173930799999995 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:08.744148+0000", 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6999999999999997e-06 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.963724+0000", 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2195756849999999 2024-09-13T14:45:08.809 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.963925+0000", 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000201728 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.964138+0000", 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00021302699999999999 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.810 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.972197+0000", 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0080584579999999992 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.981303+0000", 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0091065569999999995 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.985237+0000", 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039338899999999998 2024-09-13T14:45:08.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.985330+0000", 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2287000000000006e-05 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:37:09.986847+0000", 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0015176289999999999 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.812 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191276 8.8 8:107ce2ea:::benchmark_data_smithi003_139172_object66483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.498528+0000", 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 358.27418116400003, 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0525131189999999, 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191276 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.813 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498528+0000", 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498528+0000", 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498527+0000", 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:08.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498532+0000", 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8600000000000001e-06 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498543+0000", 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1379999999999999e-05 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.815 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498549+0000", 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6810000000000003e-06 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.508932+0000", 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010383593 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.509021+0000", 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8812000000000003e-05 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.816 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.509105+0000", 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3584999999999993e-05 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.522155+0000", 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.01304979 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.522384+0000", 2024-09-13T14:45:08.817 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000229829 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536472+0000", 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014087288999999999 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536481+0000", 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8270000000000004e-06 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.818 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.550996+0000", 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0145151139999999 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551041+0000", 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5269999999999999e-05 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191280 8.8 8:1cc7168d:::benchmark_data_smithi003_139172_object66487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.499757+0000", 2024-09-13T14:45:08.819 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 358.27295158700002, 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.051367782, 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191280 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.820 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499757+0000", 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499757+0000", 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499757+0000", 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499761+0000", 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5410000000000002e-06 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499772+0000", 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0467e-05 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499777+0000", 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5010000000000004e-06 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.515233+0000", 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015456228000000001 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.515683+0000", 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00044969399999999999 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.516086+0000", 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00040249099999999999 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.529283+0000", 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013197728000000001 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.529647+0000", 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00036357499999999999 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536551+0000", 2024-09-13T14:45:08.824 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0069043840000000004 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536559+0000", 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551079+0000", 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0145203439999999 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551125+0000", 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5878999999999999e-05 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191287 8.8 8:12bf2cc4:::benchmark_data_smithi003_139172_object66494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.511850+0000", 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 358.260858515, 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0393236480000001, 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:45:08.826 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191287 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511850+0000", 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.827 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511850+0000", 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511849+0000", 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511854+0000", 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6979999999999997e-06 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511869+0000", 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4596000000000001e-05 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511875+0000", 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7830000000000004e-06 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.523525+0000", 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011650600000000001 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.829 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.523687+0000", 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016207799999999999 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.523832+0000", 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014468900000000001 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.529668+0000", 2024-09-13T14:45:08.830 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0058362120000000003 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536588+0000", 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0069196329999999997 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536594+0000", 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0750000000000003e-06 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551140+0000", 2024-09-13T14:45:08.831 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0145458519999999 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551174+0000", 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4384999999999997e-05 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191307 8.8 8:187dddd2:::benchmark_data_smithi003_139172_object66514:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T14:45:08.832 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.539855+0000", 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 358.23285416599998, 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0656769129999999, 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191307 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539855+0000", 2024-09-13T14:45:08.833 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539855+0000", 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539854+0000", 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.834 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539858+0000", 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4809999999999998e-06 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539893+0000", 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5342999999999997e-05 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539902+0000", 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2390000000000008e-06 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.835 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539985+0000", 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2859000000000001e-05 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539994+0000", 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8310000000000007e-06 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551214+0000", 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0112197590000001 2024-09-13T14:45:08.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551314+0000", 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000100556 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551441+0000", 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012729399999999999 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.575082+0000", 2024-09-13T14:45:08.837 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.023640372 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605405+0000", 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.030322898000000001 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605441+0000", 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6359999999999997e-05 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605490+0000", 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8578999999999997e-05 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605532+0000", 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2021999999999998e-05 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.839 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191316 8.8 8:1339840c:::benchmark_data_smithi003_139172_object66523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.549840+0000", 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 358.222868909, 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.055850465, 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191316 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.840 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549840+0000", 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549840+0000", 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549839+0000", 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549844+0000", 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8980000000000004e-06 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549854+0000", 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.046e-05 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549860+0000", 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6289999999999998e-06 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549922+0000", 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1762999999999998e-05 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549931+0000", 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.4930000000000004e-06 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.556819+0000", 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.006887436 2024-09-13T14:45:08.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.556938+0000", 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000119732 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.557056+0000", 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011714699999999999 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.575089+0000", 2024-09-13T14:45:08.844 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.018033850000000001 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605570+0000", 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.030480854000000002 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605632+0000", 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2069999999999994e-05 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.845 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605663+0000", 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0624999999999999e-05 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605691+0000", 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7497000000000001e-05 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:41012 9.6 9:63d12f41:::benchmark_data_smithi003_156985_object41011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:42:10.197430+0000", 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 178.57527865899999, 2024-09-13T14:45:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.23089896600000001, 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 41012 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.197430+0000", 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.197430+0000", 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.197420+0000", 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999895 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.197458+0000", 2024-09-13T14:45:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7929000000000002e-05 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.197470+0000", 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2316e-05 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.197477+0000", 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5649999999999998e-06 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.197631+0000", 2024-09-13T14:45:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000154346 2024-09-13T14:45:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.197767+0000", 2024-09-13T14:45:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000136124 2024-09-13T14:45:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:45:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.198002+0000", 2024-09-13T14:45:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00023483 2024-09-13T14:45:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.201023+0000", 2024-09-13T14:45:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0030207189999999998 2024-09-13T14:45:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.210578+0000", 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0095554360000000005 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.428260+0000", 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21768216000000001 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.428300+0000", 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0119e-05 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.428329+0000", 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8843000000000002e-05 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:41016 9.6 9:63031843:::benchmark_data_smithi003_156985_object41015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:42:10.198865+0000", 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 178.573843813, 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.23125474300000001, 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 41016 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.198865+0000", 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.198865+0000", 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.198864+0000", 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.198920+0000", 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5699999999999999e-05 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.198934+0000", 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4273999999999999e-05 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.198940+0000", 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5990000000000001e-06 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.199384+0000", 2024-09-13T14:45:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00044434100000000001 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.199499+0000", 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000114929 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.199636+0000", 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000137381 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.201148+0000", 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0015119700000000001 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.210619+0000", 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0094702970000000008 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430028+0000", 2024-09-13T14:45:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21940950200000001 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430079+0000", 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0730999999999999e-05 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430120+0000", 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1026999999999999e-05 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:41025 9.6 9:668eb8d4:::benchmark_data_smithi003_156985_object41024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:42:10.203510+0000", 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 178.56919884800001, 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22676436699999999, 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 41025 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.203510+0000", 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.203510+0000", 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.203509+0000", 2024-09-13T14:45:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.203564+0000", 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4839000000000001e-05 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.203578+0000", 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4235e-05 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.203586+0000", 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4739999999999999e-06 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.203636+0000", 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0161999999999998e-05 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.203765+0000", 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012898400000000001 2024-09-13T14:45:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.203922+0000", 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015731299999999999 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.205286+0000", 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0013638159999999999 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.210636+0000", 2024-09-13T14:45:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0053501390000000003 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430179+0000", 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.219543036 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430232+0000", 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3196999999999999e-05 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430274+0000", 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2239e-05 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:41030 9.6 9:6b4bb61f:::benchmark_data_smithi003_156985_object41029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:42:10.206688+0000", 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 178.56602136000001, 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.223727808, 2024-09-13T14:45:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 41030 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.206688+0000", 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.206688+0000", 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.206686+0000", 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.206736+0000", 2024-09-13T14:45:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9981999999999997e-05 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.206750+0000", 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3210000000000001e-05 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.206755+0000", 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6189999999999999e-06 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.206812+0000", 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6752999999999997e-05 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.206934+0000", 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000122318 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.207074+0000", 2024-09-13T14:45:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000139569 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.208237+0000", 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001162848 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.210790+0000", 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0025537260000000001 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430329+0000", 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21953835399999999 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430373+0000", 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4685000000000003e-05 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430415+0000", 2024-09-13T14:45:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1990000000000003e-05 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:41058 9.6 9:6cddca51:::benchmark_data_smithi003_156985_object41057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:42:10.225058+0000", 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 178.547651215, 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.205555761, 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 41058 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.225058+0000", 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.225058+0000", 2024-09-13T14:45:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.225056+0000", 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.225101+0000", 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4916999999999997e-05 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.225113+0000", 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2004999999999999e-05 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.225118+0000", 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4719999999999997e-06 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.225170+0000", 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1962999999999997e-05 2024-09-13T14:45:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.225305+0000", 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013524500000000001 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.225464+0000", 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015857499999999999 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.226794+0000", 2024-09-13T14:45:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0013304 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.227094+0000", 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00029980099999999998 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430523+0000", 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.20342914300000001 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430562+0000", 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9295000000000002e-05 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:42:10.430614+0000", 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1258999999999998e-05 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:120794 9.7 9:e7706e89:::benchmark_data_smithi003_156985_object120793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:18.841809+0000", 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 109.93090016399999, 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4987142480000002, 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 120794 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841809+0000", 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841809+0000", 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841807+0000", 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841843+0000", 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.553e-05 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841859+0000", 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6364000000000001e-05 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841868+0000", 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9709999999999993e-06 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842332+0000", 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00046374099999999999 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842456+0000", 2024-09-13T14:45:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000124659 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842767+0000", 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00031060300000000001 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.847411+0000", 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0046438859999999998 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.850363+0000", 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029518190000000001 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.872547+0000", 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.022184261 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.872565+0000", 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7665000000000001e-05 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.854869+0000", 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98230381 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.854879+0000", 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0081000000000001e-05 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.329521+0000", 2024-09-13T14:45:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.474642472 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for degraded object", 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.330222+0000", 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00070061600000000004 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.340464+0000", 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010241894 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.340523+0000", 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9441999999999998e-05 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:120798 9.17 9:e86e3aa7:::benchmark_data_smithi003_156985_object120797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:18.842605+0000", 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 109.93010404499999, 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5056814049999998, 2024-09-13T14:45:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 120798 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842605+0000", 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842605+0000", 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842604+0000", 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842633+0000", 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8603e-05 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842640+0000", 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4390000000000003e-06 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842645+0000", 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2409999999999997e-06 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.844796+0000", 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00215161 2024-09-13T14:45:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.844915+0000", 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118263 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.845043+0000", 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012889600000000001 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.850380+0000", 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0053369899999999998 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.869291+0000", 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.018910756000000001 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.869300+0000", 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7430000000000008e-06 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.853609+0000", 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98430951 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.853620+0000", 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0145e-05 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.334335+0000", 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4807156269999999 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for degraded object", 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.335040+0000", 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00070517900000000001 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.348222+0000", 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013181877999999999 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.348286+0000", 2024-09-13T14:45:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4036999999999997e-05 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:120805 9.17 9:ed42f0a7:::benchmark_data_smithi003_156985_object120804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:18.855612+0000", 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 109.917096847, 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4960159339999999, 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 120805 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855612+0000", 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855612+0000", 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855611+0000", 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855642+0000", 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0188999999999998e-05 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855650+0000", 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5199999999999997e-06 2024-09-13T14:45:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855655+0000", 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3800000000000004e-06 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.856508+0000", 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00085295800000000001 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.856627+0000", 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011941499999999999 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.856764+0000", 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013728899999999999 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.869327+0000", 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012562298 2024-09-13T14:45:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.869332+0000", 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.339e-06 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.853647+0000", 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98431561999999995 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.853655+0000", 2024-09-13T14:45:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1319999999999996e-06 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.335106+0000", 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4814514729999999 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for degraded object", 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.335718+0000", 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00061208699999999998 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.351572+0000", 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015854035999999998 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.351628+0000", 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5828999999999997e-05 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:180117 9.17 9:e808b7cf:::benchmark_data_smithi003_156985_object59201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:08.602312+0000", 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.170397137, 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.11178550299999999, 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 180117 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.602312+0000", 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.602312+0000", 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.602311+0000", 2024-09-13T14:45:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.602314+0000", 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.909e-06 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.602321+0000", 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4370000000000004e-06 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.602325+0000", 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0060000000000003e-06 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.628371+0000", 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.026046059 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for degraded object", 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.629850+0000", 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0014795800000000001 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.652337+0000", 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.022486487999999999 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.652438+0000", 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000100778 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:45:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.652586+0000", 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014818000000000001 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.664298+0000", 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011711869 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.712364+0000", 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.048066272 2024-09-13T14:45:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.712396+0000", 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2450000000000003e-05 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.712456+0000", 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9633000000000003e-05 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.714097+0000", 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0016412899999999999 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:180124 9.17 9:e9e8c6fa:::benchmark_data_smithi003_156985_object59208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:08.608683+0000", 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.16402613799999999, 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.106988688, 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 180124 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.608683+0000", 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.608683+0000", 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.608681+0000", 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.608688+0000", 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.562e-06 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.608700+0000", 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2333e-05 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.608706+0000", 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2890000000000003e-06 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.631883+0000", 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.023176760000000001 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for degraded object", 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.633049+0000", 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001165595 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.660810+0000", 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.027761378 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.660906+0000", 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5600000000000006e-05 2024-09-13T14:45:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.661026+0000", 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120124 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.667852+0000", 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0068259260000000004 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.714163+0000", 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.046311591999999999 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.714261+0000", 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8094e-05 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.714335+0000", 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3885000000000002e-05 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.715672+0000", 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001336161 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:180129 9.17 9:eba6e8eb:::benchmark_data_smithi003_156985_object59213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:08.616382+0000", 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.15632736799999999, 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.104641525, 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 180129 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.616382+0000", 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.616382+0000", 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.616380+0000", 2024-09-13T14:45:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.616387+0000", 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2359999999999998e-06 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.616400+0000", 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2490999999999999e-05 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.616405+0000", 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6400000000000002e-06 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.633890+0000", 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.017484673999999999 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for degraded object", 2024-09-13T14:45:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.637916+0000", 2024-09-13T14:45:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.004026623 2024-09-13T14:45:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.667898+0000", 2024-09-13T14:45:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.029981240999999999 2024-09-13T14:45:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:08.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.667996+0000", 2024-09-13T14:45:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8480999999999994e-05 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.709268+0000", 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.041271782 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.715714+0000", 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.006446256 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.718659+0000", 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029452279999999998 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.719632+0000", 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00097285500000000001 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.719693+0000", 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0449999999999999e-05 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:08.721023+0000", 2024-09-13T14:45:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0013303659999999999 2024-09-13T14:45:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:08.928 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:45:08.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:09.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:09.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:09.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:09.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:09.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:09.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:09.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:09 smithi019 ceph-mon[91296]: pgmap v647: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 31095/191901 objects degraded (16.204%); 13811/191901 objects misplaced (7.197%); 13 MiB/s, 215 objects/s recovering 2024-09-13T14:45:09.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:09 smithi019 ceph-mon[91296]: from='client.36006 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:45:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:09 smithi003 ceph-mon[107278]: pgmap v647: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 31095/191901 objects degraded (16.204%); 13811/191901 objects misplaced (7.197%); 13 MiB/s, 215 objects/s recovering 2024-09-13T14:45:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:09 smithi003 ceph-mon[107278]: from='client.36006 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:45:09.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:09 smithi003 ceph-mon[104007]: pgmap v647: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 31095/191901 objects degraded (16.204%); 13811/191901 objects misplaced (7.197%); 13 MiB/s, 215 objects/s recovering 2024-09-13T14:45:09.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:09 smithi003 ceph-mon[104007]: from='client.36006 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:45:09.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:09.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:09.627 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.660 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:09.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:09.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:09.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:09.911 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:09.945 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:10.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:10.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:10.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:45:10.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:10.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:10.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:10.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:10.402 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:10.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:45:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:45:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:10.504 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:10.534 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:10.537 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:10.638 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.674 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:10.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:10.775 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:10.811 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:10.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:10.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:11.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:11.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:11.126 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:11 smithi003 ceph-mon[107278]: pgmap v648: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 642 op/s; 30871/187470 objects degraded (16.467%); 13189/187470 objects misplaced (7.035%); 14 MiB/s, 217 objects/s recovering 2024-09-13T14:45:11.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 30871/187470 objects degraded (16.467%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:11.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:11 smithi003 ceph-mon[104007]: pgmap v648: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 642 op/s; 30871/187470 objects degraded (16.467%); 13189/187470 objects misplaced (7.035%); 14 MiB/s, 217 objects/s recovering 2024-09-13T14:45:11.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 30871/187470 objects degraded (16.467%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:11 smithi019 ceph-mon[91296]: pgmap v648: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 642 op/s; 30871/187470 objects degraded (16.467%); 13189/187470 objects misplaced (7.035%); 14 MiB/s, 217 objects/s recovering 2024-09-13T14:45:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 30871/187470 objects degraded (16.467%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:11.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:11.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:11.258 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:11.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:11.378 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:11.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:11.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:11.518 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:11.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:11.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:11.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:11.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:11.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:11.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:11.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:12.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:12.064 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:45:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:12.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:12.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:12.235 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:12.278 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:12 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3581044570' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:12.278 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:12 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:12.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:12.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:12.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:12 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3581044570' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:12.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:12 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:12 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3581044570' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:12 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:12.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:12.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:12.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:12.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:12.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:12.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:13.036 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:13.075 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:13.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[104007]: pgmap v649: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 30035/185112 objects degraded (16.225%); 13189/185112 objects misplaced (7.125%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:45:13.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:13.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:45:13.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[104007]: osdmap e193: 8 total, 8 up, 8 in 2024-09-13T14:45:13.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:13.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:13.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[107278]: pgmap v649: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 30035/185112 objects degraded (16.225%); 13189/185112 objects misplaced (7.125%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:45:13.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:13.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:45:13.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[107278]: osdmap e193: 8 total, 8 up, 8 in 2024-09-13T14:45:13.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:13.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:13.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:13.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:13.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:13.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:13.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:13 smithi019 ceph-mon[91296]: pgmap v649: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 30035/185112 objects degraded (16.225%); 13189/185112 objects misplaced (7.125%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:45:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:13 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:13 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:45:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:13 smithi019 ceph-mon[91296]: osdmap e193: 8 total, 8 up, 8 in 2024-09-13T14:45:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:13.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:13.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:13.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:13.610 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:13.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:13.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:13.771 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:45:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:13.806 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:45:13.806 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:13.832 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:45:13.833 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:13.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:13.864 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:45:13.864 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:13.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:13.901 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:45:13.901 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:13.942 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:45:13.942 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:13.981 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:45:13.981 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:13.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:14.011 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:45:14.011 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:14.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:14.128 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:14.160 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3909716175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:14.241 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:45:14.241 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:45:14.241 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:183447 9.0 9:064c47ec:::benchmark_data_smithi003_156985_object62531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:13.813368+0000", 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.030969648999999998, 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.031001478999999998, 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183447 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:14.242 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.813368+0000", 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.813368+0000", 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.813367+0000", 2024-09-13T14:45:14.243 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.813372+0000", 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5829999999999997e-06 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.813383+0000", 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0998999999999999e-05 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.244 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.813389+0000", 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8669999999999999e-06 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.823052+0000", 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0096624229999999998 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.823231+0000", 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000179638 2024-09-13T14:45:14.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.823392+0000", 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000160567 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.828696+0000", 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.005304591 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:14.246 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.828704+0000", 2024-09-13T14:45:14.247 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9580000000000007e-06 2024-09-13T14:45:14.247 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:14.247 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:14.247 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:14.247 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:14.247 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-09-13T14:45:14.247 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 1 2024-09-13T14:45:14.247 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:45:14.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:14.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:14.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:45:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:45:14] "GET /metrics HTTP/1.1" 200 36561 "" "Prometheus/2.51.0" 2024-09-13T14:45:14.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3909716175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:14.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3909716175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:14.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:14.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:14.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:14.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:14.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:14.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:14.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.879 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:14.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:14.980 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:15.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:15.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:15.056 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:45:15.056 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:45:15.057 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:45:15.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:45:15.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:15.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:15.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:15.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:15.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:15.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:15 smithi019 ceph-mon[91296]: pgmap v651: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 30115/182556 objects degraded (16.496%); 13010/182556 objects misplaced (7.127%); 13 MiB/s, 205 objects/s recovering 2024-09-13T14:45:15.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:15 smithi003 ceph-mon[104007]: pgmap v651: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 30115/182556 objects degraded (16.496%); 13010/182556 objects misplaced (7.127%); 13 MiB/s, 205 objects/s recovering 2024-09-13T14:45:15.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:15 smithi003 ceph-mon[107278]: pgmap v651: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 30115/182556 objects degraded (16.496%); 13010/182556 objects misplaced (7.127%); 13 MiB/s, 205 objects/s recovering 2024-09-13T14:45:15.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:15.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:45:15.532 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:15.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:15.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:15.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:15.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:15.875 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:15.901 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:16.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:16.043 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:16.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:16.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:16.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:16.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 30115/182556 objects degraded (16.496%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:16.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 30115/182556 objects degraded (16.496%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:16.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:16.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:16.411 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:16.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 30115/182556 objects degraded (16.496%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:16.447 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:16.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:16.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:16.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:16.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:16.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:16.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:45:16.838 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:16.873 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:16.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:17.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:17.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:17.123 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:17.160 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:17.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:17 smithi003 ceph-mon[104007]: pgmap v652: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 635 op/s; 29131/175662 objects degraded (16.584%); 12449/175662 objects misplaced (7.087%); 13 MiB/s, 200 objects/s recovering 2024-09-13T14:45:17.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:17 smithi003 ceph-mon[107278]: pgmap v652: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 635 op/s; 29131/175662 objects degraded (16.584%); 12449/175662 objects misplaced (7.087%); 13 MiB/s, 200 objects/s recovering 2024-09-13T14:45:17.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:17.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:17.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:17.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:17 smithi019 ceph-mon[91296]: pgmap v652: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 635 op/s; 29131/175662 objects degraded (16.584%); 12449/175662 objects misplaced (7.087%); 13 MiB/s, 200 objects/s recovering 2024-09-13T14:45:17.434 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:17.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:17.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:17.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:17.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:17.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:17.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:17.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:18.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:18.024 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:18.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:18.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:18.224 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:45:18.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:18.304 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:18.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:18.457 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:18.558 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:18.590 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:18.653 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:18.692 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:18.737 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:18.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:18.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:18.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:19.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:19.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:19.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:19.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:19.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:19.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:19 smithi003 ceph-mon[104007]: pgmap v653: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 636 op/s; 29237/172806 objects degraded (16.919%); 12237/172806 objects misplaced (7.081%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:45:19.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:19 smithi003 ceph-mon[107278]: pgmap v653: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 636 op/s; 29237/172806 objects degraded (16.919%); 12237/172806 objects misplaced (7.081%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:45:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:19 smithi019 ceph-mon[91296]: pgmap v653: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 636 op/s; 29237/172806 objects degraded (16.919%); 12237/172806 objects misplaced (7.081%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:45:19.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:19.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:19.617 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:19.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:19.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:19.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:19.828 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:19.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:19.884 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:45:19.886 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:45:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:19.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:20.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:20.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:20.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:20.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:20.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:20.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:20.449 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:20.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:20.490 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:45:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:45:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:20.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:20.608 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:20.706 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:20.706 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:20.710 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:20.749 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:20.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:20.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:20.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:21.004 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:21.049 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:21.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:21.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:21 smithi003 ceph-mon[104007]: pgmap v654: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 645 op/s; 29055/168090 objects degraded (17.285%); 11569/168090 objects misplaced (6.883%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:45:21.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:21 smithi003 ceph-mon[104007]: from='client.36018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:21.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 29055/168090 objects degraded (17.285%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:21 smithi003 ceph-mon[107278]: pgmap v654: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 645 op/s; 29055/168090 objects degraded (17.285%); 11569/168090 objects misplaced (6.883%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:45:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:21 smithi003 ceph-mon[107278]: from='client.36018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 29055/168090 objects degraded (17.285%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.189 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:21.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:21.289 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:21.320 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:21.324 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2024-09-13T14:45:19.671155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3670808549,"num_objects":56030,"num_object_clones":0,"num_object_copies":168090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":11569,"num_objects_unfound":0,"num_objects_dirty":56030,"num_whiteouts":0,"num_read":293,"num_read_kb":748,"num_write":158327,"num_write_kb":5967677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32164,"num_bytes_recovered":2108294390,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186987,"ondisk_log_size":186987,"up":297,"acting":287,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":334,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19271196,"kb_used_data":14094684,"kb_used_omap":29,"kb_used_meta":5176098,"kb_avail":730690020,"statfs":{"total":767960285184,"available":748226580480,"internally_reserved":0,"allocated":14432956416,"data_stored":14425200008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":5300324878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-423362560,"num_objects":-6460,"num_object_clones":0,"num_object_copies":-19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1816,"num_objects_misplaced":-1620,"num_objects_unfound":0,"num_objects_dirty":-6460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":131137536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005519"},"pg_stats":[{"pgid":"9.19","version":"193'7309","reported_seq":7440,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.680109+0000","last_change":"2024-09-13T14:42:18.899519+0000","last_active":"2024-09-13T14:45:14.680109+0000","last_peered":"2024-09-13T14:45:14.680109+0000","last_clean":"2024-09-13T14:45:14.680109+0000","last_became_active":"2024-09-13T14:42:15.129169+0000","last_became_peered":"2024-09-13T14:42:15.129169+0000","last_unstale":"2024-09-13T14:45:14.680109+0000","last_undegraded":"2024-09-13T14:45:14.680109+0000","last_fullsized":"2024-09-13T14:45:14.680109+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3130","last_scrub_stamp":"2024-09-13T14:42:18.899277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:18.899277+0000","objects_scrubbed":1667,"log_size":7309,"log_dups_size":0,"ondisk_log_size":7309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:03.200084+0000","scrub_duration":425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123863040,"num_objects":1890,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":141376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":195,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.091866+0000","last_change":"2024-09-13T14:43:19.875328+0000","last_active":"2024-09-13T14:45:12.091866+0000","last_peered":"2024-09-13T14:45:12.091866+0000","last_clean":"2024-09-13T14:45:12.091866+0000","last_became_active":"2024-09-13T14:43:19.874507+0000","last_became_peered":"2024-09-13T14:43:19.874507+0000","last_unstale":"2024-09-13T14:45:12.091866+0000","last_undegraded":"2024-09-13T14:45:12.091866+0000","last_fullsized":"2024-09-13T14:45:12.091866+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:07.089107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":159,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.996978+0000","last_change":"2024-09-13T14:35:30.492165+0000","last_active":"2024-09-13T14:44:58.996978+0000","last_peered":"2024-09-13T14:44:58.996978+0000","last_clean":"2024-09-13T14:44:58.996978+0000","last_became_active":"2024-09-13T14:35:30.491844+0000","last_became_peered":"2024-09-13T14:35:30.491844+0000","last_unstale":"2024-09-13T14:44:58.996978+0000","last_undegraded":"2024-09-13T14:44:58.996978+0000","last_fullsized":"2024-09-13T14:44:58.996978+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:22.876890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.18","version":"193'7277","reported_seq":13382,"reported_epoch":193,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:14.709071+0000","last_change":"2024-09-13T14:43:21.506884+0000","last_active":"2024-09-13T14:45:14.709071+0000","last_peered":"2024-09-13T14:45:14.709071+0000","last_clean":"2024-09-13T14:43:18.728712+0000","last_became_active":"2024-09-13T14:43:21.499842+0000","last_became_peered":"2024-09-13T14:43:21.499842+0000","last_unstale":"2024-09-13T14:45:14.709071+0000","last_undegraded":"2024-09-13T14:43:21.342550+0000","last_fullsized":"2024-09-13T14:43:21.112483+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'3290","last_scrub_stamp":"2024-09-13T14:42:20.569479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:20.569479+0000","objects_scrubbed":1768,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":240,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116457472,"num_objects":1777,"num_object_clones":0,"num_object_copies":5331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":127872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1989,"num_bytes_recovered":130351104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,1],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,1","objects":6},{"shards":"0,3,5","objects":1771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":117,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.001950+0000","last_change":"2024-09-13T14:43:19.874928+0000","last_active":"2024-09-13T14:44:59.001950+0000","last_peered":"2024-09-13T14:44:59.001950+0000","last_clean":"2024-09-13T14:44:59.001950+0000","last_became_active":"2024-09-13T14:43:19.874518+0000","last_became_peered":"2024-09-13T14:43:19.874518+0000","last_unstale":"2024-09-13T14:44:59.001950+0000","last_undegraded":"2024-09-13T14:44:59.001950+0000","last_fullsized":"2024-09-13T14:44:59.001950+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:15:22.392336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":264,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.073079+0000","last_change":"2024-09-13T14:43:19.870143+0000","last_active":"2024-09-13T14:45:12.073079+0000","last_peered":"2024-09-13T14:45:12.073079+0000","last_clean":"2024-09-13T14:45:12.073079+0000","last_became_active":"2024-09-13T14:43:19.869829+0000","last_became_peered":"2024-09-13T14:43:19.869829+0000","last_unstale":"2024-09-13T14:45:12.073079+0000","last_undegraded":"2024-09-13T14:45:12.073079+0000","last_fullsized":"2024-09-13T14:45:12.073079+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.422457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":265,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.074522+0000","last_change":"2024-09-13T14:43:19.871195+0000","last_active":"2024-09-13T14:45:12.074522+0000","last_peered":"2024-09-13T14:45:12.074522+0000","last_clean":"2024-09-13T14:45:12.074522+0000","last_became_active":"2024-09-13T14:43:19.870496+0000","last_became_peered":"2024-09-13T14:43:19.870496+0000","last_unstale":"2024-09-13T14:45:12.074522+0000","last_undegraded":"2024-09-13T14:45:12.074522+0000","last_fullsized":"2024-09-13T14:45:12.074522+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:25.646089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":140,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.996694+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:44:58.996694+0000","last_peered":"2024-09-13T14:44:58.996694+0000","last_clean":"2024-09-13T14:44:58.996694+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:44:58.996694+0000","last_undegraded":"2024-09-13T14:44:58.996694+0000","last_fullsized":"2024-09-13T14:44:58.996694+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":221,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.091800+0000","last_change":"2024-09-13T14:43:19.872044+0000","last_active":"2024-09-13T14:45:12.091800+0000","last_peered":"2024-09-13T14:45:12.091800+0000","last_clean":"2024-09-13T14:45:12.091800+0000","last_became_active":"2024-09-13T14:43:19.870646+0000","last_became_peered":"2024-09-13T14:43:19.870646+0000","last_unstale":"2024-09-13T14:45:12.091800+0000","last_undegraded":"2024-09-13T14:45:12.091800+0000","last_fullsized":"2024-09-13T14:45:12.091800+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:33.158472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":184,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.010904+0000","last_change":"2024-09-13T14:43:19.888180+0000","last_active":"2024-09-13T14:44:59.010904+0000","last_peered":"2024-09-13T14:44:59.010904+0000","last_clean":"2024-09-13T14:44:59.010904+0000","last_became_active":"2024-09-13T14:43:19.875873+0000","last_became_peered":"2024-09-13T14:43:19.875873+0000","last_unstale":"2024-09-13T14:44:59.010904+0000","last_undegraded":"2024-09-13T14:44:59.010904+0000","last_fullsized":"2024-09-13T14:44:59.010904+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:47.078713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"193'7195","reported_seq":13216,"reported_epoch":193,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:14.727970+0000","last_change":"2024-09-13T14:43:21.358111+0000","last_active":"2024-09-13T14:45:14.727970+0000","last_peered":"2024-09-13T14:45:14.727970+0000","last_clean":"2024-09-13T14:43:18.840495+0000","last_became_active":"2024-09-13T14:43:21.306427+0000","last_became_peered":"2024-09-13T14:43:21.306427+0000","last_unstale":"2024-09-13T14:45:14.727970+0000","last_undegraded":"2024-09-13T14:43:21.093819+0000","last_fullsized":"2024-09-13T14:43:20.854552+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3036","last_scrub_stamp":"2024-09-13T14:42:16.024050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.024050+0000","objects_scrubbed":1508,"log_size":7195,"log_dups_size":0,"ondisk_log_size":7195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115474432,"num_objects":1762,"num_object_clones":0,"num_object_copies":5286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4176,"num_write_kb":141760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1962,"num_bytes_recovered":128581632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,1","objects":3},{"shards":"0,5,7","objects":1759}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":42,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062966+0000","last_change":"2024-09-13T14:43:19.883523+0000","last_active":"2024-09-13T14:44:59.062966+0000","last_peered":"2024-09-13T14:44:59.062966+0000","last_clean":"2024-09-13T14:44:59.062966+0000","last_became_active":"2024-09-13T14:43:19.871241+0000","last_became_peered":"2024-09-13T14:43:19.871241+0000","last_unstale":"2024-09-13T14:44:59.062966+0000","last_undegraded":"2024-09-13T14:44:59.062966+0000","last_fullsized":"2024-09-13T14:44:59.062966+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:04.592529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.010775+0000","last_change":"2024-09-13T14:43:19.874136+0000","last_active":"2024-09-13T14:44:59.010775+0000","last_peered":"2024-09-13T14:44:59.010775+0000","last_clean":"2024-09-13T14:44:59.010775+0000","last_became_active":"2024-09-13T14:43:19.873671+0000","last_became_peered":"2024-09-13T14:43:19.873671+0000","last_unstale":"2024-09-13T14:44:59.010775+0000","last_undegraded":"2024-09-13T14:44:59.010775+0000","last_fullsized":"2024-09-13T14:44:59.010775+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:51.386822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"193'7354","reported_seq":7485,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:16.700313+0000","last_change":"2024-09-13T14:42:15.821133+0000","last_active":"2024-09-13T14:45:16.700313+0000","last_peered":"2024-09-13T14:45:16.700313+0000","last_clean":"2024-09-13T14:45:16.700313+0000","last_became_active":"2024-09-13T14:42:15.078221+0000","last_became_peered":"2024-09-13T14:42:15.078221+0000","last_unstale":"2024-09-13T14:45:16.700313+0000","last_undegraded":"2024-09-13T14:45:16.700313+0000","last_fullsized":"2024-09-13T14:45:16.700313+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'2973","last_scrub_stamp":"2024-09-13T14:42:15.820869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:15.820869+0000","objects_scrubbed":1533,"log_size":7354,"log_dups_size":0,"ondisk_log_size":7354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:47.303703+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120455168,"num_objects":1838,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":150208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":159,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.997046+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:44:58.997046+0000","last_peered":"2024-09-13T14:44:58.997046+0000","last_clean":"2024-09-13T14:44:58.997046+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:44:58.997046+0000","last_undegraded":"2024-09-13T14:44:58.997046+0000","last_fullsized":"2024-09-13T14:44:58.997046+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":172,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062387+0000","last_change":"2024-09-13T14:37:08.752050+0000","last_active":"2024-09-13T14:44:59.062387+0000","last_peered":"2024-09-13T14:44:59.062387+0000","last_clean":"2024-09-13T14:44:59.062387+0000","last_became_active":"2024-09-13T14:37:08.751650+0000","last_became_peered":"2024-09-13T14:37:08.751650+0000","last_unstale":"2024-09-13T14:44:59.062387+0000","last_undegraded":"2024-09-13T14:44:59.062387+0000","last_fullsized":"2024-09-13T14:44:59.062387+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:17.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"193'7398","reported_seq":18779,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:18.989685+0000","last_change":"2024-09-13T14:44:59.050198+0000","last_active":"2024-09-13T14:45:18.989685+0000","last_peered":"2024-09-13T14:45:18.989685+0000","last_clean":"2024-09-13T14:45:18.989685+0000","last_became_active":"2024-09-13T14:44:58.997243+0000","last_became_peered":"2024-09-13T14:44:58.997243+0000","last_unstale":"2024-09-13T14:45:18.989685+0000","last_undegraded":"2024-09-13T14:45:18.989685+0000","last_fullsized":"2024-09-13T14:45:18.989685+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3303","last_scrub_stamp":"2024-09-13T14:42:19.788610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:19.788610+0000","objects_scrubbed":1743,"log_size":7398,"log_dups_size":0,"ondisk_log_size":7398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:24.047041+0000","scrub_duration":0.62419811800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106496000,"num_objects":1625,"num_object_clones":0,"num_object_copies":4875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4116,"num_write_kb":127936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3741,"num_bytes_recovered":245170176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":155,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.002196+0000","last_change":"2024-09-13T14:43:19.880218+0000","last_active":"2024-09-13T14:44:59.002196+0000","last_peered":"2024-09-13T14:44:59.002196+0000","last_clean":"2024-09-13T14:44:59.002196+0000","last_became_active":"2024-09-13T14:43:19.874769+0000","last_became_peered":"2024-09-13T14:43:19.874769+0000","last_unstale":"2024-09-13T14:44:59.002196+0000","last_undegraded":"2024-09-13T14:44:59.002196+0000","last_fullsized":"2024-09-13T14:44:59.002196+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:49.775209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":158,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.010060+0000","last_change":"2024-09-13T14:43:19.860813+0000","last_active":"2024-09-13T14:44:59.010060+0000","last_peered":"2024-09-13T14:44:59.010060+0000","last_clean":"2024-09-13T14:44:59.010060+0000","last_became_active":"2024-09-13T14:43:19.860518+0000","last_became_peered":"2024-09-13T14:43:19.860518+0000","last_unstale":"2024-09-13T14:44:59.010060+0000","last_undegraded":"2024-09-13T14:44:59.010060+0000","last_fullsized":"2024-09-13T14:44:59.010060+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:21.310511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"193'7327","reported_seq":7988,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:18.526703+0000","last_change":"2024-09-13T14:43:28.958578+0000","last_active":"2024-09-13T14:45:18.526703+0000","last_peered":"2024-09-13T14:45:18.526703+0000","last_clean":"2024-09-13T14:45:18.526703+0000","last_became_active":"2024-09-13T14:43:28.955368+0000","last_became_peered":"2024-09-13T14:43:28.955368+0000","last_unstale":"2024-09-13T14:45:18.526703+0000","last_undegraded":"2024-09-13T14:45:18.526703+0000","last_fullsized":"2024-09-13T14:45:18.526703+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3084","last_scrub_stamp":"2024-09-13T14:42:16.825170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.825170+0000","objects_scrubbed":1592,"log_size":7327,"log_dups_size":0,"ondisk_log_size":7327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:56.306613+0000","scrub_duration":0.667424664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109510656,"num_objects":1671,"num_object_clones":0,"num_object_copies":5013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":138816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":126484480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":155,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.990985+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:44:58.990985+0000","last_peered":"2024-09-13T14:44:58.990985+0000","last_clean":"2024-09-13T14:44:58.990985+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:44:58.990985+0000","last_undegraded":"2024-09-13T14:44:58.990985+0000","last_fullsized":"2024-09-13T14:44:58.990985+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":259,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.085508+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:45:12.085508+0000","last_peered":"2024-09-13T14:45:12.085508+0000","last_clean":"2024-09-13T14:45:12.085508+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:45:12.085508+0000","last_undegraded":"2024-09-13T14:45:12.085508+0000","last_fullsized":"2024-09-13T14:45:12.085508+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"193'7151","reported_seq":14679,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:17.436294+0000","last_change":"2024-09-13T14:44:21.618688+0000","last_active":"2024-09-13T14:45:17.436294+0000","last_peered":"2024-09-13T14:45:17.436294+0000","last_clean":"2024-09-13T14:45:17.436294+0000","last_became_active":"2024-09-13T14:44:21.602316+0000","last_became_peered":"2024-09-13T14:44:21.602316+0000","last_unstale":"2024-09-13T14:45:17.436294+0000","last_undegraded":"2024-09-13T14:45:17.436294+0000","last_fullsized":"2024-09-13T14:45:17.436294+0000","mapping_epoch":186,"log_start":"156'262","ondisk_log_start":"156'262","created":152,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'2995","last_scrub_stamp":"2024-09-13T14:42:16.817217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.817217+0000","objects_scrubbed":1525,"log_size":6889,"log_dups_size":0,"ondisk_log_size":6889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:34.225750+0000","scrub_duration":583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111149056,"num_objects":1696,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4169,"num_write_kb":138880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3692,"num_bytes_recovered":241958912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":219,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.090777+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:45:12.090777+0000","last_peered":"2024-09-13T14:45:12.090777+0000","last_clean":"2024-09-13T14:45:12.090777+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:45:12.090777+0000","last_undegraded":"2024-09-13T14:45:12.090777+0000","last_fullsized":"2024-09-13T14:45:12.090777+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":155,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.063203+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:44:59.063203+0000","last_peered":"2024-09-13T14:44:59.063203+0000","last_clean":"2024-09-13T14:44:59.063203+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:44:59.063203+0000","last_undegraded":"2024-09-13T14:44:59.063203+0000","last_fullsized":"2024-09-13T14:44:59.063203+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"193'7473","reported_seq":9176,"reported_epoch":193,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:18.705141+0000","last_change":"2024-09-13T14:44:57.094029+0000","last_active":"2024-09-13T14:45:18.705141+0000","last_peered":"2024-09-13T14:45:18.705141+0000","last_clean":"2024-09-13T14:43:18.804350+0000","last_became_active":"2024-09-13T14:43:21.502161+0000","last_became_peered":"2024-09-13T14:43:21.502161+0000","last_unstale":"2024-09-13T14:45:18.705141+0000","last_undegraded":"2024-09-13T14:43:20.876998+0000","last_fullsized":"2024-09-13T14:43:20.853040+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3278","last_scrub_stamp":"2024-09-13T14:42:18.806999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:18.806999+0000","objects_scrubbed":1737,"log_size":7473,"log_dups_size":0,"ondisk_log_size":7473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:59.799396+0000","scrub_duration":487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113049600,"num_objects":1725,"num_object_clones":0,"num_object_copies":5175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1700,"num_objects_misplaced":1346,"num_objects_unfound":0,"num_objects_dirty":1725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4206,"num_write_kb":134208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":51511296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"1,2,6","objects":1346},{"shards":"2,6","objects":-1321}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":166,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.010221+0000","last_change":"2024-09-13T14:43:19.860723+0000","last_active":"2024-09-13T14:44:59.010221+0000","last_peered":"2024-09-13T14:44:59.010221+0000","last_clean":"2024-09-13T14:44:59.010221+0000","last_became_active":"2024-09-13T14:43:19.860401+0000","last_became_peered":"2024-09-13T14:43:19.860401+0000","last_unstale":"2024-09-13T14:44:59.010221+0000","last_undegraded":"2024-09-13T14:44:59.010221+0000","last_fullsized":"2024-09-13T14:44:59.010221+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:27.920262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":172,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.064838+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:44:59.064838+0000","last_peered":"2024-09-13T14:44:59.064838+0000","last_clean":"2024-09-13T14:44:59.064838+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:44:59.064838+0000","last_undegraded":"2024-09-13T14:44:59.064838+0000","last_fullsized":"2024-09-13T14:44:59.064838+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"193'7263","reported_seq":7323,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:18.839832+0000","last_change":"2024-09-13T14:42:15.076493+0000","last_active":"2024-09-13T14:45:18.839832+0000","last_peered":"2024-09-13T14:45:18.839832+0000","last_clean":"2024-09-13T14:45:18.839832+0000","last_became_active":"2024-09-13T14:42:15.074126+0000","last_became_peered":"2024-09-13T14:42:15.074126+0000","last_unstale":"2024-09-13T14:45:18.839832+0000","last_undegraded":"2024-09-13T14:45:18.839832+0000","last_fullsized":"2024-09-13T14:45:18.839832+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":7263,"log_dups_size":0,"ondisk_log_size":7263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113704960,"num_objects":1735,"num_object_clones":0,"num_object_copies":5205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5781,"num_write_kb":240512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":156,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062510+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:44:59.062510+0000","last_peered":"2024-09-13T14:44:59.062510+0000","last_clean":"2024-09-13T14:44:59.062510+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:44:59.062510+0000","last_undegraded":"2024-09-13T14:44:59.062510+0000","last_fullsized":"2024-09-13T14:44:59.062510+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":165,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.063272+0000","last_change":"2024-09-13T14:43:19.869693+0000","last_active":"2024-09-13T14:44:59.063272+0000","last_peered":"2024-09-13T14:44:59.063272+0000","last_clean":"2024-09-13T14:44:59.063272+0000","last_became_active":"2024-09-13T14:43:19.869343+0000","last_became_peered":"2024-09-13T14:43:19.869343+0000","last_unstale":"2024-09-13T14:44:59.063272+0000","last_undegraded":"2024-09-13T14:44:59.063272+0000","last_fullsized":"2024-09-13T14:44:59.063272+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:37.370378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"193'7266","reported_seq":7399,"reported_epoch":193,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:16.828763+0000","last_change":"2024-09-13T14:43:28.023481+0000","last_active":"2024-09-13T14:45:16.828763+0000","last_peered":"2024-09-13T14:45:16.828763+0000","last_clean":"2024-09-13T14:43:18.778117+0000","last_became_active":"2024-09-13T14:43:26.253579+0000","last_became_peered":"2024-09-13T14:43:26.253579+0000","last_unstale":"2024-09-13T14:45:16.828763+0000","last_undegraded":"2024-09-13T14:43:21.106616+0000","last_fullsized":"2024-09-13T14:43:20.987194+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3027","last_scrub_stamp":"2024-09-13T14:42:15.818474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:15.818474+0000","objects_scrubbed":1489,"log_size":7266,"log_dups_size":0,"ondisk_log_size":7266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116916224,"num_objects":1784,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4249,"num_write_kb":145408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2","objects":-1972},{"shards":"1,2,3","objects":1779}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":189,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.089758+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:45:12.089758+0000","last_peered":"2024-09-13T14:45:12.089758+0000","last_clean":"2024-09-13T14:45:12.089758+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:45:12.089758+0000","last_undegraded":"2024-09-13T14:45:12.089758+0000","last_fullsized":"2024-09-13T14:45:12.089758+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":229,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.089222+0000","last_change":"2024-09-13T14:37:08.758818+0000","last_active":"2024-09-13T14:45:12.089222+0000","last_peered":"2024-09-13T14:45:12.089222+0000","last_clean":"2024-09-13T14:45:12.089222+0000","last_became_active":"2024-09-13T14:37:08.758225+0000","last_became_peered":"2024-09-13T14:37:08.758225+0000","last_unstale":"2024-09-13T14:45:12.089222+0000","last_undegraded":"2024-09-13T14:45:12.089222+0000","last_fullsized":"2024-09-13T14:45:12.089222+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:18.890207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"193'7301","reported_seq":13441,"reported_epoch":193,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:14.715438+0000","last_change":"2024-09-13T14:43:21.498879+0000","last_active":"2024-09-13T14:45:14.715438+0000","last_peered":"2024-09-13T14:45:14.715438+0000","last_clean":"2024-09-13T14:43:18.793932+0000","last_became_active":"2024-09-13T14:43:21.445419+0000","last_became_peered":"2024-09-13T14:43:21.445419+0000","last_unstale":"2024-09-13T14:45:14.715438+0000","last_undegraded":"2024-09-13T14:43:21.110245+0000","last_fullsized":"2024-09-13T14:43:20.854131+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3248","last_scrub_stamp":"2024-09-13T14:42:19.745287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:19.745287+0000","objects_scrubbed":1743,"log_size":7301,"log_dups_size":0,"ondisk_log_size":7301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118030336,"num_objects":1801,"num_object_clones":0,"num_object_copies":5403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":131840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":131203072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,1],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,1","objects":4},{"shards":"0,3,5","objects":1797}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":187,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.072956+0000","last_change":"2024-09-13T14:43:19.888775+0000","last_active":"2024-09-13T14:45:12.072956+0000","last_peered":"2024-09-13T14:45:12.072956+0000","last_clean":"2024-09-13T14:45:12.072956+0000","last_became_active":"2024-09-13T14:43:19.871952+0000","last_became_peered":"2024-09-13T14:43:19.871952+0000","last_unstale":"2024-09-13T14:45:12.072956+0000","last_undegraded":"2024-09-13T14:45:12.072956+0000","last_fullsized":"2024-09-13T14:45:12.072956+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:18:03.133314+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:46.454803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":167,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.991144+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:44:58.991144+0000","last_peered":"2024-09-13T14:44:58.991144+0000","last_clean":"2024-09-13T14:44:58.991144+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:44:58.991144+0000","last_undegraded":"2024-09-13T14:44:58.991144+0000","last_fullsized":"2024-09-13T14:44:58.991144+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":178,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.090596+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:45:12.090596+0000","last_peered":"2024-09-13T14:45:12.090596+0000","last_clean":"2024-09-13T14:45:12.090596+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:45:12.090596+0000","last_undegraded":"2024-09-13T14:45:12.090596+0000","last_fullsized":"2024-09-13T14:45:12.090596+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":138,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.010681+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:44:59.010681+0000","last_peered":"2024-09-13T14:44:59.010681+0000","last_clean":"2024-09-13T14:44:59.010681+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:44:59.010681+0000","last_undegraded":"2024-09-13T14:44:59.010681+0000","last_fullsized":"2024-09-13T14:44:59.010681+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"193'11485","reported_seq":11586,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.706999+0000","last_change":"2024-09-13T14:41:41.821076+0000","last_active":"2024-09-13T14:45:14.706999+0000","last_peered":"2024-09-13T14:45:14.706999+0000","last_clean":"2024-09-13T14:45:14.706999+0000","last_became_active":"2024-09-13T14:41:41.820767+0000","last_became_peered":"2024-09-13T14:41:41.820767+0000","last_unstale":"2024-09-13T14:45:14.706999+0000","last_undegraded":"2024-09-13T14:45:14.706999+0000","last_fullsized":"2024-09-13T14:45:14.706999+0000","mapping_epoch":152,"log_start":"171'3900","ondisk_log_start":"171'3900","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:48.859865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234291200,"num_objects":3575,"num_object_clones":0,"num_object_copies":10725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11485,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":195,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.996521+0000","last_change":"2024-09-13T14:43:19.872365+0000","last_active":"2024-09-13T14:44:58.996521+0000","last_peered":"2024-09-13T14:44:58.996521+0000","last_clean":"2024-09-13T14:44:58.996521+0000","last_became_active":"2024-09-13T14:43:19.871956+0000","last_became_peered":"2024-09-13T14:43:19.871956+0000","last_unstale":"2024-09-13T14:44:58.996521+0000","last_undegraded":"2024-09-13T14:44:58.996521+0000","last_fullsized":"2024-09-13T14:44:58.996521+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:48.443596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":265,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.072956+0000","last_change":"2024-09-13T14:43:19.867565+0000","last_active":"2024-09-13T14:45:12.072956+0000","last_peered":"2024-09-13T14:45:12.072956+0000","last_clean":"2024-09-13T14:45:12.072956+0000","last_became_active":"2024-09-13T14:43:19.858361+0000","last_became_peered":"2024-09-13T14:43:19.858361+0000","last_unstale":"2024-09-13T14:45:12.072956+0000","last_undegraded":"2024-09-13T14:45:12.072956+0000","last_fullsized":"2024-09-13T14:45:12.072956+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:03.954121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":270,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.074343+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:45:12.074343+0000","last_peered":"2024-09-13T14:45:12.074343+0000","last_clean":"2024-09-13T14:45:12.074343+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:45:12.074343+0000","last_undegraded":"2024-09-13T14:45:12.074343+0000","last_fullsized":"2024-09-13T14:45:12.074343+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"193'11508","reported_seq":15639,"reported_epoch":193,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:16.798349+0000","last_change":"2024-09-13T14:43:35.834292+0000","last_active":"2024-09-13T14:45:16.798349+0000","last_peered":"2024-09-13T14:45:16.798349+0000","last_clean":"2024-09-13T14:43:18.829456+0000","last_became_active":"2024-09-13T14:43:26.226481+0000","last_became_peered":"2024-09-13T14:43:26.226481+0000","last_unstale":"2024-09-13T14:45:16.798349+0000","last_undegraded":"2024-09-13T14:43:35.827368+0000","last_fullsized":"2024-09-13T14:43:20.850342+0000","mapping_epoch":179,"log_start":"172'5232","ondisk_log_start":"172'5232","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":6276,"log_dups_size":0,"ondisk_log_size":6276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227409920,"num_objects":3470,"num_object_clones":0,"num_object_copies":10410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1988,"num_objects_misplaced":3463,"num_objects_unfound":0,"num_objects_dirty":3470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11508,"num_write_kb":479296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2031,"num_bytes_recovered":133103616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":3463},{"shards":"1,7","objects":-1981}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":246,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.061165+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:45:12.061165+0000","last_peered":"2024-09-13T14:45:12.061165+0000","last_clean":"2024-09-13T14:45:12.061165+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:45:12.061165+0000","last_undegraded":"2024-09-13T14:45:12.061165+0000","last_fullsized":"2024-09-13T14:45:12.061165+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":230,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.085173+0000","last_change":"2024-09-13T14:35:30.473739+0000","last_active":"2024-09-13T14:45:12.085173+0000","last_peered":"2024-09-13T14:45:12.085173+0000","last_clean":"2024-09-13T14:45:12.085173+0000","last_became_active":"2024-09-13T14:35:30.473448+0000","last_became_peered":"2024-09-13T14:35:30.473448+0000","last_unstale":"2024-09-13T14:45:12.085173+0000","last_undegraded":"2024-09-13T14:45:12.085173+0000","last_fullsized":"2024-09-13T14:45:12.085173+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:52.239598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":78,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.992529+0000","last_change":"2024-09-13T14:43:19.860282+0000","last_active":"2024-09-13T14:44:58.992529+0000","last_peered":"2024-09-13T14:44:58.992529+0000","last_clean":"2024-09-13T14:44:58.992529+0000","last_became_active":"2024-09-13T14:43:19.859966+0000","last_became_peered":"2024-09-13T14:43:19.859966+0000","last_unstale":"2024-09-13T14:44:58.992529+0000","last_undegraded":"2024-09-13T14:44:58.992529+0000","last_fullsized":"2024-09-13T14:44:58.992529+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:25.735714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"193'7194","reported_seq":7324,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.682327+0000","last_change":"2024-09-13T14:42:16.895536+0000","last_active":"2024-09-13T14:45:14.682327+0000","last_peered":"2024-09-13T14:45:14.682327+0000","last_clean":"2024-09-13T14:45:14.682327+0000","last_became_active":"2024-09-13T14:42:15.123602+0000","last_became_peered":"2024-09-13T14:42:15.123602+0000","last_unstale":"2024-09-13T14:45:14.682327+0000","last_undegraded":"2024-09-13T14:45:14.682327+0000","last_fullsized":"2024-09-13T14:45:14.682327+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3040","last_scrub_stamp":"2024-09-13T14:42:16.895299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.895299+0000","objects_scrubbed":1521,"log_size":7194,"log_dups_size":0,"ondisk_log_size":7194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:54.632739+0000","scrub_duration":483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116719616,"num_objects":1781,"num_object_clones":0,"num_object_copies":5343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":141440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":168,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.990955+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:44:58.990955+0000","last_peered":"2024-09-13T14:44:58.990955+0000","last_clean":"2024-09-13T14:44:58.990955+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:44:58.990955+0000","last_undegraded":"2024-09-13T14:44:58.990955+0000","last_fullsized":"2024-09-13T14:44:58.990955+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":156,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062663+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:44:59.062663+0000","last_peered":"2024-09-13T14:44:59.062663+0000","last_clean":"2024-09-13T14:44:59.062663+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:44:59.062663+0000","last_undegraded":"2024-09-13T14:44:59.062663+0000","last_fullsized":"2024-09-13T14:44:59.062663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":241,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.083739+0000","last_change":"2024-09-13T14:37:08.794198+0000","last_active":"2024-09-13T14:45:12.083739+0000","last_peered":"2024-09-13T14:45:12.083739+0000","last_clean":"2024-09-13T14:45:12.083739+0000","last_became_active":"2024-09-13T14:37:08.793689+0000","last_became_peered":"2024-09-13T14:37:08.793689+0000","last_unstale":"2024-09-13T14:45:12.083739+0000","last_undegraded":"2024-09-13T14:45:12.083739+0000","last_fullsized":"2024-09-13T14:45:12.083739+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:19.865901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"193'11484","reported_seq":11533,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:19.023292+0000","last_change":"2024-09-13T14:41:41.691642+0000","last_active":"2024-09-13T14:45:19.023292+0000","last_peered":"2024-09-13T14:45:19.023292+0000","last_clean":"2024-09-13T14:45:19.023292+0000","last_became_active":"2024-09-13T14:41:41.691265+0000","last_became_peered":"2024-09-13T14:41:41.691265+0000","last_unstale":"2024-09-13T14:45:19.023292+0000","last_undegraded":"2024-09-13T14:45:19.023292+0000","last_fullsized":"2024-09-13T14:45:19.023292+0000","mapping_epoch":152,"log_start":"157'2600","ondisk_log_start":"157'2600","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":8884,"log_dups_size":0,"ondisk_log_size":8884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:07.758310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215351296,"num_objects":3286,"num_object_clones":0,"num_object_copies":9858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11484,"num_write_kb":472640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":172,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.991298+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:44:58.991298+0000","last_peered":"2024-09-13T14:44:58.991298+0000","last_clean":"2024-09-13T14:44:58.991298+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:44:58.991298+0000","last_undegraded":"2024-09-13T14:44:58.991298+0000","last_fullsized":"2024-09-13T14:44:58.991298+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":269,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.083936+0000","last_change":"2024-09-13T14:32:09.488979+0000","last_active":"2024-09-13T14:45:12.083936+0000","last_peered":"2024-09-13T14:45:12.083936+0000","last_clean":"2024-09-13T14:45:12.083936+0000","last_became_active":"2024-09-13T14:32:09.488320+0000","last_became_peered":"2024-09-13T14:32:09.488320+0000","last_unstale":"2024-09-13T14:45:12.083936+0000","last_undegraded":"2024-09-13T14:45:12.083936+0000","last_fullsized":"2024-09-13T14:45:12.083936+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:28.662506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":40,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062342+0000","last_change":"2024-09-13T14:43:19.874797+0000","last_active":"2024-09-13T14:44:59.062342+0000","last_peered":"2024-09-13T14:44:59.062342+0000","last_clean":"2024-09-13T14:44:59.062342+0000","last_became_active":"2024-09-13T14:43:19.874460+0000","last_became_peered":"2024-09-13T14:43:19.874460+0000","last_unstale":"2024-09-13T14:44:59.062342+0000","last_undegraded":"2024-09-13T14:44:59.062342+0000","last_fullsized":"2024-09-13T14:44:59.062342+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:55.075477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"193'11584","reported_seq":11633,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:18.883596+0000","last_change":"2024-09-13T14:41:41.950035+0000","last_active":"2024-09-13T14:45:18.883596+0000","last_peered":"2024-09-13T14:45:18.883596+0000","last_clean":"2024-09-13T14:45:18.883596+0000","last_became_active":"2024-09-13T14:41:41.949672+0000","last_became_peered":"2024-09-13T14:41:41.949672+0000","last_unstale":"2024-09-13T14:45:18.883596+0000","last_undegraded":"2024-09-13T14:45:18.883596+0000","last_fullsized":"2024-09-13T14:45:18.883596+0000","mapping_epoch":152,"log_start":"172'5389","ondisk_log_start":"172'5389","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":6195,"log_dups_size":0,"ondisk_log_size":6195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:43.973874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226099200,"num_objects":3450,"num_object_clones":0,"num_object_copies":10350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":237,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.083835+0000","last_change":"2024-09-13T14:43:19.858770+0000","last_active":"2024-09-13T14:45:12.083835+0000","last_peered":"2024-09-13T14:45:12.083835+0000","last_clean":"2024-09-13T14:45:12.083835+0000","last_became_active":"2024-09-13T14:43:19.857709+0000","last_became_peered":"2024-09-13T14:43:19.857709+0000","last_unstale":"2024-09-13T14:45:12.083835+0000","last_undegraded":"2024-09-13T14:45:12.083835+0000","last_fullsized":"2024-09-13T14:45:12.083835+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:09:56.959220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":168,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.002281+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:44:59.002281+0000","last_peered":"2024-09-13T14:44:59.002281+0000","last_clean":"2024-09-13T14:44:59.002281+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:44:59.002281+0000","last_undegraded":"2024-09-13T14:44:59.002281+0000","last_fullsized":"2024-09-13T14:44:59.002281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":248,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.085606+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:45:12.085606+0000","last_peered":"2024-09-13T14:45:12.085606+0000","last_clean":"2024-09-13T14:45:12.085606+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:45:12.085606+0000","last_undegraded":"2024-09-13T14:45:12.085606+0000","last_fullsized":"2024-09-13T14:45:12.085606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"193'11751","reported_seq":20226,"reported_epoch":193,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:45:18.889214+0000","last_change":"2024-09-13T14:44:20.420585+0000","last_active":"2024-09-13T14:45:18.889214+0000","last_peered":"2024-09-13T14:45:18.889214+0000","last_clean":"2024-09-13T14:42:27.191777+0000","last_became_active":"2024-09-13T14:43:21.205187+0000","last_became_peered":"2024-09-13T14:43:21.205187+0000","last_unstale":"2024-09-13T14:45:18.889214+0000","last_undegraded":"2024-09-13T14:45:18.889214+0000","last_fullsized":"2024-09-13T14:43:20.852083+0000","mapping_epoch":179,"log_start":"172'5560","ondisk_log_start":"172'5560","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":6191,"log_dups_size":0,"ondisk_log_size":6191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:03.541405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224657432,"num_objects":3429,"num_object_clones":0,"num_object_copies":10287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6760,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11751,"num_write_kb":485697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":275316736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[7,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":3380},{"shards":"4,7","objects":49}],"blocked_by":[],"up_primary":5,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":219,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.092108+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:45:12.092108+0000","last_peered":"2024-09-13T14:45:12.092108+0000","last_clean":"2024-09-13T14:45:12.092108+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:45:12.092108+0000","last_undegraded":"2024-09-13T14:45:12.092108+0000","last_fullsized":"2024-09-13T14:45:12.092108+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":229,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.086180+0000","last_change":"2024-09-13T14:32:09.490012+0000","last_active":"2024-09-13T14:45:12.086180+0000","last_peered":"2024-09-13T14:45:12.086180+0000","last_clean":"2024-09-13T14:45:12.086180+0000","last_became_active":"2024-09-13T14:32:09.489652+0000","last_became_peered":"2024-09-13T14:32:09.489652+0000","last_unstale":"2024-09-13T14:45:12.086180+0000","last_undegraded":"2024-09-13T14:45:12.086180+0000","last_fullsized":"2024-09-13T14:45:12.086180+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:10.587217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":224,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.089504+0000","last_change":"2024-09-13T14:43:19.871833+0000","last_active":"2024-09-13T14:45:12.089504+0000","last_peered":"2024-09-13T14:45:12.089504+0000","last_clean":"2024-09-13T14:45:12.089504+0000","last_became_active":"2024-09-13T14:43:19.871024+0000","last_became_peered":"2024-09-13T14:43:19.871024+0000","last_unstale":"2024-09-13T14:45:12.089504+0000","last_undegraded":"2024-09-13T14:45:12.089504+0000","last_fullsized":"2024-09-13T14:45:12.089504+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:18:12.342823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"193'11576","reported_seq":11677,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:17.426011+0000","last_change":"2024-09-13T14:41:41.950797+0000","last_active":"2024-09-13T14:45:17.426011+0000","last_peered":"2024-09-13T14:45:17.426011+0000","last_clean":"2024-09-13T14:45:17.426011+0000","last_became_active":"2024-09-13T14:41:41.950412+0000","last_became_peered":"2024-09-13T14:41:41.950412+0000","last_unstale":"2024-09-13T14:45:17.426011+0000","last_undegraded":"2024-09-13T14:45:17.426011+0000","last_fullsized":"2024-09-13T14:45:17.426011+0000","mapping_epoch":152,"log_start":"173'5994","ondisk_log_start":"173'5994","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":5582,"log_dups_size":0,"ondisk_log_size":5582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:09.662164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231342080,"num_objects":3530,"num_object_clones":0,"num_object_copies":10590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":262,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.074600+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:45:12.074600+0000","last_peered":"2024-09-13T14:45:12.074600+0000","last_clean":"2024-09-13T14:45:12.074600+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:45:12.074600+0000","last_undegraded":"2024-09-13T14:45:12.074600+0000","last_fullsized":"2024-09-13T14:45:12.074600+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":178,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.010433+0000","last_change":"2024-09-13T14:43:19.860737+0000","last_active":"2024-09-13T14:44:59.010433+0000","last_peered":"2024-09-13T14:44:59.010433+0000","last_clean":"2024-09-13T14:44:59.010433+0000","last_became_active":"2024-09-13T14:43:19.860451+0000","last_became_peered":"2024-09-13T14:43:19.860451+0000","last_unstale":"2024-09-13T14:44:59.010433+0000","last_undegraded":"2024-09-13T14:44:59.010433+0000","last_fullsized":"2024-09-13T14:44:59.010433+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:14.888211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":138,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062282+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:44:59.062282+0000","last_peered":"2024-09-13T14:44:59.062282+0000","last_clean":"2024-09-13T14:44:59.062282+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:44:59.062282+0000","last_undegraded":"2024-09-13T14:44:59.062282+0000","last_fullsized":"2024-09-13T14:44:59.062282+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":155,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062592+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:44:59.062592+0000","last_peered":"2024-09-13T14:44:59.062592+0000","last_clean":"2024-09-13T14:44:59.062592+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:44:59.062592+0000","last_undegraded":"2024-09-13T14:44:59.062592+0000","last_fullsized":"2024-09-13T14:44:59.062592+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":162,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.063398+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:44:59.063398+0000","last_peered":"2024-09-13T14:44:59.063398+0000","last_clean":"2024-09-13T14:44:59.063398+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:44:59.063398+0000","last_undegraded":"2024-09-13T14:44:59.063398+0000","last_fullsized":"2024-09-13T14:44:59.063398+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":84,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.010139+0000","last_change":"2024-09-13T14:43:19.870397+0000","last_active":"2024-09-13T14:44:59.010139+0000","last_peered":"2024-09-13T14:44:59.010139+0000","last_clean":"2024-09-13T14:44:59.010139+0000","last_became_active":"2024-09-13T14:43:19.870037+0000","last_became_peered":"2024-09-13T14:43:19.870037+0000","last_unstale":"2024-09-13T14:44:59.010139+0000","last_undegraded":"2024-09-13T14:44:59.010139+0000","last_fullsized":"2024-09-13T14:44:59.010139+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:36.094674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":58,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.095488+0000","last_change":"2024-09-13T14:43:19.881173+0000","last_active":"2024-09-13T14:45:12.095488+0000","last_peered":"2024-09-13T14:45:12.095488+0000","last_clean":"2024-09-13T14:45:12.095488+0000","last_became_active":"2024-09-13T14:43:19.870848+0000","last_became_peered":"2024-09-13T14:43:19.870848+0000","last_unstale":"2024-09-13T14:45:12.095488+0000","last_undegraded":"2024-09-13T14:45:12.095488+0000","last_fullsized":"2024-09-13T14:45:12.095488+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:00.809391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":78,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.996984+0000","last_change":"2024-09-13T14:43:19.871796+0000","last_active":"2024-09-13T14:44:58.996984+0000","last_peered":"2024-09-13T14:44:58.996984+0000","last_clean":"2024-09-13T14:44:58.996984+0000","last_became_active":"2024-09-13T14:43:19.871130+0000","last_became_peered":"2024-09-13T14:43:19.871130+0000","last_unstale":"2024-09-13T14:44:58.996984+0000","last_undegraded":"2024-09-13T14:44:58.996984+0000","last_fullsized":"2024-09-13T14:44:58.996984+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:33.743486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":186,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.091444+0000","last_change":"2024-09-13T14:43:19.871328+0000","last_active":"2024-09-13T14:45:12.091444+0000","last_peered":"2024-09-13T14:45:12.091444+0000","last_clean":"2024-09-13T14:45:12.091444+0000","last_became_active":"2024-09-13T14:43:19.869668+0000","last_became_peered":"2024-09-13T14:43:19.869668+0000","last_unstale":"2024-09-13T14:45:12.091444+0000","last_undegraded":"2024-09-13T14:45:12.091444+0000","last_fullsized":"2024-09-13T14:45:12.091444+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:18.042885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":257,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.075738+0000","last_change":"2024-09-13T14:37:08.759355+0000","last_active":"2024-09-13T14:45:12.075738+0000","last_peered":"2024-09-13T14:45:12.075738+0000","last_clean":"2024-09-13T14:45:12.075738+0000","last_became_active":"2024-09-13T14:37:08.759058+0000","last_became_peered":"2024-09-13T14:37:08.759058+0000","last_unstale":"2024-09-13T14:45:12.075738+0000","last_undegraded":"2024-09-13T14:45:12.075738+0000","last_fullsized":"2024-09-13T14:45:12.075738+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:42.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":188,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.089134+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:45:12.089134+0000","last_peered":"2024-09-13T14:45:12.089134+0000","last_clean":"2024-09-13T14:45:12.089134+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:45:12.089134+0000","last_undegraded":"2024-09-13T14:45:12.089134+0000","last_fullsized":"2024-09-13T14:45:12.089134+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.12","version":"193'7345","reported_seq":13163,"reported_epoch":193,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:16.770461+0000","last_change":"2024-09-13T14:43:29.841339+0000","last_active":"2024-09-13T14:45:16.770461+0000","last_peered":"2024-09-13T14:45:16.770461+0000","last_clean":"2024-09-13T14:43:18.835862+0000","last_became_active":"2024-09-13T14:43:26.266307+0000","last_became_peered":"2024-09-13T14:43:26.266307+0000","last_unstale":"2024-09-13T14:45:16.770461+0000","last_undegraded":"2024-09-13T14:43:26.134821+0000","last_fullsized":"2024-09-13T14:43:20.850997+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3181","last_scrub_stamp":"2024-09-13T14:42:18.816556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:18.816556+0000","objects_scrubbed":1693,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114491392,"num_objects":1747,"num_object_clones":0,"num_object_copies":5241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4178,"num_write_kb":135424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1892,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0","3","5"],"object_location_counts":[{"shards":"0,1","objects":-2060},{"shards":"0,1,5","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":155,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.010371+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:44:59.010371+0000","last_peered":"2024-09-13T14:44:59.010371+0000","last_clean":"2024-09-13T14:44:59.010371+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:44:59.010371+0000","last_undegraded":"2024-09-13T14:44:59.010371+0000","last_fullsized":"2024-09-13T14:44:59.010371+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":213,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.089804+0000","last_change":"2024-09-13T14:43:19.872068+0000","last_active":"2024-09-13T14:45:12.089804+0000","last_peered":"2024-09-13T14:45:12.089804+0000","last_clean":"2024-09-13T14:45:12.089804+0000","last_became_active":"2024-09-13T14:43:19.871529+0000","last_became_peered":"2024-09-13T14:43:19.871529+0000","last_unstale":"2024-09-13T14:45:12.089804+0000","last_undegraded":"2024-09-13T14:45:12.089804+0000","last_fullsized":"2024-09-13T14:45:12.089804+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:11.995530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"193'7321","reported_seq":12955,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:18.802839+0000","last_change":"2024-09-13T14:43:29.963964+0000","last_active":"2024-09-13T14:45:18.802839+0000","last_peered":"2024-09-13T14:45:18.802839+0000","last_clean":"2024-09-13T14:45:18.802839+0000","last_became_active":"2024-09-13T14:43:29.959212+0000","last_became_peered":"2024-09-13T14:43:29.959212+0000","last_unstale":"2024-09-13T14:45:18.802839+0000","last_undegraded":"2024-09-13T14:45:18.802839+0000","last_fullsized":"2024-09-13T14:45:18.802839+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":7321,"log_dups_size":0,"ondisk_log_size":7321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113180672,"num_objects":1727,"num_object_clones":0,"num_object_copies":5181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5839,"num_write_kb":242112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1850,"num_bytes_recovered":121241600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":266,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.074440+0000","last_change":"2024-09-13T14:43:19.870848+0000","last_active":"2024-09-13T14:45:12.074440+0000","last_peered":"2024-09-13T14:45:12.074440+0000","last_clean":"2024-09-13T14:45:12.074440+0000","last_became_active":"2024-09-13T14:43:19.870037+0000","last_became_peered":"2024-09-13T14:43:19.870037+0000","last_unstale":"2024-09-13T14:45:12.074440+0000","last_undegraded":"2024-09-13T14:45:12.074440+0000","last_fullsized":"2024-09-13T14:45:12.074440+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:03.760717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":229,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.083267+0000","last_change":"2024-09-13T14:37:08.758601+0000","last_active":"2024-09-13T14:45:12.083267+0000","last_peered":"2024-09-13T14:45:12.083267+0000","last_clean":"2024-09-13T14:45:12.083267+0000","last_became_active":"2024-09-13T14:37:08.758031+0000","last_became_peered":"2024-09-13T14:37:08.758031+0000","last_unstale":"2024-09-13T14:45:12.083267+0000","last_undegraded":"2024-09-13T14:45:12.083267+0000","last_fullsized":"2024-09-13T14:45:12.083267+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:12.269600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"193'7634","reported_seq":13940,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:18.867269+0000","last_change":"2024-09-13T14:43:16.600263+0000","last_active":"2024-09-13T14:45:18.867269+0000","last_peered":"2024-09-13T14:45:18.867269+0000","last_clean":"2024-09-13T14:45:18.867269+0000","last_became_active":"2024-09-13T14:43:16.578306+0000","last_became_peered":"2024-09-13T14:43:16.578306+0000","last_unstale":"2024-09-13T14:45:18.867269+0000","last_undegraded":"2024-09-13T14:45:18.867269+0000","last_fullsized":"2024-09-13T14:45:18.867269+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3245","last_scrub_stamp":"2024-09-13T14:42:17.833200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:17.833200+0000","objects_scrubbed":1661,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:16.805530+0000","scrub_duration":483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120324096,"num_objects":1836,"num_object_clones":0,"num_object_copies":5508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4411,"num_write_kb":146752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1973,"num_bytes_recovered":129302528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":188,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.095303+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:45:12.095303+0000","last_peered":"2024-09-13T14:45:12.095303+0000","last_clean":"2024-09-13T14:45:12.095303+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:45:12.095303+0000","last_undegraded":"2024-09-13T14:45:12.095303+0000","last_fullsized":"2024-09-13T14:45:12.095303+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":174,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062427+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:44:59.062427+0000","last_peered":"2024-09-13T14:44:59.062427+0000","last_clean":"2024-09-13T14:44:59.062427+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:44:59.062427+0000","last_undegraded":"2024-09-13T14:44:59.062427+0000","last_fullsized":"2024-09-13T14:44:59.062427+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"193'7169","reported_seq":7274,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:18.895515+0000","last_change":"2024-09-13T14:42:23.582272+0000","last_active":"2024-09-13T14:45:18.895515+0000","last_peered":"2024-09-13T14:45:18.895515+0000","last_clean":"2024-09-13T14:45:18.895515+0000","last_became_active":"2024-09-13T14:42:23.579475+0000","last_became_peered":"2024-09-13T14:42:23.579475+0000","last_unstale":"2024-09-13T14:45:18.895515+0000","last_undegraded":"2024-09-13T14:45:18.895515+0000","last_fullsized":"2024-09-13T14:45:18.895515+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3023","last_scrub_stamp":"2024-09-13T14:42:17.713316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:17.713316+0000","objects_scrubbed":1552,"log_size":7169,"log_dups_size":0,"ondisk_log_size":7169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:08.367012+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":5091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4153,"num_write_kb":137536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":245,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.061554+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:45:12.061554+0000","last_peered":"2024-09-13T14:45:12.061554+0000","last_clean":"2024-09-13T14:45:12.061554+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:45:12.061554+0000","last_undegraded":"2024-09-13T14:45:12.061554+0000","last_fullsized":"2024-09-13T14:45:12.061554+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":250,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.085695+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:45:12.085695+0000","last_peered":"2024-09-13T14:45:12.085695+0000","last_clean":"2024-09-13T14:45:12.085695+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:45:12.085695+0000","last_undegraded":"2024-09-13T14:45:12.085695+0000","last_fullsized":"2024-09-13T14:45:12.085695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"193'7352","reported_seq":7491,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:18.959796+0000","last_change":"2024-09-13T14:42:25.288996+0000","last_active":"2024-09-13T14:45:18.959796+0000","last_peered":"2024-09-13T14:45:18.959796+0000","last_clean":"2024-09-13T14:45:18.959796+0000","last_became_active":"2024-09-13T14:42:15.125321+0000","last_became_peered":"2024-09-13T14:42:15.125321+0000","last_unstale":"2024-09-13T14:45:18.959796+0000","last_undegraded":"2024-09-13T14:45:18.959796+0000","last_fullsized":"2024-09-13T14:45:18.959796+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"166'3354","last_scrub_stamp":"2024-09-13T14:42:25.288934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:25.288934+0000","objects_scrubbed":1820,"log_size":7352,"log_dups_size":0,"ondisk_log_size":7352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:26.394629+0000","scrub_duration":0.191515928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117047296,"num_objects":1786,"num_object_clones":0,"num_object_copies":5358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":126848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":182,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062524+0000","last_change":"2024-09-13T14:37:08.749549+0000","last_active":"2024-09-13T14:44:59.062524+0000","last_peered":"2024-09-13T14:44:59.062524+0000","last_clean":"2024-09-13T14:44:59.062524+0000","last_became_active":"2024-09-13T14:37:08.749013+0000","last_became_peered":"2024-09-13T14:37:08.749013+0000","last_unstale":"2024-09-13T14:44:59.062524+0000","last_undegraded":"2024-09-13T14:44:59.062524+0000","last_fullsized":"2024-09-13T14:44:59.062524+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:35.239989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":157,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.990899+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:44:58.990899+0000","last_peered":"2024-09-13T14:44:58.990899+0000","last_clean":"2024-09-13T14:44:58.990899+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:44:58.990899+0000","last_undegraded":"2024-09-13T14:44:58.990899+0000","last_fullsized":"2024-09-13T14:44:58.990899+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"193'7507","reported_seq":13974,"reported_epoch":193,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:19.007486+0000","last_change":"2024-09-13T14:43:21.335815+0000","last_active":"2024-09-13T14:45:19.007486+0000","last_peered":"2024-09-13T14:45:19.007486+0000","last_clean":"2024-09-13T14:43:18.842274+0000","last_became_active":"2024-09-13T14:43:21.328599+0000","last_became_peered":"2024-09-13T14:43:21.328599+0000","last_unstale":"2024-09-13T14:45:19.007486+0000","last_undegraded":"2024-09-13T14:43:21.055960+0000","last_fullsized":"2024-09-13T14:43:20.850323+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":7507,"log_dups_size":0,"ondisk_log_size":7507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112984064,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5984,"num_write_kb":246656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2132,"num_bytes_recovered":139722752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,1],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"1,5","objects":-14},{"shards":"3,5,7","objects":1738}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":172,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062257+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:44:59.062257+0000","last_peered":"2024-09-13T14:44:59.062257+0000","last_clean":"2024-09-13T14:44:59.062257+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:44:59.062257+0000","last_undegraded":"2024-09-13T14:44:59.062257+0000","last_fullsized":"2024-09-13T14:44:59.062257+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":140,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:58.991295+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:44:58.991295+0000","last_peered":"2024-09-13T14:44:58.991295+0000","last_clean":"2024-09-13T14:44:58.991295+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:44:58.991295+0000","last_undegraded":"2024-09-13T14:44:58.991295+0000","last_fullsized":"2024-09-13T14:44:58.991295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"193'7279","reported_seq":7410,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:16.759856+0000","last_change":"2024-09-13T14:42:16.817489+0000","last_active":"2024-09-13T14:45:16.759856+0000","last_peered":"2024-09-13T14:45:16.759856+0000","last_clean":"2024-09-13T14:45:16.759856+0000","last_became_active":"2024-09-13T14:42:15.082970+0000","last_became_peered":"2024-09-13T14:42:15.082970+0000","last_unstale":"2024-09-13T14:45:16.759856+0000","last_undegraded":"2024-09-13T14:45:16.759856+0000","last_fullsized":"2024-09-13T14:45:16.759856+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3026","last_scrub_stamp":"2024-09-13T14:42:16.817418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.817418+0000","objects_scrubbed":1500,"log_size":7279,"log_dups_size":0,"ondisk_log_size":7279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:36.060283+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119275520,"num_objects":1820,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4260,"num_write_kb":146560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":221,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.091715+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:45:12.091715+0000","last_peered":"2024-09-13T14:45:12.091715+0000","last_clean":"2024-09-13T14:45:12.091715+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:45:12.091715+0000","last_undegraded":"2024-09-13T14:45:12.091715+0000","last_fullsized":"2024-09-13T14:45:12.091715+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":207,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.091307+0000","last_change":"2024-09-13T14:37:08.750424+0000","last_active":"2024-09-13T14:45:12.091307+0000","last_peered":"2024-09-13T14:45:12.091307+0000","last_clean":"2024-09-13T14:45:12.091307+0000","last_became_active":"2024-09-13T14:37:08.749577+0000","last_became_peered":"2024-09-13T14:37:08.749577+0000","last_unstale":"2024-09-13T14:45:12.091307+0000","last_undegraded":"2024-09-13T14:45:12.091307+0000","last_fullsized":"2024-09-13T14:45:12.091307+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:16.435452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"193'7262","reported_seq":13310,"reported_epoch":193,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:14.729470+0000","last_change":"2024-09-13T14:43:21.354297+0000","last_active":"2024-09-13T14:45:14.729470+0000","last_peered":"2024-09-13T14:45:14.729470+0000","last_clean":"2024-09-13T14:43:18.829968+0000","last_became_active":"2024-09-13T14:43:21.292689+0000","last_became_peered":"2024-09-13T14:43:21.292689+0000","last_unstale":"2024-09-13T14:45:14.729470+0000","last_undegraded":"2024-09-13T14:43:21.072736+0000","last_fullsized":"2024-09-13T14:43:20.855615+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3124","last_scrub_stamp":"2024-09-13T14:42:17.909091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:17.909091+0000","objects_scrubbed":1626,"log_size":7262,"log_dups_size":0,"ondisk_log_size":7262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120717312,"num_objects":1842,"num_object_clones":0,"num_object_copies":5526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4154,"num_write_kb":139840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":129171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,1","objects":5},{"shards":"0,5,7","objects":1837}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":155,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T14:44:59.062191+0000","last_change":"2024-09-13T14:32:09.484255+0000","last_active":"2024-09-13T14:44:59.062191+0000","last_peered":"2024-09-13T14:44:59.062191+0000","last_clean":"2024-09-13T14:44:59.062191+0000","last_became_active":"2024-09-13T14:32:09.483903+0000","last_became_peered":"2024-09-13T14:32:09.483903+0000","last_unstale":"2024-09-13T14:44:59.062191+0000","last_undegraded":"2024-09-13T14:44:59.062191+0000","last_fullsized":"2024-09-13T14:44:59.062191+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:41.931522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":262,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T14:45:12.074712+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:45:12.074712+0000","last_peered":"2024-09-13T14:45:12.074712+0000","last_clean":"2024-09-13T14:45:12.074712+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:45:12.074712+0000","last_undegraded":"2024-09-13T14:45:12.074712+0000","last_fullsized":"2024-09-13T14:45:12.074712+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":3670212632,"num_objects":56004,"num_object_clones":0,"num_object_copies":168012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29055,"num_objects_misplaced":11569,"num_objects_unfound":0,"num_objects_dirty":56004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158114,"num_write_kb":5966017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32152,"num_bytes_recovered":2107113472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14351216640,"data_stored":14351204424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186828,"ondisk_log_size":186828,"up":78,"acting":68,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":182,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365107,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3603908,"kb_used_data":2690364,"kb_used_omap":1,"kb_used_meta":913534,"kb_avail":90141244,"statfs":{"total":95995035648,"available":92304633856,"internally_reserved":0,"allocated":2754932736,"data_stored":2753949466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":935459160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561337,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1361912,"kb_used_data":763696,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":92383240,"statfs":{"total":95995035648,"available":94600437760,"internally_reserved":0,"allocated":782024704,"data_stored":781053521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790269,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3814252,"kb_used_data":2913188,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":89930900,"statfs":{"total":95995035648,"available":92089241600,"internally_reserved":0,"allocated":2983104512,"data_stored":2982140623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986497,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2021376,"kb_used_data":1122808,"kb_used_omap":1,"kb_used_meta":898558,"kb_avail":91723776,"statfs":{"total":95995035648,"available":93925146624,"internally_reserved":0,"allocated":1149755392,"data_stored":1148775869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920123736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519652,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2668588,"kb_used_data":2274084,"kb_used_omap":4,"kb_used_meta":394427,"kb_avail":91076564,"statfs":{"total":95995035648,"available":93262401536,"internally_reserved":0,"allocated":2328662016,"data_stored":2327698231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":403893696},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304568,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1001324,"kb_used_data":672164,"kb_used_omap":6,"kb_used_meta":329145,"kb_avail":92743828,"statfs":{"total":95995035648,"available":94969679872,"internally_reserved":0,"allocated":688295936,"data_stored":687332595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":337044935},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827871,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3104428,"kb_used_data":2461860,"kb_used_omap":5,"kb_used_meta":642298,"kb_avail":90640724,"statfs":{"total":95995035648,"available":92816101376,"internally_reserved":0,"allocated":2520944640,"data_stored":2519981266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":657713388},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285735,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1695408,"kb_used_data":1196520,"kb_used_omap":5,"kb_used_meta":498874,"kb_avail":92049744,"statfs":{"total":95995035648,"available":94258937856,"internally_reserved":0,"allocated":1225236480,"data_stored":1224268437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":510847035},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1209008128,"data_stored":1209008128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2506162176,"data_stored":2506162176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":684261376,"data_stored":684261376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2320629760,"data_stored":2320629760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1139937280,"data_stored":1139933208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2973569024,"data_stored":2973564952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":778567680,"data_stored":778567680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2739081216,"data_stored":2739077144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:45:21.329 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:45:21.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-13T14:45:21.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:21.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:21 smithi019 ceph-mon[91296]: pgmap v654: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 645 op/s; 29055/168090 objects degraded (17.285%); 11569/168090 objects misplaced (6.883%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:45:21.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:21 smithi019 ceph-mon[91296]: from='client.36018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:21.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 29055/168090 objects degraded (17.285%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:21.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:21.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:45:21.568 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:21.612 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:21.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:21.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:21.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:21.876 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:21.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:22.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:22.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:22.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:22.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:22.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:22.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:22.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:22.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:22.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:22.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:22.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:22.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:22.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:22.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:23.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:23.037 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:23.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:23.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:23.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:23.315 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:23.417 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:23.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:23 smithi019 ceph-mon[91296]: pgmap v655: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 28237/165540 objects degraded (17.058%); 11569/165540 objects misplaced (6.989%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:45:23.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:23 smithi003 ceph-mon[107278]: pgmap v655: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 28237/165540 objects degraded (17.058%); 11569/165540 objects misplaced (6.989%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:45:23.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:23 smithi003 ceph-mon[104007]: pgmap v655: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 28237/165540 objects degraded (17.058%); 11569/165540 objects misplaced (6.989%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:45:23.450 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:23.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:23.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:23.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:23.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:23.824 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:23.861 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:23.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:24.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:24.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:24.113 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:45:24.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:24.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:24 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/720025689' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:24.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:45:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:45:24] "GET /metrics HTTP/1.1" 200 36561 "" "Prometheus/2.51.0" 2024-09-13T14:45:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:24 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/720025689' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:24.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:24.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:24.419 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:24.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:24 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/720025689' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:24.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:24.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:24.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:24.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:24.807 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:24.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:24.935 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:24.963 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:25.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.103 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:25.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:25 smithi003 ceph-mon[104007]: pgmap v656: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 593 op/s; 28321/162765 objects degraded (17.400%); 11385/162765 objects misplaced (6.995%); 11 MiB/s, 171 objects/s recovering 2024-09-13T14:45:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:25 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:25 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/720025689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:25 smithi003 ceph-mon[104007]: osdmap e194: 8 total, 8 up, 8 in 2024-09-13T14:45:25.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:25 smithi003 ceph-mon[107278]: pgmap v656: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 593 op/s; 28321/162765 objects degraded (17.400%); 11385/162765 objects misplaced (6.995%); 11 MiB/s, 171 objects/s recovering 2024-09-13T14:45:25.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:25 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:25.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:25 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/720025689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:25.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:25 smithi003 ceph-mon[107278]: osdmap e194: 8 total, 8 up, 8 in 2024-09-13T14:45:25.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:25.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:25.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.390 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:25.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:25.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:25 smithi019 ceph-mon[91296]: pgmap v656: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 593 op/s; 28321/162765 objects degraded (17.400%); 11385/162765 objects misplaced (6.995%); 11 MiB/s, 171 objects/s recovering 2024-09-13T14:45:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:25 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:25 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/720025689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:25 smithi019 ceph-mon[91296]: osdmap e194: 8 total, 8 up, 8 in 2024-09-13T14:45:25.491 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:25.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:25.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:25.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:25.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:25.766 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.805 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:25.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:25.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:25.935 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:26.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:26.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:26.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:26.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:26.360 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 28321/162765 objects degraded (17.400%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:26.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 28321/162765 objects degraded (17.400%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:26.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:26.416 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:26.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 28321/162765 objects degraded (17.400%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:26.463 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:26.492 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:26.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:26.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:26.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:26.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:26.881 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:26.913 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:27.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:27.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:27.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:27.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:27 smithi003 ceph-mon[107278]: pgmap v658: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 27165/155013 objects degraded (17.524%); 10720/155013 objects misplaced (6.916%); 13 MiB/s, 212 objects/s recovering 2024-09-13T14:45:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:27 smithi003 ceph-mon[104007]: pgmap v658: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 27165/155013 objects degraded (17.524%); 10720/155013 objects misplaced (6.916%); 13 MiB/s, 212 objects/s recovering 2024-09-13T14:45:27.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:27.241 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:45:27.241 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:27.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:45:27.279 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:27.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:27.316 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:45:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:27.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:27.351 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:45:27.351 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:27.379 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:27 smithi019 ceph-mon[91296]: pgmap v658: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 27165/155013 objects degraded (17.524%); 10720/155013 objects misplaced (6.916%); 13 MiB/s, 212 objects/s recovering 2024-09-13T14:45:27.385 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:45:27.385 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:27.421 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:45:27.422 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:27.450 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:27.454 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:45:27.454 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.486 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:27.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:27.489 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:45:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:27.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:27.624 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:27.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:170801 8.5 8:a7ef04f1:::benchmark_data_smithi003_139172_object46008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:38:47.007953+0000", 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 400.74333976899999, 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.14155104299999999, 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.757 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 170801 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.007953+0000", 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.758 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.007953+0000", 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.007951+0000", 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.007958+0000", 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1790000000000002e-06 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.759 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.116290+0000", 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.108331447 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.116306+0000", 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5919000000000001e-05 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.116351+0000", 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5534000000000001e-05 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.116471+0000", 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000119547 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.116601+0000", 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000129741 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.139747+0000", 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.023146047999999999 2024-09-13T14:45:27.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.139778+0000", 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1690000000000003e-05 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.149421+0000", 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0096429389999999997 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.149465+0000", 2024-09-13T14:45:27.762 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4132999999999999e-05 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:38:47.149504+0000", 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8506e-05 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:221534 8.9 8:9a63baea:::benchmark_data_smithi003_139172_object96741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:46.132030+0000", 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 341.61926299300001, 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.14028480700000001, 2024-09-13T14:45:27.763 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 221534 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.132030+0000", 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.764 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.132030+0000", 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.132028+0000", 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.132037+0000", 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3610000000000001e-06 2024-09-13T14:45:27.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.132053+0000", 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6612999999999998e-05 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.132060+0000", 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4239999999999998e-06 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.766 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.132112+0000", 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2274999999999997e-05 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.132239+0000", 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012706699999999999 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.132355+0000", 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000115946 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.767 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.136812+0000", 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0044572300000000004 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.136903+0000", 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1038000000000005e-05 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.272232+0000", 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.135328913 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.272278+0000", 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6446999999999998e-05 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.272314+0000", 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5989999999999999e-05 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:221538 8.9 8:9fe2d129:::benchmark_data_smithi003_139172_object96745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:45:27.769 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:46.138521+0000", 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 341.61277161800001, 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.15437456199999999, 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 221538 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.138521+0000", 2024-09-13T14:45:27.770 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.138521+0000", 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.138520+0000", 2024-09-13T14:45:27.771 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.138526+0000", 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0909999999999998e-06 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.138537+0000", 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0779e-05 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.772 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.138542+0000", 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6740000000000002e-06 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.138577+0000", 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4876000000000003e-05 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.138680+0000", 2024-09-13T14:45:27.773 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000102404 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.138804+0000", 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012484400000000001 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.142567+0000", 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0037628729999999999 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.774 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.142714+0000", 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000147052 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.292803+0000", 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.15008860299999999 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.292851+0000", 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8350999999999998e-05 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:46.292896+0000", 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4249999999999998e-05 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:120832 9.0 9:01790219:::benchmark_data_smithi003_156985_object120831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:21.355891+0000", 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 126.395401385, 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.166246166, 2024-09-13T14:45:27.776 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 120832 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.355891+0000", 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.777 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.355891+0000", 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.355884+0000", 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999924 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.356045+0000", 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016164799999999999 2024-09-13T14:45:27.778 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.356055+0000", 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8339999999999994e-06 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.356060+0000", 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3810000000000001e-06 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.356116+0000", 2024-09-13T14:45:27.779 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6437000000000002e-05 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.356124+0000", 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8420000000000005e-06 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.506462+0000", 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.150337731 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.780 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.506559+0000", 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7746999999999998e-05 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.506681+0000", 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012163 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.512421+0000", 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057400089999999999 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.781 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.512430+0000", 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5709999999999998e-06 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.519006+0000", 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.006576096 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.519066+0000", 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0096999999999997e-05 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.782 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.522137+0000", 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0030717079999999998 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:140161 9.0 9:04841747:::benchmark_data_smithi003_156985_object19245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:44:04.309173+0000", 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 83.442119696000006, 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.078954811, 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.783 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 140161 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.309173+0000", 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.784 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.309173+0000", 2024-09-13T14:45:27.785 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.785 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.785 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.785 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.785 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.309171+0000", 2024-09-13T14:45:27.785 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:27.785 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.309182+0000", 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0902e-05 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.309205+0000", 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3399000000000001e-05 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.786 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.309213+0000", 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7600000000000002e-06 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.309270+0000", 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7260999999999999e-05 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.309452+0000", 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00018182800000000001 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.787 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.309613+0000", 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016118 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.311840+0000", 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022265980000000002 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.312652+0000", 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00081271800000000003 2024-09-13T14:45:27.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.385616+0000", 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.072963816000000001 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.385669+0000", 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2840000000000002e-05 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.789 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:04.388128+0000", 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002458619 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:148885 9.0 9:03d3b7e6:::benchmark_data_smithi003_156985_object27969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:44:18.569296+0000", 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 69.181996514999994, 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.100818449, 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.790 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 148885 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.569296+0000", 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.569296+0000", 2024-09-13T14:45:27.791 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.569294+0000", 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.569305+0000", 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1443999999999999e-05 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.792 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.569327+0000", 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2127999999999999e-05 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.569336+0000", 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3960000000000006e-06 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.569402+0000", 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6124999999999995e-05 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.794 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.794 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.569585+0000", 2024-09-13T14:45:27.794 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000183158 2024-09-13T14:45:27.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.794 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.794 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.569750+0000", 2024-09-13T14:45:27.794 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000164897 2024-09-13T14:45:27.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.574621+0000", 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.004870859 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.577041+0000", 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00242034 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.796 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.667759+0000", 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.090717790000000006 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.667821+0000", 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2256999999999997e-05 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.670115+0000", 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022936129999999999 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.797 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:148887 9.0 9:075e95af:::benchmark_data_smithi003_156985_object27971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:44:18.570267+0000", 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 69.181025531000003, 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.104434026, 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 148887 2024-09-13T14:45:27.798 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.570267+0000", 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.570267+0000", 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.799 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.570266+0000", 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.570275+0000", 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7839999999999992e-06 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.800 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.801 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.570289+0000", 2024-09-13T14:45:27.801 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4105e-05 2024-09-13T14:45:27.801 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.801 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.801 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.801 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.570295+0000", 2024-09-13T14:45:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.802 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:27.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:27.803 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8109999999999999e-06 2024-09-13T14:45:27.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.803 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.803 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.572189+0000", 2024-09-13T14:45:27.803 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001893837 2024-09-13T14:45:27.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.803 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.572350+0000", 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016136699999999999 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.572497+0000", 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000146632 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.574729+0000", 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022324300000000001 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.804 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.805 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.805 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.577193+0000", 2024-09-13T14:45:27.805 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002463888 2024-09-13T14:45:27.805 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.805 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.806 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.806 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.670170+0000", 2024-09-13T14:45:27.806 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.092976750999999996 2024-09-13T14:45:27.806 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.806 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.806 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.807 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.670258+0000", 2024-09-13T14:45:27.807 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8770999999999994e-05 2024-09-13T14:45:27.807 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.807 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.807 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.808 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:18.674701+0000", 2024-09-13T14:45:27.808 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0044428089999999998 2024-09-13T14:45:27.808 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.808 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.809 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.809 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.809 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.809 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:149331 9.0 9:039720a2:::benchmark_data_smithi003_156985_object28415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T14:45:27.809 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:44:19.281929+0000", 2024-09-13T14:45:27.810 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 68.469363912000006, 2024-09-13T14:45:27.810 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22040004599999999, 2024-09-13T14:45:27.810 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.810 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.810 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.811 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.811 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.811 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149331 2024-09-13T14:45:27.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.811 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.812 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.812 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.812 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.281929+0000", 2024-09-13T14:45:27.812 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.812 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.813 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.813 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.813 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.281929+0000", 2024-09-13T14:45:27.813 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.813 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.813 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.813 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.281927+0000", 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.281934+0000", 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4889999999999998e-06 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.814 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.281953+0000", 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8275e-05 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.281959+0000", 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1349999999999997e-06 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.282008+0000", 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9459999999999997e-05 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.282180+0000", 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017216200000000001 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.282336+0000", 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015576000000000001 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.288582+0000", 2024-09-13T14:45:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.006245947 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.288595+0000", 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2612999999999999e-05 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.499799+0000", 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21120388700000001 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.817 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.499874+0000", 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4833999999999995e-05 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.502329+0000", 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002455273 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:149332 9.0 9:052a100e:::benchmark_data_smithi003_156985_object28416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T14:45:27.818 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:44:19.282294+0000", 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 68.468998661000001, 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22612290700000001, 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149332 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.819 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.282294+0000", 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.282294+0000", 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.820 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.282293+0000", 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.282299+0000", 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7010000000000001e-06 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.282311+0000", 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1695000000000001e-05 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.282316+0000", 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7069999999999997e-06 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.285086+0000", 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027693409999999998 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.285238+0000", 2024-09-13T14:45:27.822 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000152595 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.285379+0000", 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014127000000000001 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.288688+0000", 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0033084870000000001 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.290197+0000", 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0015091919999999999 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.506276+0000", 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.216079142 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.506401+0000", 2024-09-13T14:45:27.824 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012428100000000001 2024-09-13T14:45:27.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.825 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.825 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.825 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:19.508417+0000", 2024-09-13T14:45:27.825 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0020163709999999999 2024-09-13T14:45:27.825 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.825 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.825 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:159654 9.0 9:05723524:::benchmark_data_smithi003_156985_object38738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:44:35.771773+0000", 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 51.979519688000003, 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.085388622999999997, 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.826 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 159654 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.771773+0000", 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.771773+0000", 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.771771+0000", 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.771784+0000", 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2404e-05 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.771800+0000", 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6283999999999999e-05 2024-09-13T14:45:27.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.771806+0000", 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5590000000000002e-06 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.771864+0000", 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7509000000000001e-05 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.772056+0000", 2024-09-13T14:45:27.829 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000192153 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.772219+0000", 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000162905 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.775475+0000", 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003256498 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.830 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.779222+0000", 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0037469370000000001 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.855124+0000", 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.075901971999999998 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.855180+0000", 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5699000000000003e-05 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.831 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.857162+0000", 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0019815140000000002 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:159660 9.0 9:077b1473:::benchmark_data_smithi003_156985_object38744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:44:35.785694+0000", 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 51.965599122999997, 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.074283737000000002, 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.832 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 159660 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.785694+0000", 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.833 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.785694+0000", 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.785692+0000", 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.785701+0000", 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4829999999999999e-06 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.785718+0000", 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7368999999999999e-05 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.785724+0000", 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.015e-06 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.785775+0000", 2024-09-13T14:45:27.835 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1366999999999998e-05 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.785943+0000", 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000168016 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.786085+0000", 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014177100000000001 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.836 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.791261+0000", 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0051754979999999997 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.791269+0000", 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4190000000000002e-06 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.858405+0000", 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.067135619999999993 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.858460+0000", 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4922e-05 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:35.859977+0000", 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001517577 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:160195 9.0 9:073b40a4:::benchmark_data_smithi003_156985_object39279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:44:36.693264+0000", 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 51.058028845000003, 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.082200969999999998, 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 160195 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.839 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.693264+0000", 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.693264+0000", 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.693262+0000", 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:27.840 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.841 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.841 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.693269+0000", 2024-09-13T14:45:27.841 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8929999999999999e-06 2024-09-13T14:45:27.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.841 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.841 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.693280+0000", 2024-09-13T14:45:27.841 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0329999999999999e-05 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.693286+0000", 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2879999999999998e-06 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.698490+0000", 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0052044389999999999 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.698681+0000", 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00019095599999999999 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.698841+0000", 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000159619 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.701836+0000", 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029947490000000001 2024-09-13T14:45:27.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.701865+0000", 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9655999999999999e-05 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.773419+0000", 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.071553585000000003 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.773457+0000", 2024-09-13T14:45:27.844 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8359000000000003e-05 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:44:36.775465+0000", 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002007497 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:175077 9.0 9:05dcd9c2:::benchmark_data_smithi003_156985_object54161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:00.780495+0000", 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 26.970797310000002, 2024-09-13T14:45:27.845 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.096307769000000001, 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 175077 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.780495+0000", 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.846 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.780495+0000", 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.780493+0000", 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.780501+0000", 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8259999999999993e-06 2024-09-13T14:45:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.780516+0000", 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5160000000000001e-05 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.780522+0000", 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.648e-06 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.786720+0000", 2024-09-13T14:45:27.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0061984409999999998 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.786864+0000", 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014322900000000001 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.787001+0000", 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013760899999999999 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.789847+0000", 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028456990000000001 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.793066+0000", 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0032191450000000001 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.875064+0000", 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.081997569000000006 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.875110+0000", 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6199000000000003e-05 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.876803+0000", 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001693321 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:175087 9.0 9:05b9eb3a:::benchmark_data_smithi003_156985_object54171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:00.799578+0000", 2024-09-13T14:45:27.852 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 26.951714711000001, 2024-09-13T14:45:27.852 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.078922205999999995, 2024-09-13T14:45:27.852 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.852 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.852 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 175087 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.799578+0000", 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.799578+0000", 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.799576+0000", 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.799585+0000", 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7590000000000003e-06 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.799603+0000", 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7577000000000001e-05 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.799610+0000", 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5599999999999996e-06 2024-09-13T14:45:27.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.799669+0000", 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8349e-05 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.799836+0000", 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016736999999999999 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.800003+0000", 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000166622 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.802415+0000", 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024124910000000001 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.802466+0000", 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1275e-05 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.857 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.876879+0000", 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.074412269000000003 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.876941+0000", 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1928999999999996e-05 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:00.878500+0000", 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0015595260000000001 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.858 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:175308 9.6 9:63a764b5:::benchmark_data_smithi003_156985_object54392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:01.158943+0000", 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 26.592349894000002, 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.072586968000000002, 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.859 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 175308 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.158943+0000", 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.158943+0000", 2024-09-13T14:45:27.860 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.861 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.861 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.861 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.861 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.158941+0000", 2024-09-13T14:45:27.861 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:27.861 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.861 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.861 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.861 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.158949+0000", 2024-09-13T14:45:27.862 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4610000000000008e-06 2024-09-13T14:45:27.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.862 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.862 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.158970+0000", 2024-09-13T14:45:27.862 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0879000000000001e-05 2024-09-13T14:45:27.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.158978+0000", 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.571e-06 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.159027+0000", 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9357999999999997e-05 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.159145+0000", 2024-09-13T14:45:27.863 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011768 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.159250+0000", 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010518699999999999 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.161777+0000", 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002527119 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.864 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.161964+0000", 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000187207 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.231457+0000", 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.069492987000000006 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.231491+0000", 2024-09-13T14:45:27.865 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3927999999999998e-05 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:01.231530+0000", 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8390999999999999e-05 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.866 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:177200 9.0 9:03299dba:::benchmark_data_smithi003_156985_object56284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:03.924310+0000", 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 23.826982520000001, 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.087943861999999998, 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 177200 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.867 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.924310+0000", 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.924310+0000", 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.924308+0000", 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:27.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.924317+0000", 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4940000000000003e-06 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.924335+0000", 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8257000000000002e-05 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.924342+0000", 2024-09-13T14:45:27.869 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2130000000000002e-06 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.924485+0000", 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000143127 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.924635+0000", 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014967299999999999 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.924778+0000", 2024-09-13T14:45:27.870 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014329499999999999 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.927464+0000", 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026856470000000002 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.935983+0000", 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0085184530000000005 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.871 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:04.010120+0000", 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.074137403000000004 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:04.010158+0000", 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8436000000000003e-05 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:04.012254+0000", 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0020956579999999998 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.872 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:177205 9.0 9:02dc534d:::benchmark_data_smithi003_156985_object56289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:03.933326+0000", 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 23.817967110000001, 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.083384172000000006, 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.873 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 177205 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.933326+0000", 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.933326+0000", 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.874 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.933324+0000", 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.933335+0000", 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0662999999999999e-05 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.933350+0000", 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4769e-05 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.933356+0000", 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.93e-06 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.933418+0000", 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1953999999999997e-05 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.933593+0000", 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017509399999999999 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.933703+0000", 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011025 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.936063+0000", 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0023598009999999999 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:03.937905+0000", 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001842265 2024-09-13T14:45:27.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:04.015627+0000", 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.077722037999999993 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:04.015667+0000", 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9660999999999999e-05 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:04.016710+0000", 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001043069 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:179057 9.0 9:07ef4c8b:::benchmark_data_smithi003_156985_object58141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:06.845808+0000", 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 20.90548433, 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.090041277000000003, 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.879 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 179057 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.845808+0000", 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.880 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.845808+0000", 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.845807+0000", 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.845815+0000", 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9640000000000003e-06 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.881 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.845828+0000", 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3548e-05 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.845835+0000", 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3080000000000004e-06 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.846820+0000", 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00098499999999999998 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.846960+0000", 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013990899999999999 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.847089+0000", 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000128799 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.855430+0000", 2024-09-13T14:45:27.884 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0083410270000000009 2024-09-13T14:45:27.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.884 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.884 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.884 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.855489+0000", 2024-09-13T14:45:27.884 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9117999999999999e-05 2024-09-13T14:45:27.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.885 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.934298+0000", 2024-09-13T14:45:27.885 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.078809441999999993 2024-09-13T14:45:27.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.934337+0000", 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8797999999999997e-05 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:06.935850+0000", 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0015127140000000001 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.887 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.887 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:179626 9.0 9:06e29047:::benchmark_data_smithi003_156985_object58710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:45:27.887 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:07.754728+0000", 2024-09-13T14:45:27.887 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 19.996564281000001, 2024-09-13T14:45:27.887 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.084541637000000003, 2024-09-13T14:45:27.887 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 179626 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.754728+0000", 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.888 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.754728+0000", 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.754727+0000", 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.754733+0000", 2024-09-13T14:45:27.889 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0759999999999999e-06 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.754742+0000", 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1360000000000003e-06 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.754747+0000", 2024-09-13T14:45:27.890 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5480000000000003e-06 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.764044+0000", 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0092972839999999994 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.764196+0000", 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000151605 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.764334+0000", 2024-09-13T14:45:27.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013787900000000001 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.767074+0000", 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027402799999999999 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.836956+0000", 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.069882545000000004 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:27.893 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.837212+0000", 2024-09-13T14:45:27.893 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00025560099999999999 2024-09-13T14:45:27.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.837308+0000", 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5866000000000007e-05 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.839270+0000", 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0019621859999999999 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.895 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.895 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.45869.0:179633 9.d e192/179 9:be15f414:::benchmark_data_smithi003_156985_object58717:head v 192'11285, mlcod=0'0)", 2024-09-13T14:45:27.895 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:07.767661+0000", 2024-09-13T14:45:27.895 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 19.983631603999999, 2024-09-13T14:45:27.895 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.069373629000000006, 2024-09-13T14:45:27.895 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:45:27.896 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:27.896 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:45:27.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:45:27.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.767661+0000", 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.767661+0000", 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:45:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.767660+0000", 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.767666+0000", 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5729999999999998e-06 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.767674+0000", 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.524e-06 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.767680+0000", 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5929999999999996e-06 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.767725+0000", 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4654999999999999e-05 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.767744+0000", 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9159000000000001e-05 2024-09-13T14:45:27.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.836996+0000", 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.069252522999999996 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:07.837035+0000", 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8386000000000001e-05 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.900 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.901 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:45:27.901 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:45:27.901 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:45:27.903 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:27.935 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:28.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:28.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:28.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:28.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:28.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:28.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:28.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:45:28.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3626358832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:28.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3626358832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:28.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3626358832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:28.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:28.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:28.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:28.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:28.679 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:28.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:28.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:28.833 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:45:28.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:28.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:29.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:29.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:29.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:29.277 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:29.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:29.395 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:29.410 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:29 smithi019 ceph-mon[91296]: pgmap v659: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 683 op/s; 27259/152292 objects degraded (17.899%); 10521/152292 objects misplaced (6.908%); 13 MiB/s, 212 objects/s recovering 2024-09-13T14:45:29.410 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:45:29.429 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:29 smithi003 ceph-mon[104007]: pgmap v659: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 683 op/s; 27259/152292 objects degraded (17.899%); 10521/152292 objects misplaced (6.908%); 13 MiB/s, 212 objects/s recovering 2024-09-13T14:45:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:45:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:29 smithi003 ceph-mon[107278]: pgmap v659: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 683 op/s; 27259/152292 objects degraded (17.899%); 10521/152292 objects misplaced (6.908%); 13 MiB/s, 212 objects/s recovering 2024-09-13T14:45:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:45:29.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:29.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:29.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:29.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:29.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:29.844 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:29.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:29.979 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:30.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:30.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:30.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:30.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:30.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:30.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:30.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:30.431 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:45:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:45:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:30.482 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:30.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:30.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:30.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:30.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:30.796 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.827 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:30.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:30.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:30.928 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:31.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:31.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:31 smithi003 ceph-mon[104007]: pgmap v660: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 27065/147747 objects degraded (18.318%); 9799/147747 objects misplaced (6.632%); 14 MiB/s, 216 objects/s recovering 2024-09-13T14:45:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/65973564' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:45:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:31 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:45:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 27065/147747 objects degraded (18.318%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:31 smithi003 ceph-mon[107278]: pgmap v660: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 27065/147747 objects degraded (18.318%); 9799/147747 objects misplaced (6.632%); 14 MiB/s, 216 objects/s recovering 2024-09-13T14:45:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/65973564' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:45:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:31 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:45:31.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 27065/147747 objects degraded (18.318%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:31.184 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pg_num to 11 2024-09-13T14:45:31.211 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:31.240 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:31.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:31.372 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:31.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:31 smithi019 ceph-mon[91296]: pgmap v660: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 27065/147747 objects degraded (18.318%); 9799/147747 objects misplaced (6.632%); 14 MiB/s, 216 objects/s recovering 2024-09-13T14:45:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/65973564' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:45:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:31 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:45:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 27065/147747 objects degraded (18.318%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:31.476 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:31.508 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:31.609 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:31.640 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:31.741 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:31.769 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:31.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:31.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:31.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:32.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.056 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:32.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:32.157 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:32.194 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:32.229 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:32 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T14:45:32.229 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:32 smithi019 ceph-mon[91296]: osdmap e195: 8 total, 8 up, 8 in 2024-09-13T14:45:32.229 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:32 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T14:45:32.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:32.327 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:32.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:32.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:32 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T14:45:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:32 smithi003 ceph-mon[104007]: osdmap e195: 8 total, 8 up, 8 in 2024-09-13T14:45:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:32 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T14:45:32.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:32 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T14:45:32.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:32 smithi003 ceph-mon[107278]: osdmap e195: 8 total, 8 up, 8 in 2024-09-13T14:45:32.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:32 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T14:45:32.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:32.534 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:32.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:32.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:32.722 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:32.754 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:32.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:32.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:32.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:32.912 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:45:32.912 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:45:32.912 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T14:45:32.912 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:45:32.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:33.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:33.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:33.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:33.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:33.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:33.408 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:33 smithi003 ceph-mon[104007]: pgmap v662: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 26256/145281 objects degraded (18.073%); 9799/145281 objects misplaced (6.745%); 16 MiB/s, 257 objects/s recovering 2024-09-13T14:45:33.408 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T14:45:33.408 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:33 smithi003 ceph-mon[104007]: osdmap e196: 8 total, 8 up, 8 in 2024-09-13T14:45:33.408 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:33 smithi003 ceph-mon[107278]: pgmap v662: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 26256/145281 objects degraded (18.073%); 9799/145281 objects misplaced (6.745%); 16 MiB/s, 257 objects/s recovering 2024-09-13T14:45:33.408 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T14:45:33.408 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:33 smithi003 ceph-mon[107278]: osdmap e196: 8 total, 8 up, 8 in 2024-09-13T14:45:33.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:33.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:33.574 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:33 smithi019 ceph-mon[91296]: pgmap v662: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 26256/145281 objects degraded (18.073%); 9799/145281 objects misplaced (6.745%); 16 MiB/s, 257 objects/s recovering 2024-09-13T14:45:33.574 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T14:45:33.574 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:33 smithi019 ceph-mon[91296]: osdmap e196: 8 total, 8 up, 8 in 2024-09-13T14:45:33.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:33.612 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:45:33.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:33.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:33.757 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:33.796 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:33.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:33.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:34.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:34.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:34.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:34.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:34.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:34.380 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:34 smithi003 ceph-mon[107278]: osdmap e197: 8 total, 8 up, 8 in 2024-09-13T14:45:34.380 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:34 smithi003 ceph-mon[107278]: pgmap v665: 109 pgs: 10 unknown, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 2.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 26372/142398 objects degraded (18.520%); 9596/142398 objects misplaced (6.739%); 11 MiB/s, 174 objects/s recovering 2024-09-13T14:45:34.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:34 smithi003 ceph-mon[104007]: osdmap e197: 8 total, 8 up, 8 in 2024-09-13T14:45:34.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:34 smithi003 ceph-mon[104007]: pgmap v665: 109 pgs: 10 unknown, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 2.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 26372/142398 objects degraded (18.520%); 9596/142398 objects misplaced (6.739%); 11 MiB/s, 174 objects/s recovering 2024-09-13T14:45:34.380 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:45:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:45:34] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:45:34.468 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:34.499 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:34.517 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:34 smithi019 ceph-mon[91296]: osdmap e197: 8 total, 8 up, 8 in 2024-09-13T14:45:34.518 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:34 smithi019 ceph-mon[91296]: pgmap v665: 109 pgs: 10 unknown, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 2.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 26372/142398 objects degraded (18.520%); 9596/142398 objects misplaced (6.739%); 11 MiB/s, 174 objects/s recovering 2024-09-13T14:45:34.601 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.677 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:34.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:34.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:34.812 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:34.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:34.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.947 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:34.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:35.036 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:35.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:35.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:35.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:35.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:35.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:35.357 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:35 smithi019 ceph-mon[91296]: 1.3 scrub starts 2024-09-13T14:45:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:35 smithi019 ceph-mon[91296]: 1.1 scrub starts 2024-09-13T14:45:35.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:35.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:35.611 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:35.642 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:35.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:35.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:35 smithi003 ceph-mon[107278]: 1.3 scrub starts 2024-09-13T14:45:35.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:35 smithi003 ceph-mon[107278]: 1.1 scrub starts 2024-09-13T14:45:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:35 smithi003 ceph-mon[104007]: 1.3 scrub starts 2024-09-13T14:45:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:35 smithi003 ceph-mon[104007]: 1.1 scrub starts 2024-09-13T14:45:35.744 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:35.775 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:35.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:35.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:36.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:36.176 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:36.207 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:36.254 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:45:36.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:36.339 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:36 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 26372/142398 objects degraded (18.520%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:36.340 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:36 smithi019 ceph-mon[91296]: 1.0 scrub starts 2024-09-13T14:45:36.340 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:36 smithi019 ceph-mon[91296]: pgmap v666: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 25279/135348 objects degraded (18.677%); 8808/135348 objects misplaced (6.508%); 16 MiB/s, 255 objects/s recovering 2024-09-13T14:45:36.340 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:36 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3893543026' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:36.459 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:36.493 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:36.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:36.622 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:36 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 26372/142398 objects degraded (18.520%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:36.622 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:36 smithi003 ceph-mon[107278]: 1.0 scrub starts 2024-09-13T14:45:36.622 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:36 smithi003 ceph-mon[107278]: pgmap v666: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 25279/135348 objects degraded (18.677%); 8808/135348 objects misplaced (6.508%); 16 MiB/s, 255 objects/s recovering 2024-09-13T14:45:36.622 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:36 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3893543026' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:36.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:36 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 26372/142398 objects degraded (18.520%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:36.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:36 smithi003 ceph-mon[104007]: 1.0 scrub starts 2024-09-13T14:45:36.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:36 smithi003 ceph-mon[104007]: pgmap v666: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 25279/135348 objects degraded (18.677%); 8808/135348 objects misplaced (6.508%); 16 MiB/s, 255 objects/s recovering 2024-09-13T14:45:36.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:36 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3893543026' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:36.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:36.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.801 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:36.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:36.902 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:36.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.933 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:36.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:37.034 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:37.039 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:45:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:45:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:45:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:45:37.039 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:45:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.073 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:37.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:37.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:37.196 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:45:37.196 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:45:37.196 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T14:45:37.196 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2024-09-13T14:45:37.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 0 2024-09-13T14:45:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:37.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:37.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:37.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:37.365 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:37 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:37.366 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3893543026' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:37.366 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:37 smithi019 ceph-mon[91296]: osdmap e198: 8 total, 8 up, 8 in 2024-09-13T14:45:37.366 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:37 smithi019 ceph-mon[91296]: 1.7 scrub starts 2024-09-13T14:45:37.366 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:37 smithi019 ceph-mon[91296]: 1.7 scrub ok 2024-09-13T14:45:37.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:37.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3893543026' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:37.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[104007]: osdmap e198: 8 total, 8 up, 8 in 2024-09-13T14:45:37.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[104007]: 1.7 scrub starts 2024-09-13T14:45:37.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[104007]: 1.7 scrub ok 2024-09-13T14:45:37.380 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:37.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3893543026' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:37.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[107278]: osdmap e198: 8 total, 8 up, 8 in 2024-09-13T14:45:37.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[107278]: 1.7 scrub starts 2024-09-13T14:45:37.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:37 smithi003 ceph-mon[107278]: 1.7 scrub ok 2024-09-13T14:45:37.452 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:37.481 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:37.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:37.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:37.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:45:37.736 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.810 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:37.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:37.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:37.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:38.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:38.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:38.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:38.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:38.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:38.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:38.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:38 smithi019 ceph-mon[91296]: 1.6 scrub starts 2024-09-13T14:45:38.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:38 smithi019 ceph-mon[91296]: 1.6 scrub ok 2024-09-13T14:45:38.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:38 smithi019 ceph-mon[91296]: pgmap v668: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 25403/132483 objects degraded (19.175%); 8580/132483 objects misplaced (6.476%); 13 MiB/s, 205 objects/s recovering 2024-09-13T14:45:38.452 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.485 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:38.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:38.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:38.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:38.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:38.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:38 smithi003 ceph-mon[107278]: 1.6 scrub starts 2024-09-13T14:45:38.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:38 smithi003 ceph-mon[107278]: 1.6 scrub ok 2024-09-13T14:45:38.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:38 smithi003 ceph-mon[107278]: pgmap v668: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 25403/132483 objects degraded (19.175%); 8580/132483 objects misplaced (6.476%); 13 MiB/s, 205 objects/s recovering 2024-09-13T14:45:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:38 smithi003 ceph-mon[104007]: 1.6 scrub starts 2024-09-13T14:45:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:38 smithi003 ceph-mon[104007]: 1.6 scrub ok 2024-09-13T14:45:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:38 smithi003 ceph-mon[104007]: pgmap v668: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 25403/132483 objects degraded (19.175%); 8580/132483 objects misplaced (6.476%); 13 MiB/s, 205 objects/s recovering 2024-09-13T14:45:38.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:38.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:38.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:39.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:39.051 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:39.058 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:45:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:39.061 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:45:39.105 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:39.142 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:45:39.142 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:39.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:39.193 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:45:39.194 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:39.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:39.212 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:45:39.212 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:39.274 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:45:39.274 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:39.305 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:39.307 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:45:39.307 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:39.307 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:39 smithi003 ceph-mon[107278]: 1.4 scrub starts 2024-09-13T14:45:39.307 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:39 smithi003 ceph-mon[107278]: 1.4 scrub ok 2024-09-13T14:45:39.307 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/738059385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:39.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:39 smithi003 ceph-mon[104007]: 1.4 scrub starts 2024-09-13T14:45:39.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:39 smithi003 ceph-mon[104007]: 1.4 scrub ok 2024-09-13T14:45:39.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/738059385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:39.339 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:45:39.339 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:39.340 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:39.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:39.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:39.590 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:39.623 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:39.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:39 smithi019 ceph-mon[91296]: 1.4 scrub starts 2024-09-13T14:45:39.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:39 smithi019 ceph-mon[91296]: 1.4 scrub ok 2024-09-13T14:45:39.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/738059385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:39.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:39.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:39.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:40.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:40.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:40.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:40.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:40.298 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:40.328 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:40.430 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:40.456 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:40.558 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:40.582 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:45:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:45:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:40.582 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:40 smithi019 ceph-mon[91296]: pgmap v669: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 25193/127902 objects degraded (19.697%); 7725/127902 objects misplaced (6.040%); 15 MiB/s, 238 objects/s recovering 2024-09-13T14:45:40.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:40 smithi019 ceph-mon[91296]: 1.8 scrub starts 2024-09-13T14:45:40.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:40 smithi019 ceph-mon[91296]: 1.8 scrub ok 2024-09-13T14:45:40.593 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:40.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:40 smithi003 ceph-mon[107278]: pgmap v669: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 25193/127902 objects degraded (19.697%); 7725/127902 objects misplaced (6.040%); 15 MiB/s, 238 objects/s recovering 2024-09-13T14:45:40.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:40 smithi003 ceph-mon[107278]: 1.8 scrub starts 2024-09-13T14:45:40.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:40 smithi003 ceph-mon[107278]: 1.8 scrub ok 2024-09-13T14:45:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:40 smithi003 ceph-mon[104007]: pgmap v669: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 25193/127902 objects degraded (19.697%); 7725/127902 objects misplaced (6.040%); 15 MiB/s, 238 objects/s recovering 2024-09-13T14:45:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:40 smithi003 ceph-mon[104007]: 1.8 scrub starts 2024-09-13T14:45:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:40 smithi003 ceph-mon[104007]: 1.8 scrub ok 2024-09-13T14:45:40.695 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:40.733 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:40.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:45:40.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:40.865 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:40.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:41.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:41.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:41.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:41.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:41.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:41.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:41.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:41.478 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 25193/127902 objects degraded (19.697%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:41.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:41 smithi003 ceph-mon[107278]: 1.9 scrub starts 2024-09-13T14:45:41.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:41 smithi003 ceph-mon[107278]: 1.9 scrub ok 2024-09-13T14:45:41.479 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 25193/127902 objects degraded (19.697%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:41.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:41 smithi003 ceph-mon[104007]: 1.9 scrub starts 2024-09-13T14:45:41.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:41 smithi003 ceph-mon[104007]: 1.9 scrub ok 2024-09-13T14:45:41.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:41.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:41.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:41.650 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:41.689 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 25193/127902 objects degraded (19.697%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:41.689 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:41 smithi019 ceph-mon[91296]: 1.9 scrub starts 2024-09-13T14:45:41.689 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:41 smithi019 ceph-mon[91296]: 1.9 scrub ok 2024-09-13T14:45:41.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:41.795 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:41.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.796 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:41.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:41.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:41.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:42.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:42.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:42.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:42.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:42.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:42.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:42 smithi003 ceph-mon[107278]: 1.2 scrub starts 2024-09-13T14:45:42.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:42 smithi003 ceph-mon[107278]: pgmap v670: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 24379/125424 objects degraded (19.437%); 7725/125424 objects misplaced (6.159%); 16 MiB/s, 257 objects/s recovering 2024-09-13T14:45:42.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:42 smithi003 ceph-mon[107278]: 1.2 scrub ok 2024-09-13T14:45:42.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:42 smithi003 ceph-mon[104007]: 1.2 scrub starts 2024-09-13T14:45:42.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:42 smithi003 ceph-mon[104007]: pgmap v670: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 24379/125424 objects degraded (19.437%); 7725/125424 objects misplaced (6.159%); 16 MiB/s, 257 objects/s recovering 2024-09-13T14:45:42.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:42 smithi003 ceph-mon[104007]: 1.2 scrub ok 2024-09-13T14:45:42.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:42.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:42.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:42.665 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:42 smithi019 ceph-mon[91296]: 1.2 scrub starts 2024-09-13T14:45:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:42 smithi019 ceph-mon[91296]: pgmap v670: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 24379/125424 objects degraded (19.437%); 7725/125424 objects misplaced (6.159%); 16 MiB/s, 257 objects/s recovering 2024-09-13T14:45:42.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:42 smithi019 ceph-mon[91296]: 1.2 scrub ok 2024-09-13T14:45:42.767 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:42.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:45:42.802 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:42.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:42.904 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:42.934 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:42.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:43.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:43.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:43.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:43.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:43.359 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:43.471 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.1 primary-affinity to 0 (802) 2024-09-13T14:45:43.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:43.562 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1149837772' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-13T14:45:43.562 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-13T14:45:43.563 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[107278]: 1.a scrub starts 2024-09-13T14:45:43.563 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[107278]: 1.a scrub ok 2024-09-13T14:45:43.563 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:43.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1149837772' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-13T14:45:43.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-13T14:45:43.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[104007]: 1.a scrub starts 2024-09-13T14:45:43.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[104007]: 1.a scrub ok 2024-09-13T14:45:43.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:43.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:43.621 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:43.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1149837772' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-13T14:45:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:43 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-13T14:45:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:43 smithi019 ceph-mon[91296]: 1.a scrub starts 2024-09-13T14:45:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:43 smithi019 ceph-mon[91296]: 1.a scrub ok 2024-09-13T14:45:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:43.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:43.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:43.849 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:43.874 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:43.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:43.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:44.100 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:44.124 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:44.225 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:44.248 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:44.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:44.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:44.373 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:44.421 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:45:44] "GET /metrics HTTP/1.1" 200 36558 "" "Prometheus/2.51.0" 2024-09-13T14:45:44.474 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:44.505 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:44.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:44.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:44.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:44 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-13T14:45:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:44 smithi019 ceph-mon[91296]: osdmap e199: 8 total, 8 up, 8 in 2024-09-13T14:45:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:44 smithi019 ceph-mon[91296]: 1.5 scrub starts 2024-09-13T14:45:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:44 smithi019 ceph-mon[91296]: 1.5 scrub ok 2024-09-13T14:45:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:44 smithi019 ceph-mon[91296]: pgmap v672: 109 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 521 op/s; 24448/122676 objects degraded (19.929%); 7538/122676 objects misplaced (6.145%); 9.8 MiB/s, 157 objects/s recovering 2024-09-13T14:45:44.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-13T14:45:44.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[107278]: osdmap e199: 8 total, 8 up, 8 in 2024-09-13T14:45:44.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[107278]: 1.5 scrub starts 2024-09-13T14:45:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[107278]: 1.5 scrub ok 2024-09-13T14:45:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[107278]: pgmap v672: 109 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 521 op/s; 24448/122676 objects degraded (19.929%); 7538/122676 objects misplaced (6.145%); 9.8 MiB/s, 157 objects/s recovering 2024-09-13T14:45:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-13T14:45:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[104007]: osdmap e199: 8 total, 8 up, 8 in 2024-09-13T14:45:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[104007]: 1.5 scrub starts 2024-09-13T14:45:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[104007]: 1.5 scrub ok 2024-09-13T14:45:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:44 smithi003 ceph-mon[104007]: pgmap v672: 109 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 521 op/s; 24448/122676 objects degraded (19.929%); 7538/122676 objects misplaced (6.145%); 9.8 MiB/s, 157 objects/s recovering 2024-09-13T14:45:44.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:44.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:44.859 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:44.885 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:44.987 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:45.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:45.112 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:45.136 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:45.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:45.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:45.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:45.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:45.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:45.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:45.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:45.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:45.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:45 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:45:45.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:45 smithi019 ceph-mon[91296]: osdmap e200: 8 total, 8 up, 8 in 2024-09-13T14:45:45.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:45 smithi019 ceph-mon[91296]: 1.3 scrub starts 2024-09-13T14:45:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:45 smithi019 ceph-mon[91296]: 1.3 scrub ok 2024-09-13T14:45:45.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:45 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:45:45.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:45 smithi003 ceph-mon[107278]: osdmap e200: 8 total, 8 up, 8 in 2024-09-13T14:45:45.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:45 smithi003 ceph-mon[107278]: 1.3 scrub starts 2024-09-13T14:45:45.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:45 smithi003 ceph-mon[107278]: 1.3 scrub ok 2024-09-13T14:45:45.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:45 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:45:45.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:45 smithi003 ceph-mon[104007]: osdmap e200: 8 total, 8 up, 8 in 2024-09-13T14:45:45.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:45 smithi003 ceph-mon[104007]: 1.3 scrub starts 2024-09-13T14:45:45.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:45 smithi003 ceph-mon[104007]: 1.3 scrub ok 2024-09-13T14:45:45.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:45.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:45.910 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:45.949 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:46.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:46.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:46.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:46.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:46.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:46.440 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:46.488 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:46.590 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:46.625 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:46.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[107278]: osdmap e201: 8 total, 8 up, 8 in 2024-09-13T14:45:46.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 24448/122676 objects degraded (19.929%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:46.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[107278]: 1.1 scrub starts 2024-09-13T14:45:46.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[107278]: 1.1 scrub ok 2024-09-13T14:45:46.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[107278]: pgmap v675: 109 pgs: 2 active+recovering+undersized+remapped, 7 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 17305/116247 objects degraded (14.886%); 8652/116247 objects misplaced (7.443%); 18 MiB/s, 291 objects/s recovering 2024-09-13T14:45:46.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[104007]: osdmap e201: 8 total, 8 up, 8 in 2024-09-13T14:45:46.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 24448/122676 objects degraded (19.929%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:46.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[104007]: 1.1 scrub starts 2024-09-13T14:45:46.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[104007]: 1.1 scrub ok 2024-09-13T14:45:46.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:46 smithi003 ceph-mon[104007]: pgmap v675: 109 pgs: 2 active+recovering+undersized+remapped, 7 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 17305/116247 objects degraded (14.886%); 8652/116247 objects misplaced (7.443%); 18 MiB/s, 291 objects/s recovering 2024-09-13T14:45:46.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:46.751 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:46.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:46.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:46.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:46 smithi019 ceph-mon[91296]: osdmap e201: 8 total, 8 up, 8 in 2024-09-13T14:45:46.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 24448/122676 objects degraded (19.929%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:46.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:46 smithi019 ceph-mon[91296]: 1.1 scrub starts 2024-09-13T14:45:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:46 smithi019 ceph-mon[91296]: 1.1 scrub ok 2024-09-13T14:45:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:46 smithi019 ceph-mon[91296]: pgmap v675: 109 pgs: 2 active+recovering+undersized+remapped, 7 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 17305/116247 objects degraded (14.886%); 8652/116247 objects misplaced (7.443%); 18 MiB/s, 291 objects/s recovering 2024-09-13T14:45:47.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:47.042 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:47.144 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:47.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:47.278 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:47.308 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:47.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:47.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:47.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:47.603 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:47 smithi003 ceph-mon[104007]: 1.0 scrub starts 2024-09-13T14:45:47.603 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:47 smithi003 ceph-mon[104007]: 1.0 scrub ok 2024-09-13T14:45:47.604 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:47 smithi003 ceph-mon[107278]: 1.0 scrub starts 2024-09-13T14:45:47.604 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:47 smithi003 ceph-mon[107278]: 1.0 scrub ok 2024-09-13T14:45:47.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:47.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:47.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:47.895 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:47.930 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:47.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:47 smithi019 ceph-mon[91296]: 1.0 scrub starts 2024-09-13T14:45:47.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:47 smithi019 ceph-mon[91296]: 1.0 scrub ok 2024-09-13T14:45:48.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:48.118 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:48.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:48.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:48.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:48.463 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:45:48.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:48.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:48 smithi003 ceph-mon[104007]: pgmap v676: 109 pgs: 2 active+recovering+undersized+remapped, 7 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 530 op/s; 17305/115659 objects degraded (14.962%); 8652/115659 objects misplaced (7.481%); 14 MiB/s, 223 objects/s recovering 2024-09-13T14:45:48.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/762730508' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:48.521 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:48 smithi003 ceph-mon[107278]: pgmap v676: 109 pgs: 2 active+recovering+undersized+remapped, 7 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 530 op/s; 17305/115659 objects degraded (14.962%); 8652/115659 objects misplaced (7.481%); 14 MiB/s, 223 objects/s recovering 2024-09-13T14:45:48.521 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/762730508' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:48.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:48.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:48 smithi019 ceph-mon[91296]: pgmap v676: 109 pgs: 2 active+recovering+undersized+remapped, 7 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 530 op/s; 17305/115659 objects degraded (14.962%); 8652/115659 objects misplaced (7.481%); 14 MiB/s, 223 objects/s recovering 2024-09-13T14:45:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/762730508' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:48.684 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:48.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:48.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:48.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:48.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:49.060 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:49.094 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:49.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:49.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:49.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:49.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:49.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:49.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:49.617 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:49.660 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:49.765 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:49.798 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:49.806 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:49 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:49.806 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/762730508' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:49.806 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:49 smithi019 ceph-mon[91296]: osdmap e202: 8 total, 8 up, 8 in 2024-09-13T14:45:49.845 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:49 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:49.845 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/762730508' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:49.845 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:49 smithi003 ceph-mon[104007]: osdmap e202: 8 total, 8 up, 8 in 2024-09-13T14:45:49.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:49 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:49.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/762730508' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:49.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:49 smithi003 ceph-mon[107278]: osdmap e202: 8 total, 8 up, 8 in 2024-09-13T14:45:49.901 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.937 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:49.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:50.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:50.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:50.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:50.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:50.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.376 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:50.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:50.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:50.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:50.635 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.665 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:50.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:50.684 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:45:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:45:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:50.685 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:50 smithi019 ceph-mon[91296]: pgmap v678: 109 pgs: 2 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 19439/110748 objects degraded (17.552%); 4872/110748 objects misplaced (4.399%); 23 MiB/s, 382 objects/s recovering 2024-09-13T14:45:50.767 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:50.804 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:50.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:50.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:50 smithi003 ceph-mon[104007]: pgmap v678: 109 pgs: 2 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 19439/110748 objects degraded (17.552%); 4872/110748 objects misplaced (4.399%); 23 MiB/s, 382 objects/s recovering 2024-09-13T14:45:50.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:50 smithi003 ceph-mon[107278]: pgmap v678: 109 pgs: 2 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 19439/110748 objects degraded (17.552%); 4872/110748 objects misplaced (4.399%); 23 MiB/s, 382 objects/s recovering 2024-09-13T14:45:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.945 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:50.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:51.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:51.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:51.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:51.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:51.221 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:51.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:51.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:51.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.553 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:51.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:51.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:51.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:51 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-13T14:45:51.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 19439/110748 objects degraded (17.552%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:51 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-13T14:45:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 19439/110748 objects degraded (17.552%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:51.694 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:51.736 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:45:51.736 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:45:51.737 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:45:51.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:45:51.789 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:51 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-13T14:45:51.789 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 19439/110748 objects degraded (17.552%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:51.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:51.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:51.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:45:51.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:51.986 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:52.088 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:52.123 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:52.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:52.243 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:52.244 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:52.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:52.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:52.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:52.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:52.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:52.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:52.681 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:45:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:52.696 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:52.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:52.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:52 smithi003 ceph-mon[104007]: pgmap v679: 109 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 446 op/s; 22563/108186 objects degraded (20.856%); 4272/108186 objects misplaced (3.949%); 19 MiB/s, 309 objects/s recovering 2024-09-13T14:45:52.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:52.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:52.725 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:52 smithi003 ceph-mon[107278]: pgmap v679: 109 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 446 op/s; 22563/108186 objects degraded (20.856%); 4272/108186 objects misplaced (3.949%); 19 MiB/s, 309 objects/s recovering 2024-09-13T14:45:52.725 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:52.725 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:52.725 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:45:52.725 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:52.773 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:45:52.774 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:52.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:52.815 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:45:52.816 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:52.829 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:52 smithi019 ceph-mon[91296]: pgmap v679: 109 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 446 op/s; 22563/108186 objects degraded (20.856%); 4272/108186 objects misplaced (3.949%); 19 MiB/s, 309 objects/s recovering 2024-09-13T14:45:52.829 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:52.829 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:52.848 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:45:52.848 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:52.882 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:45:52.883 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:52.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:52.917 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:45:52.917 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:52.948 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:45:52.948 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:52.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:53.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:53.099 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:53.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:45:53.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:53.228 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:53.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:53.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:53.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:53.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:53.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:53.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:53.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.832 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:53.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:53.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1931264470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:53.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:53.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:53.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1931264470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:53.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:53.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:53.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1931264470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:53.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:53.967 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:54.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:54.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:54.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:45:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:45:54] "GET /metrics HTTP/1.1" 200 36558 "" "Prometheus/2.51.0" 2024-09-13T14:45:54.212 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:54.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:54.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:54.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:54.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:54.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:54.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:54.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:54.779 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:54.807 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:54.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:54.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:54.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:54 smithi019 ceph-mon[91296]: pgmap v680: 109 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 372 op/s; 22451/106932 objects degraded (20.996%); 4272/106932 objects misplaced (3.995%); 14 MiB/s, 225 objects/s recovering 2024-09-13T14:45:54.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:45:54.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:45:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:54.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:54 smithi003 ceph-mon[104007]: pgmap v680: 109 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 372 op/s; 22451/106932 objects degraded (20.996%); 4272/106932 objects misplaced (3.995%); 14 MiB/s, 225 objects/s recovering 2024-09-13T14:45:54.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:45:54.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:45:54.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:54 smithi003 ceph-mon[107278]: pgmap v680: 109 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 372 op/s; 22451/106932 objects degraded (20.996%); 4272/106932 objects misplaced (3.995%); 14 MiB/s, 225 objects/s recovering 2024-09-13T14:45:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:45:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:45:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:54.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:55.049 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:55.079 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:55.181 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:55.212 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:55.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:55.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:55.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:55.501 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:55.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:55.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:55.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:55.785 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:55.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:55.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:55.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:55.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 22451/106932 objects degraded (20.996%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:55.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 22451/106932 objects degraded (20.996%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:55.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 22451/106932 objects degraded (20.996%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:56.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:56.061 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:56.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:56.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:56.291 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:56.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:56.422 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:56.446 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:56.547 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:56.582 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:56.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:56.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:56.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:56.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:56.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:56 smithi019 ceph-mon[91296]: pgmap v681: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 21316/97398 objects degraded (21.885%); 2924/97398 objects misplaced (3.002%); 19 MiB/s, 326 objects/s recovering 2024-09-13T14:45:56.932 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:45:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=cleanup t=2024-09-13T14:45:56.525321837Z level=info msg="Completed cleanup jobs" duration=119.30816ms 2024-09-13T14:45:56.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:56 smithi003 ceph-mon[104007]: pgmap v681: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 21316/97398 objects degraded (21.885%); 2924/97398 objects misplaced (3.002%); 19 MiB/s, 326 objects/s recovering 2024-09-13T14:45:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:56 smithi003 ceph-mon[107278]: pgmap v681: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 21316/97398 objects degraded (21.885%); 2924/97398 objects misplaced (3.002%); 19 MiB/s, 326 objects/s recovering 2024-09-13T14:45:56.975 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:57.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:57.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:57.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:57.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:57.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:57.286 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:57.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:57.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:57.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:57.553 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:57.655 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:57.688 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:57.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:57.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:57.834 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:57.936 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:57.970 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:58.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:58.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:58.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:58.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:58.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:58.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:58.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:58.532 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:58.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:58.581 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:58.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:58.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:58.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:58.849 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:58.861 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:58 smithi003 ceph-mon[104007]: pgmap v682: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 21316/96330 objects degraded (22.128%); 2924/96330 objects misplaced (3.035%); 19 MiB/s, 326 objects/s recovering 2024-09-13T14:45:58.861 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:58.861 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:58.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:58 smithi003 ceph-mon[107278]: pgmap v682: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 21316/96330 objects degraded (22.128%); 2924/96330 objects misplaced (3.035%); 19 MiB/s, 326 objects/s recovering 2024-09-13T14:45:58.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:58.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:58.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:59.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:59.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:59.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:59.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:58 smithi019 ceph-mon[91296]: pgmap v682: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 21316/96330 objects degraded (22.128%); 2924/96330 objects misplaced (3.035%); 19 MiB/s, 326 objects/s recovering 2024-09-13T14:45:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:45:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:59.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:59.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:59.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:59.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:59.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:59.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:59.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:59.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:45:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:59.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:59.836 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:45:59.869 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:45:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:59.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:00.004 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:00.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:00.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:00.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/519734469' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:45:59 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:00.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/519734469' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:45:59 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:00.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/519734469' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:45:59 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:00.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:00.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:00.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:00.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:00.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:00.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:00.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:00.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:00.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:46:00.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:46:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:46:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:00.693 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:00.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:00.795 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:00.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:00.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:00.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:01.058 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:01.080 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:01.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:01.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:00 smithi019 ceph-mon[91296]: pgmap v683: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 1.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 20752/89268 objects degraded (23.247%); 1604/89268 objects misplaced (1.797%); 22 MiB/s, 390 objects/s recovering 2024-09-13T14:46:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:00 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/519734469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:00 smithi019 ceph-mon[91296]: osdmap e203: 8 total, 8 up, 8 in 2024-09-13T14:46:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 20752/89268 objects degraded (23.247%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:00 smithi019 ceph-mon[91296]: osdmap e204: 8 total, 8 up, 8 in 2024-09-13T14:46:01.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[104007]: pgmap v683: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 1.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 20752/89268 objects degraded (23.247%); 1604/89268 objects misplaced (1.797%); 22 MiB/s, 390 objects/s recovering 2024-09-13T14:46:01.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/519734469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:01.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[104007]: osdmap e203: 8 total, 8 up, 8 in 2024-09-13T14:46:01.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 20752/89268 objects degraded (23.247%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:01.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[104007]: osdmap e204: 8 total, 8 up, 8 in 2024-09-13T14:46:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[107278]: pgmap v683: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 1.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 20752/89268 objects degraded (23.247%); 1604/89268 objects misplaced (1.797%); 22 MiB/s, 390 objects/s recovering 2024-09-13T14:46:01.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/519734469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:01.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[107278]: osdmap e203: 8 total, 8 up, 8 in 2024-09-13T14:46:01.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 20752/89268 objects degraded (23.247%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:01.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:00 smithi003 ceph-mon[107278]: osdmap e204: 8 total, 8 up, 8 in 2024-09-13T14:46:01.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:01.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:01.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:01.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:01.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:01.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:01.694 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:01.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.725 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:01.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:01.826 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:01.853 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:01.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:01.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:02.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:02.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:02.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.262 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:02.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:02.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:02.395 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:02.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:02.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:02.522 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:02.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:02.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:02.770 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:02.801 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:02.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:02.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:02 smithi019 ceph-mon[91296]: osdmap e205: 8 total, 8 up, 8 in 2024-09-13T14:46:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:02 smithi019 ceph-mon[91296]: 9.d continuing backfill to osd.6 from (184'8344,203'13461] MIN to 203'13461 2024-09-13T14:46:02.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:02 smithi019 ceph-mon[91296]: pgmap v687: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 1.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 635 op/s; 19875/85962 objects degraded (23.121%); 1152/85962 objects misplaced (1.340%); 17 MiB/s, 305 objects/s recovering 2024-09-13T14:46:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:02 smithi003 ceph-mon[107278]: osdmap e205: 8 total, 8 up, 8 in 2024-09-13T14:46:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:02 smithi003 ceph-mon[107278]: 9.d continuing backfill to osd.6 from (184'8344,203'13461] MIN to 203'13461 2024-09-13T14:46:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:02 smithi003 ceph-mon[107278]: pgmap v687: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 1.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 635 op/s; 19875/85962 objects degraded (23.121%); 1152/85962 objects misplaced (1.340%); 17 MiB/s, 305 objects/s recovering 2024-09-13T14:46:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:02 smithi003 ceph-mon[104007]: osdmap e205: 8 total, 8 up, 8 in 2024-09-13T14:46:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:02 smithi003 ceph-mon[104007]: 9.d continuing backfill to osd.6 from (184'8344,203'13461] MIN to 203'13461 2024-09-13T14:46:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:02 smithi003 ceph-mon[104007]: pgmap v687: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 1.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 635 op/s; 19875/85962 objects degraded (23.121%); 1152/85962 objects misplaced (1.340%); 17 MiB/s, 305 objects/s recovering 2024-09-13T14:46:02.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:03.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:03.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:03.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:03.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:03.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:03.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:03.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:03.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:03.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:03.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:03.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:03.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:03.863 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:03.897 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:03.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:04.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:04.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:04.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:04.224 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:04.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:04.297 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:04.358 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:46:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:46:04] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.51.0" 2024-09-13T14:46:04.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:04.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:04.524 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:04.547 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:04.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:04.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:04.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:04.804 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:04.887 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:04 smithi003 ceph-mon[107278]: osdmap e206: 8 total, 8 up, 8 in 2024-09-13T14:46:04.887 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:04 smithi003 ceph-mon[107278]: pgmap v689: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 1.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 376 op/s; 19769/84750 objects degraded (23.326%); 1152/84750 objects misplaced (1.359%); 13 MiB/s, 206 objects/s recovering 2024-09-13T14:46:04.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:04 smithi003 ceph-mon[104007]: osdmap e206: 8 total, 8 up, 8 in 2024-09-13T14:46:04.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:04 smithi003 ceph-mon[104007]: pgmap v689: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 1.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 376 op/s; 19769/84750 objects degraded (23.326%); 1152/84750 objects misplaced (1.359%); 13 MiB/s, 206 objects/s recovering 2024-09-13T14:46:04.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:04.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:04 smithi019 ceph-mon[91296]: osdmap e206: 8 total, 8 up, 8 in 2024-09-13T14:46:04.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:04 smithi019 ceph-mon[91296]: pgmap v689: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 1.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 376 op/s; 19769/84750 objects degraded (23.326%); 1152/84750 objects misplaced (1.359%); 13 MiB/s, 206 objects/s recovering 2024-09-13T14:46:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.941 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:04.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:05.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:05.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:05.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:05.219 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:05.225 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:05.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:05.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:05.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:05.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:05.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:05.641 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:05.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:05.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:46:05.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:05.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:05.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:05.913 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:05.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:05 smithi003 ceph-mon[107278]: osdmap e207: 8 total, 8 up, 8 in 2024-09-13T14:46:05.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 19769/84750 objects degraded (23.326%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:05.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:05 smithi003 ceph-mon[104007]: osdmap e207: 8 total, 8 up, 8 in 2024-09-13T14:46:05.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 19769/84750 objects degraded (23.326%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:46:05.929 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:46:05.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:05 smithi019 ceph-mon[91296]: osdmap e207: 8 total, 8 up, 8 in 2024-09-13T14:46:05.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 19769/84750 objects degraded (23.326%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:06.016 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:06.047 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:06.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:06.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:06.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:06.314 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:06.416 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:06.450 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:06.552 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:06.591 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:06.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:06.726 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:06.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:06.829 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:06.857 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:06.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:06 smithi019 ceph-mon[91296]: pgmap v691: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 18501/77088 objects degraded (24.000%); 1615/77088 objects misplaced (2.095%); 14 MiB/s, 310 objects/s recovering 2024-09-13T14:46:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:06 smithi019 ceph-mon[91296]: from='client.46085 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:06.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:06 smithi003 ceph-mon[104007]: pgmap v691: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 18501/77088 objects degraded (24.000%); 1615/77088 objects misplaced (2.095%); 14 MiB/s, 310 objects/s recovering 2024-09-13T14:46:06.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:06 smithi003 ceph-mon[104007]: from='client.46085 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:06 smithi003 ceph-mon[107278]: pgmap v691: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 18501/77088 objects degraded (24.000%); 1615/77088 objects misplaced (2.095%); 14 MiB/s, 310 objects/s recovering 2024-09-13T14:46:06.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:06 smithi003 ceph-mon[107278]: from='client.46085 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:06.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:07.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:07.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:07.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:07.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:07.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:07.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:07.549 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:07.578 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:07.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:07.708 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:07.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:07.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:07.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:07.926 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:46:07.926 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:46:07.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:218715 9.12 9:4ee89088:::benchmark_data_smithi003_156985_object97799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.608774+0000", 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.02064363, 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020675635000000001, 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 218715 2024-09-13T14:46:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.608774+0000", 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.608774+0000", 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.608776+0000", 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5269999999999999e-06 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.608781+0000", 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7729999999999996e-06 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.608794+0000", 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3033999999999999e-05 2024-09-13T14:46:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.608802+0000", 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0190000000000006e-06 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.608867+0000", 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4988999999999997e-05 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.610390+0000", 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001522928 2024-09-13T14:46:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.627548+0000", 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017157591 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.627641+0000", 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3059999999999993e-05 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.627767+0000", 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0001261 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGRecoveryDelete(9.0 e207,179 9:024b021e:::benchmark_data_smithi003_156985_object39750:head,188'6644)", 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.613657+0000", 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.015760873000000002, 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015876218000000001, 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.613657+0000", 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.613657+0000", 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.613658+0000", 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6599999999999995e-07 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.613663+0000", 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3449999999999996e-06 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.613675+0000", 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2081e-05 2024-09-13T14:46:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.613684+0000", 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8540000000000003e-06 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.45869.0:218682 9.8 e207/179)", 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.616173+0000", 2024-09-13T14:46:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.013244557000000001, 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013402084, 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.616173+0000", 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.616173+0000", 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.616175+0000", 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.362e-06 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.616183+0000", 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5779999999999999e-06 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.616201+0000", 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7914000000000002e-05 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.616209+0000", 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3350000000000007e-06 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:218723 9.8 9.837e0b68 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.626359+0000", 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0030586049999999998, 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032591339999999999, 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 218723 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.626359+0000", 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.626359+0000", 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.626361+0000", 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.99e-06 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.626367+0000", 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3099999999999997e-06 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.626381+0000", 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3628999999999999e-05 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.626389+0000", 2024-09-13T14:46:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5889999999999997e-06 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.45869.0:218695 9.18 e207/179)", 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.624188+0000", 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0052295350000000004, 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054791390000000001, 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.624188+0000", 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.624188+0000", 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.624189+0000", 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0130000000000001e-06 2024-09-13T14:46:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.624197+0000", 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8310000000000001e-06 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.627831+0000", 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036341469999999999 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.627839+0000", 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5619999999999998e-06 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.45869.0:218708 9.15 e207/206)", 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.629176+0000", 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00024220700000000001, 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053098900000000005, 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.629176+0000", 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.629176+0000", 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.629176+0000", 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8599999999999997e-07 2024-09-13T14:46:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.629181+0000", 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8960000000000002e-06 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.629192+0000", 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1087000000000001e-05 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.629199+0000", 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5570000000000001e-06 2024-09-13T14:46:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:218682 9.8 9:11f6768d:::benchmark_data_smithi003_156985_object97766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.565875+0000", 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.063542812000000004, 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.063876825999999998, 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 218682 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.565875+0000", 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.565875+0000", 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.565876+0000", 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.511e-06 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.565883+0000", 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3749999999999999e-06 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.565899+0000", 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6435000000000001e-05 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.565907+0000", 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0290000000000005e-06 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.565966+0000", 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8891000000000002e-05 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:46:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.575730+0000", 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0097637409999999994 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.612230+0000", 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036500013999999997 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.612492+0000", 2024-09-13T14:46:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000261609 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.613352+0000", 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00086089999999999995 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:218695 9.18 9:1e85a4a2:::benchmark_data_smithi003_156985_object97779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.581446+0000", 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.047971715999999998, 2024-09-13T14:46:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.048383061999999998, 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 218695 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.581446+0000", 2024-09-13T14:46:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.581446+0000", 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.581448+0000", 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5650000000000001e-06 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.581455+0000", 2024-09-13T14:46:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7100000000000007e-06 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.581476+0000", 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0865000000000001e-05 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.581485+0000", 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3049999999999993e-06 2024-09-13T14:46:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.581611+0000", 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000126983 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.582412+0000", 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00080071199999999997 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.616265+0000", 2024-09-13T14:46:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033853016 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.616553+0000", 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028824800000000001 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.618517+0000", 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019631100000000001 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:218697 9.8 9:17c4ba45:::benchmark_data_smithi003_156985_object97781:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.582254+0000", 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.047163736999999997, 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.047662989000000003, 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 218697 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.582254+0000", 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.582254+0000", 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.582255+0000", 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6899999999999996e-07 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.582259+0000", 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2899999999999996e-06 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.582271+0000", 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1647999999999999e-05 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.582278+0000", 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2949999999999996e-06 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.582450+0000", 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000171626 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.583160+0000", 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070974799999999995 2024-09-13T14:46:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.601899+0000", 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018739116 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.601987+0000", 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8215999999999997e-05 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.602514+0000", 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052736600000000003 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.611964+0000", 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094499630000000005 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:218708 9.15 9:ab9968a6:::benchmark_data_smithi003_156985_object97792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.604183+0000", 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.025234289999999999, 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025816342999999999, 2024-09-13T14:46:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 218708 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604183+0000", 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604183+0000", 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604185+0000", 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.792e-06 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604191+0000", 2024-09-13T14:46:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2179999999999996e-06 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604237+0000", 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5389000000000002e-05 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604247+0000", 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6369999999999994e-06 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604305+0000", 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8600999999999997e-05 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.607493+0000", 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031878850000000001 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.608073+0000", 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058035499999999996 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.616920+0000", 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0088467830000000004 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:218712 9.8 9.983b50a8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:07.604271+0000", 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.025147098999999999, 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025797256000000001, 2024-09-13T14:46:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 218712 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604271+0000", 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604271+0000", 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604271+0000", 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2900000000000003e-07 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604275+0000", 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.399e-06 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604284+0000", 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1630000000000002e-06 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:07.604290+0000", 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0850000000000002e-06 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:07.973 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.973 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:07.973 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-13T14:46:07.973 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 11 2024-09-13T14:46:07.973 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:46:07.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.007 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:08.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:08.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:08.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:08.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:08.193 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":691,"stamp":"2024-09-13T14:46:05.690668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1682839525,"num_objects":25696,"num_object_clones":0,"num_object_copies":77088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18501,"num_objects_misplaced":1615,"num_objects_unfound":0,"num_objects_dirty":25696,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188512,"num_write_kb":5966049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45032,"num_bytes_recovered":2890137782,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":173024,"ondisk_log_size":173024,"up":327,"acting":321,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":372,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14813672,"kb_used_data":9489820,"kb_used_omap":29,"kb_used_meta":5323682,"kb_avail":735147544,"statfs":{"total":767960285184,"available":752791085056,"internally_reserved":0,"allocated":9717575680,"data_stored":9709075264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":5451450894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,5,5,3,3,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-193855488,"num_objects":-2958,"num_object_clones":0,"num_object_copies":-8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1374,"num_objects_misplaced":463,"num_objects_unfound":0,"num_objects_dirty":-2958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2958,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1577,"num_bytes_recovered":75300864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.079275"},"pg_stats":[{"pgid":"9.19","version":"207'8366","reported_seq":8525,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.800236+0000","last_change":"2024-09-13T14:42:18.899519+0000","last_active":"2024-09-13T14:46:04.800236+0000","last_peered":"2024-09-13T14:46:04.800236+0000","last_clean":"2024-09-13T14:46:04.800236+0000","last_became_active":"2024-09-13T14:42:15.129169+0000","last_became_peered":"2024-09-13T14:42:15.129169+0000","last_unstale":"2024-09-13T14:46:04.800236+0000","last_undegraded":"2024-09-13T14:46:04.800236+0000","last_fullsized":"2024-09-13T14:46:04.800236+0000","mapping_epoch":159,"log_start":"157'1778","ondisk_log_start":"157'1778","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3130","last_scrub_stamp":"2024-09-13T14:42:18.899277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:18.899277+0000","objects_scrubbed":1667,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:03.200084+0000","scrub_duration":425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54591488,"num_objects":833,"num_object_clones":0,"num_object_copies":2499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5252,"num_write_kb":141376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":219,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.615389+0000","last_change":"2024-09-13T14:43:19.875328+0000","last_active":"2024-09-13T14:46:01.615389+0000","last_peered":"2024-09-13T14:46:01.615389+0000","last_clean":"2024-09-13T14:46:01.615389+0000","last_became_active":"2024-09-13T14:43:19.874507+0000","last_became_peered":"2024-09-13T14:43:19.874507+0000","last_unstale":"2024-09-13T14:46:01.615389+0000","last_undegraded":"2024-09-13T14:46:01.615389+0000","last_fullsized":"2024-09-13T14:46:01.615389+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:07.089107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":172,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620073+0000","last_change":"2024-09-13T14:35:30.492165+0000","last_active":"2024-09-13T14:46:01.620073+0000","last_peered":"2024-09-13T14:46:01.620073+0000","last_clean":"2024-09-13T14:46:01.620073+0000","last_became_active":"2024-09-13T14:35:30.491844+0000","last_became_peered":"2024-09-13T14:35:30.491844+0000","last_unstale":"2024-09-13T14:46:01.620073+0000","last_undegraded":"2024-09-13T14:46:01.620073+0000","last_fullsized":"2024-09-13T14:46:01.620073+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:22.876890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.18","version":"207'8297","reported_seq":17491,"reported_epoch":207,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:46:04.861274+0000","last_change":"2024-09-13T14:43:21.506884+0000","last_active":"2024-09-13T14:46:04.861274+0000","last_peered":"2024-09-13T14:46:04.861274+0000","last_clean":"2024-09-13T14:43:18.728712+0000","last_became_active":"2024-09-13T14:43:21.499842+0000","last_became_peered":"2024-09-13T14:43:21.499842+0000","last_unstale":"2024-09-13T14:46:04.861274+0000","last_undegraded":"2024-09-13T14:43:21.342550+0000","last_fullsized":"2024-09-13T14:43:21.112483+0000","mapping_epoch":179,"log_start":"157'1566","ondisk_log_start":"157'1566","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'3290","last_scrub_stamp":"2024-09-13T14:42:20.569479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:20.569479+0000","objects_scrubbed":1768,"log_size":6731,"log_dups_size":0,"ondisk_log_size":6731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":240,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49610752,"num_objects":757,"num_object_clones":0,"num_object_copies":2271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":127872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3009,"num_bytes_recovered":197197824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,1],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,1","objects":6},{"shards":"0,3,5","objects":751}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":130,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.619835+0000","last_change":"2024-09-13T14:43:19.874928+0000","last_active":"2024-09-13T14:46:01.619835+0000","last_peered":"2024-09-13T14:46:01.619835+0000","last_clean":"2024-09-13T14:46:01.619835+0000","last_became_active":"2024-09-13T14:43:19.874518+0000","last_became_peered":"2024-09-13T14:43:19.874518+0000","last_unstale":"2024-09-13T14:46:01.619835+0000","last_undegraded":"2024-09-13T14:46:01.619835+0000","last_fullsized":"2024-09-13T14:46:01.619835+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:15:22.392336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":293,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.636720+0000","last_change":"2024-09-13T14:43:19.870143+0000","last_active":"2024-09-13T14:46:04.636720+0000","last_peered":"2024-09-13T14:46:04.636720+0000","last_clean":"2024-09-13T14:46:04.636720+0000","last_became_active":"2024-09-13T14:43:19.869829+0000","last_became_peered":"2024-09-13T14:43:19.869829+0000","last_unstale":"2024-09-13T14:46:04.636720+0000","last_undegraded":"2024-09-13T14:46:04.636720+0000","last_fullsized":"2024-09-13T14:46:04.636720+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.422457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":294,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.636133+0000","last_change":"2024-09-13T14:43:19.871195+0000","last_active":"2024-09-13T14:46:04.636133+0000","last_peered":"2024-09-13T14:46:04.636133+0000","last_clean":"2024-09-13T14:46:04.636133+0000","last_became_active":"2024-09-13T14:43:19.870496+0000","last_became_peered":"2024-09-13T14:43:19.870496+0000","last_unstale":"2024-09-13T14:46:04.636133+0000","last_undegraded":"2024-09-13T14:46:04.636133+0000","last_fullsized":"2024-09-13T14:46:04.636133+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:25.646089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":153,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620095+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:46:01.620095+0000","last_peered":"2024-09-13T14:46:01.620095+0000","last_clean":"2024-09-13T14:46:01.620095+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:46:01.620095+0000","last_undegraded":"2024-09-13T14:46:01.620095+0000","last_fullsized":"2024-09-13T14:46:01.620095+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":245,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.615024+0000","last_change":"2024-09-13T14:43:19.872044+0000","last_active":"2024-09-13T14:46:01.615024+0000","last_peered":"2024-09-13T14:46:01.615024+0000","last_clean":"2024-09-13T14:46:01.615024+0000","last_became_active":"2024-09-13T14:43:19.870646+0000","last_became_peered":"2024-09-13T14:43:19.870646+0000","last_unstale":"2024-09-13T14:46:01.615024+0000","last_undegraded":"2024-09-13T14:46:01.615024+0000","last_fullsized":"2024-09-13T14:46:01.615024+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:33.158472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":197,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620635+0000","last_change":"2024-09-13T14:43:19.888180+0000","last_active":"2024-09-13T14:46:01.620635+0000","last_peered":"2024-09-13T14:46:01.620635+0000","last_clean":"2024-09-13T14:46:01.620635+0000","last_became_active":"2024-09-13T14:43:19.875873+0000","last_became_peered":"2024-09-13T14:43:19.875873+0000","last_unstale":"2024-09-13T14:46:01.620635+0000","last_undegraded":"2024-09-13T14:46:01.620635+0000","last_fullsized":"2024-09-13T14:46:01.620635+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:47.078713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"207'8231","reported_seq":17386,"reported_epoch":207,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:46:04.884554+0000","last_change":"2024-09-13T14:43:21.358111+0000","last_active":"2024-09-13T14:46:04.884554+0000","last_peered":"2024-09-13T14:46:04.884554+0000","last_clean":"2024-09-13T14:43:18.840495+0000","last_became_active":"2024-09-13T14:43:21.306427+0000","last_became_peered":"2024-09-13T14:43:21.306427+0000","last_unstale":"2024-09-13T14:46:04.884554+0000","last_undegraded":"2024-09-13T14:43:21.093819+0000","last_fullsized":"2024-09-13T14:43:20.854552+0000","mapping_epoch":179,"log_start":"157'1505","ondisk_log_start":"157'1505","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3036","last_scrub_stamp":"2024-09-13T14:42:16.024050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.024050+0000","objects_scrubbed":1508,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47579136,"num_objects":726,"num_object_clones":0,"num_object_copies":2178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5212,"num_write_kb":141760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2997,"num_bytes_recovered":196411392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,1","objects":2},{"shards":"0,5,7","objects":724}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":55,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.619375+0000","last_change":"2024-09-13T14:43:19.883523+0000","last_active":"2024-09-13T14:46:01.619375+0000","last_peered":"2024-09-13T14:46:01.619375+0000","last_clean":"2024-09-13T14:46:01.619375+0000","last_became_active":"2024-09-13T14:43:19.871241+0000","last_became_peered":"2024-09-13T14:43:19.871241+0000","last_unstale":"2024-09-13T14:46:01.619375+0000","last_undegraded":"2024-09-13T14:46:01.619375+0000","last_fullsized":"2024-09-13T14:46:01.619375+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:04.592529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":40,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620503+0000","last_change":"2024-09-13T14:43:19.874136+0000","last_active":"2024-09-13T14:46:01.620503+0000","last_peered":"2024-09-13T14:46:01.620503+0000","last_clean":"2024-09-13T14:46:01.620503+0000","last_became_active":"2024-09-13T14:43:19.873671+0000","last_became_peered":"2024-09-13T14:43:19.873671+0000","last_unstale":"2024-09-13T14:46:01.620503+0000","last_undegraded":"2024-09-13T14:46:01.620503+0000","last_fullsized":"2024-09-13T14:46:01.620503+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:51.386822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"206'8339","reported_seq":8497,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:03.908291+0000","last_change":"2024-09-13T14:45:44.488534+0000","last_active":"2024-09-13T14:46:03.908291+0000","last_peered":"2024-09-13T14:46:03.908291+0000","last_clean":"2024-09-13T14:46:03.908291+0000","last_became_active":"2024-09-13T14:45:44.444948+0000","last_became_peered":"2024-09-13T14:45:44.444948+0000","last_unstale":"2024-09-13T14:46:03.908291+0000","last_undegraded":"2024-09-13T14:46:03.908291+0000","last_fullsized":"2024-09-13T14:46:03.908291+0000","mapping_epoch":199,"log_start":"157'2328","ondisk_log_start":"157'2328","created":152,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'2973","last_scrub_stamp":"2024-09-13T14:42:15.820869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:15.820869+0000","objects_scrubbed":1533,"log_size":6011,"log_dups_size":0,"ondisk_log_size":6011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:24.564016+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55902208,"num_objects":853,"num_object_clones":0,"num_object_copies":2559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5374,"num_write_kb":150208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":172,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620410+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:46:01.620410+0000","last_peered":"2024-09-13T14:46:01.620410+0000","last_clean":"2024-09-13T14:46:01.620410+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:46:01.620410+0000","last_undegraded":"2024-09-13T14:46:01.620410+0000","last_fullsized":"2024-09-13T14:46:01.620410+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":185,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618432+0000","last_change":"2024-09-13T14:37:08.752050+0000","last_active":"2024-09-13T14:46:01.618432+0000","last_peered":"2024-09-13T14:46:01.618432+0000","last_clean":"2024-09-13T14:46:01.618432+0000","last_became_active":"2024-09-13T14:37:08.751650+0000","last_became_peered":"2024-09-13T14:37:08.751650+0000","last_unstale":"2024-09-13T14:46:01.618432+0000","last_undegraded":"2024-09-13T14:46:01.618432+0000","last_fullsized":"2024-09-13T14:46:01.618432+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:17.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"206'8232","reported_seq":19626,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:03.997841+0000","last_change":"2024-09-13T14:44:59.050198+0000","last_active":"2024-09-13T14:46:03.997841+0000","last_peered":"2024-09-13T14:46:03.997841+0000","last_clean":"2024-09-13T14:46:03.997841+0000","last_became_active":"2024-09-13T14:44:58.997243+0000","last_became_peered":"2024-09-13T14:44:58.997243+0000","last_unstale":"2024-09-13T14:46:03.997841+0000","last_undegraded":"2024-09-13T14:46:03.997841+0000","last_fullsized":"2024-09-13T14:46:03.997841+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3303","last_scrub_stamp":"2024-09-13T14:42:19.788610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:19.788610+0000","objects_scrubbed":1743,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:24.047041+0000","scrub_duration":0.62419811800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51838976,"num_objects":791,"num_object_clones":0,"num_object_copies":2373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4950,"num_write_kb":127936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3741,"num_bytes_recovered":245170176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":168,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620374+0000","last_change":"2024-09-13T14:43:19.880218+0000","last_active":"2024-09-13T14:46:01.620374+0000","last_peered":"2024-09-13T14:46:01.620374+0000","last_clean":"2024-09-13T14:46:01.620374+0000","last_became_active":"2024-09-13T14:43:19.874769+0000","last_became_peered":"2024-09-13T14:43:19.874769+0000","last_unstale":"2024-09-13T14:46:01.620374+0000","last_undegraded":"2024-09-13T14:46:01.620374+0000","last_fullsized":"2024-09-13T14:46:01.620374+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:49.775209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":171,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620064+0000","last_change":"2024-09-13T14:43:19.860813+0000","last_active":"2024-09-13T14:46:01.620064+0000","last_peered":"2024-09-13T14:46:01.620064+0000","last_clean":"2024-09-13T14:46:01.620064+0000","last_became_active":"2024-09-13T14:43:19.860518+0000","last_became_peered":"2024-09-13T14:43:19.860518+0000","last_unstale":"2024-09-13T14:46:01.620064+0000","last_undegraded":"2024-09-13T14:46:01.620064+0000","last_fullsized":"2024-09-13T14:46:01.620064+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:21.310511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"206'8204","reported_seq":8878,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:03.643232+0000","last_change":"2024-09-13T14:43:28.958578+0000","last_active":"2024-09-13T14:46:03.643232+0000","last_peered":"2024-09-13T14:46:03.643232+0000","last_clean":"2024-09-13T14:46:03.643232+0000","last_became_active":"2024-09-13T14:43:28.955368+0000","last_became_peered":"2024-09-13T14:43:28.955368+0000","last_unstale":"2024-09-13T14:46:03.643232+0000","last_undegraded":"2024-09-13T14:46:03.643232+0000","last_fullsized":"2024-09-13T14:46:03.643232+0000","mapping_epoch":181,"log_start":"156'119","ondisk_log_start":"156'119","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3084","last_scrub_stamp":"2024-09-13T14:42:16.825170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.825170+0000","objects_scrubbed":1592,"log_size":8085,"log_dups_size":0,"ondisk_log_size":8085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:56.306613+0000","scrub_duration":0.667424664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52035584,"num_objects":794,"num_object_clones":0,"num_object_copies":2382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":138816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":126484480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":168,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.617088+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:46:01.617088+0000","last_peered":"2024-09-13T14:46:01.617088+0000","last_clean":"2024-09-13T14:46:01.617088+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:46:01.617088+0000","last_undegraded":"2024-09-13T14:46:01.617088+0000","last_fullsized":"2024-09-13T14:46:01.617088+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":287,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635530+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:46:04.635530+0000","last_peered":"2024-09-13T14:46:04.635530+0000","last_clean":"2024-09-13T14:46:04.635530+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:46:04.635530+0000","last_undegraded":"2024-09-13T14:46:04.635530+0000","last_fullsized":"2024-09-13T14:46:04.635530+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":233,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.634259+0000","last_change":"2024-09-13T14:45:40.724301+0000","last_active":"2024-09-13T14:46:04.634259+0000","last_peered":"2024-09-13T14:46:04.634259+0000","last_clean":"2024-09-13T14:46:04.634259+0000","last_became_active":"2024-09-13T14:45:33.246028+0000","last_became_peered":"2024-09-13T14:45:33.246028+0000","last_unstale":"2024-09-13T14:46:04.634259+0000","last_undegraded":"2024-09-13T14:46:04.634259+0000","last_fullsized":"2024-09-13T14:46:04.634259+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:38.017638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"205'8011","reported_seq":15564,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:02.549739+0000","last_change":"2024-09-13T14:44:21.618688+0000","last_active":"2024-09-13T14:46:02.549739+0000","last_peered":"2024-09-13T14:46:02.549739+0000","last_clean":"2024-09-13T14:46:02.549739+0000","last_became_active":"2024-09-13T14:44:21.602316+0000","last_became_peered":"2024-09-13T14:44:21.602316+0000","last_unstale":"2024-09-13T14:46:02.549739+0000","last_undegraded":"2024-09-13T14:46:02.549739+0000","last_fullsized":"2024-09-13T14:46:02.549739+0000","mapping_epoch":186,"log_start":"157'1754","ondisk_log_start":"157'1754","created":152,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'2995","last_scrub_stamp":"2024-09-13T14:42:16.817217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.817217+0000","objects_scrubbed":1525,"log_size":6257,"log_dups_size":0,"ondisk_log_size":6257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:34.225750+0000","scrub_duration":583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54788096,"num_objects":836,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":138880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3692,"num_bytes_recovered":241958912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":243,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.615122+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:46:01.615122+0000","last_peered":"2024-09-13T14:46:01.615122+0000","last_clean":"2024-09-13T14:46:01.615122+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:46:01.615122+0000","last_undegraded":"2024-09-13T14:46:01.615122+0000","last_fullsized":"2024-09-13T14:46:01.615122+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":168,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618768+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:46:01.618768+0000","last_peered":"2024-09-13T14:46:01.618768+0000","last_clean":"2024-09-13T14:46:01.618768+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:46:01.618768+0000","last_undegraded":"2024-09-13T14:46:01.618768+0000","last_fullsized":"2024-09-13T14:46:01.618768+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":233,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.634135+0000","last_change":"2024-09-13T14:45:39.694414+0000","last_active":"2024-09-13T14:46:04.634135+0000","last_peered":"2024-09-13T14:46:04.634135+0000","last_clean":"2024-09-13T14:46:04.634135+0000","last_became_active":"2024-09-13T14:45:33.242795+0000","last_became_peered":"2024-09-13T14:45:33.242795+0000","last_unstale":"2024-09-13T14:46:04.634135+0000","last_undegraded":"2024-09-13T14:46:04.634135+0000","last_fullsized":"2024-09-13T14:46:04.634135+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:11.766495+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"206'8374","reported_seq":14626,"reported_epoch":206,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:46:03.736924+0000","last_change":"2024-09-13T14:44:57.094029+0000","last_active":"2024-09-13T14:46:03.736924+0000","last_peered":"2024-09-13T14:46:03.736924+0000","last_clean":"2024-09-13T14:43:18.804350+0000","last_became_active":"2024-09-13T14:43:21.502161+0000","last_became_peered":"2024-09-13T14:43:21.502161+0000","last_unstale":"2024-09-13T14:46:03.736924+0000","last_undegraded":"2024-09-13T14:43:20.876998+0000","last_fullsized":"2024-09-13T14:43:20.853040+0000","mapping_epoch":179,"log_start":"156'355","ondisk_log_start":"156'355","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3278","last_scrub_stamp":"2024-09-13T14:42:18.806999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:18.806999+0000","objects_scrubbed":1737,"log_size":8019,"log_dups_size":0,"ondisk_log_size":8019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:59.799396+0000","scrub_duration":487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54001664,"num_objects":824,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":134208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3054,"num_bytes_recovered":139722752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"2,6","objects":46}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":179,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620226+0000","last_change":"2024-09-13T14:43:19.860723+0000","last_active":"2024-09-13T14:46:01.620226+0000","last_peered":"2024-09-13T14:46:01.620226+0000","last_clean":"2024-09-13T14:46:01.620226+0000","last_became_active":"2024-09-13T14:43:19.860401+0000","last_became_peered":"2024-09-13T14:43:19.860401+0000","last_unstale":"2024-09-13T14:46:01.620226+0000","last_undegraded":"2024-09-13T14:46:01.620226+0000","last_fullsized":"2024-09-13T14:46:01.620226+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:27.920262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":185,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.619300+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:46:01.619300+0000","last_peered":"2024-09-13T14:46:01.619300+0000","last_clean":"2024-09-13T14:46:01.619300+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:46:01.619300+0000","last_undegraded":"2024-09-13T14:46:01.619300+0000","last_fullsized":"2024-09-13T14:46:01.619300+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"206'8174","reported_seq":8247,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:03.776943+0000","last_change":"2024-09-13T14:42:15.076493+0000","last_active":"2024-09-13T14:46:03.776943+0000","last_peered":"2024-09-13T14:46:03.776943+0000","last_clean":"2024-09-13T14:46:03.776943+0000","last_became_active":"2024-09-13T14:42:15.074126+0000","last_became_peered":"2024-09-13T14:42:15.074126+0000","last_unstale":"2024-09-13T14:46:03.776943+0000","last_undegraded":"2024-09-13T14:46:03.776943+0000","last_fullsized":"2024-09-13T14:46:03.776943+0000","mapping_epoch":152,"log_start":"160'3086","ondisk_log_start":"160'3086","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54001664,"num_objects":824,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":240512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":169,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618538+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:46:01.618538+0000","last_peered":"2024-09-13T14:46:01.618538+0000","last_clean":"2024-09-13T14:46:01.618538+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:46:01.618538+0000","last_undegraded":"2024-09-13T14:46:01.618538+0000","last_fullsized":"2024-09-13T14:46:01.618538+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":178,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.619131+0000","last_change":"2024-09-13T14:43:19.869693+0000","last_active":"2024-09-13T14:46:01.619131+0000","last_peered":"2024-09-13T14:46:01.619131+0000","last_clean":"2024-09-13T14:46:01.619131+0000","last_became_active":"2024-09-13T14:43:19.869343+0000","last_became_peered":"2024-09-13T14:43:19.869343+0000","last_unstale":"2024-09-13T14:46:01.619131+0000","last_undegraded":"2024-09-13T14:46:01.619131+0000","last_fullsized":"2024-09-13T14:46:01.619131+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:37.370378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":233,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635610+0000","last_change":"2024-09-13T14:45:42.662478+0000","last_active":"2024-09-13T14:46:04.635610+0000","last_peered":"2024-09-13T14:46:04.635610+0000","last_clean":"2024-09-13T14:46:04.635610+0000","last_became_active":"2024-09-13T14:45:33.261545+0000","last_became_peered":"2024-09-13T14:45:33.261545+0000","last_unstale":"2024-09-13T14:46:04.635610+0000","last_undegraded":"2024-09-13T14:46:04.635610+0000","last_fullsized":"2024-09-13T14:46:04.635610+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:40.315726+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"205'8197","reported_seq":9179,"reported_epoch":205,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:46:03.447063+0000","last_change":"2024-09-13T14:45:45.511123+0000","last_active":"2024-09-13T14:46:03.447063+0000","last_peered":"2024-09-13T14:46:03.447063+0000","last_clean":"2024-09-13T14:43:18.778117+0000","last_became_active":"2024-09-13T14:45:45.449803+0000","last_became_peered":"2024-09-13T14:45:45.449803+0000","last_unstale":"2024-09-13T14:46:03.447063+0000","last_undegraded":"2024-09-13T14:45:45.421570+0000","last_fullsized":"2024-09-13T14:46:03.447063+0000","mapping_epoch":200,"log_start":"157'2015","ondisk_log_start":"157'2015","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3027","last_scrub_stamp":"2024-09-13T14:42:15.818474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:15.818474+0000","objects_scrubbed":1489,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55902208,"num_objects":853,"num_object_clones":0,"num_object_copies":2559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5180,"num_write_kb":145408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":411,"num_bytes_recovered":26804224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":848},{"shards":"1,2,7","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":214,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.613946+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:46:01.613946+0000","last_peered":"2024-09-13T14:46:01.613946+0000","last_clean":"2024-09-13T14:46:01.613946+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:46:01.613946+0000","last_undegraded":"2024-09-13T14:46:01.613946+0000","last_fullsized":"2024-09-13T14:46:01.613946+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":129,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.615066+0000","last_change":"2024-09-13T14:45:44.445221+0000","last_active":"2024-09-13T14:46:01.615066+0000","last_peered":"2024-09-13T14:46:01.615066+0000","last_clean":"2024-09-13T14:46:01.615066+0000","last_became_active":"2024-09-13T14:45:44.444956+0000","last_became_peered":"2024-09-13T14:45:44.444956+0000","last_unstale":"2024-09-13T14:46:01.615066+0000","last_undegraded":"2024-09-13T14:46:01.615066+0000","last_fullsized":"2024-09-13T14:46:01.615066+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:22.175285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"207'8349","reported_seq":17662,"reported_epoch":207,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:46:04.889673+0000","last_change":"2024-09-13T14:43:21.498879+0000","last_active":"2024-09-13T14:46:04.889673+0000","last_peered":"2024-09-13T14:46:04.889673+0000","last_clean":"2024-09-13T14:43:18.793932+0000","last_became_active":"2024-09-13T14:43:21.445419+0000","last_became_peered":"2024-09-13T14:43:21.445419+0000","last_unstale":"2024-09-13T14:46:04.889673+0000","last_undegraded":"2024-09-13T14:43:21.110245+0000","last_fullsized":"2024-09-13T14:43:20.854131+0000","mapping_epoch":179,"log_start":"157'1681","ondisk_log_start":"157'1681","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3248","last_scrub_stamp":"2024-09-13T14:42:19.745287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:19.745287+0000","objects_scrubbed":1743,"log_size":6668,"log_dups_size":0,"ondisk_log_size":6668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49348608,"num_objects":753,"num_object_clones":0,"num_object_copies":2259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5110,"num_write_kb":131840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3050,"num_bytes_recovered":199884800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,1],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,1","objects":4},{"shards":"0,3,5","objects":749}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":236,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.636628+0000","last_change":"2024-09-13T14:45:46.634166+0000","last_active":"2024-09-13T14:46:04.636628+0000","last_peered":"2024-09-13T14:46:04.636628+0000","last_clean":"2024-09-13T14:46:04.636628+0000","last_became_active":"2024-09-13T14:45:33.239497+0000","last_became_peered":"2024-09-13T14:45:33.239497+0000","last_unstale":"2024-09-13T14:46:04.636628+0000","last_undegraded":"2024-09-13T14:46:04.636628+0000","last_fullsized":"2024-09-13T14:46:04.636628+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:46.634098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:46.634098+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:18.086459+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":180,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.617120+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:46:01.617120+0000","last_peered":"2024-09-13T14:46:01.617120+0000","last_clean":"2024-09-13T14:46:01.617120+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:46:01.617120+0000","last_undegraded":"2024-09-13T14:46:01.617120+0000","last_fullsized":"2024-09-13T14:46:01.617120+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":203,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.614945+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:46:01.614945+0000","last_peered":"2024-09-13T14:46:01.614945+0000","last_clean":"2024-09-13T14:46:01.614945+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:46:01.614945+0000","last_undegraded":"2024-09-13T14:46:01.614945+0000","last_fullsized":"2024-09-13T14:46:01.614945+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":151,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620739+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:46:01.620739+0000","last_peered":"2024-09-13T14:46:01.620739+0000","last_clean":"2024-09-13T14:46:01.620739+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:46:01.620739+0000","last_undegraded":"2024-09-13T14:46:01.620739+0000","last_fullsized":"2024-09-13T14:46:01.620739+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":236,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635124+0000","last_change":"2024-09-13T14:45:44.649642+0000","last_active":"2024-09-13T14:46:04.635124+0000","last_peered":"2024-09-13T14:46:04.635124+0000","last_clean":"2024-09-13T14:46:04.635124+0000","last_became_active":"2024-09-13T14:45:33.237495+0000","last_became_peered":"2024-09-13T14:45:33.237495+0000","last_unstale":"2024-09-13T14:46:04.635124+0000","last_undegraded":"2024-09-13T14:46:04.635124+0000","last_fullsized":"2024-09-13T14:46:04.635124+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:44.649596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:44.649596+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:40:17.028455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"207'13522","reported_seq":13651,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.852793+0000","last_change":"2024-09-13T14:41:41.821076+0000","last_active":"2024-09-13T14:46:04.852793+0000","last_peered":"2024-09-13T14:46:04.852793+0000","last_clean":"2024-09-13T14:46:04.852793+0000","last_became_active":"2024-09-13T14:41:41.820767+0000","last_became_peered":"2024-09-13T14:41:41.820767+0000","last_unstale":"2024-09-13T14:46:04.852793+0000","last_undegraded":"2024-09-13T14:46:04.852793+0000","last_fullsized":"2024-09-13T14:46:04.852793+0000","mapping_epoch":152,"log_start":"177'7434","ondisk_log_start":"177'7434","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":6088,"log_dups_size":0,"ondisk_log_size":6088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:48.859865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100794368,"num_objects":1538,"num_object_clones":0,"num_object_copies":4614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13522,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":208,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.619508+0000","last_change":"2024-09-13T14:43:19.872365+0000","last_active":"2024-09-13T14:46:01.619508+0000","last_peered":"2024-09-13T14:46:01.619508+0000","last_clean":"2024-09-13T14:46:01.619508+0000","last_became_active":"2024-09-13T14:43:19.871956+0000","last_became_peered":"2024-09-13T14:43:19.871956+0000","last_unstale":"2024-09-13T14:46:01.619508+0000","last_undegraded":"2024-09-13T14:46:01.619508+0000","last_fullsized":"2024-09-13T14:46:01.619508+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:48.443596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":294,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635307+0000","last_change":"2024-09-13T14:43:19.867565+0000","last_active":"2024-09-13T14:46:04.635307+0000","last_peered":"2024-09-13T14:46:04.635307+0000","last_clean":"2024-09-13T14:46:04.635307+0000","last_became_active":"2024-09-13T14:43:19.858361+0000","last_became_peered":"2024-09-13T14:43:19.858361+0000","last_unstale":"2024-09-13T14:46:04.635307+0000","last_undegraded":"2024-09-13T14:46:04.635307+0000","last_fullsized":"2024-09-13T14:46:04.635307+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:03.954121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":298,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635716+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:46:04.635716+0000","last_peered":"2024-09-13T14:46:04.635716+0000","last_clean":"2024-09-13T14:46:04.635716+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:46:04.635716+0000","last_undegraded":"2024-09-13T14:46:04.635716+0000","last_fullsized":"2024-09-13T14:46:04.635716+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":233,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635737+0000","last_change":"2024-09-13T14:45:41.695182+0000","last_active":"2024-09-13T14:46:04.635737+0000","last_peered":"2024-09-13T14:46:04.635737+0000","last_clean":"2024-09-13T14:46:04.635737+0000","last_became_active":"2024-09-13T14:45:33.247188+0000","last_became_peered":"2024-09-13T14:45:33.247188+0000","last_unstale":"2024-09-13T14:46:04.635737+0000","last_undegraded":"2024-09-13T14:46:04.635737+0000","last_fullsized":"2024-09-13T14:46:04.635737+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:41.695089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:41.695089+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:40.995554+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"206'13380","reported_seq":18618,"reported_epoch":206,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:46:03.899907+0000","last_change":"2024-09-13T14:45:45.538904+0000","last_active":"2024-09-13T14:46:03.899907+0000","last_peered":"2024-09-13T14:46:03.899907+0000","last_clean":"2024-09-13T14:43:18.829456+0000","last_became_active":"2024-09-13T14:45:45.454351+0000","last_became_peered":"2024-09-13T14:45:45.454351+0000","last_unstale":"2024-09-13T14:46:03.899907+0000","last_undegraded":"2024-09-13T14:45:45.427342+0000","last_fullsized":"2024-09-13T14:45:45.419919+0000","mapping_epoch":200,"log_start":"184'8276","ondisk_log_start":"184'8276","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":5104,"log_dups_size":0,"ondisk_log_size":5104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:38.864880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104726528,"num_objects":1598,"num_object_clones":0,"num_object_copies":4794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13380,"num_write_kb":479296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2572,"num_bytes_recovered":168558592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1],"avail_no_missing":["7","1","5"],"object_location_counts":[{"shards":"1,5,7","objects":1591},{"shards":"1,7","objects":-808}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":271,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.705077+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:46:01.705077+0000","last_peered":"2024-09-13T14:46:01.705077+0000","last_clean":"2024-09-13T14:46:01.705077+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:46:01.705077+0000","last_undegraded":"2024-09-13T14:46:01.705077+0000","last_fullsized":"2024-09-13T14:46:01.705077+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":80,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.709209+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:46:01.709209+0000","last_peered":"2024-09-13T14:46:01.709209+0000","last_clean":"2024-09-13T14:46:01.709209+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:46:01.709209+0000","last_undegraded":"2024-09-13T14:46:01.709209+0000","last_fullsized":"2024-09-13T14:46:01.709209+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":91,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.617521+0000","last_change":"2024-09-13T14:43:19.860282+0000","last_active":"2024-09-13T14:46:01.617521+0000","last_peered":"2024-09-13T14:46:01.617521+0000","last_clean":"2024-09-13T14:46:01.617521+0000","last_became_active":"2024-09-13T14:43:19.859966+0000","last_became_peered":"2024-09-13T14:43:19.859966+0000","last_unstale":"2024-09-13T14:46:01.617521+0000","last_undegraded":"2024-09-13T14:46:01.617521+0000","last_fullsized":"2024-09-13T14:46:01.617521+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:25.735714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":235,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.634245+0000","last_change":"2024-09-13T14:45:45.659359+0000","last_active":"2024-09-13T14:46:04.634245+0000","last_peered":"2024-09-13T14:46:04.634245+0000","last_clean":"2024-09-13T14:46:04.634245+0000","last_became_active":"2024-09-13T14:45:33.238989+0000","last_became_peered":"2024-09-13T14:45:33.238989+0000","last_unstale":"2024-09-13T14:46:04.634245+0000","last_undegraded":"2024-09-13T14:46:04.634245+0000","last_fullsized":"2024-09-13T14:46:04.634245+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:45.659271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:45.659271+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:02:00.161304+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"207'8210","reported_seq":8369,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.800091+0000","last_change":"2024-09-13T14:42:16.895536+0000","last_active":"2024-09-13T14:46:04.800091+0000","last_peered":"2024-09-13T14:46:04.800091+0000","last_clean":"2024-09-13T14:46:04.800091+0000","last_became_active":"2024-09-13T14:42:15.123602+0000","last_became_peered":"2024-09-13T14:42:15.123602+0000","last_unstale":"2024-09-13T14:46:04.800091+0000","last_undegraded":"2024-09-13T14:46:04.800091+0000","last_fullsized":"2024-09-13T14:46:04.800091+0000","mapping_epoch":152,"log_start":"157'1454","ondisk_log_start":"157'1454","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3040","last_scrub_stamp":"2024-09-13T14:42:16.895299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.895299+0000","objects_scrubbed":1521,"log_size":6756,"log_dups_size":0,"ondisk_log_size":6756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:54.632739+0000","scrub_duration":483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5176,"num_write_kb":141440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":181,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.617040+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:46:01.617040+0000","last_peered":"2024-09-13T14:46:01.617040+0000","last_clean":"2024-09-13T14:46:01.617040+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:46:01.617040+0000","last_undegraded":"2024-09-13T14:46:01.617040+0000","last_fullsized":"2024-09-13T14:46:01.617040+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":169,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618758+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:46:01.618758+0000","last_peered":"2024-09-13T14:46:01.618758+0000","last_clean":"2024-09-13T14:46:01.618758+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:46:01.618758+0000","last_undegraded":"2024-09-13T14:46:01.618758+0000","last_fullsized":"2024-09-13T14:46:01.618758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":140,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.705836+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:46:01.705836+0000","last_peered":"2024-09-13T14:46:01.705836+0000","last_clean":"2024-09-13T14:46:01.705836+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:46:01.705836+0000","last_undegraded":"2024-09-13T14:46:01.705836+0000","last_fullsized":"2024-09-13T14:46:01.705836+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":233,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635760+0000","last_change":"2024-09-13T14:45:36.649035+0000","last_active":"2024-09-13T14:46:04.635760+0000","last_peered":"2024-09-13T14:46:04.635760+0000","last_clean":"2024-09-13T14:46:04.635760+0000","last_became_active":"2024-09-13T14:45:33.241773+0000","last_became_peered":"2024-09-13T14:45:33.241773+0000","last_unstale":"2024-09-13T14:46:04.635760+0000","last_undegraded":"2024-09-13T14:46:04.635760+0000","last_fullsized":"2024-09-13T14:46:04.635760+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:36.648956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.648956+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:54:22.397291+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"206'13284","reported_seq":13346,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.080905+0000","last_change":"2024-09-13T14:41:41.691642+0000","last_active":"2024-09-13T14:46:04.080905+0000","last_peered":"2024-09-13T14:46:04.080905+0000","last_clean":"2024-09-13T14:46:04.080905+0000","last_became_active":"2024-09-13T14:41:41.691265+0000","last_became_peered":"2024-09-13T14:41:41.691265+0000","last_unstale":"2024-09-13T14:46:04.080905+0000","last_undegraded":"2024-09-13T14:46:04.080905+0000","last_fullsized":"2024-09-13T14:46:04.080905+0000","mapping_epoch":152,"log_start":"171'4400","ondisk_log_start":"171'4400","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":8884,"log_dups_size":0,"ondisk_log_size":8884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:07.758310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97386496,"num_objects":1486,"num_object_clones":0,"num_object_copies":4458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13284,"num_write_kb":472640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":185,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.617414+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:46:01.617414+0000","last_peered":"2024-09-13T14:46:01.617414+0000","last_clean":"2024-09-13T14:46:01.617414+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:46:01.617414+0000","last_undegraded":"2024-09-13T14:46:01.617414+0000","last_fullsized":"2024-09-13T14:46:01.617414+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635416+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:46:04.635416+0000","last_peered":"2024-09-13T14:46:04.635416+0000","last_clean":"2024-09-13T14:46:04.635416+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:46:04.635416+0000","last_undegraded":"2024-09-13T14:46:04.635416+0000","last_fullsized":"2024-09-13T14:46:04.635416+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":53,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618454+0000","last_change":"2024-09-13T14:43:19.874797+0000","last_active":"2024-09-13T14:46:01.618454+0000","last_peered":"2024-09-13T14:46:01.618454+0000","last_clean":"2024-09-13T14:46:01.618454+0000","last_became_active":"2024-09-13T14:43:19.874460+0000","last_became_peered":"2024-09-13T14:43:19.874460+0000","last_unstale":"2024-09-13T14:46:01.618454+0000","last_undegraded":"2024-09-13T14:46:01.618454+0000","last_fullsized":"2024-09-13T14:46:01.618454+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:55.075477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":233,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635671+0000","last_change":"2024-09-13T14:45:37.637113+0000","last_active":"2024-09-13T14:46:04.635671+0000","last_peered":"2024-09-13T14:46:04.635671+0000","last_clean":"2024-09-13T14:46:04.635671+0000","last_became_active":"2024-09-13T14:45:33.242091+0000","last_became_peered":"2024-09-13T14:45:33.242091+0000","last_unstale":"2024-09-13T14:46:04.635671+0000","last_undegraded":"2024-09-13T14:46:04.635671+0000","last_fullsized":"2024-09-13T14:46:04.635671+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:37.637037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:37.637037+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:11:44.061062+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"206'13408","reported_seq":13470,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:03.899703+0000","last_change":"2024-09-13T14:41:41.950035+0000","last_active":"2024-09-13T14:46:03.899703+0000","last_peered":"2024-09-13T14:46:03.899703+0000","last_clean":"2024-09-13T14:46:03.899703+0000","last_became_active":"2024-09-13T14:41:41.949672+0000","last_became_peered":"2024-09-13T14:41:41.949672+0000","last_unstale":"2024-09-13T14:46:03.899703+0000","last_undegraded":"2024-09-13T14:46:03.899703+0000","last_fullsized":"2024-09-13T14:46:03.899703+0000","mapping_epoch":152,"log_start":"184'8258","ondisk_log_start":"184'8258","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":5150,"log_dups_size":0,"ondisk_log_size":5150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:43.973874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":4878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13408,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":221,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.705683+0000","last_change":"2024-09-13T14:45:44.440956+0000","last_active":"2024-09-13T14:46:01.705683+0000","last_peered":"2024-09-13T14:46:01.705683+0000","last_clean":"2024-09-13T14:46:01.705683+0000","last_became_active":"2024-09-13T14:45:44.440574+0000","last_became_peered":"2024-09-13T14:45:44.440574+0000","last_unstale":"2024-09-13T14:46:01.705683+0000","last_undegraded":"2024-09-13T14:46:01.705683+0000","last_fullsized":"2024-09-13T14:46:01.705683+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:08.477427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":181,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620621+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:46:01.620621+0000","last_peered":"2024-09-13T14:46:01.620621+0000","last_clean":"2024-09-13T14:46:01.620621+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:46:01.620621+0000","last_undegraded":"2024-09-13T14:46:01.620621+0000","last_fullsized":"2024-09-13T14:46:01.620621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":277,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635664+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:46:04.635664+0000","last_peered":"2024-09-13T14:46:04.635664+0000","last_clean":"2024-09-13T14:46:04.635664+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:46:04.635664+0000","last_undegraded":"2024-09-13T14:46:04.635664+0000","last_fullsized":"2024-09-13T14:46:04.635664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":233,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.634882+0000","last_change":"2024-09-13T14:45:43.677930+0000","last_active":"2024-09-13T14:46:04.634882+0000","last_peered":"2024-09-13T14:46:04.634882+0000","last_clean":"2024-09-13T14:46:04.634882+0000","last_became_active":"2024-09-13T14:45:33.268281+0000","last_became_peered":"2024-09-13T14:45:33.268281+0000","last_unstale":"2024-09-13T14:46:04.634882+0000","last_undegraded":"2024-09-13T14:46:04.634882+0000","last_fullsized":"2024-09-13T14:46:04.634882+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:43.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:43.677882+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:36.648701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"206'13565","reported_seq":28826,"reported_epoch":206,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:04.060026+0000","last_change":"2024-09-13T14:46:01.698412+0000","last_active":"2024-09-13T14:46:04.060026+0000","last_peered":"2024-09-13T14:46:04.060026+0000","last_clean":"2024-09-13T14:42:27.191777+0000","last_became_active":"2024-09-13T14:46:01.621254+0000","last_became_peered":"2024-09-13T14:46:01.621254+0000","last_unstale":"2024-09-13T14:46:04.060026+0000","last_undegraded":"2024-09-13T14:46:04.060026+0000","last_fullsized":"2024-09-13T14:46:04.060026+0000","mapping_epoch":204,"log_start":"184'8344","ondisk_log_start":"184'8344","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:51.535593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105775128,"num_objects":1615,"num_object_clones":0,"num_object_copies":4845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1615,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13565,"num_write_kb":485697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7581,"num_bytes_recovered":496828416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,7],"avail_no_missing":["5","1","4","7"],"object_location_counts":[{"shards":"4,5,7","objects":1615}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":244,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.615174+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:46:01.615174+0000","last_peered":"2024-09-13T14:46:01.615174+0000","last_clean":"2024-09-13T14:46:01.615174+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:46:01.615174+0000","last_undegraded":"2024-09-13T14:46:01.615174+0000","last_fullsized":"2024-09-13T14:46:01.615174+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":42,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620992+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:46:01.620992+0000","last_peered":"2024-09-13T14:46:01.620992+0000","last_clean":"2024-09-13T14:46:01.620992+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:46:01.620992+0000","last_undegraded":"2024-09-13T14:46:01.620992+0000","last_fullsized":"2024-09-13T14:46:01.620992+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":248,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.613649+0000","last_change":"2024-09-13T14:43:19.871833+0000","last_active":"2024-09-13T14:46:01.613649+0000","last_peered":"2024-09-13T14:46:01.613649+0000","last_clean":"2024-09-13T14:46:01.613649+0000","last_became_active":"2024-09-13T14:43:19.871024+0000","last_became_peered":"2024-09-13T14:43:19.871024+0000","last_unstale":"2024-09-13T14:46:01.613649+0000","last_undegraded":"2024-09-13T14:46:01.613649+0000","last_fullsized":"2024-09-13T14:46:01.613649+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:18:12.342823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":233,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635250+0000","last_change":"2024-09-13T14:45:38.648178+0000","last_active":"2024-09-13T14:46:04.635250+0000","last_peered":"2024-09-13T14:46:04.635250+0000","last_clean":"2024-09-13T14:46:04.635250+0000","last_became_active":"2024-09-13T14:45:33.242650+0000","last_became_peered":"2024-09-13T14:45:33.242650+0000","last_unstale":"2024-09-13T14:46:04.635250+0000","last_undegraded":"2024-09-13T14:46:04.635250+0000","last_fullsized":"2024-09-13T14:46:04.635250+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:38.648108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:38.648108+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:35.941747+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"205'13405","reported_seq":13530,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:02.565628+0000","last_change":"2024-09-13T14:41:41.950797+0000","last_active":"2024-09-13T14:46:02.565628+0000","last_peered":"2024-09-13T14:46:02.565628+0000","last_clean":"2024-09-13T14:46:02.565628+0000","last_became_active":"2024-09-13T14:41:41.950412+0000","last_became_peered":"2024-09-13T14:41:41.950412+0000","last_unstale":"2024-09-13T14:46:02.565628+0000","last_undegraded":"2024-09-13T14:46:02.565628+0000","last_fullsized":"2024-09-13T14:46:02.565628+0000","mapping_epoch":152,"log_start":"183'7694","ondisk_log_start":"183'7694","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":5711,"log_dups_size":0,"ondisk_log_size":5711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:09.662164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111476736,"num_objects":1701,"num_object_clones":0,"num_object_copies":5103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13405,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":290,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.636222+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:46:04.636222+0000","last_peered":"2024-09-13T14:46:04.636222+0000","last_clean":"2024-09-13T14:46:04.636222+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:46:04.636222+0000","last_undegraded":"2024-09-13T14:46:04.636222+0000","last_fullsized":"2024-09-13T14:46:04.636222+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620671+0000","last_change":"2024-09-13T14:43:19.860737+0000","last_active":"2024-09-13T14:46:01.620671+0000","last_peered":"2024-09-13T14:46:01.620671+0000","last_clean":"2024-09-13T14:46:01.620671+0000","last_became_active":"2024-09-13T14:43:19.860451+0000","last_became_peered":"2024-09-13T14:43:19.860451+0000","last_unstale":"2024-09-13T14:46:01.620671+0000","last_undegraded":"2024-09-13T14:46:01.620671+0000","last_fullsized":"2024-09-13T14:46:01.620671+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:14.888211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":151,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618433+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:46:01.618433+0000","last_peered":"2024-09-13T14:46:01.618433+0000","last_clean":"2024-09-13T14:46:01.618433+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:46:01.618433+0000","last_undegraded":"2024-09-13T14:46:01.618433+0000","last_fullsized":"2024-09-13T14:46:01.618433+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":168,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618669+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:46:01.618669+0000","last_peered":"2024-09-13T14:46:01.618669+0000","last_clean":"2024-09-13T14:46:01.618669+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:46:01.618669+0000","last_undegraded":"2024-09-13T14:46:01.618669+0000","last_fullsized":"2024-09-13T14:46:01.618669+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":175,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618915+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:46:01.618915+0000","last_peered":"2024-09-13T14:46:01.618915+0000","last_clean":"2024-09-13T14:46:01.618915+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:46:01.618915+0000","last_undegraded":"2024-09-13T14:46:01.618915+0000","last_fullsized":"2024-09-13T14:46:01.618915+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":97,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620144+0000","last_change":"2024-09-13T14:43:19.870397+0000","last_active":"2024-09-13T14:46:01.620144+0000","last_peered":"2024-09-13T14:46:01.620144+0000","last_clean":"2024-09-13T14:46:01.620144+0000","last_became_active":"2024-09-13T14:43:19.870037+0000","last_became_peered":"2024-09-13T14:43:19.870037+0000","last_unstale":"2024-09-13T14:46:01.620144+0000","last_undegraded":"2024-09-13T14:46:01.620144+0000","last_fullsized":"2024-09-13T14:46:01.620144+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:36.094674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":83,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.615175+0000","last_change":"2024-09-13T14:43:19.881173+0000","last_active":"2024-09-13T14:46:01.615175+0000","last_peered":"2024-09-13T14:46:01.615175+0000","last_clean":"2024-09-13T14:46:01.615175+0000","last_became_active":"2024-09-13T14:43:19.870848+0000","last_became_peered":"2024-09-13T14:43:19.870848+0000","last_unstale":"2024-09-13T14:46:01.615175+0000","last_undegraded":"2024-09-13T14:46:01.615175+0000","last_fullsized":"2024-09-13T14:46:01.615175+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:00.809391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":91,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620612+0000","last_change":"2024-09-13T14:43:19.871796+0000","last_active":"2024-09-13T14:46:01.620612+0000","last_peered":"2024-09-13T14:46:01.620612+0000","last_clean":"2024-09-13T14:46:01.620612+0000","last_became_active":"2024-09-13T14:43:19.871130+0000","last_became_peered":"2024-09-13T14:43:19.871130+0000","last_unstale":"2024-09-13T14:46:01.620612+0000","last_undegraded":"2024-09-13T14:46:01.620612+0000","last_fullsized":"2024-09-13T14:46:01.620612+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:33.743486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":210,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.614284+0000","last_change":"2024-09-13T14:43:19.871328+0000","last_active":"2024-09-13T14:46:01.614284+0000","last_peered":"2024-09-13T14:46:01.614284+0000","last_clean":"2024-09-13T14:46:01.614284+0000","last_became_active":"2024-09-13T14:43:19.869668+0000","last_became_peered":"2024-09-13T14:43:19.869668+0000","last_unstale":"2024-09-13T14:46:01.614284+0000","last_undegraded":"2024-09-13T14:46:01.614284+0000","last_fullsized":"2024-09-13T14:46:01.614284+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:18.042885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":286,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.636012+0000","last_change":"2024-09-13T14:37:08.759355+0000","last_active":"2024-09-13T14:46:04.636012+0000","last_peered":"2024-09-13T14:46:04.636012+0000","last_clean":"2024-09-13T14:46:04.636012+0000","last_became_active":"2024-09-13T14:37:08.759058+0000","last_became_peered":"2024-09-13T14:37:08.759058+0000","last_unstale":"2024-09-13T14:46:04.636012+0000","last_undegraded":"2024-09-13T14:46:04.636012+0000","last_fullsized":"2024-09-13T14:46:04.636012+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:42.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":212,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.613475+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:46:01.613475+0000","last_peered":"2024-09-13T14:46:01.613475+0000","last_clean":"2024-09-13T14:46:01.613475+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:46:01.613475+0000","last_undegraded":"2024-09-13T14:46:01.613475+0000","last_fullsized":"2024-09-13T14:46:01.613475+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.12","version":"207'8366","reported_seq":15026,"reported_epoch":207,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:46:04.908349+0000","last_change":"2024-09-13T14:45:45.572526+0000","last_active":"2024-09-13T14:46:04.908349+0000","last_peered":"2024-09-13T14:46:04.908349+0000","last_clean":"2024-09-13T14:43:18.835862+0000","last_became_active":"2024-09-13T14:45:45.538023+0000","last_became_peered":"2024-09-13T14:45:45.538023+0000","last_unstale":"2024-09-13T14:46:04.908349+0000","last_undegraded":"2024-09-13T14:45:45.458299+0000","last_fullsized":"2024-09-13T14:46:04.908349+0000","mapping_epoch":200,"log_start":"157'2424","ondisk_log_start":"157'2424","created":159,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3181","last_scrub_stamp":"2024-09-13T14:42:18.816556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:18.816556+0000","objects_scrubbed":1693,"log_size":5942,"log_dups_size":0,"ondisk_log_size":5942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47579136,"num_objects":726,"num_object_clones":0,"num_object_copies":2178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5199,"num_write_kb":135424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2298,"num_bytes_recovered":150601728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","3","5"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,1,5","objects":723}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":168,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.620581+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:46:01.620581+0000","last_peered":"2024-09-13T14:46:01.620581+0000","last_clean":"2024-09-13T14:46:01.620581+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:46:01.620581+0000","last_undegraded":"2024-09-13T14:46:01.620581+0000","last_fullsized":"2024-09-13T14:46:01.620581+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":237,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.613814+0000","last_change":"2024-09-13T14:43:19.872068+0000","last_active":"2024-09-13T14:46:01.613814+0000","last_peered":"2024-09-13T14:46:01.613814+0000","last_clean":"2024-09-13T14:46:01.613814+0000","last_became_active":"2024-09-13T14:43:19.871529+0000","last_became_peered":"2024-09-13T14:43:19.871529+0000","last_unstale":"2024-09-13T14:46:01.613814+0000","last_undegraded":"2024-09-13T14:46:01.613814+0000","last_fullsized":"2024-09-13T14:46:01.613814+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:11.995530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"206'8250","reported_seq":13897,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:03.871773+0000","last_change":"2024-09-13T14:43:29.963964+0000","last_active":"2024-09-13T14:46:03.871773+0000","last_peered":"2024-09-13T14:46:03.871773+0000","last_clean":"2024-09-13T14:46:03.871773+0000","last_became_active":"2024-09-13T14:43:29.959212+0000","last_became_peered":"2024-09-13T14:43:29.959212+0000","last_unstale":"2024-09-13T14:46:03.871773+0000","last_undegraded":"2024-09-13T14:46:03.871773+0000","last_fullsized":"2024-09-13T14:46:03.871773+0000","mapping_epoch":182,"log_start":"160'3113","ondisk_log_start":"160'3113","created":159,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52297728,"num_objects":798,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6768,"num_write_kb":242112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1850,"num_bytes_recovered":121241600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":295,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635812+0000","last_change":"2024-09-13T14:43:19.870848+0000","last_active":"2024-09-13T14:46:04.635812+0000","last_peered":"2024-09-13T14:46:04.635812+0000","last_clean":"2024-09-13T14:46:04.635812+0000","last_became_active":"2024-09-13T14:43:19.870037+0000","last_became_peered":"2024-09-13T14:43:19.870037+0000","last_unstale":"2024-09-13T14:46:04.635812+0000","last_undegraded":"2024-09-13T14:46:04.635812+0000","last_fullsized":"2024-09-13T14:46:04.635812+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:03.760717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":129,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.614440+0000","last_change":"2024-09-13T14:45:44.445815+0000","last_active":"2024-09-13T14:46:01.614440+0000","last_peered":"2024-09-13T14:46:01.614440+0000","last_clean":"2024-09-13T14:46:01.614440+0000","last_became_active":"2024-09-13T14:45:44.445562+0000","last_became_peered":"2024-09-13T14:45:44.445562+0000","last_unstale":"2024-09-13T14:46:01.614440+0000","last_undegraded":"2024-09-13T14:46:01.614440+0000","last_fullsized":"2024-09-13T14:46:01.614440+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:31.178220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"206'8636","reported_seq":14955,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:03.893270+0000","last_change":"2024-09-13T14:43:16.600263+0000","last_active":"2024-09-13T14:46:03.893270+0000","last_peered":"2024-09-13T14:46:03.893270+0000","last_clean":"2024-09-13T14:46:03.893270+0000","last_became_active":"2024-09-13T14:43:16.578306+0000","last_became_peered":"2024-09-13T14:43:16.578306+0000","last_unstale":"2024-09-13T14:46:03.893270+0000","last_undegraded":"2024-09-13T14:46:03.893270+0000","last_fullsized":"2024-09-13T14:46:03.893270+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3245","last_scrub_stamp":"2024-09-13T14:42:17.833200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:17.833200+0000","objects_scrubbed":1661,"log_size":8636,"log_dups_size":0,"ondisk_log_size":8636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:16.805530+0000","scrub_duration":483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54657024,"num_objects":834,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5413,"num_write_kb":146752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1973,"num_bytes_recovered":129302528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":212,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.615017+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:46:01.615017+0000","last_peered":"2024-09-13T14:46:01.615017+0000","last_clean":"2024-09-13T14:46:01.615017+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:46:01.615017+0000","last_undegraded":"2024-09-13T14:46:01.615017+0000","last_fullsized":"2024-09-13T14:46:01.615017+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":187,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618474+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:46:01.618474+0000","last_peered":"2024-09-13T14:46:01.618474+0000","last_clean":"2024-09-13T14:46:01.618474+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:46:01.618474+0000","last_undegraded":"2024-09-13T14:46:01.618474+0000","last_fullsized":"2024-09-13T14:46:01.618474+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"206'8045","reported_seq":8163,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:03.870003+0000","last_change":"2024-09-13T14:42:23.582272+0000","last_active":"2024-09-13T14:46:03.870003+0000","last_peered":"2024-09-13T14:46:03.870003+0000","last_clean":"2024-09-13T14:46:03.870003+0000","last_became_active":"2024-09-13T14:42:23.579475+0000","last_became_peered":"2024-09-13T14:42:23.579475+0000","last_unstale":"2024-09-13T14:46:03.870003+0000","last_undegraded":"2024-09-13T14:46:03.870003+0000","last_fullsized":"2024-09-13T14:46:03.870003+0000","mapping_epoch":164,"log_start":"160'2924","ondisk_log_start":"160'2924","created":159,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3023","last_scrub_stamp":"2024-09-13T14:42:17.713316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:17.713316+0000","objects_scrubbed":1552,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:08.367012+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53805056,"num_objects":821,"num_object_clones":0,"num_object_copies":2463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":137536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":270,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.705764+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:46:01.705764+0000","last_peered":"2024-09-13T14:46:01.705764+0000","last_clean":"2024-09-13T14:46:01.705764+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:46:01.705764+0000","last_undegraded":"2024-09-13T14:46:01.705764+0000","last_fullsized":"2024-09-13T14:46:01.705764+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":278,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635624+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:46:04.635624+0000","last_peered":"2024-09-13T14:46:04.635624+0000","last_clean":"2024-09-13T14:46:04.635624+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:46:04.635624+0000","last_undegraded":"2024-09-13T14:46:04.635624+0000","last_fullsized":"2024-09-13T14:46:04.635624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"206'8298","reported_seq":8450,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.042089+0000","last_change":"2024-09-13T14:42:25.288996+0000","last_active":"2024-09-13T14:46:04.042089+0000","last_peered":"2024-09-13T14:46:04.042089+0000","last_clean":"2024-09-13T14:46:04.042089+0000","last_became_active":"2024-09-13T14:42:15.125321+0000","last_became_peered":"2024-09-13T14:42:15.125321+0000","last_unstale":"2024-09-13T14:46:04.042089+0000","last_undegraded":"2024-09-13T14:46:04.042089+0000","last_fullsized":"2024-09-13T14:46:04.042089+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"166'3354","last_scrub_stamp":"2024-09-13T14:42:25.288934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:25.288934+0000","objects_scrubbed":1820,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:26.394629+0000","scrub_duration":0.191515928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55050240,"num_objects":840,"num_object_clones":0,"num_object_copies":2520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4944,"num_write_kb":126848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":195,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618671+0000","last_change":"2024-09-13T14:37:08.749549+0000","last_active":"2024-09-13T14:46:01.618671+0000","last_peered":"2024-09-13T14:46:01.618671+0000","last_clean":"2024-09-13T14:46:01.618671+0000","last_became_active":"2024-09-13T14:37:08.749013+0000","last_became_peered":"2024-09-13T14:37:08.749013+0000","last_unstale":"2024-09-13T14:46:01.618671+0000","last_undegraded":"2024-09-13T14:46:01.618671+0000","last_fullsized":"2024-09-13T14:46:01.618671+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:35.239989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":170,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.617089+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:46:01.617089+0000","last_peered":"2024-09-13T14:46:01.617089+0000","last_clean":"2024-09-13T14:46:01.617089+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:46:01.617089+0000","last_undegraded":"2024-09-13T14:46:01.617089+0000","last_fullsized":"2024-09-13T14:46:01.617089+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"206'8433","reported_seq":17691,"reported_epoch":206,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:46:04.052940+0000","last_change":"2024-09-13T14:43:21.335815+0000","last_active":"2024-09-13T14:46:04.052940+0000","last_peered":"2024-09-13T14:46:04.052940+0000","last_clean":"2024-09-13T14:43:18.842274+0000","last_became_active":"2024-09-13T14:43:21.328599+0000","last_became_peered":"2024-09-13T14:43:21.328599+0000","last_unstale":"2024-09-13T14:46:04.052940+0000","last_undegraded":"2024-09-13T14:43:21.055960+0000","last_fullsized":"2024-09-13T14:43:20.850323+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":8433,"log_dups_size":0,"ondisk_log_size":8433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52297728,"num_objects":798,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":246656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3058,"num_bytes_recovered":200409088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,1],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"1,5","objects":-14},{"shards":"3,5,7","objects":812}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":185,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618306+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:46:01.618306+0000","last_peered":"2024-09-13T14:46:01.618306+0000","last_clean":"2024-09-13T14:46:01.618306+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:46:01.618306+0000","last_undegraded":"2024-09-13T14:46:01.618306+0000","last_fullsized":"2024-09-13T14:46:01.618306+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":153,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.617373+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:46:01.617373+0000","last_peered":"2024-09-13T14:46:01.617373+0000","last_clean":"2024-09-13T14:46:01.617373+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:46:01.617373+0000","last_undegraded":"2024-09-13T14:46:01.617373+0000","last_fullsized":"2024-09-13T14:46:01.617373+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"206'8248","reported_seq":8405,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-13T14:46:03.850413+0000","last_change":"2024-09-13T14:45:44.488830+0000","last_active":"2024-09-13T14:46:03.850413+0000","last_peered":"2024-09-13T14:46:03.850413+0000","last_clean":"2024-09-13T14:46:03.850413+0000","last_became_active":"2024-09-13T14:45:44.445102+0000","last_became_peered":"2024-09-13T14:45:44.445102+0000","last_unstale":"2024-09-13T14:46:03.850413+0000","last_undegraded":"2024-09-13T14:46:03.850413+0000","last_fullsized":"2024-09-13T14:46:03.850413+0000","mapping_epoch":199,"log_start":"157'2211","ondisk_log_start":"157'2211","created":159,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3026","last_scrub_stamp":"2024-09-13T14:42:16.817418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.817418+0000","objects_scrubbed":1500,"log_size":6037,"log_dups_size":0,"ondisk_log_size":6037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:18.635831+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55771136,"num_objects":851,"num_object_clones":0,"num_object_copies":2553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":146560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":245,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.614537+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:46:01.614537+0000","last_peered":"2024-09-13T14:46:01.614537+0000","last_clean":"2024-09-13T14:46:01.614537+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:46:01.614537+0000","last_undegraded":"2024-09-13T14:46:01.614537+0000","last_fullsized":"2024-09-13T14:46:01.614537+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":231,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.615656+0000","last_change":"2024-09-13T14:37:08.750424+0000","last_active":"2024-09-13T14:46:01.615656+0000","last_peered":"2024-09-13T14:46:01.615656+0000","last_clean":"2024-09-13T14:46:01.615656+0000","last_became_active":"2024-09-13T14:37:08.749577+0000","last_became_peered":"2024-09-13T14:37:08.749577+0000","last_unstale":"2024-09-13T14:46:01.615656+0000","last_undegraded":"2024-09-13T14:46:01.615656+0000","last_fullsized":"2024-09-13T14:46:01.615656+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:16.435452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"207'8275","reported_seq":19870,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.849910+0000","last_change":"2024-09-13T14:46:04.656169+0000","last_active":"2024-09-13T14:46:04.849910+0000","last_peered":"2024-09-13T14:46:04.849910+0000","last_clean":"2024-09-13T14:46:04.849910+0000","last_became_active":"2024-09-13T14:46:04.637594+0000","last_became_peered":"2024-09-13T14:46:04.637594+0000","last_unstale":"2024-09-13T14:46:04.849910+0000","last_undegraded":"2024-09-13T14:46:04.849910+0000","last_fullsized":"2024-09-13T14:46:04.849910+0000","mapping_epoch":206,"log_start":"157'1515","ondisk_log_start":"157'1515","created":159,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3124","last_scrub_stamp":"2024-09-13T14:42:17.909091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:17.909091+0000","objects_scrubbed":1626,"log_size":6760,"log_dups_size":0,"ondisk_log_size":6760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:52.939922+0000","scrub_duration":450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54329344,"num_objects":829,"num_object_clones":0,"num_object_copies":2487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":139840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3808,"num_bytes_recovered":249561088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":168,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T14:46:01.618262+0000","last_change":"2024-09-13T14:32:09.484255+0000","last_active":"2024-09-13T14:46:01.618262+0000","last_peered":"2024-09-13T14:46:01.618262+0000","last_clean":"2024-09-13T14:46:01.618262+0000","last_became_active":"2024-09-13T14:32:09.483903+0000","last_became_peered":"2024-09-13T14:32:09.483903+0000","last_unstale":"2024-09-13T14:46:01.618262+0000","last_undegraded":"2024-09-13T14:46:01.618262+0000","last_fullsized":"2024-09-13T14:46:01.618262+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:41.931522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":291,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T14:46:04.635929+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:46:04.635929+0000","last_peered":"2024-09-13T14:46:04.635929+0000","last_clean":"2024-09-13T14:46:04.635929+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:46:04.635929+0000","last_undegraded":"2024-09-13T14:46:04.635929+0000","last_fullsized":"2024-09-13T14:46:04.635929+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":1682243608,"num_objects":25670,"num_object_clones":0,"num_object_copies":77010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18501,"num_objects_misplaced":1615,"num_objects_unfound":0,"num_objects_dirty":25670,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188448,"num_write_kb":5966017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45024,"num_bytes_recovered":2890137600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9639440384,"data_stored":9639428168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171865,"ondisk_log_size":171865,"up":78,"acting":72,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":182,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365116,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2844612,"kb_used_data":1931068,"kb_used_omap":1,"kb_used_meta":913534,"kb_avail":90900540,"statfs":{"total":95995035648,"available":93082152960,"internally_reserved":0,"allocated":1977413632,"data_stored":1976337362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":935459160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561346,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":969464,"kb_used_data":371184,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":92775688,"statfs":{"total":95995035648,"available":95002304512,"internally_reserved":0,"allocated":380092416,"data_stored":379028233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790278,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3341932,"kb_used_data":2432548,"kb_used_omap":1,"kb_used_meta":909374,"kb_avail":90403220,"statfs":{"total":95995035648,"available":92572897280,"internally_reserved":0,"allocated":2490929152,"data_stored":2489872263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931199320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986506,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1437120,"kb_used_data":534392,"kb_used_omap":1,"kb_used_meta":902718,"kb_avail":92308032,"statfs":{"total":95995035648,"available":94523424768,"internally_reserved":0,"allocated":547217408,"data_stored":546144885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924383576},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519661,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2114480,"kb_used_data":1714468,"kb_used_omap":4,"kb_used_meta":399995,"kb_avail":91630672,"statfs":{"total":95995035648,"available":93829808128,"internally_reserved":0,"allocated":1755615232,"data_stored":1754558447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":409595328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304577,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707052,"kb_used_data":323748,"kb_used_omap":6,"kb_used_meta":383289,"kb_avail":93038100,"statfs":{"total":95995035648,"available":95271014400,"internally_reserved":0,"allocated":331517952,"data_stored":330461611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":392488391},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827880,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2322532,"kb_used_data":1627028,"kb_used_omap":5,"kb_used_meta":695482,"kb_avail":91422620,"statfs":{"total":95995035648,"available":93616762880,"internally_reserved":0,"allocated":1666076672,"data_stored":1665028132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":712173804},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285745,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1076480,"kb_used_data":555384,"kb_used_omap":5,"kb_used_meta":521082,"kb_avail":92668672,"statfs":{"total":95995035648,"available":94892720128,"internally_reserved":0,"allocated":568713216,"data_stored":567644331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":533588027},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":560005120,"data_stored":560005120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1659174912,"data_stored":1659174912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326828032,"data_stored":326828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1743585280,"data_stored":1743585280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":538644480,"data_stored":538640408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2478903296,"data_stored":2478899224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":368836608,"data_stored":368836608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1963462656,"data_stored":1963458584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:46:08.198 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:46:08.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-13T14:46:08.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:08.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:08.386 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:08.410 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:08.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:08.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:08.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:08.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:08.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:08.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:08.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:08.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:08.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:08 smithi003 ceph-mon[107278]: pgmap v692: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 18501/75933 objects degraded (24.365%); 1615/75933 objects misplaced (2.127%); 12 MiB/s, 259 objects/s recovering 2024-09-13T14:46:08.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:08 smithi003 ceph-mon[104007]: pgmap v692: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 18501/75933 objects degraded (24.365%); 1615/75933 objects misplaced (2.127%); 12 MiB/s, 259 objects/s recovering 2024-09-13T14:46:08.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:46:08.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:09.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:09.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:09.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:08 smithi019 ceph-mon[91296]: pgmap v692: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 18501/75933 objects degraded (24.365%); 1615/75933 objects misplaced (2.127%); 12 MiB/s, 259 objects/s recovering 2024-09-13T14:46:09.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:09.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:09.324 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:09.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:09.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:09.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:09.605 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:09.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:09.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:09.849 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:09.886 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:09.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:10.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:10.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:10.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:10.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:10.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:10.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:10.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:10.593 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:46:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:46:10.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:10.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:10.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:10.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.904 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:10 smithi003 ceph-mon[107278]: pgmap v693: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 17778/69270 objects degraded (25.665%); 1393/69270 objects misplaced (2.011%); 11 MiB/s, 273 objects/s recovering 2024-09-13T14:46:10.904 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 17778/69270 objects degraded (25.665%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:10.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:10 smithi003 ceph-mon[104007]: pgmap v693: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 17778/69270 objects degraded (25.665%); 1393/69270 objects misplaced (2.011%); 11 MiB/s, 273 objects/s recovering 2024-09-13T14:46:10.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 17778/69270 objects degraded (25.665%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:11.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:11.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:11.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:10 smithi019 ceph-mon[91296]: pgmap v693: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 17778/69270 objects degraded (25.665%); 1393/69270 objects misplaced (2.011%); 11 MiB/s, 273 objects/s recovering 2024-09-13T14:46:11.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 17778/69270 objects degraded (25.665%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:11.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:11.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:11.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:11.329 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:11.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:11.475 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:11.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:11.608 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:11.710 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:11.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:11.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:11.868 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:11.970 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:11.994 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:12.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:12.131 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:12.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:12.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:12.412 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:12.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:12.553 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:12.655 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:12.684 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:12.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:12.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:12.884 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:12 smithi019 ceph-mon[91296]: pgmap v694: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 16976/66186 objects degraded (25.649%); 1393/66186 objects misplaced (2.105%); 13 MiB/s, 313 objects/s recovering 2024-09-13T14:46:12.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:12 smithi003 ceph-mon[104007]: pgmap v694: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 16976/66186 objects degraded (25.649%); 1393/66186 objects misplaced (2.105%); 13 MiB/s, 313 objects/s recovering 2024-09-13T14:46:12.917 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:12 smithi003 ceph-mon[107278]: pgmap v694: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 16976/66186 objects degraded (25.649%); 1393/66186 objects misplaced (2.105%); 13 MiB/s, 313 objects/s recovering 2024-09-13T14:46:12.931 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:12.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:13.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:13.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:13.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.236 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:13.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:13.337 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:13.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:13.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:13.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:13.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:13.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:13.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:13.845 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:46:13.845 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:46:13.845 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:46:13.846 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:13.846 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:13.846 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:13.846 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:46:13.846 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:46:13.846 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:46:13.846 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:46:13.846 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:13.846 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:13.847 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:13.847 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:13.854 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:13.863 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:46:13.863 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:13.901 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:13.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:13.932 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:13 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1924374028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:13.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:13.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:13.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:13 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1924374028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:14.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:14.057 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:14.057 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:14.058 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:13 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1924374028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:14.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:14.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:14.368 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:46:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:46:14] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-13T14:46:14.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:14.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:14.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:14.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:14.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:14.780 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:14.815 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:46:14.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:14.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:14 smithi003 ceph-mon[107278]: pgmap v695: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 16837/64641 objects degraded (26.047%); 1393/64641 objects misplaced (2.155%); 12 MiB/s, 266 objects/s recovering 2024-09-13T14:46:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2230877832' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:14 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:14 smithi003 ceph-mon[104007]: pgmap v695: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 16837/64641 objects degraded (26.047%); 1393/64641 objects misplaced (2.155%); 12 MiB/s, 266 objects/s recovering 2024-09-13T14:46:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2230877832' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:14 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:15.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:15.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:15.164 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:14 smithi019 ceph-mon[91296]: pgmap v695: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 16837/64641 objects degraded (26.047%); 1393/64641 objects misplaced (2.155%); 12 MiB/s, 266 objects/s recovering 2024-09-13T14:46:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2230877832' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:15.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:14 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:15.192 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:15.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:15.323 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:15.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:15.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:15.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:15.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:15.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:15.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:15.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:15.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:16.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:16.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:16.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:16.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:16.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:16.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[107278]: osdmap e208: 8 total, 8 up, 8 in 2024-09-13T14:46:16.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 16837/64641 objects degraded (26.047%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:16.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[107278]: osdmap e209: 8 total, 8 up, 8 in 2024-09-13T14:46:16.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:16.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:16.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[104007]: osdmap e208: 8 total, 8 up, 8 in 2024-09-13T14:46:16.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 16837/64641 objects degraded (26.047%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:16.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:15 smithi003 ceph-mon[104007]: osdmap e209: 8 total, 8 up, 8 in 2024-09-13T14:46:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:15 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:15 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:15 smithi019 ceph-mon[91296]: osdmap e208: 8 total, 8 up, 8 in 2024-09-13T14:46:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:15 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 16837/64641 objects degraded (26.047%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:16.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:15 smithi019 ceph-mon[91296]: osdmap e209: 8 total, 8 up, 8 in 2024-09-13T14:46:16.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:16.289 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:16.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:16.413 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:16.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:16.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:16.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:16.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:16.663 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:16.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:16.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:46:16.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:16.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:16.899 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:16 smithi003 ceph-mon[104007]: pgmap v698: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 15429/54144 objects degraded (28.496%); 1175/54144 objects misplaced (2.170%); 11 MiB/s, 264 objects/s recovering 2024-09-13T14:46:16.899 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:16 smithi003 ceph-mon[104007]: osdmap e210: 8 total, 8 up, 8 in 2024-09-13T14:46:16.899 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:16 smithi003 ceph-mon[107278]: pgmap v698: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 15429/54144 objects degraded (28.496%); 1175/54144 objects misplaced (2.170%); 11 MiB/s, 264 objects/s recovering 2024-09-13T14:46:16.899 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:16 smithi003 ceph-mon[107278]: osdmap e210: 8 total, 8 up, 8 in 2024-09-13T14:46:16.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:17.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:17.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:17.153 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:16 smithi019 ceph-mon[91296]: pgmap v698: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 15429/54144 objects degraded (28.496%); 1175/54144 objects misplaced (2.170%); 11 MiB/s, 264 objects/s recovering 2024-09-13T14:46:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:16 smithi019 ceph-mon[91296]: osdmap e210: 8 total, 8 up, 8 in 2024-09-13T14:46:17.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:17.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:17.394 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:17.432 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:17.534 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:17.564 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:17.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.696 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:17.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:17.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:17.826 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:17.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:17.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:18.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:18.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:18.226 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:18.254 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:18.356 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:18.386 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:18.419 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:18.419 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:18.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:18.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:18.631 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:18.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:18.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:18.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:18.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:18.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:19.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:19.074 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:19.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:19.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:18 smithi019 ceph-mon[91296]: pgmap v700: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 15429/53088 objects degraded (29.063%); 1175/53088 objects misplaced (2.213%); 7.9 MiB/s, 175 objects/s recovering 2024-09-13T14:46:19.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:18 smithi003 ceph-mon[104007]: pgmap v700: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 15429/53088 objects degraded (29.063%); 1175/53088 objects misplaced (2.213%); 7.9 MiB/s, 175 objects/s recovering 2024-09-13T14:46:19.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:18 smithi003 ceph-mon[107278]: pgmap v700: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 1.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 15429/53088 objects degraded (29.063%); 1175/53088 objects misplaced (2.213%); 7.9 MiB/s, 175 objects/s recovering 2024-09-13T14:46:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:19.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:19.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:19.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:19.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:46:19.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:19.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:19.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:19.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:19.795 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:19.827 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:19.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:19.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:20.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:20.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:20.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:20.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:20.230 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:20.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:20.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:20.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:20.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:20.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:20.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:20.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:46:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:46:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:20.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:20.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:46:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:20.910 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:20.937 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:21.014 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:20 smithi003 ceph-mon[107278]: pgmap v701: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 952 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 14118/45750 objects degraded (30.859%); 1376/45750 objects misplaced (3.008%); 11 MiB/s, 239 objects/s recovering 2024-09-13T14:46:21.015 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 14118/45750 objects degraded (30.859%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:21.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:20 smithi003 ceph-mon[104007]: pgmap v701: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 952 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 14118/45750 objects degraded (30.859%); 1376/45750 objects misplaced (3.008%); 11 MiB/s, 239 objects/s recovering 2024-09-13T14:46:21.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 14118/45750 objects degraded (30.859%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:21.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:21.067 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:21.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:20 smithi019 ceph-mon[91296]: pgmap v701: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 952 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 14118/45750 objects degraded (30.859%); 1376/45750 objects misplaced (3.008%); 11 MiB/s, 239 objects/s recovering 2024-09-13T14:46:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 14118/45750 objects degraded (30.859%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:21.199 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:21.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:21.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:21.427 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:21.453 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:21.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:21.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:21.714 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:21.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pg_num to 21 2024-09-13T14:46:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:21.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.008 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:22.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:22.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:22.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:22.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/961597263' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:46:22.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/961597263' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:46:22.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/961597263' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:46:22.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:22.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:22.392 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:22.428 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:22.510 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:22.529 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.562 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:22.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:22.663 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:22.697 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:22.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:22.834 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:22.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:22 smithi003 ceph-mon[107278]: pgmap v702: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 886 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 13352/42585 objects degraded (31.354%); 1376/42585 objects misplaced (3.231%); 12 MiB/s, 207 objects/s recovering 2024-09-13T14:46:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/961597263' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T14:46:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:22 smithi003 ceph-mon[107278]: osdmap e211: 8 total, 8 up, 8 in 2024-09-13T14:46:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:22 smithi003 ceph-mon[104007]: pgmap v702: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 886 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 13352/42585 objects degraded (31.354%); 1376/42585 objects misplaced (3.231%); 12 MiB/s, 207 objects/s recovering 2024-09-13T14:46:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/961597263' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T14:46:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:22 smithi003 ceph-mon[104007]: osdmap e211: 8 total, 8 up, 8 in 2024-09-13T14:46:22.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:22.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:23.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:23.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:22 smithi019 ceph-mon[91296]: pgmap v702: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 886 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 13352/42585 objects degraded (31.354%); 1376/42585 objects misplaced (3.231%); 12 MiB/s, 207 objects/s recovering 2024-09-13T14:46:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/961597263' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T14:46:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:22 smithi019 ceph-mon[91296]: osdmap e211: 8 total, 8 up, 8 in 2024-09-13T14:46:23.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:23.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:23.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:23.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.390 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:23.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:23.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:23.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:23.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:23.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:23.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:23.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:23.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:23.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:23.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:23.904 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:23 smithi003 ceph-mon[107278]: osdmap e212: 8 total, 8 up, 8 in 2024-09-13T14:46:23.904 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:23 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T14:46:23.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:23 smithi003 ceph-mon[104007]: osdmap e212: 8 total, 8 up, 8 in 2024-09-13T14:46:23.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:23 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T14:46:24.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:24.155 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:23 smithi019 ceph-mon[91296]: osdmap e212: 8 total, 8 up, 8 in 2024-09-13T14:46:24.155 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:23 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T14:46:24.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:24.165 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:46:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:46:24] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-13T14:46:24.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:24.306 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:24.337 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:24.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:24.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:24.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:24.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:24.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:24.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:24.748 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:24.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:24.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:24.981 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:25.003 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:25.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:25.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:25.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:24 smithi019 ceph-mon[91296]: pgmap v705: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 101 active+clean; 853 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 10363/41013 objects degraded (25.268%); 1700/41013 objects misplaced (4.145%); 9.2 MiB/s, 158 objects/s recovering 2024-09-13T14:46:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T14:46:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:24 smithi019 ceph-mon[91296]: osdmap e213: 8 total, 8 up, 8 in 2024-09-13T14:46:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:24 smithi003 ceph-mon[104007]: pgmap v705: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 101 active+clean; 853 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 10363/41013 objects degraded (25.268%); 1700/41013 objects misplaced (4.145%); 9.2 MiB/s, 158 objects/s recovering 2024-09-13T14:46:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T14:46:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:24 smithi003 ceph-mon[104007]: osdmap e213: 8 total, 8 up, 8 in 2024-09-13T14:46:25.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:24 smithi003 ceph-mon[107278]: pgmap v705: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 101 active+clean; 853 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 10363/41013 objects degraded (25.268%); 1700/41013 objects misplaced (4.145%); 9.2 MiB/s, 158 objects/s recovering 2024-09-13T14:46:25.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T14:46:25.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:24 smithi003 ceph-mon[107278]: osdmap e213: 8 total, 8 up, 8 in 2024-09-13T14:46:25.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:25.273 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:25.375 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:25.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:25.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:25.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:25.647 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:25.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:25.771 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:25.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:25.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:25.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:25.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:26.024 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:26.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:25 smithi019 ceph-mon[91296]: osdmap e214: 8 total, 8 up, 8 in 2024-09-13T14:46:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:25 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2179918337' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:25 smithi019 ceph-mon[91296]: 1.0 scrub starts 2024-09-13T14:46:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10363/41013 objects degraded (25.268%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:25 smithi019 ceph-mon[91296]: osdmap e215: 8 total, 8 up, 8 in 2024-09-13T14:46:26.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[107278]: osdmap e214: 8 total, 8 up, 8 in 2024-09-13T14:46:26.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2179918337' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:26.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[107278]: 1.0 scrub starts 2024-09-13T14:46:26.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10363/41013 objects degraded (25.268%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:26.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[107278]: osdmap e215: 8 total, 8 up, 8 in 2024-09-13T14:46:26.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[104007]: osdmap e214: 8 total, 8 up, 8 in 2024-09-13T14:46:26.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2179918337' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:26.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[104007]: 1.0 scrub starts 2024-09-13T14:46:26.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10363/41013 objects degraded (25.268%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:26.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:25 smithi003 ceph-mon[104007]: osdmap e215: 8 total, 8 up, 8 in 2024-09-13T14:46:26.210 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:46:26.210 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:46:26.210 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:46:26.210 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:46:26.211 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:46:26.211 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:26.211 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:26.211 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:26.211 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:26.211 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:26.211 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:46:26.212 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:46:26.212 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:46:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:26.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:26.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:26.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:26.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:26.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:26.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:26.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:26.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:26.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:26.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:26.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:26.960 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:26.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:27.089 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:27.121 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:27.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:26 smithi019 ceph-mon[91296]: pgmap v708: 119 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfilling, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 112 active+clean; 657 MiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 916 op/s; 9666/31584 objects degraded (30.604%); 378/31584 objects misplaced (1.197%); 23 MiB/s, 360 objects/s recovering 2024-09-13T14:46:27.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:26 smithi019 ceph-mon[91296]: 1.7 scrub starts 2024-09-13T14:46:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:26 smithi003 ceph-mon[104007]: pgmap v708: 119 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfilling, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 112 active+clean; 657 MiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 916 op/s; 9666/31584 objects degraded (30.604%); 378/31584 objects misplaced (1.197%); 23 MiB/s, 360 objects/s recovering 2024-09-13T14:46:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:26 smithi003 ceph-mon[104007]: 1.7 scrub starts 2024-09-13T14:46:27.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:26 smithi003 ceph-mon[107278]: pgmap v708: 119 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfilling, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 112 active+clean; 657 MiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 916 op/s; 9666/31584 objects degraded (30.604%); 378/31584 objects misplaced (1.197%); 23 MiB/s, 360 objects/s recovering 2024-09-13T14:46:27.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:26 smithi003 ceph-mon[107278]: 1.7 scrub starts 2024-09-13T14:46:27.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:27.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:27.348 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:27.382 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:27.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:27.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:27.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:27.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:27.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:27.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:27.789 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:27.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:27.923 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:28.025 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:28.054 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:28.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:28.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:27 smithi019 ceph-mon[91296]: 1.4 scrub starts 2024-09-13T14:46:28.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:27 smithi003 ceph-mon[107278]: 1.4 scrub starts 2024-09-13T14:46:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:27 smithi003 ceph-mon[104007]: 1.4 scrub starts 2024-09-13T14:46:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:28.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:28.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:28.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:28.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:28.430 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:28.467 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:28.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:28.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:28.684 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:46:28.684 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:46:28.684 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-13T14:46:28.684 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:46:28.684 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:46:28.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-13T14:46:28.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:28.774 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:28.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:28.900 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:28.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:29.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:29.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:29.106 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:46:29.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:28 smithi019 ceph-mon[91296]: pgmap v710: 119 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfilling, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 112 active+clean; 638 MiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 734 op/s; 9666/30690 objects degraded (31.496%); 378/30690 objects misplaced (1.232%); 19 MiB/s, 296 objects/s recovering 2024-09-13T14:46:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:28 smithi019 ceph-mon[91296]: 1.14 scrub starts 2024-09-13T14:46:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:28 smithi003 ceph-mon[107278]: pgmap v710: 119 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfilling, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 112 active+clean; 638 MiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 734 op/s; 9666/30690 objects degraded (31.496%); 378/30690 objects misplaced (1.232%); 19 MiB/s, 296 objects/s recovering 2024-09-13T14:46:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:28 smithi003 ceph-mon[107278]: 1.14 scrub starts 2024-09-13T14:46:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:28 smithi003 ceph-mon[104007]: pgmap v710: 119 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfilling, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 112 active+clean; 638 MiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 734 op/s; 9666/30690 objects degraded (31.496%); 378/30690 objects misplaced (1.232%); 19 MiB/s, 296 objects/s recovering 2024-09-13T14:46:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:28 smithi003 ceph-mon[104007]: 1.14 scrub starts 2024-09-13T14:46:29.187 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:29.289 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:29.326 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:29.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:29.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:29.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:29.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:29.604 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:29.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:29.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:29.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:29.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:30.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:30.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:30.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:30.180 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:30.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:29 smithi019 ceph-mon[91296]: 1.6 scrub starts 2024-09-13T14:46:30.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:29 smithi003 ceph-mon[107278]: 1.6 scrub starts 2024-09-13T14:46:30.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:29 smithi003 ceph-mon[104007]: 1.6 scrub starts 2024-09-13T14:46:30.282 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:30.311 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:30.338 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:30.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:30.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:30.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:30.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:30.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:46:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:46:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:30.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:30.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:30.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:30.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:30.875 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:30.977 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:31.008 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:31.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:31.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:30 smithi003 ceph-mon[104007]: pgmap v711: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 497 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 950 op/s; 9666/23907 objects degraded (40.432%); 17 MiB/s, 300 objects/s recovering 2024-09-13T14:46:31.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:30 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T14:46:31.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9666/23907 objects degraded (40.432%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:31.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:30 smithi003 ceph-mon[107278]: pgmap v711: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 497 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 950 op/s; 9666/23907 objects degraded (40.432%); 17 MiB/s, 300 objects/s recovering 2024-09-13T14:46:31.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:30 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T14:46:31.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9666/23907 objects degraded (40.432%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:31.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:31.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:30 smithi019 ceph-mon[91296]: pgmap v711: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 497 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 950 op/s; 9666/23907 objects degraded (40.432%); 17 MiB/s, 300 objects/s recovering 2024-09-13T14:46:31.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:30 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T14:46:31.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9666/23907 objects degraded (40.432%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:31.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:31.304 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:31.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:31.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:31.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:31.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:31.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.586 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:31.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:31.687 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:31.732 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:31.834 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:31.876 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:31.936 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:46:31.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:32.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:32.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:32.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:32.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:32.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:31 smithi003 ceph-mon[107278]: 1.5 scrub starts 2024-09-13T14:46:32.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2032817917' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:32.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:31 smithi003 ceph-mon[104007]: 1.5 scrub starts 2024-09-13T14:46:32.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2032817917' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:32.189 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:31 smithi019 ceph-mon[91296]: 1.5 scrub starts 2024-09-13T14:46:32.189 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2032817917' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:32.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:32.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:32.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:46:32.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:32.470 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:46:32.470 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:46:32.471 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:46:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:32.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:32.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:32.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:32.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:32.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:32.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.916 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:32.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:33.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:33.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[104007]: pgmap v712: 119 pgs: 1 active+recovery_wait+degraded, 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 109 active+clean; 430 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 8824/20703 objects degraded (42.622%); 17 MiB/s, 289 objects/s recovering 2024-09-13T14:46:33.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:33.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2032817917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:33.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[104007]: osdmap e216: 8 total, 8 up, 8 in 2024-09-13T14:46:33.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[104007]: 1.10 scrub starts 2024-09-13T14:46:33.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[104007]: 1.10 scrub ok 2024-09-13T14:46:33.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[107278]: pgmap v712: 119 pgs: 1 active+recovery_wait+degraded, 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 109 active+clean; 430 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 8824/20703 objects degraded (42.622%); 17 MiB/s, 289 objects/s recovering 2024-09-13T14:46:33.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:33.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2032817917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:33.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[107278]: osdmap e216: 8 total, 8 up, 8 in 2024-09-13T14:46:33.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[107278]: 1.10 scrub starts 2024-09-13T14:46:33.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:32 smithi003 ceph-mon[107278]: 1.10 scrub ok 2024-09-13T14:46:33.156 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:33.166 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:32 smithi019 ceph-mon[91296]: pgmap v712: 119 pgs: 1 active+recovery_wait+degraded, 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 109 active+clean; 430 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 8824/20703 objects degraded (42.622%); 17 MiB/s, 289 objects/s recovering 2024-09-13T14:46:33.166 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:32 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:33.166 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2032817917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:33.166 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:32 smithi019 ceph-mon[91296]: osdmap e216: 8 total, 8 up, 8 in 2024-09-13T14:46:33.166 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:32 smithi019 ceph-mon[91296]: 1.10 scrub starts 2024-09-13T14:46:33.166 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:32 smithi019 ceph-mon[91296]: 1.10 scrub ok 2024-09-13T14:46:33.192 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:33.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:33.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:33.437 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:33.470 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:33.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:33.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.602 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:33.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:33.700 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:33.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:33.761 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:46:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:46:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:46:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:46:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:53919 8.c 8:3c8894e3:::benchmark_data_smithi003_139172_object53918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:46:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:36:46.522566+0000", 2024-09-13T14:46:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 587.21886970100002, 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.030694461, 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 53919 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.522566+0000", 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.522566+0000", 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.522566+0000", 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6000000000000003e-07 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.523754+0000", 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001187848 2024-09-13T14:46:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.523773+0000", 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8754999999999999e-05 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.523781+0000", 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999999e-06 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.523908+0000", 2024-09-13T14:46:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012730799999999999 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.524036+0000", 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012819099999999999 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.524226+0000", 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018940800000000001 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549262+0000", 2024-09-13T14:46:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025035999 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549273+0000", 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0954e-05 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553061+0000", 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.003788667 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553260+0000", 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019860099999999999 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:53924 8.c 8:3b907ba2:::benchmark_data_smithi003_139172_object53923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:36:46.526036+0000", 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 587.21539912100002, 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0276176050000001, 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 53924 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526036+0000", 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526036+0000", 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526038+0000", 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.387e-06 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526061+0000", 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.304e-05 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526077+0000", 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6300999999999999e-05 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526084+0000", 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5109999999999997e-06 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527077+0000", 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0009923480000000001 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527347+0000", 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000270661 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527602+0000", 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025452299999999999 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549305+0000", 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021703324 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549314+0000", 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2439999999999993e-06 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553519+0000", 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.004205867 2024-09-13T14:46:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553654+0000", 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000134399 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:53925 8.c 8:32910906:::benchmark_data_smithi003_139172_object53924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:36:46.526979+0000", 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 587.21445657899994, 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0417181719999999, 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 53925 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526979+0000", 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526979+0000", 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.526980+0000", 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.187e-06 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527001+0000", 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0908e-05 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527013+0000", 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2203e-05 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.527020+0000", 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1119999999999998e-06 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.541776+0000", 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014755819999999999 2024-09-13T14:46:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.541947+0000", 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017083699999999999 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.542127+0000", 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000180003 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549346+0000", 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072187060000000001 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:46.549354+0000", 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4910000000000005e-06 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.553698+0000", 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0043439009999999 2024-09-13T14:46:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.554556+0000", 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00085828399999999996 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.568579+0000", 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014023066000000001 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:36:47.568697+0000", 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117654 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85389 8.c 8:32251d57:::benchmark_data_smithi003_139172_object85388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:07.765151+0000", 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 565.97628474199996, 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1862126320000002, 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85389 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765151+0000", 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765151+0000", 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765152+0000", 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.486e-06 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765419+0000", 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000266588 2024-09-13T14:46:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765435+0000", 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6041000000000001e-05 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765443+0000", 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3010000000000007e-06 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765544+0000", 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101245 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.765557+0000", 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3161e-05 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.739019+0000", 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97346153300000005 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.739031+0000", 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2259000000000001e-05 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.941457+0000", 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.202425351 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.941615+0000", 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015840799999999999 2024-09-13T14:46:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.941794+0000", 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178684 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.949747+0000", 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079533920000000001 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.949769+0000", 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2130999999999999e-05 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.949818+0000", 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9157999999999999e-05 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.949901+0000", 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2967999999999996e-05 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.951363+0000", 2024-09-13T14:46:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014619259999999999 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:85397 8.c 8:37504a20:::benchmark_data_smithi003_139172_object85396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:37:07.783684+0000", 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 565.95775121700001, 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1708010880000002, 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85397 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783684+0000", 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783684+0000", 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783685+0000", 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.299e-06 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783709+0000", 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3929000000000001e-05 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783724+0000", 2024-09-13T14:46:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4225e-05 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.783731+0000", 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2570000000000001e-06 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.784233+0000", 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050265399999999995 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:07.784242+0000", 2024-09-13T14:46:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9490000000000002e-06 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.739057+0000", 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95481418100000004 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:08.739064+0000", 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7950000000000008e-06 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.946256+0000", 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2071917219999999 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.946424+0000", 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016787 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.946771+0000", 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000346588 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.951533+0000", 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047621060000000003 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.951546+0000", 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3358e-05 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.952975+0000", 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014288339999999999 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.953033+0000", 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7717999999999999e-05 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:37:09.954485+0000", 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001452603 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35454.0:156289 8.c 8:3a176f0a:::benchmark_data_smithi003_139172_object31496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-13T14:46:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:38:28.608616+0000", 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 485.13281933799999, 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0276335910000001, 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35454", 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 156289 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608616+0000", 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608616+0000", 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608617+0000", 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4079999999999999e-06 2024-09-13T14:46:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608623+0000", 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8379999999999998e-06 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608640+0000", 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7215e-05 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608648+0000", 2024-09-13T14:46:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8280000000000003e-06 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608712+0000", 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4113000000000001e-05 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:28.608724+0000", 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1267000000000001e-05 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.623532+0000", 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0148086999999999 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.623608+0000", 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5431000000000003e-05 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.623726+0000", 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118579 2024-09-13T14:46:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636094+0000", 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012368048 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636109+0000", 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4365999999999999e-05 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636145+0000", 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6273000000000003e-05 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636202+0000", 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7065999999999999e-05 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:38:29.636250+0000", 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7459000000000001e-05 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120809 9.15 9:ad532cb5:::benchmark_data_smithi003_156985_object120808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-13T14:46:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.868760+0000", 2024-09-13T14:46:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 194.87267546999999, 2024-09-13T14:46:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4990805589999998, 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120809 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868760+0000", 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868760+0000", 2024-09-13T14:46:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868762+0000", 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.012e-06 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868793+0000", 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1041000000000003e-05 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868810+0000", 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7215999999999999e-05 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868820+0000", 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0104e-05 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.871590+0000", 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027697830000000001 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.871718+0000", 2024-09-13T14:46:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127603 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.871855+0000", 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013778600000000001 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.886738+0000", 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014882948999999999 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.886755+0000", 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6398000000000001e-05 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.857773+0000", 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97101844299999995 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.857785+0000", 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2305e-05 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.301380+0000", 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4435946200000001 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.302932+0000", 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001552166 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.367740+0000", 2024-09-13T14:46:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.064808097999999995 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.367840+0000", 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100035 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120816 9.8 9:11d2346f:::benchmark_data_smithi003_156985_object120815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.887707+0000", 2024-09-13T14:46:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 194.85372803800001, 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6405986719999999, 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120816 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887707+0000", 2024-09-13T14:46:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887707+0000", 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887709+0000", 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.483e-06 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887734+0000", 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.561e-05 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887750+0000", 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5675000000000002e-05 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887758+0000", 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2730000000000002e-06 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.888218+0000", 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00046011899999999999 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.888229+0000", 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0944000000000001e-05 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856495+0000", 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96826583099999997 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856507+0000", 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1977999999999999e-05 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517113+0000", 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.660605506 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517220+0000", 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107624 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517354+0000", 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013387300000000001 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.521270+0000", 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039158910000000003 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:33.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.521574+0000", 2024-09-13T14:46:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030395699999999999 2024-09-13T14:46:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.525399+0000", 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038248459999999998 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.525464+0000", 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5418000000000003e-05 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.528306+0000", 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028416439999999999 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120817 9.18 9:1a90c8b1:::benchmark_data_smithi003_156985_object120816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.887776+0000", 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 194.853659642, 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6247730489999999, 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120817 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887776+0000", 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887776+0000", 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887776+0000", 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8899999999999999e-07 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887799+0000", 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2682000000000001e-05 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887807+0000", 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2290000000000003e-06 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887814+0000", 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3319999999999997e-06 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.888257+0000", 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000443218 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.888265+0000", 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3799999999999994e-06 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856535+0000", 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96826975900000001 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856544+0000", 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7420000000000003e-06 2024-09-13T14:46:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.504430+0000", 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6478863850000001 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.504557+0000", 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127195 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.504670+0000", 2024-09-13T14:46:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011225700000000001 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.510528+0000", 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058583569999999998 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.510581+0000", 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2958999999999999e-05 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.510680+0000", 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8884999999999999e-05 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.510757+0000", 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7373999999999995e-05 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.512549+0000", 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017916060000000001 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120822 9.18 9:1e2c2cd9:::benchmark_data_smithi003_156985_object120821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.895452+0000", 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 194.84598301899999, 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.623319607, 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120822 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895452+0000", 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895452+0000", 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895454+0000", 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.409e-06 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895477+0000", 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2963999999999999e-05 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895490+0000", 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3627e-05 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895499+0000", 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3599999999999996e-06 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895564+0000", 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5571e-05 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895576+0000", 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2102e-05 2024-09-13T14:46:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856568+0000", 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96099208400000002 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856576+0000", 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8879999999999997e-06 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.508342+0000", 2024-09-13T14:46:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6517653860000001 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.508521+0000", 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000179218 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.508709+0000", 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000188443 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.512606+0000", 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003896254 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.512658+0000", 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2846000000000002e-05 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517156+0000", 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044978889999999997 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517196+0000", 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9301999999999997e-05 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.518772+0000", 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001576264 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120825 9.15 9:aa3ed837:::benchmark_data_smithi003_156985_object120824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.899645+0000", 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 194.84179067400001, 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.477297343, 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120825 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899645+0000", 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899645+0000", 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899647+0000", 2024-09-13T14:46:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9379999999999999e-06 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899671+0000", 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4493000000000001e-05 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899686+0000", 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5183e-05 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899695+0000", 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6419999999999996e-06 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899772+0000", 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7247000000000001e-05 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899785+0000", 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3063e-05 2024-09-13T14:46:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.857816+0000", 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95803042299999996 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.858198+0000", 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00038188000000000002 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.367951+0000", 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5097530529999998 2024-09-13T14:46:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.368107+0000", 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156074 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.368287+0000", 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000180416 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.372843+0000", 2024-09-13T14:46:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045564170000000001 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.373071+0000", 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000227963 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.374047+0000", 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00097598800000000001 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.374110+0000", 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2889e-05 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.376942+0000", 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028316740000000002 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:136803 9.8 9:11e973c6:::benchmark_data_smithi003_156985_object15887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-13T14:46:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:58.441150+0000", 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 155.30028563299999, 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.53725493300000005, 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136803 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441150+0000", 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441150+0000", 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441150+0000", 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.16e-07 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441155+0000", 2024-09-13T14:46:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0249999999999996e-06 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441165+0000", 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0662000000000001e-05 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441173+0000", 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2330000000000006e-06 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.461177+0000", 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020003204 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.546296+0000", 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.085119427999999997 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.956515+0000", 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.41021918899999998 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.956747+0000", 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000231771 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.957329+0000", 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058164000000000002 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.963091+0000", 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057619639999999996 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.972590+0000", 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094997450000000004 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.975825+0000", 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032349420000000002 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.975889+0000", 2024-09-13T14:46:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3830000000000004e-05 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.978405+0000", 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025154840000000001 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:215903 9.15 9:add94de9:::benchmark_data_smithi003_156985_object94987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.677703+0000", 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 30.063732121000001, 2024-09-13T14:46:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97887381799999995, 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 215903 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677703+0000", 2024-09-13T14:46:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677703+0000", 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677704+0000", 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.265e-06 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677709+0000", 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9609999999999996e-06 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677725+0000", 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5999999999999999e-05 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677733+0000", 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6399999999999997e-06 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677802+0000", 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9051e-05 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677814+0000", 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.208e-05 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.653049+0000", 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97523451400000005 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.653161+0000", 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112428 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.653586+0000", 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042491799999999999 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656399+0000", 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028127040000000001 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656414+0000", 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5299999999999999e-05 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656477+0000", 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2767000000000004e-05 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656534+0000", 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6839999999999998e-05 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656577+0000", 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3350000000000003e-05 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:215952 9.15 9:a81f33b3:::benchmark_data_smithi003_156985_object95036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.737954+0000", 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 30.003481234999999, 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.932049456, 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 215952 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737954+0000", 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737954+0000", 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737956+0000", 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7379999999999999e-06 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737961+0000", 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3009999999999997e-06 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737972+0000", 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0822000000000001e-05 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737979+0000", 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9460000000000001e-06 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.748544+0000", 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010564726999999999 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.748552+0000", 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1820000000000006e-06 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656616+0000", 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90806452999999998 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656726+0000", 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010941400000000001 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.657068+0000", 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000342274 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663538+0000", 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064696969999999999 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663609+0000", 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1557000000000006e-05 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.669908+0000", 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062991239999999997 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.669960+0000", 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1600000000000001e-05 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670004+0000", 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3544000000000002e-05 2024-09-13T14:46:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216095 9.15 9:ae47eb7d:::benchmark_data_smithi003_156985_object95179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.897037+0000", 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 29.844398760000001, 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.77305581400000001, 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216095 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897037+0000", 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897037+0000", 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897038+0000", 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3400000000000001e-06 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897043+0000", 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8189999999999999e-06 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897054+0000", 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1031e-05 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897060+0000", 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4989999999999999e-06 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897107+0000", 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6891000000000002e-05 2024-09-13T14:46:33.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897115+0000", 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7689999999999996e-06 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.660201+0000", 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.76308632899999995 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.660327+0000", 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012528999999999999 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.660801+0000", 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000474184 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663648+0000", 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028476769999999998 2024-09-13T14:46:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.666408+0000", 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002759863 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670013+0000", 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036042719999999999 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670055+0000", 2024-09-13T14:46:33.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2967999999999999e-05 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670092+0000", 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6881999999999997e-05 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216126 9.15 9:aeb4fb92:::benchmark_data_smithi003_156985_object95210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.937787+0000", 2024-09-13T14:46:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 29.803648614, 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.73238416799999995, 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216126 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937787+0000", 2024-09-13T14:46:33.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937787+0000", 2024-09-13T14:46:33.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937788+0000", 2024-09-13T14:46:33.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.581e-06 2024-09-13T14:46:33.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937794+0000", 2024-09-13T14:46:33.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2229999999999998e-06 2024-09-13T14:46:33.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937807+0000", 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3628999999999999e-05 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937814+0000", 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1300000000000003e-06 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937879+0000", 2024-09-13T14:46:33.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5115000000000003e-05 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937888+0000", 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8929999999999994e-06 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663676+0000", 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.72578801999999998 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663788+0000", 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111539 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.664178+0000", 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039035099999999999 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.669828+0000", 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056497170000000003 2024-09-13T14:46:33.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.669895+0000", 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6657999999999998e-05 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670102+0000", 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020704999999999999 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670138+0000", 2024-09-13T14:46:33.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6746999999999999e-05 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670171+0000", 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2515000000000003e-05 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216152 9.15 9:aaebaddc:::benchmark_data_smithi003_156985_object95236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.973029+0000", 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 29.768406676000001, 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.70077610199999996, 2024-09-13T14:46:33.882 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216152 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973029+0000", 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973029+0000", 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973030+0000", 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1400000000000001e-06 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973035+0000", 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1059999999999999e-06 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973046+0000", 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1299e-05 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973053+0000", 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7700000000000004e-06 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973111+0000", 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8301000000000003e-05 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973120+0000", 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1300000000000007e-06 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.666444+0000", 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.69332316500000002 2024-09-13T14:46:33.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.666570+0000", 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012622499999999999 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.666968+0000", 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039803899999999999 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670221+0000", 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032530189999999998 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670280+0000", 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9024000000000003e-05 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673518+0000", 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032381599999999999 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673759+0000", 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000241308 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673805+0000", 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5416e-05 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216195 9.15 9:a896c16a:::benchmark_data_smithi003_156985_object95279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:46:33.889 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:04.016195+0000", 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 29.725239984000002, 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.66127026700000002, 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216195 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.890 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016195+0000", 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016195+0000", 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016197+0000", 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8279999999999999e-06 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016205+0000", 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8369999999999997e-06 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016227+0000", 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1956000000000001e-05 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016238+0000", 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.062e-05 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016319+0000", 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1804000000000005e-05 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016331+0000", 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1819e-05 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670325+0000", 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.65399382699999997 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670436+0000", 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111094 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670811+0000", 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000374741 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677294+0000", 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064832329999999997 2024-09-13T14:46:33.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677352+0000", 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7731e-05 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677363+0000", 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1659e-05 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677397+0000", 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3451000000000002e-05 2024-09-13T14:46:33.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677466+0000", 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8666999999999998e-05 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216200 9.15 9:af738736:::benchmark_data_smithi003_156985_object95284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:04.025460+0000", 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 29.715975084, 2024-09-13T14:46:33.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.65968305100000002, 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216200 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025460+0000", 2024-09-13T14:46:33.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025460+0000", 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025462+0000", 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8619999999999999e-06 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025469+0000", 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0720000000000002e-06 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025487+0000", 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7937000000000001e-05 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025499+0000", 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2038000000000001e-05 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025585+0000", 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5599000000000005e-05 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025597+0000", 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2665e-05 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673839+0000", 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.64824168299999996 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673947+0000", 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107537 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.674396+0000", 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044932300000000001 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.684929+0000", 2024-09-13T14:46:33.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01053345 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.684986+0000", 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6527999999999998e-05 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685046+0000", 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9769000000000003e-05 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685101+0000", 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4866e-05 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685143+0000", 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2722000000000002e-05 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216286 9.15 9:ad9baf3d:::benchmark_data_smithi003_156985_object95370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:04.132237+0000", 2024-09-13T14:46:33.903 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 29.609198085999999, 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.55320481799999999, 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216286 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132237+0000", 2024-09-13T14:46:33.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132237+0000", 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132238+0000", 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3499999999999996e-07 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132240+0000", 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3539999999999998e-06 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132246+0000", 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1689999999999997e-06 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132251+0000", 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7269999999999998e-06 2024-09-13T14:46:33.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132412+0000", 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016087499999999999 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132422+0000", 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7589999999999992e-06 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:46:33.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677499+0000", 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.54507772099999996 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677613+0000", 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114028 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.678090+0000", 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047650500000000001 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685159+0000", 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070695580000000001 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685199+0000", 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0033000000000001e-05 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685359+0000", 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015973499999999999 2024-09-13T14:46:33.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685408+0000", 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8856000000000001e-05 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685442+0000", 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3963e-05 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:46:33.911 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:46:33.911 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:46:33.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:33.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:34.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:34.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:34.191 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:34.217 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:34.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:34.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:33 smithi019 ceph-mon[91296]: 9.17 scrub starts 2024-09-13T14:46:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:33 smithi019 ceph-mon[91296]: 9.17 scrub ok 2024-09-13T14:46:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:33 smithi019 ceph-mon[91296]: 1.1 scrub starts 2024-09-13T14:46:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:33 smithi019 ceph-mon[91296]: 1.1 scrub ok 2024-09-13T14:46:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:46:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:46:34] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-13T14:46:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:33 smithi003 ceph-mon[107278]: 9.17 scrub starts 2024-09-13T14:46:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:33 smithi003 ceph-mon[107278]: 9.17 scrub ok 2024-09-13T14:46:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:33 smithi003 ceph-mon[107278]: 1.1 scrub starts 2024-09-13T14:46:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:33 smithi003 ceph-mon[107278]: 1.1 scrub ok 2024-09-13T14:46:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:33 smithi003 ceph-mon[104007]: 9.17 scrub starts 2024-09-13T14:46:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:33 smithi003 ceph-mon[104007]: 9.17 scrub ok 2024-09-13T14:46:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:33 smithi003 ceph-mon[104007]: 1.1 scrub starts 2024-09-13T14:46:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:33 smithi003 ceph-mon[104007]: 1.1 scrub ok 2024-09-13T14:46:34.466 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:34.491 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:34.593 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:34.621 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:34.723 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:34.753 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:34.855 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:34.886 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:34.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:35.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:35.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:46:35.128 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:35.156 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:35.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:35.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:35.384 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:35.414 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:35.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:34 smithi019 ceph-mon[91296]: pgmap v714: 119 pgs: 1 active+recovery_wait+degraded, 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 109 active+clean; 397 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 520 op/s; 8824/19098 objects degraded (46.204%); 6.6 MiB/s, 168 objects/s recovering 2024-09-13T14:46:35.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:34 smithi019 ceph-mon[91296]: 1.f scrub starts 2024-09-13T14:46:35.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:34 smithi019 ceph-mon[91296]: 1.f scrub ok 2024-09-13T14:46:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:34 smithi003 ceph-mon[107278]: pgmap v714: 119 pgs: 1 active+recovery_wait+degraded, 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 109 active+clean; 397 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 520 op/s; 8824/19098 objects degraded (46.204%); 6.6 MiB/s, 168 objects/s recovering 2024-09-13T14:46:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:34 smithi003 ceph-mon[107278]: 1.f scrub starts 2024-09-13T14:46:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:34 smithi003 ceph-mon[107278]: 1.f scrub ok 2024-09-13T14:46:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:34 smithi003 ceph-mon[104007]: pgmap v714: 119 pgs: 1 active+recovery_wait+degraded, 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 109 active+clean; 397 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 520 op/s; 8824/19098 objects degraded (46.204%); 6.6 MiB/s, 168 objects/s recovering 2024-09-13T14:46:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:34 smithi003 ceph-mon[104007]: 1.f scrub starts 2024-09-13T14:46:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:34 smithi003 ceph-mon[104007]: 1.f scrub ok 2024-09-13T14:46:35.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:35.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:35.659 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:35.693 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:35.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:35.795 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:35.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:35.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:35.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:36.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:36.083 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:35 smithi003 ceph-mon[107278]: 1.d scrub starts 2024-09-13T14:46:36.084 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8824/19098 objects degraded (46.204%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:36.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:35 smithi003 ceph-mon[104007]: 1.d scrub starts 2024-09-13T14:46:36.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8824/19098 objects degraded (46.204%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:36.096 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:36.198 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:36.233 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:36.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:36.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:36.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:35 smithi019 ceph-mon[91296]: 1.d scrub starts 2024-09-13T14:46:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8824/19098 objects degraded (46.204%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:36.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:36.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:36.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:36.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:36.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:36.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:36.909 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:36.942 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:37.044 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:37.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 120913 objects 2024-09-13T14:46:37.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :186.193 2024-09-13T14:46:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:37.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:37.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:37.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:37.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:37.302 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:37.324 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:37.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:37.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:36 smithi019 ceph-mon[91296]: 1.d scrub ok 2024-09-13T14:46:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:36 smithi019 ceph-mon[91296]: pgmap v715: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 160 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 36 op/s; 7910/7731 objects degraded (102.315%) 2024-09-13T14:46:37.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:36 smithi003 ceph-mon[107278]: 1.d scrub ok 2024-09-13T14:46:37.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:36 smithi003 ceph-mon[107278]: pgmap v715: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 160 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 36 op/s; 7910/7731 objects degraded (102.315%) 2024-09-13T14:46:37.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:36 smithi003 ceph-mon[104007]: 1.d scrub ok 2024-09-13T14:46:37.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:36 smithi003 ceph-mon[104007]: pgmap v715: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 160 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 36 op/s; 7910/7731 objects degraded (102.315%) 2024-09-13T14:46:37.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:37.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:37.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:37.674 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:37.696 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:37.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:37.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:37.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:37.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:38.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:38.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:38.173 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:38.195 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:38.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:38.205 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:46:38.205 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:46:38.206 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-09-13T14:46:38.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:46:38.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:38.318 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:38.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:38.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:37 smithi019 ceph-mon[91296]: 1.11 scrub starts 2024-09-13T14:46:38.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:37 smithi019 ceph-mon[91296]: 1.11 scrub ok 2024-09-13T14:46:38.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:38 smithi003 ceph-mon[107278]: 1.11 scrub starts 2024-09-13T14:46:38.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:38 smithi003 ceph-mon[107278]: 1.11 scrub ok 2024-09-13T14:46:38.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:37 smithi003 ceph-mon[104007]: 1.11 scrub starts 2024-09-13T14:46:38.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:37 smithi003 ceph-mon[104007]: 1.11 scrub ok 2024-09-13T14:46:38.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:38.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:38.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:38.566 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:38.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:38.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:38.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:38.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:38.916 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:38.938 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:39.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:39.062 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:39.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:39.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:39.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:39.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:39.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:39 smithi019 ceph-mon[91296]: 1.3 scrub starts 2024-09-13T14:46:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:39 smithi019 ceph-mon[91296]: 1.3 scrub ok 2024-09-13T14:46:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:39 smithi019 ceph-mon[91296]: pgmap v716: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 138 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 41 op/s; 7910/6681 objects degraded (118.395%) 2024-09-13T14:46:39.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:39 smithi003 ceph-mon[104007]: 1.3 scrub starts 2024-09-13T14:46:39.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:39 smithi003 ceph-mon[104007]: 1.3 scrub ok 2024-09-13T14:46:39.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:39 smithi003 ceph-mon[104007]: pgmap v716: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 138 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 41 op/s; 7910/6681 objects degraded (118.395%) 2024-09-13T14:46:39.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:39 smithi003 ceph-mon[107278]: 1.3 scrub starts 2024-09-13T14:46:39.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:39 smithi003 ceph-mon[107278]: 1.3 scrub ok 2024-09-13T14:46:39.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:39 smithi003 ceph-mon[107278]: pgmap v716: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 138 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 41 op/s; 7910/6681 objects degraded (118.395%) 2024-09-13T14:46:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:39.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:39.561 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:39.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:39.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:46:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:39.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:39.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:39.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:39.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:40.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.065 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:40.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:40.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:40.188 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:40 smithi003 ceph-mon[107278]: 1.2 scrub starts 2024-09-13T14:46:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:40 smithi003 ceph-mon[107278]: 1.2 scrub ok 2024-09-13T14:46:40.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:40 smithi003 ceph-mon[104007]: 1.2 scrub starts 2024-09-13T14:46:40.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:40 smithi003 ceph-mon[104007]: 1.2 scrub ok 2024-09-13T14:46:40.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:40.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:40.332 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:46:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:46:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:40.332 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:40 smithi019 ceph-mon[91296]: 1.2 scrub starts 2024-09-13T14:46:40.333 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:40 smithi019 ceph-mon[91296]: 1.2 scrub ok 2024-09-13T14:46:40.416 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:40.440 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:40.508 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:40.511 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:40.512 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:40.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:40.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:40.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:40.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:40.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:40.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:40.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:40.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:41.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:41.063 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:41.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:41.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:41.290 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:41.313 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:41.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:41 smithi003 ceph-mon[107278]: 1.12 scrub starts 2024-09-13T14:46:41.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:41 smithi003 ceph-mon[107278]: 1.12 scrub ok 2024-09-13T14:46:41.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:41 smithi003 ceph-mon[107278]: pgmap v717: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 32 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 7910/1587 objects degraded (498.425%) 2024-09-13T14:46:41.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 7910/1587 objects degraded (498.425%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:41.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:41 smithi003 ceph-mon[104007]: 1.12 scrub starts 2024-09-13T14:46:41.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:41 smithi003 ceph-mon[104007]: 1.12 scrub ok 2024-09-13T14:46:41.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:41 smithi003 ceph-mon[104007]: pgmap v717: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 32 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 7910/1587 objects degraded (498.425%) 2024-09-13T14:46:41.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 7910/1587 objects degraded (498.425%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:41.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:41.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:41 smithi019 ceph-mon[91296]: 1.12 scrub starts 2024-09-13T14:46:41.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:41 smithi019 ceph-mon[91296]: 1.12 scrub ok 2024-09-13T14:46:41.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:41 smithi019 ceph-mon[91296]: pgmap v717: 119 pgs: 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 32 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 7910/1587 objects degraded (498.425%) 2024-09-13T14:46:41.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 7910/1587 objects degraded (498.425%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:41.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:41.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:41.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:41.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:41.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:41.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:41.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:41.813 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:41.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:41.937 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:42.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:42.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:42.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:42.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:42 smithi003 ceph-mon[107278]: 1.e scrub starts 2024-09-13T14:46:42.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:42 smithi003 ceph-mon[107278]: 1.e scrub ok 2024-09-13T14:46:42.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:42 smithi003 ceph-mon[107278]: 1.c scrub starts 2024-09-13T14:46:42.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:42 smithi003 ceph-mon[107278]: 1.c scrub ok 2024-09-13T14:46:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:42 smithi003 ceph-mon[104007]: 1.e scrub starts 2024-09-13T14:46:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:42 smithi003 ceph-mon[104007]: 1.e scrub ok 2024-09-13T14:46:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:42 smithi003 ceph-mon[104007]: 1.c scrub starts 2024-09-13T14:46:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:42 smithi003 ceph-mon[104007]: 1.c scrub ok 2024-09-13T14:46:42.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:42.288 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:42.289 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:46:42.289 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:46:42.289 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:46:42.289 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:46:42.289 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:46:42.289 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:46:42.289 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:42.289 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:42.289 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:42.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:42.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:42.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:42.296 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:46:42.296 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:42.311 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:42.313 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:46:42.313 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:42.414 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:42.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:42 smithi019 ceph-mon[91296]: 1.e scrub starts 2024-09-13T14:46:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:42 smithi019 ceph-mon[91296]: 1.e scrub ok 2024-09-13T14:46:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:42 smithi019 ceph-mon[91296]: 1.c scrub starts 2024-09-13T14:46:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:42 smithi019 ceph-mon[91296]: 1.c scrub ok 2024-09-13T14:46:42.436 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:42.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:42.560 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:42.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:42.661 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:42.684 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:42.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:42.807 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:42.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:42.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:43.034 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:43.057 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:43.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:43.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:43.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:43.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:43.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:43.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:43 smithi003 ceph-mon[107278]: pgmap v718: 119 pgs: 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 7583/78 objects degraded (9721.795%) 2024-09-13T14:46:43.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1783316277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:43.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:43.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:43 smithi003 ceph-mon[104007]: pgmap v718: 119 pgs: 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 7583/78 objects degraded (9721.795%) 2024-09-13T14:46:43.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1783316277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:43.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:43 smithi019 ceph-mon[91296]: pgmap v718: 119 pgs: 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 7583/78 objects degraded (9721.795%) 2024-09-13T14:46:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1783316277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.433 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:43.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:43.534 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:43.556 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:43.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:43.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:43.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:43.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:43.928 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:44.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:44.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:44.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:44.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:44.270 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:44 smithi003 ceph-mon[104007]: 1.13 scrub starts 2024-09-13T14:46:44.270 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:44 smithi003 ceph-mon[104007]: 1.13 scrub ok 2024-09-13T14:46:44.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:44 smithi003 ceph-mon[107278]: 1.13 scrub starts 2024-09-13T14:46:44.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:44 smithi003 ceph-mon[107278]: 1.13 scrub ok 2024-09-13T14:46:44.270 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:46:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:46:44] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:46:44.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:44.301 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:44.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:44.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:44 smithi019 ceph-mon[91296]: 1.13 scrub starts 2024-09-13T14:46:44.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:44 smithi019 ceph-mon[91296]: 1.13 scrub ok 2024-09-13T14:46:44.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:44.550 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:44.651 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:44.674 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:44.775 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:44.798 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:44.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:44.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:45.049 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:45.071 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:45.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:45.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:45.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:45.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:45.360 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:45 smithi003 ceph-mon[107278]: 1.0 scrub starts 2024-09-13T14:46:45.360 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:45 smithi003 ceph-mon[107278]: 1.0 scrub ok 2024-09-13T14:46:45.360 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:45 smithi003 ceph-mon[107278]: pgmap v719: 119 pgs: 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 7583/78 objects degraded (9721.795%) 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:45 smithi003 ceph-mon[104007]: 1.0 scrub starts 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:45 smithi003 ceph-mon[104007]: 1.0 scrub ok 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:45 smithi003 ceph-mon[104007]: pgmap v719: 119 pgs: 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 7583/78 objects degraded (9721.795%) 2024-09-13T14:46:45.422 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:45 smithi019 ceph-mon[91296]: 1.0 scrub starts 2024-09-13T14:46:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:45 smithi019 ceph-mon[91296]: 1.0 scrub ok 2024-09-13T14:46:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:45 smithi019 ceph-mon[91296]: pgmap v719: 119 pgs: 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 7583/78 objects degraded (9721.795%) 2024-09-13T14:46:45.445 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:45.549 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:45.572 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:45.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:45.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:45.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:45.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:45.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:45.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:45.946 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:46.048 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:46.070 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:46.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:46 smithi003 ceph-mon[107278]: 1.7 scrub starts 2024-09-13T14:46:46.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:46 smithi003 ceph-mon[107278]: 1.7 scrub ok 2024-09-13T14:46:46.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 7583/78 objects degraded (9721.795%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:46.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:46 smithi003 ceph-mon[104007]: 1.7 scrub starts 2024-09-13T14:46:46.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:46 smithi003 ceph-mon[104007]: 1.7 scrub ok 2024-09-13T14:46:46.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 7583/78 objects degraded (9721.795%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:46.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:46.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:46.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:46.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:46 smithi019 ceph-mon[91296]: 1.7 scrub starts 2024-09-13T14:46:46.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:46 smithi019 ceph-mon[91296]: 1.7 scrub ok 2024-09-13T14:46:46.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 7583/78 objects degraded (9721.795%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:46.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:46.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:46.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:46.671 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:46.693 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:46.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:46.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:46.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:47.032 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:46:47.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:47.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:47.129 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:46:47.130 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:46:47.173 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:47.196 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:47.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:47.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:47.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:47.421 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:47 smithi019 ceph-mon[91296]: 1.4 scrub starts 2024-09-13T14:46:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:47 smithi019 ceph-mon[91296]: 1.4 scrub ok 2024-09-13T14:46:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:47 smithi019 ceph-mon[91296]: pgmap v720: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 795 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 5084/78 objects degraded (6517.949%) 2024-09-13T14:46:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2487615460' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:47 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[107278]: 1.4 scrub starts 2024-09-13T14:46:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[107278]: 1.4 scrub ok 2024-09-13T14:46:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[107278]: pgmap v720: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 795 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 5084/78 objects degraded (6517.949%) 2024-09-13T14:46:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2487615460' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[104007]: 1.4 scrub starts 2024-09-13T14:46:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[104007]: 1.4 scrub ok 2024-09-13T14:46:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[104007]: pgmap v720: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 795 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 5084/78 objects degraded (6517.949%) 2024-09-13T14:46:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2487615460' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:47.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:47 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.445 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:47.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:47.546 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:47.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:47.602 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:47.603 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:47.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:47.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:47.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:47.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:46:47.820 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:47.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:47.944 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:48.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:48.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:48.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[107278]: 1.14 scrub starts 2024-09-13T14:46:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[107278]: 1.14 scrub ok 2024-09-13T14:46:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:46:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:48.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[107278]: osdmap e217: 8 total, 8 up, 8 in 2024-09-13T14:46:48.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[107278]: from='client.36087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[104007]: 1.14 scrub starts 2024-09-13T14:46:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[104007]: 1.14 scrub ok 2024-09-13T14:46:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:46:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[104007]: osdmap e217: 8 total, 8 up, 8 in 2024-09-13T14:46:48.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:48 smithi003 ceph-mon[104007]: from='client.36087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:48.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:48.295 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:48.320 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:48.385 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2024-09-13T14:46:47.028336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":814370,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206591,"num_write_kb":5719393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50029,"num_bytes_recovered":2983395534,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159089,"ondisk_log_size":159089,"up":357,"acting":353,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":412,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10153268,"kb_used_data":4743412,"kb_used_omap":29,"kb_used_meta":5409762,"kb_avail":739807948,"statfs":{"total":767960285184,"available":757563338752,"internally_reserved":0,"allocated":4857253888,"data_stored":4848170704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":5539596814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-166898371,"num_objects":-2551,"num_object_clones":0,"num_object_copies":-7653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2551,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2476,"num_bytes_recovered":12124160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.317988"},"pg_stats":[{"pgid":"1.11","version":"79'100","reported_seq":269,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:36.384655+0000","last_change":"2024-09-13T14:46:36.384655+0000","last_active":"2024-09-13T14:46:36.384655+0000","last_peered":"2024-09-13T14:46:36.384655+0000","last_clean":"2024-09-13T14:46:36.384655+0000","last_became_active":"2024-09-13T14:46:24.879116+0000","last_became_peered":"2024-09-13T14:46:24.879116+0000","last_unstale":"2024-09-13T14:46:36.384655+0000","last_undegraded":"2024-09-13T14:46:36.384655+0000","last_fullsized":"2024-09-13T14:46:36.384655+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:25:08.340894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.19","version":"216'9199","reported_seq":9376,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:36.977171+0000","last_change":"2024-09-13T14:42:18.899519+0000","last_active":"2024-09-13T14:46:36.977171+0000","last_peered":"2024-09-13T14:46:36.977171+0000","last_clean":"2024-09-13T14:46:36.977171+0000","last_became_active":"2024-09-13T14:42:15.129169+0000","last_became_peered":"2024-09-13T14:42:15.129169+0000","last_unstale":"2024-09-13T14:46:36.977171+0000","last_undegraded":"2024-09-13T14:46:36.977171+0000","last_fullsized":"2024-09-13T14:46:36.977171+0000","mapping_epoch":159,"log_start":"172'4158","ondisk_log_start":"172'4158","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3130","last_scrub_stamp":"2024-09-13T14:42:18.899277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:18.899277+0000","objects_scrubbed":1667,"log_size":5041,"log_dups_size":0,"ondisk_log_size":5041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:03.200084+0000","scrub_duration":425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":141376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":242,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.962386+0000","last_change":"2024-09-13T14:43:19.875328+0000","last_active":"2024-09-13T14:46:31.962386+0000","last_peered":"2024-09-13T14:46:31.962386+0000","last_clean":"2024-09-13T14:46:31.962386+0000","last_became_active":"2024-09-13T14:43:19.874507+0000","last_became_peered":"2024-09-13T14:43:19.874507+0000","last_unstale":"2024-09-13T14:46:31.962386+0000","last_undegraded":"2024-09-13T14:46:31.962386+0000","last_fullsized":"2024-09-13T14:46:31.962386+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:07.089107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":182,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.952575+0000","last_change":"2024-09-13T14:35:30.492165+0000","last_active":"2024-09-13T14:46:25.952575+0000","last_peered":"2024-09-13T14:46:25.952575+0000","last_clean":"2024-09-13T14:46:25.952575+0000","last_became_active":"2024-09-13T14:35:30.491844+0000","last_became_peered":"2024-09-13T14:35:30.491844+0000","last_unstale":"2024-09-13T14:46:25.952575+0000","last_undegraded":"2024-09-13T14:46:25.952575+0000","last_fullsized":"2024-09-13T14:46:25.952575+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:22.876890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":270,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:32.271341+0000","last_change":"2024-09-13T14:46:32.271341+0000","last_active":"2024-09-13T14:46:32.271341+0000","last_peered":"2024-09-13T14:46:32.271341+0000","last_clean":"2024-09-13T14:46:32.271341+0000","last_became_active":"2024-09-13T14:46:24.878995+0000","last_became_peered":"2024-09-13T14:46:24.878995+0000","last_unstale":"2024-09-13T14:46:32.271341+0000","last_undegraded":"2024-09-13T14:46:32.271341+0000","last_fullsized":"2024-09-13T14:46:32.271341+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:42.845875+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"216'9054","reported_seq":20519,"reported_epoch":216,"state":"active+recovery_wait+undersized+remapped","last_fresh":"2024-09-13T14:46:37.059438+0000","last_change":"2024-09-13T14:46:36.969373+0000","last_active":"2024-09-13T14:46:37.059438+0000","last_peered":"2024-09-13T14:46:37.059438+0000","last_clean":"2024-09-13T14:43:18.728712+0000","last_became_active":"2024-09-13T14:43:21.499842+0000","last_became_peered":"2024-09-13T14:43:21.499842+0000","last_unstale":"2024-09-13T14:46:37.059438+0000","last_undegraded":"2024-09-13T14:46:37.059438+0000","last_fullsized":"2024-09-13T14:43:21.112483+0000","mapping_epoch":179,"log_start":"171'4025","ondisk_log_start":"171'4025","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'3290","last_scrub_stamp":"2024-09-13T14:42:20.569479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:20.569479+0000","objects_scrubbed":1768,"log_size":5029,"log_dups_size":0,"ondisk_log_size":5029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":240,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":127872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3760,"num_bytes_recovered":246415360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,1],"avail_no_missing":["0","1","3","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":140,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.957596+0000","last_change":"2024-09-13T14:43:19.874928+0000","last_active":"2024-09-13T14:46:25.957596+0000","last_peered":"2024-09-13T14:46:25.957596+0000","last_clean":"2024-09-13T14:46:25.957596+0000","last_became_active":"2024-09-13T14:43:19.874518+0000","last_became_peered":"2024-09-13T14:43:19.874518+0000","last_unstale":"2024-09-13T14:46:25.957596+0000","last_undegraded":"2024-09-13T14:46:25.957596+0000","last_fullsized":"2024-09-13T14:46:25.957596+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:15:22.392336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":311,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.947381+0000","last_change":"2024-09-13T14:43:19.870143+0000","last_active":"2024-09-13T14:46:31.947381+0000","last_peered":"2024-09-13T14:46:31.947381+0000","last_clean":"2024-09-13T14:46:31.947381+0000","last_became_active":"2024-09-13T14:43:19.869829+0000","last_became_peered":"2024-09-13T14:43:19.869829+0000","last_unstale":"2024-09-13T14:46:31.947381+0000","last_undegraded":"2024-09-13T14:46:31.947381+0000","last_fullsized":"2024-09-13T14:46:31.947381+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.422457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":270,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:42.355175+0000","last_change":"2024-09-13T14:46:42.355175+0000","last_active":"2024-09-13T14:46:42.355175+0000","last_peered":"2024-09-13T14:46:42.355175+0000","last_clean":"2024-09-13T14:46:42.355175+0000","last_became_active":"2024-09-13T14:46:24.883838+0000","last_became_peered":"2024-09-13T14:46:24.883838+0000","last_unstale":"2024-09-13T14:46:42.355175+0000","last_undegraded":"2024-09-13T14:46:42.355175+0000","last_fullsized":"2024-09-13T14:46:42.355175+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:31.858048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":312,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.940131+0000","last_change":"2024-09-13T14:43:19.871195+0000","last_active":"2024-09-13T14:46:31.940131+0000","last_peered":"2024-09-13T14:46:31.940131+0000","last_clean":"2024-09-13T14:46:31.940131+0000","last_became_active":"2024-09-13T14:43:19.870496+0000","last_became_peered":"2024-09-13T14:43:19.870496+0000","last_unstale":"2024-09-13T14:46:31.940131+0000","last_undegraded":"2024-09-13T14:46:31.940131+0000","last_fullsized":"2024-09-13T14:46:31.940131+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:25.646089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":163,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.952542+0000","last_change":"2024-09-13T14:21:52.611482+0000","last_active":"2024-09-13T14:46:25.952542+0000","last_peered":"2024-09-13T14:46:25.952542+0000","last_clean":"2024-09-13T14:46:25.952542+0000","last_became_active":"2024-09-13T14:21:52.611213+0000","last_became_peered":"2024-09-13T14:21:52.611213+0000","last_unstale":"2024-09-13T14:46:25.952542+0000","last_undegraded":"2024-09-13T14:46:25.952542+0000","last_fullsized":"2024-09-13T14:46:25.952542+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:55.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":267,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:39.383066+0000","last_change":"2024-09-13T14:46:39.383066+0000","last_active":"2024-09-13T14:46:39.383066+0000","last_peered":"2024-09-13T14:46:39.383066+0000","last_clean":"2024-09-13T14:46:39.383066+0000","last_became_active":"2024-09-13T14:46:24.882618+0000","last_became_peered":"2024-09-13T14:46:24.882618+0000","last_unstale":"2024-09-13T14:46:39.383066+0000","last_undegraded":"2024-09-13T14:46:39.383066+0000","last_fullsized":"2024-09-13T14:46:39.383066+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:58.813263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":267,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956803+0000","last_change":"2024-09-13T14:43:19.872044+0000","last_active":"2024-09-13T14:46:31.956803+0000","last_peered":"2024-09-13T14:46:31.956803+0000","last_clean":"2024-09-13T14:46:31.956803+0000","last_became_active":"2024-09-13T14:43:19.870646+0000","last_became_peered":"2024-09-13T14:43:19.870646+0000","last_unstale":"2024-09-13T14:46:31.956803+0000","last_undegraded":"2024-09-13T14:46:31.956803+0000","last_fullsized":"2024-09-13T14:46:31.956803+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:33.158472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":207,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.867667+0000","last_change":"2024-09-13T14:43:19.888180+0000","last_active":"2024-09-13T14:46:25.867667+0000","last_peered":"2024-09-13T14:46:25.867667+0000","last_clean":"2024-09-13T14:46:25.867667+0000","last_became_active":"2024-09-13T14:43:19.875873+0000","last_became_peered":"2024-09-13T14:43:19.875873+0000","last_unstale":"2024-09-13T14:46:25.867667+0000","last_undegraded":"2024-09-13T14:46:25.867667+0000","last_fullsized":"2024-09-13T14:46:25.867667+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:47.078713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":267,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:35.387752+0000","last_change":"2024-09-13T14:46:35.387752+0000","last_active":"2024-09-13T14:46:35.387752+0000","last_peered":"2024-09-13T14:46:35.387752+0000","last_clean":"2024-09-13T14:46:35.387752+0000","last_became_active":"2024-09-13T14:46:24.879415+0000","last_became_peered":"2024-09-13T14:46:24.879415+0000","last_unstale":"2024-09-13T14:46:35.387752+0000","last_undegraded":"2024-09-13T14:46:35.387752+0000","last_fullsized":"2024-09-13T14:46:35.387752+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:42.176681+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"216'8957","reported_seq":20302,"reported_epoch":216,"state":"active+recovery_wait+undersized+remapped","last_fresh":"2024-09-13T14:46:37.011932+0000","last_change":"2024-09-13T14:46:36.972351+0000","last_active":"2024-09-13T14:46:37.011932+0000","last_peered":"2024-09-13T14:46:37.011932+0000","last_clean":"2024-09-13T14:43:18.840495+0000","last_became_active":"2024-09-13T14:43:21.306427+0000","last_became_peered":"2024-09-13T14:43:21.306427+0000","last_unstale":"2024-09-13T14:46:37.011932+0000","last_undegraded":"2024-09-13T14:46:37.011932+0000","last_fullsized":"2024-09-13T14:43:20.854552+0000","mapping_epoch":179,"log_start":"171'3954","ondisk_log_start":"171'3954","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3036","last_scrub_stamp":"2024-09-13T14:42:16.024050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.024050+0000","objects_scrubbed":1508,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5938,"num_write_kb":141760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3721,"num_bytes_recovered":243859456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1],"avail_no_missing":["0","1","3","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":65,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868373+0000","last_change":"2024-09-13T14:43:19.883523+0000","last_active":"2024-09-13T14:46:25.868373+0000","last_peered":"2024-09-13T14:46:25.868373+0000","last_clean":"2024-09-13T14:46:25.868373+0000","last_became_active":"2024-09-13T14:43:19.871241+0000","last_became_peered":"2024-09-13T14:43:19.871241+0000","last_unstale":"2024-09-13T14:46:25.868373+0000","last_undegraded":"2024-09-13T14:46:25.868373+0000","last_fullsized":"2024-09-13T14:46:25.868373+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:04.592529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":50,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.867526+0000","last_change":"2024-09-13T14:43:19.874136+0000","last_active":"2024-09-13T14:46:25.867526+0000","last_peered":"2024-09-13T14:46:25.867526+0000","last_clean":"2024-09-13T14:46:25.867526+0000","last_became_active":"2024-09-13T14:43:19.873671+0000","last_became_peered":"2024-09-13T14:43:19.873671+0000","last_unstale":"2024-09-13T14:46:25.867526+0000","last_undegraded":"2024-09-13T14:46:25.867526+0000","last_fullsized":"2024-09-13T14:46:25.867526+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:51.386822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":268,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:41.340234+0000","last_change":"2024-09-13T14:46:41.340234+0000","last_active":"2024-09-13T14:46:41.340234+0000","last_peered":"2024-09-13T14:46:41.340234+0000","last_clean":"2024-09-13T14:46:41.340234+0000","last_became_active":"2024-09-13T14:46:24.884430+0000","last_became_peered":"2024-09-13T14:46:24.884430+0000","last_unstale":"2024-09-13T14:46:41.340234+0000","last_undegraded":"2024-09-13T14:46:41.340234+0000","last_fullsized":"2024-09-13T14:46:41.340234+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:28.225074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"216'9192","reported_seq":9360,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.002315+0000","last_change":"2024-09-13T14:45:44.488534+0000","last_active":"2024-09-13T14:46:37.002315+0000","last_peered":"2024-09-13T14:46:37.002315+0000","last_clean":"2024-09-13T14:46:37.002315+0000","last_became_active":"2024-09-13T14:45:44.444948+0000","last_became_peered":"2024-09-13T14:45:44.444948+0000","last_unstale":"2024-09-13T14:46:37.002315+0000","last_undegraded":"2024-09-13T14:46:37.002315+0000","last_fullsized":"2024-09-13T14:46:37.002315+0000","mapping_epoch":199,"log_start":"157'2328","ondisk_log_start":"157'2328","created":152,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'2973","last_scrub_stamp":"2024-09-13T14:42:15.820869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:15.820869+0000","objects_scrubbed":1533,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:24.564016+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6227,"num_write_kb":150208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":182,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.956499+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:46:25.956499+0000","last_peered":"2024-09-13T14:46:25.956499+0000","last_clean":"2024-09-13T14:46:25.956499+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:46:25.956499+0000","last_undegraded":"2024-09-13T14:46:25.956499+0000","last_fullsized":"2024-09-13T14:46:25.956499+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":195,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868164+0000","last_change":"2024-09-13T14:37:08.752050+0000","last_active":"2024-09-13T14:46:25.868164+0000","last_peered":"2024-09-13T14:46:25.868164+0000","last_clean":"2024-09-13T14:46:25.868164+0000","last_became_active":"2024-09-13T14:37:08.751650+0000","last_became_peered":"2024-09-13T14:37:08.751650+0000","last_unstale":"2024-09-13T14:46:25.868164+0000","last_undegraded":"2024-09-13T14:46:25.868164+0000","last_fullsized":"2024-09-13T14:46:25.868164+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:17.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":267,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:34.297919+0000","last_change":"2024-09-13T14:46:34.297919+0000","last_active":"2024-09-13T14:46:34.297919+0000","last_peered":"2024-09-13T14:46:34.297919+0000","last_clean":"2024-09-13T14:46:34.297919+0000","last_became_active":"2024-09-13T14:46:24.877554+0000","last_became_peered":"2024-09-13T14:46:24.877554+0000","last_unstale":"2024-09-13T14:46:34.297919+0000","last_undegraded":"2024-09-13T14:46:34.297919+0000","last_fullsized":"2024-09-13T14:46:34.297919+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:34.297813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:34.297813+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:34.753670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"216'9023","reported_seq":20427,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:36.974174+0000","last_change":"2024-09-13T14:44:59.050198+0000","last_active":"2024-09-13T14:46:36.974174+0000","last_peered":"2024-09-13T14:46:36.974174+0000","last_clean":"2024-09-13T14:46:36.974174+0000","last_became_active":"2024-09-13T14:44:58.997243+0000","last_became_peered":"2024-09-13T14:44:58.997243+0000","last_unstale":"2024-09-13T14:46:36.974174+0000","last_undegraded":"2024-09-13T14:46:36.974174+0000","last_fullsized":"2024-09-13T14:46:36.974174+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3303","last_scrub_stamp":"2024-09-13T14:42:19.788610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:19.788610+0000","objects_scrubbed":1743,"log_size":9023,"log_dups_size":0,"ondisk_log_size":9023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:24.047041+0000","scrub_duration":0.62419811800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5741,"num_write_kb":127936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3741,"num_bytes_recovered":245170176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":178,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.957776+0000","last_change":"2024-09-13T14:43:19.880218+0000","last_active":"2024-09-13T14:46:25.957776+0000","last_peered":"2024-09-13T14:46:25.957776+0000","last_clean":"2024-09-13T14:46:25.957776+0000","last_became_active":"2024-09-13T14:43:19.874769+0000","last_became_peered":"2024-09-13T14:43:19.874769+0000","last_unstale":"2024-09-13T14:46:25.957776+0000","last_undegraded":"2024-09-13T14:46:25.957776+0000","last_fullsized":"2024-09-13T14:46:25.957776+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:49.775209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":181,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868173+0000","last_change":"2024-09-13T14:43:19.860813+0000","last_active":"2024-09-13T14:46:25.868173+0000","last_peered":"2024-09-13T14:46:25.868173+0000","last_clean":"2024-09-13T14:46:25.868173+0000","last_became_active":"2024-09-13T14:43:19.860518+0000","last_became_peered":"2024-09-13T14:43:19.860518+0000","last_unstale":"2024-09-13T14:46:25.868173+0000","last_undegraded":"2024-09-13T14:46:25.868173+0000","last_fullsized":"2024-09-13T14:46:25.868173+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:21.310511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":267,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:40.349553+0000","last_change":"2024-09-13T14:46:40.349553+0000","last_active":"2024-09-13T14:46:40.349553+0000","last_peered":"2024-09-13T14:46:40.349553+0000","last_clean":"2024-09-13T14:46:40.349553+0000","last_became_active":"2024-09-13T14:46:24.883713+0000","last_became_peered":"2024-09-13T14:46:24.883713+0000","last_unstale":"2024-09-13T14:46:40.349553+0000","last_undegraded":"2024-09-13T14:46:40.349553+0000","last_fullsized":"2024-09-13T14:46:40.349553+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:28.208975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"216'8998","reported_seq":9682,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.023407+0000","last_change":"2024-09-13T14:43:28.958578+0000","last_active":"2024-09-13T14:46:37.023407+0000","last_peered":"2024-09-13T14:46:37.023407+0000","last_clean":"2024-09-13T14:46:37.023407+0000","last_became_active":"2024-09-13T14:43:28.955368+0000","last_became_peered":"2024-09-13T14:43:28.955368+0000","last_unstale":"2024-09-13T14:46:37.023407+0000","last_undegraded":"2024-09-13T14:46:37.023407+0000","last_fullsized":"2024-09-13T14:46:37.023407+0000","mapping_epoch":181,"log_start":"171'3518","ondisk_log_start":"171'3518","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3084","last_scrub_stamp":"2024-09-13T14:42:16.825170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.825170+0000","objects_scrubbed":1592,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:56.306613+0000","scrub_duration":0.667424664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":138816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":126484480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":178,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.865659+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:46:25.865659+0000","last_peered":"2024-09-13T14:46:25.865659+0000","last_clean":"2024-09-13T14:46:25.865659+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:46:25.865659+0000","last_undegraded":"2024-09-13T14:46:25.865659+0000","last_fullsized":"2024-09-13T14:46:25.865659+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":305,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.950408+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:46:31.950408+0000","last_peered":"2024-09-13T14:46:31.950408+0000","last_clean":"2024-09-13T14:46:31.950408+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:46:31.950408+0000","last_undegraded":"2024-09-13T14:46:31.950408+0000","last_fullsized":"2024-09-13T14:46:31.950408+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":253,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.948763+0000","last_change":"2024-09-13T14:45:40.724301+0000","last_active":"2024-09-13T14:46:31.948763+0000","last_peered":"2024-09-13T14:46:31.948763+0000","last_clean":"2024-09-13T14:46:31.948763+0000","last_became_active":"2024-09-13T14:45:33.246028+0000","last_became_peered":"2024-09-13T14:45:33.246028+0000","last_unstale":"2024-09-13T14:46:31.948763+0000","last_undegraded":"2024-09-13T14:46:31.948763+0000","last_fullsized":"2024-09-13T14:46:31.948763+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:30.500946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"216'8847","reported_seq":16422,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.020379+0000","last_change":"2024-09-13T14:44:21.618688+0000","last_active":"2024-09-13T14:46:37.020379+0000","last_peered":"2024-09-13T14:46:37.020379+0000","last_clean":"2024-09-13T14:46:37.020379+0000","last_became_active":"2024-09-13T14:44:21.602316+0000","last_became_peered":"2024-09-13T14:44:21.602316+0000","last_unstale":"2024-09-13T14:46:37.020379+0000","last_undegraded":"2024-09-13T14:46:37.020379+0000","last_fullsized":"2024-09-13T14:46:37.020379+0000","mapping_epoch":186,"log_start":"160'3162","ondisk_log_start":"160'3162","created":152,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'2995","last_scrub_stamp":"2024-09-13T14:42:16.817217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.817217+0000","objects_scrubbed":1525,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:34.225750+0000","scrub_duration":583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5865,"num_write_kb":138880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3692,"num_bytes_recovered":241958912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":266,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.955770+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:46:31.955770+0000","last_peered":"2024-09-13T14:46:31.955770+0000","last_clean":"2024-09-13T14:46:31.955770+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:46:31.955770+0000","last_undegraded":"2024-09-13T14:46:31.955770+0000","last_fullsized":"2024-09-13T14:46:31.955770+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":178,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868064+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:46:25.868064+0000","last_peered":"2024-09-13T14:46:25.868064+0000","last_clean":"2024-09-13T14:46:25.868064+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:46:25.868064+0000","last_undegraded":"2024-09-13T14:46:25.868064+0000","last_fullsized":"2024-09-13T14:46:25.868064+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":253,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.940594+0000","last_change":"2024-09-13T14:45:39.694414+0000","last_active":"2024-09-13T14:46:31.940594+0000","last_peered":"2024-09-13T14:46:31.940594+0000","last_clean":"2024-09-13T14:46:31.940594+0000","last_became_active":"2024-09-13T14:45:33.242795+0000","last_became_peered":"2024-09-13T14:45:33.242795+0000","last_unstale":"2024-09-13T14:46:31.940594+0000","last_undegraded":"2024-09-13T14:46:31.940594+0000","last_fullsized":"2024-09-13T14:46:31.940594+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:33.693079+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"216'9198","reported_seq":17027,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.006702+0000","last_change":"2024-09-13T14:46:16.674442+0000","last_active":"2024-09-13T14:46:37.006702+0000","last_peered":"2024-09-13T14:46:37.006702+0000","last_clean":"2024-09-13T14:46:37.006702+0000","last_became_active":"2024-09-13T14:46:16.627004+0000","last_became_peered":"2024-09-13T14:46:16.627004+0000","last_unstale":"2024-09-13T14:46:37.006702+0000","last_undegraded":"2024-09-13T14:46:37.006702+0000","last_fullsized":"2024-09-13T14:46:37.006702+0000","mapping_epoch":209,"log_start":"171'3706","ondisk_log_start":"171'3706","created":152,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3278","last_scrub_stamp":"2024-09-13T14:42:18.806999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:18.806999+0000","objects_scrubbed":1737,"log_size":5492,"log_dups_size":0,"ondisk_log_size":5492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:30.702388+0000","scrub_duration":487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5931,"num_write_kb":134208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3832,"num_bytes_recovered":139722752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":189,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868455+0000","last_change":"2024-09-13T14:43:19.860723+0000","last_active":"2024-09-13T14:46:25.868455+0000","last_peered":"2024-09-13T14:46:25.868455+0000","last_clean":"2024-09-13T14:46:25.868455+0000","last_became_active":"2024-09-13T14:43:19.860401+0000","last_became_peered":"2024-09-13T14:43:19.860401+0000","last_unstale":"2024-09-13T14:46:25.868455+0000","last_undegraded":"2024-09-13T14:46:25.868455+0000","last_fullsized":"2024-09-13T14:46:25.868455+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:27.920262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":195,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868510+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:46:25.868510+0000","last_peered":"2024-09-13T14:46:25.868510+0000","last_clean":"2024-09-13T14:46:25.868510+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:46:25.868510+0000","last_undegraded":"2024-09-13T14:46:25.868510+0000","last_fullsized":"2024-09-13T14:46:25.868510+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":265,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.945464+0000","last_change":"2024-09-13T14:46:31.945464+0000","last_active":"2024-09-13T14:46:31.945464+0000","last_peered":"2024-09-13T14:46:31.945464+0000","last_clean":"2024-09-13T14:46:31.945464+0000","last_became_active":"2024-09-13T14:46:24.878793+0000","last_became_peered":"2024-09-13T14:46:24.878793+0000","last_unstale":"2024-09-13T14:46:31.945464+0000","last_undegraded":"2024-09-13T14:46:31.945464+0000","last_fullsized":"2024-09-13T14:46:31.945464+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:44.649596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:44.649596+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:46:35.250620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3","version":"216'8998","reported_seq":9081,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.071040+0000","last_change":"2024-09-13T14:42:15.076493+0000","last_active":"2024-09-13T14:46:37.071040+0000","last_peered":"2024-09-13T14:46:37.071040+0000","last_clean":"2024-09-13T14:46:37.071040+0000","last_became_active":"2024-09-13T14:42:15.074126+0000","last_became_peered":"2024-09-13T14:42:15.074126+0000","last_unstale":"2024-09-13T14:46:37.071040+0000","last_undegraded":"2024-09-13T14:46:37.071040+0000","last_fullsized":"2024-09-13T14:46:37.071040+0000","mapping_epoch":152,"log_start":"174'4606","ondisk_log_start":"174'4606","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7516,"num_write_kb":240512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":179,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868300+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:46:25.868300+0000","last_peered":"2024-09-13T14:46:25.868300+0000","last_clean":"2024-09-13T14:46:25.868300+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:46:25.868300+0000","last_undegraded":"2024-09-13T14:46:25.868300+0000","last_fullsized":"2024-09-13T14:46:25.868300+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":188,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868756+0000","last_change":"2024-09-13T14:43:19.869693+0000","last_active":"2024-09-13T14:46:25.868756+0000","last_peered":"2024-09-13T14:46:25.868756+0000","last_clean":"2024-09-13T14:46:25.868756+0000","last_became_active":"2024-09-13T14:43:19.869343+0000","last_became_peered":"2024-09-13T14:43:19.869343+0000","last_unstale":"2024-09-13T14:46:25.868756+0000","last_undegraded":"2024-09-13T14:46:25.868756+0000","last_fullsized":"2024-09-13T14:46:25.868756+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:37.370378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":253,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.945970+0000","last_change":"2024-09-13T14:45:42.662478+0000","last_active":"2024-09-13T14:46:31.945970+0000","last_peered":"2024-09-13T14:46:31.945970+0000","last_clean":"2024-09-13T14:46:31.945970+0000","last_became_active":"2024-09-13T14:45:33.261545+0000","last_became_peered":"2024-09-13T14:45:33.261545+0000","last_unstale":"2024-09-13T14:46:31.945970+0000","last_undegraded":"2024-09-13T14:46:31.945970+0000","last_fullsized":"2024-09-13T14:46:31.945970+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:29.083540+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"216'9050","reported_seq":16749,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:41.933165+0000","last_change":"2024-09-13T14:46:41.933050+0000","last_active":"2024-09-13T14:46:41.933165+0000","last_peered":"2024-09-13T14:46:41.933165+0000","last_clean":"2024-09-13T14:46:41.933165+0000","last_became_active":"2024-09-13T14:45:45.449803+0000","last_became_peered":"2024-09-13T14:45:45.449803+0000","last_unstale":"2024-09-13T14:46:41.933165+0000","last_undegraded":"2024-09-13T14:46:41.933165+0000","last_fullsized":"2024-09-13T14:46:41.933165+0000","mapping_epoch":200,"log_start":"157'2015","ondisk_log_start":"157'2015","created":152,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3027","last_scrub_stamp":"2024-09-13T14:42:15.818474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:15.818474+0000","objects_scrubbed":1489,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:27.805615+0000","scrub_duration":683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6033,"num_write_kb":145408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":82378752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":236,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956462+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:46:31.956462+0000","last_peered":"2024-09-13T14:46:31.956462+0000","last_clean":"2024-09-13T14:46:31.956462+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:46:31.956462+0000","last_undegraded":"2024-09-13T14:46:31.956462+0000","last_fullsized":"2024-09-13T14:46:31.956462+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":151,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956905+0000","last_change":"2024-09-13T14:45:44.445221+0000","last_active":"2024-09-13T14:46:31.956905+0000","last_peered":"2024-09-13T14:46:31.956905+0000","last_clean":"2024-09-13T14:46:31.956905+0000","last_became_active":"2024-09-13T14:45:44.444956+0000","last_became_peered":"2024-09-13T14:45:44.444956+0000","last_unstale":"2024-09-13T14:46:31.956905+0000","last_undegraded":"2024-09-13T14:46:31.956905+0000","last_fullsized":"2024-09-13T14:46:31.956905+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:22.175285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"216'9102","reported_seq":20680,"reported_epoch":216,"state":"active+recovery_wait+undersized+remapped","last_fresh":"2024-09-13T14:46:37.046328+0000","last_change":"2024-09-13T14:46:36.957197+0000","last_active":"2024-09-13T14:46:37.046328+0000","last_peered":"2024-09-13T14:46:37.046328+0000","last_clean":"2024-09-13T14:43:18.793932+0000","last_became_active":"2024-09-13T14:43:21.445419+0000","last_became_peered":"2024-09-13T14:43:21.445419+0000","last_unstale":"2024-09-13T14:46:37.046328+0000","last_undegraded":"2024-09-13T14:46:37.046328+0000","last_fullsized":"2024-09-13T14:43:20.854131+0000","mapping_epoch":179,"log_start":"171'4064","ondisk_log_start":"171'4064","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3248","last_scrub_stamp":"2024-09-13T14:42:19.745287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:19.745287+0000","objects_scrubbed":1743,"log_size":5038,"log_dups_size":0,"ondisk_log_size":5038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5863,"num_write_kb":131840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3799,"num_bytes_recovered":248971264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,1],"avail_no_missing":["0","1","3","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":273,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:43.363696+0000","last_change":"2024-09-13T14:46:43.363696+0000","last_active":"2024-09-13T14:46:43.363696+0000","last_peered":"2024-09-13T14:46:43.363696+0000","last_clean":"2024-09-13T14:46:43.363696+0000","last_became_active":"2024-09-13T14:46:24.875002+0000","last_became_peered":"2024-09-13T14:46:24.875002+0000","last_unstale":"2024-09-13T14:46:43.363696+0000","last_undegraded":"2024-09-13T14:46:43.363696+0000","last_fullsized":"2024-09-13T14:46:43.363696+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:17.355674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":190,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.865780+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:46:25.865780+0000","last_peered":"2024-09-13T14:46:25.865780+0000","last_clean":"2024-09-13T14:46:25.865780+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:46:25.865780+0000","last_undegraded":"2024-09-13T14:46:25.865780+0000","last_fullsized":"2024-09-13T14:46:25.865780+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":225,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.965567+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:46:31.965567+0000","last_peered":"2024-09-13T14:46:31.965567+0000","last_clean":"2024-09-13T14:46:31.965567+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:46:31.965567+0000","last_undegraded":"2024-09-13T14:46:31.965567+0000","last_fullsized":"2024-09-13T14:46:31.965567+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":161,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.869197+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:46:25.869197+0000","last_peered":"2024-09-13T14:46:25.869197+0000","last_clean":"2024-09-13T14:46:25.869197+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:46:25.869197+0000","last_undegraded":"2024-09-13T14:46:25.869197+0000","last_fullsized":"2024-09-13T14:46:25.869197+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":270,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.351663+0000","last_change":"2024-09-13T14:46:37.351663+0000","last_active":"2024-09-13T14:46:37.351663+0000","last_peered":"2024-09-13T14:46:37.351663+0000","last_clean":"2024-09-13T14:46:37.351663+0000","last_became_active":"2024-09-13T14:46:24.879437+0000","last_became_peered":"2024-09-13T14:46:24.879437+0000","last_unstale":"2024-09-13T14:46:37.351663+0000","last_undegraded":"2024-09-13T14:46:37.351663+0000","last_fullsized":"2024-09-13T14:46:37.351663+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:37.351612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:37.351612+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:16:49.624579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"216'15060","reported_seq":15208,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.050178+0000","last_change":"2024-09-13T14:41:41.821076+0000","last_active":"2024-09-13T14:46:37.050178+0000","last_peered":"2024-09-13T14:46:37.050178+0000","last_clean":"2024-09-13T14:46:37.050178+0000","last_became_active":"2024-09-13T14:41:41.820767+0000","last_became_peered":"2024-09-13T14:41:41.820767+0000","last_unstale":"2024-09-13T14:46:37.050178+0000","last_undegraded":"2024-09-13T14:46:37.050178+0000","last_fullsized":"2024-09-13T14:46:37.050178+0000","mapping_epoch":152,"log_start":"188'10011","ondisk_log_start":"188'10011","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:48.859865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15060,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":218,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.955484+0000","last_change":"2024-09-13T14:43:19.872365+0000","last_active":"2024-09-13T14:46:25.955484+0000","last_peered":"2024-09-13T14:46:25.955484+0000","last_clean":"2024-09-13T14:46:25.955484+0000","last_became_active":"2024-09-13T14:43:19.871956+0000","last_became_peered":"2024-09-13T14:43:19.871956+0000","last_unstale":"2024-09-13T14:46:25.955484+0000","last_undegraded":"2024-09-13T14:46:25.955484+0000","last_fullsized":"2024-09-13T14:46:25.955484+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:48.443596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":312,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.949108+0000","last_change":"2024-09-13T14:43:19.867565+0000","last_active":"2024-09-13T14:46:31.949108+0000","last_peered":"2024-09-13T14:46:31.949108+0000","last_clean":"2024-09-13T14:46:31.949108+0000","last_became_active":"2024-09-13T14:43:19.858361+0000","last_became_peered":"2024-09-13T14:43:19.858361+0000","last_unstale":"2024-09-13T14:46:31.949108+0000","last_undegraded":"2024-09-13T14:46:31.949108+0000","last_fullsized":"2024-09-13T14:46:31.949108+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:03.954121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":317,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.939993+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:46:31.939993+0000","last_peered":"2024-09-13T14:46:31.939993+0000","last_clean":"2024-09-13T14:46:31.939993+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:46:31.939993+0000","last_undegraded":"2024-09-13T14:46:31.939993+0000","last_fullsized":"2024-09-13T14:46:31.939993+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":267,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:38.353107+0000","last_change":"2024-09-13T14:46:38.353107+0000","last_active":"2024-09-13T14:46:38.353107+0000","last_peered":"2024-09-13T14:46:38.353107+0000","last_clean":"2024-09-13T14:46:38.353107+0000","last_became_active":"2024-09-13T14:46:24.877082+0000","last_became_peered":"2024-09-13T14:46:24.877082+0000","last_unstale":"2024-09-13T14:46:38.353107+0000","last_undegraded":"2024-09-13T14:46:38.353107+0000","last_fullsized":"2024-09-13T14:46:38.353107+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:05.934975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"216'14978","reported_seq":20824,"reported_epoch":216,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:46:43.952668+0000","last_change":"2024-09-13T14:46:41.933655+0000","last_active":"2024-09-13T14:46:43.952668+0000","last_peered":"2024-09-13T14:46:43.952668+0000","last_clean":"2024-09-13T14:43:18.829456+0000","last_became_active":"2024-09-13T14:45:45.454351+0000","last_became_peered":"2024-09-13T14:45:45.454351+0000","last_unstale":"2024-09-13T14:46:43.952668+0000","last_undegraded":"2024-09-13T14:45:45.427342+0000","last_fullsized":"2024-09-13T14:45:45.419919+0000","mapping_epoch":200,"log_start":"189'10569","ondisk_log_start":"189'10569","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:38.864880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14978,"num_write_kb":479296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2870,"num_bytes_recovered":168558592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1],"avail_no_missing":["7","1","5"],"object_location_counts":[{"shards":"1,7","objects":-2406}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":293,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.947506+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:46:31.947506+0000","last_peered":"2024-09-13T14:46:31.947506+0000","last_clean":"2024-09-13T14:46:31.947506+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:46:31.947506+0000","last_undegraded":"2024-09-13T14:46:31.947506+0000","last_fullsized":"2024-09-13T14:46:31.947506+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":102,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.948588+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:46:31.948588+0000","last_peered":"2024-09-13T14:46:31.948588+0000","last_clean":"2024-09-13T14:46:31.948588+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:46:31.948588+0000","last_undegraded":"2024-09-13T14:46:31.948588+0000","last_fullsized":"2024-09-13T14:46:31.948588+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":101,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.865955+0000","last_change":"2024-09-13T14:43:19.860282+0000","last_active":"2024-09-13T14:46:25.865955+0000","last_peered":"2024-09-13T14:46:25.865955+0000","last_clean":"2024-09-13T14:46:25.865955+0000","last_became_active":"2024-09-13T14:43:19.859966+0000","last_became_peered":"2024-09-13T14:43:19.859966+0000","last_unstale":"2024-09-13T14:46:25.865955+0000","last_undegraded":"2024-09-13T14:46:25.865955+0000","last_fullsized":"2024-09-13T14:46:25.865955+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:25.735714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":269,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:33.277560+0000","last_change":"2024-09-13T14:46:33.277560+0000","last_active":"2024-09-13T14:46:33.277560+0000","last_peered":"2024-09-13T14:46:33.277560+0000","last_clean":"2024-09-13T14:46:33.277560+0000","last_became_active":"2024-09-13T14:46:24.876970+0000","last_became_peered":"2024-09-13T14:46:24.876970+0000","last_unstale":"2024-09-13T14:46:33.277560+0000","last_undegraded":"2024-09-13T14:46:33.277560+0000","last_fullsized":"2024-09-13T14:46:33.277560+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:18.375980+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"216'8975","reported_seq":9152,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.020053+0000","last_change":"2024-09-13T14:42:16.895536+0000","last_active":"2024-09-13T14:46:37.020053+0000","last_peered":"2024-09-13T14:46:37.020053+0000","last_clean":"2024-09-13T14:46:37.020053+0000","last_became_active":"2024-09-13T14:42:15.123602+0000","last_became_peered":"2024-09-13T14:42:15.123602+0000","last_unstale":"2024-09-13T14:46:37.020053+0000","last_undegraded":"2024-09-13T14:46:37.020053+0000","last_fullsized":"2024-09-13T14:46:37.020053+0000","mapping_epoch":152,"log_start":"171'3954","ondisk_log_start":"171'3954","created":152,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3040","last_scrub_stamp":"2024-09-13T14:42:16.895299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.895299+0000","objects_scrubbed":1521,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:54.632739+0000","scrub_duration":483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5941,"num_write_kb":141440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":191,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.865660+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:46:25.865660+0000","last_peered":"2024-09-13T14:46:25.865660+0000","last_clean":"2024-09-13T14:46:25.865660+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:46:25.865660+0000","last_undegraded":"2024-09-13T14:46:25.865660+0000","last_fullsized":"2024-09-13T14:46:25.865660+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":179,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868166+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:46:25.868166+0000","last_peered":"2024-09-13T14:46:25.868166+0000","last_clean":"2024-09-13T14:46:25.868166+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:46:25.868166+0000","last_undegraded":"2024-09-13T14:46:25.868166+0000","last_fullsized":"2024-09-13T14:46:25.868166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":162,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.947359+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:46:31.947359+0000","last_peered":"2024-09-13T14:46:31.947359+0000","last_clean":"2024-09-13T14:46:31.947359+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:46:31.947359+0000","last_undegraded":"2024-09-13T14:46:31.947359+0000","last_fullsized":"2024-09-13T14:46:31.947359+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":270,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:44.369164+0000","last_change":"2024-09-13T14:46:44.369164+0000","last_active":"2024-09-13T14:46:44.369164+0000","last_peered":"2024-09-13T14:46:44.369164+0000","last_clean":"2024-09-13T14:46:44.369164+0000","last_became_active":"2024-09-13T14:46:24.876761+0000","last_became_peered":"2024-09-13T14:46:24.876761+0000","last_unstale":"2024-09-13T14:46:44.369164+0000","last_undegraded":"2024-09-13T14:46:44.369164+0000","last_fullsized":"2024-09-13T14:46:44.369164+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:44.369116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:44.369116+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:23.866692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"216'14770","reported_seq":14842,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.059847+0000","last_change":"2024-09-13T14:41:41.691642+0000","last_active":"2024-09-13T14:46:37.059847+0000","last_peered":"2024-09-13T14:46:37.059847+0000","last_clean":"2024-09-13T14:46:37.059847+0000","last_became_active":"2024-09-13T14:41:41.691265+0000","last_became_peered":"2024-09-13T14:41:41.691265+0000","last_unstale":"2024-09-13T14:46:37.059847+0000","last_undegraded":"2024-09-13T14:46:37.059847+0000","last_fullsized":"2024-09-13T14:46:37.059847+0000","mapping_epoch":152,"log_start":"172'5900","ondisk_log_start":"172'5900","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":8870,"log_dups_size":0,"ondisk_log_size":8870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:07.758310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14770,"num_write_kb":472640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":195,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.865950+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:46:25.865950+0000","last_peered":"2024-09-13T14:46:25.865950+0000","last_clean":"2024-09-13T14:46:25.865950+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:46:25.865950+0000","last_undegraded":"2024-09-13T14:46:25.865950+0000","last_fullsized":"2024-09-13T14:46:25.865950+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":108,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.948039+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:46:31.948039+0000","last_peered":"2024-09-13T14:46:31.948039+0000","last_clean":"2024-09-13T14:46:31.948039+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:46:31.948039+0000","last_undegraded":"2024-09-13T14:46:31.948039+0000","last_fullsized":"2024-09-13T14:46:31.948039+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":63,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.867525+0000","last_change":"2024-09-13T14:43:19.874797+0000","last_active":"2024-09-13T14:46:25.867525+0000","last_peered":"2024-09-13T14:46:25.867525+0000","last_clean":"2024-09-13T14:46:25.867525+0000","last_became_active":"2024-09-13T14:43:19.874460+0000","last_became_peered":"2024-09-13T14:43:19.874460+0000","last_unstale":"2024-09-13T14:46:25.867525+0000","last_undegraded":"2024-09-13T14:46:25.867525+0000","last_fullsized":"2024-09-13T14:46:25.867525+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:55.075477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":262,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.939739+0000","last_change":"2024-09-13T14:46:31.939739+0000","last_active":"2024-09-13T14:46:31.939739+0000","last_peered":"2024-09-13T14:46:31.939739+0000","last_clean":"2024-09-13T14:46:31.939739+0000","last_became_active":"2024-09-13T14:46:24.878928+0000","last_became_peered":"2024-09-13T14:46:24.878928+0000","last_unstale":"2024-09-13T14:46:31.939739+0000","last_undegraded":"2024-09-13T14:46:31.939739+0000","last_fullsized":"2024-09-13T14:46:31.939739+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:37.637037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:37.637037+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:46:34.211675+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"216'15034","reported_seq":15106,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.031393+0000","last_change":"2024-09-13T14:41:41.950035+0000","last_active":"2024-09-13T14:46:37.031393+0000","last_peered":"2024-09-13T14:46:37.031393+0000","last_clean":"2024-09-13T14:46:37.031393+0000","last_became_active":"2024-09-13T14:41:41.949672+0000","last_became_peered":"2024-09-13T14:41:41.949672+0000","last_unstale":"2024-09-13T14:46:37.031393+0000","last_undegraded":"2024-09-13T14:46:37.031393+0000","last_fullsized":"2024-09-13T14:46:37.031393+0000","mapping_epoch":152,"log_start":"189'10596","ondisk_log_start":"189'10596","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:43.973874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15034,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":243,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.947439+0000","last_change":"2024-09-13T14:45:44.440956+0000","last_active":"2024-09-13T14:46:31.947439+0000","last_peered":"2024-09-13T14:46:31.947439+0000","last_clean":"2024-09-13T14:46:31.947439+0000","last_became_active":"2024-09-13T14:45:44.440574+0000","last_became_peered":"2024-09-13T14:45:44.440574+0000","last_unstale":"2024-09-13T14:46:31.947439+0000","last_undegraded":"2024-09-13T14:46:31.947439+0000","last_fullsized":"2024-09-13T14:46:31.947439+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:08.477427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":191,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.957841+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:46:25.957841+0000","last_peered":"2024-09-13T14:46:25.957841+0000","last_clean":"2024-09-13T14:46:25.957841+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:46:25.957841+0000","last_undegraded":"2024-09-13T14:46:25.957841+0000","last_fullsized":"2024-09-13T14:46:25.957841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":295,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.950555+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:46:31.950555+0000","last_peered":"2024-09-13T14:46:31.950555+0000","last_clean":"2024-09-13T14:46:31.950555+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:46:31.950555+0000","last_undegraded":"2024-09-13T14:46:31.950555+0000","last_fullsized":"2024-09-13T14:46:31.950555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":262,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.939695+0000","last_change":"2024-09-13T14:46:31.939695+0000","last_active":"2024-09-13T14:46:31.939695+0000","last_peered":"2024-09-13T14:46:31.939695+0000","last_clean":"2024-09-13T14:46:31.939695+0000","last_became_active":"2024-09-13T14:46:24.879042+0000","last_became_peered":"2024-09-13T14:46:24.879042+0000","last_unstale":"2024-09-13T14:46:31.939695+0000","last_undegraded":"2024-09-13T14:46:31.939695+0000","last_fullsized":"2024-09-13T14:46:31.939695+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:43.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:43.677882+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:46:36.241260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"216'15180","reported_seq":31840,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.071062+0000","last_change":"2024-09-13T14:46:25.948914+0000","last_active":"2024-09-13T14:46:37.071062+0000","last_peered":"2024-09-13T14:46:37.071062+0000","last_clean":"2024-09-13T14:46:37.071062+0000","last_became_active":"2024-09-13T14:46:25.869298+0000","last_became_peered":"2024-09-13T14:46:25.869298+0000","last_unstale":"2024-09-13T14:46:37.071062+0000","last_undegraded":"2024-09-13T14:46:37.071062+0000","last_fullsized":"2024-09-13T14:46:37.071062+0000","mapping_epoch":214,"log_start":"184'8344","ondisk_log_start":"184'8344","created":152,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":6836,"log_dups_size":0,"ondisk_log_size":6836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:00.545585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15180,"num_write_kb":485697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8268,"num_bytes_recovered":541786136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":266,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.957062+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:46:31.957062+0000","last_peered":"2024-09-13T14:46:31.957062+0000","last_clean":"2024-09-13T14:46:31.957062+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:46:31.957062+0000","last_undegraded":"2024-09-13T14:46:31.957062+0000","last_fullsized":"2024-09-13T14:46:31.957062+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.869461+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:46:25.869461+0000","last_peered":"2024-09-13T14:46:25.869461+0000","last_clean":"2024-09-13T14:46:25.869461+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:46:25.869461+0000","last_undegraded":"2024-09-13T14:46:25.869461+0000","last_fullsized":"2024-09-13T14:46:25.869461+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":270,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956206+0000","last_change":"2024-09-13T14:43:19.871833+0000","last_active":"2024-09-13T14:46:31.956206+0000","last_peered":"2024-09-13T14:46:31.956206+0000","last_clean":"2024-09-13T14:46:31.956206+0000","last_became_active":"2024-09-13T14:43:19.871024+0000","last_became_peered":"2024-09-13T14:43:19.871024+0000","last_unstale":"2024-09-13T14:46:31.956206+0000","last_undegraded":"2024-09-13T14:46:31.956206+0000","last_fullsized":"2024-09-13T14:46:31.956206+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:18:12.342823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":262,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.947899+0000","last_change":"2024-09-13T14:46:31.947899+0000","last_active":"2024-09-13T14:46:31.947899+0000","last_peered":"2024-09-13T14:46:31.947899+0000","last_clean":"2024-09-13T14:46:31.947899+0000","last_became_active":"2024-09-13T14:46:24.876652+0000","last_became_peered":"2024-09-13T14:46:24.876652+0000","last_unstale":"2024-09-13T14:46:31.947899+0000","last_undegraded":"2024-09-13T14:46:31.947899+0000","last_fullsized":"2024-09-13T14:46:31.947899+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:38.648108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:38.648108+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:46:32.228141+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"216'15106","reported_seq":15254,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.062825+0000","last_change":"2024-09-13T14:41:41.950797+0000","last_active":"2024-09-13T14:46:37.062825+0000","last_peered":"2024-09-13T14:46:37.062825+0000","last_clean":"2024-09-13T14:46:37.062825+0000","last_became_active":"2024-09-13T14:41:41.950412+0000","last_became_peered":"2024-09-13T14:41:41.950412+0000","last_unstale":"2024-09-13T14:46:37.062825+0000","last_undegraded":"2024-09-13T14:46:37.062825+0000","last_fullsized":"2024-09-13T14:46:37.062825+0000","mapping_epoch":152,"log_start":"186'9394","ondisk_log_start":"186'9394","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:09.662164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15106,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":309,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.940270+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:46:31.940270+0000","last_peered":"2024-09-13T14:46:31.940270+0000","last_clean":"2024-09-13T14:46:31.940270+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:46:31.940270+0000","last_undegraded":"2024-09-13T14:46:31.940270+0000","last_fullsized":"2024-09-13T14:46:31.940270+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":201,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.869501+0000","last_change":"2024-09-13T14:43:19.860737+0000","last_active":"2024-09-13T14:46:25.869501+0000","last_peered":"2024-09-13T14:46:25.869501+0000","last_clean":"2024-09-13T14:46:25.869501+0000","last_became_active":"2024-09-13T14:43:19.860451+0000","last_became_peered":"2024-09-13T14:43:19.860451+0000","last_unstale":"2024-09-13T14:46:25.869501+0000","last_undegraded":"2024-09-13T14:46:25.869501+0000","last_fullsized":"2024-09-13T14:46:25.869501+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:14.888211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":161,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868465+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:46:25.868465+0000","last_peered":"2024-09-13T14:46:25.868465+0000","last_clean":"2024-09-13T14:46:25.868465+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:46:25.868465+0000","last_undegraded":"2024-09-13T14:46:25.868465+0000","last_fullsized":"2024-09-13T14:46:25.868465+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":178,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868084+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:46:25.868084+0000","last_peered":"2024-09-13T14:46:25.868084+0000","last_clean":"2024-09-13T14:46:25.868084+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:46:25.868084+0000","last_undegraded":"2024-09-13T14:46:25.868084+0000","last_fullsized":"2024-09-13T14:46:25.868084+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":185,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.867770+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:46:25.867770+0000","last_peered":"2024-09-13T14:46:25.867770+0000","last_clean":"2024-09-13T14:46:25.867770+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:46:25.867770+0000","last_undegraded":"2024-09-13T14:46:25.867770+0000","last_fullsized":"2024-09-13T14:46:25.867770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":107,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.868379+0000","last_change":"2024-09-13T14:43:19.870397+0000","last_active":"2024-09-13T14:46:25.868379+0000","last_peered":"2024-09-13T14:46:25.868379+0000","last_clean":"2024-09-13T14:46:25.868379+0000","last_became_active":"2024-09-13T14:43:19.870037+0000","last_became_peered":"2024-09-13T14:43:19.870037+0000","last_unstale":"2024-09-13T14:46:25.868379+0000","last_undegraded":"2024-09-13T14:46:25.868379+0000","last_fullsized":"2024-09-13T14:46:25.868379+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:36.094674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":105,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.961486+0000","last_change":"2024-09-13T14:43:19.881173+0000","last_active":"2024-09-13T14:46:31.961486+0000","last_peered":"2024-09-13T14:46:31.961486+0000","last_clean":"2024-09-13T14:46:31.961486+0000","last_became_active":"2024-09-13T14:43:19.870848+0000","last_became_peered":"2024-09-13T14:43:19.870848+0000","last_unstale":"2024-09-13T14:46:31.961486+0000","last_undegraded":"2024-09-13T14:46:31.961486+0000","last_fullsized":"2024-09-13T14:46:31.961486+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:00.809391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":262,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.947691+0000","last_change":"2024-09-13T14:46:31.947691+0000","last_active":"2024-09-13T14:46:31.947691+0000","last_peered":"2024-09-13T14:46:31.947691+0000","last_clean":"2024-09-13T14:46:31.947691+0000","last_became_active":"2024-09-13T14:46:24.878436+0000","last_became_peered":"2024-09-13T14:46:24.878436+0000","last_unstale":"2024-09-13T14:46:31.947691+0000","last_undegraded":"2024-09-13T14:46:31.947691+0000","last_fullsized":"2024-09-13T14:46:31.947691+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:38.648108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:38.648108+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:46:33.187888+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":101,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.956356+0000","last_change":"2024-09-13T14:43:19.871796+0000","last_active":"2024-09-13T14:46:25.956356+0000","last_peered":"2024-09-13T14:46:25.956356+0000","last_clean":"2024-09-13T14:46:25.956356+0000","last_became_active":"2024-09-13T14:43:19.871130+0000","last_became_peered":"2024-09-13T14:43:19.871130+0000","last_unstale":"2024-09-13T14:46:25.956356+0000","last_undegraded":"2024-09-13T14:46:25.956356+0000","last_fullsized":"2024-09-13T14:46:25.956356+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:33.743486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":233,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956522+0000","last_change":"2024-09-13T14:43:19.871328+0000","last_active":"2024-09-13T14:46:31.956522+0000","last_peered":"2024-09-13T14:46:31.956522+0000","last_clean":"2024-09-13T14:46:31.956522+0000","last_became_active":"2024-09-13T14:43:19.869668+0000","last_became_peered":"2024-09-13T14:43:19.869668+0000","last_unstale":"2024-09-13T14:46:31.956522+0000","last_undegraded":"2024-09-13T14:46:31.956522+0000","last_fullsized":"2024-09-13T14:46:31.956522+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:18.042885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":304,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.940842+0000","last_change":"2024-09-13T14:37:08.759355+0000","last_active":"2024-09-13T14:46:31.940842+0000","last_peered":"2024-09-13T14:46:31.940842+0000","last_clean":"2024-09-13T14:46:31.940842+0000","last_became_active":"2024-09-13T14:37:08.759058+0000","last_became_peered":"2024-09-13T14:37:08.759058+0000","last_unstale":"2024-09-13T14:46:31.940842+0000","last_undegraded":"2024-09-13T14:46:31.940842+0000","last_fullsized":"2024-09-13T14:46:31.940842+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:42.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":235,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956049+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:46:31.956049+0000","last_peered":"2024-09-13T14:46:31.956049+0000","last_clean":"2024-09-13T14:46:31.956049+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:46:31.956049+0000","last_undegraded":"2024-09-13T14:46:31.956049+0000","last_fullsized":"2024-09-13T14:46:31.956049+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.12","version":"216'9092","reported_seq":17216,"reported_epoch":216,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:46:37.011384+0000","last_change":"2024-09-13T14:45:45.572526+0000","last_active":"2024-09-13T14:46:37.011384+0000","last_peered":"2024-09-13T14:46:37.011384+0000","last_clean":"2024-09-13T14:43:18.835862+0000","last_became_active":"2024-09-13T14:45:45.538023+0000","last_became_peered":"2024-09-13T14:45:45.538023+0000","last_unstale":"2024-09-13T14:46:37.011384+0000","last_undegraded":"2024-09-13T14:45:45.458299+0000","last_fullsized":"2024-09-13T14:46:37.011384+0000","mapping_epoch":200,"log_start":"171'3998","ondisk_log_start":"171'3998","created":159,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3181","last_scrub_stamp":"2024-09-13T14:42:18.816556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:18.816556+0000","objects_scrubbed":1693,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":135424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":197984256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":178,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.869430+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:46:25.869430+0000","last_peered":"2024-09-13T14:46:25.869430+0000","last_clean":"2024-09-13T14:46:25.869430+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:46:25.869430+0000","last_undegraded":"2024-09-13T14:46:25.869430+0000","last_fullsized":"2024-09-13T14:46:25.869430+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":259,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956366+0000","last_change":"2024-09-13T14:43:19.872068+0000","last_active":"2024-09-13T14:46:31.956366+0000","last_peered":"2024-09-13T14:46:31.956366+0000","last_clean":"2024-09-13T14:46:31.956366+0000","last_became_active":"2024-09-13T14:43:19.871529+0000","last_became_peered":"2024-09-13T14:43:19.871529+0000","last_unstale":"2024-09-13T14:46:31.956366+0000","last_undegraded":"2024-09-13T14:46:31.956366+0000","last_fullsized":"2024-09-13T14:46:31.956366+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:11.995530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"216'9048","reported_seq":14705,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.023457+0000","last_change":"2024-09-13T14:43:29.963964+0000","last_active":"2024-09-13T14:46:37.023457+0000","last_peered":"2024-09-13T14:46:37.023457+0000","last_clean":"2024-09-13T14:46:37.023457+0000","last_became_active":"2024-09-13T14:43:29.959212+0000","last_became_peered":"2024-09-13T14:43:29.959212+0000","last_unstale":"2024-09-13T14:46:37.023457+0000","last_undegraded":"2024-09-13T14:46:37.023457+0000","last_fullsized":"2024-09-13T14:46:37.023457+0000","mapping_epoch":182,"log_start":"174'4675","ondisk_log_start":"174'4675","created":159,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:41:40.658691+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":242112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1850,"num_bytes_recovered":121241600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":313,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.940044+0000","last_change":"2024-09-13T14:43:19.870848+0000","last_active":"2024-09-13T14:46:31.940044+0000","last_peered":"2024-09-13T14:46:31.940044+0000","last_clean":"2024-09-13T14:46:31.940044+0000","last_became_active":"2024-09-13T14:43:19.870037+0000","last_became_peered":"2024-09-13T14:43:19.870037+0000","last_unstale":"2024-09-13T14:46:31.940044+0000","last_undegraded":"2024-09-13T14:46:31.940044+0000","last_fullsized":"2024-09-13T14:46:31.940044+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:03.760717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":151,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956655+0000","last_change":"2024-09-13T14:45:44.445815+0000","last_active":"2024-09-13T14:46:31.956655+0000","last_peered":"2024-09-13T14:46:31.956655+0000","last_clean":"2024-09-13T14:46:31.956655+0000","last_became_active":"2024-09-13T14:45:44.445562+0000","last_became_peered":"2024-09-13T14:45:44.445562+0000","last_unstale":"2024-09-13T14:46:31.956655+0000","last_undegraded":"2024-09-13T14:46:31.956655+0000","last_fullsized":"2024-09-13T14:46:31.956655+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:31.178220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"216'9470","reported_seq":15799,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.060012+0000","last_change":"2024-09-13T14:43:16.600263+0000","last_active":"2024-09-13T14:46:37.060012+0000","last_peered":"2024-09-13T14:46:37.060012+0000","last_clean":"2024-09-13T14:46:37.060012+0000","last_became_active":"2024-09-13T14:43:16.578306+0000","last_became_peered":"2024-09-13T14:43:16.578306+0000","last_unstale":"2024-09-13T14:46:37.060012+0000","last_undegraded":"2024-09-13T14:46:37.060012+0000","last_fullsized":"2024-09-13T14:46:37.060012+0000","mapping_epoch":176,"log_start":"157'1145","ondisk_log_start":"157'1145","created":159,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3245","last_scrub_stamp":"2024-09-13T14:42:17.833200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:17.833200+0000","objects_scrubbed":1661,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:16.805530+0000","scrub_duration":483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6247,"num_write_kb":146752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1973,"num_bytes_recovered":129302528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":234,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.961337+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:46:31.961337+0000","last_peered":"2024-09-13T14:46:31.961337+0000","last_clean":"2024-09-13T14:46:31.961337+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:46:31.961337+0000","last_undegraded":"2024-09-13T14:46:31.961337+0000","last_fullsized":"2024-09-13T14:46:31.961337+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":197,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.867634+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:46:25.867634+0000","last_peered":"2024-09-13T14:46:25.867634+0000","last_clean":"2024-09-13T14:46:25.867634+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:46:25.867634+0000","last_undegraded":"2024-09-13T14:46:25.867634+0000","last_fullsized":"2024-09-13T14:46:25.867634+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"216'8866","reported_seq":8994,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:36.945638+0000","last_change":"2024-09-13T14:42:23.582272+0000","last_active":"2024-09-13T14:46:36.945638+0000","last_peered":"2024-09-13T14:46:36.945638+0000","last_clean":"2024-09-13T14:46:36.945638+0000","last_became_active":"2024-09-13T14:42:23.579475+0000","last_became_peered":"2024-09-13T14:42:23.579475+0000","last_unstale":"2024-09-13T14:46:36.945638+0000","last_undegraded":"2024-09-13T14:46:36.945638+0000","last_fullsized":"2024-09-13T14:46:36.945638+0000","mapping_epoch":164,"log_start":"174'4477","ondisk_log_start":"174'4477","created":159,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3023","last_scrub_stamp":"2024-09-13T14:42:17.713316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:17.713316+0000","objects_scrubbed":1552,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:08.367012+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5850,"num_write_kb":137536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":292,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.947272+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:46:31.947272+0000","last_peered":"2024-09-13T14:46:31.947272+0000","last_clean":"2024-09-13T14:46:31.947272+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:46:31.947272+0000","last_undegraded":"2024-09-13T14:46:31.947272+0000","last_fullsized":"2024-09-13T14:46:31.947272+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":297,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.951343+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:46:31.951343+0000","last_peered":"2024-09-13T14:46:31.951343+0000","last_clean":"2024-09-13T14:46:31.951343+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:46:31.951343+0000","last_undegraded":"2024-09-13T14:46:31.951343+0000","last_fullsized":"2024-09-13T14:46:31.951343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"216'9138","reported_seq":9300,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.062796+0000","last_change":"2024-09-13T14:42:25.288996+0000","last_active":"2024-09-13T14:46:37.062796+0000","last_peered":"2024-09-13T14:46:37.062796+0000","last_clean":"2024-09-13T14:46:37.062796+0000","last_became_active":"2024-09-13T14:42:15.125321+0000","last_became_peered":"2024-09-13T14:42:15.125321+0000","last_unstale":"2024-09-13T14:46:37.062796+0000","last_undegraded":"2024-09-13T14:46:37.062796+0000","last_fullsized":"2024-09-13T14:46:37.062796+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"166'3354","last_scrub_stamp":"2024-09-13T14:42:25.288934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:25.288934+0000","objects_scrubbed":1820,"log_size":9138,"log_dups_size":0,"ondisk_log_size":9138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:26.394629+0000","scrub_duration":0.191515928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5784,"num_write_kb":126848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":205,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.867994+0000","last_change":"2024-09-13T14:37:08.749549+0000","last_active":"2024-09-13T14:46:25.867994+0000","last_peered":"2024-09-13T14:46:25.867994+0000","last_clean":"2024-09-13T14:46:25.867994+0000","last_became_active":"2024-09-13T14:37:08.749013+0000","last_became_peered":"2024-09-13T14:37:08.749013+0000","last_unstale":"2024-09-13T14:46:25.867994+0000","last_undegraded":"2024-09-13T14:46:25.867994+0000","last_fullsized":"2024-09-13T14:46:25.867994+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:35.239989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":180,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.865649+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:46:25.865649+0000","last_peered":"2024-09-13T14:46:25.865649+0000","last_clean":"2024-09-13T14:46:25.865649+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:46:25.865649+0000","last_undegraded":"2024-09-13T14:46:25.865649+0000","last_fullsized":"2024-09-13T14:46:25.865649+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"216'9247","reported_seq":20977,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:36.974436+0000","last_change":"2024-09-13T14:46:32.573716+0000","last_active":"2024-09-13T14:46:36.974436+0000","last_peered":"2024-09-13T14:46:36.974436+0000","last_clean":"2024-09-13T14:46:36.974436+0000","last_became_active":"2024-09-13T14:46:23.877130+0000","last_became_peered":"2024-09-13T14:46:23.877130+0000","last_unstale":"2024-09-13T14:46:36.974436+0000","last_undegraded":"2024-09-13T14:46:36.974436+0000","last_fullsized":"2024-09-13T14:46:36.974436+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'9132","last_scrub_stamp":"2024-09-13T14:46:32.573630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.573630+0000","objects_scrubbed":117,"log_size":9247,"log_dups_size":0,"ondisk_log_size":9247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:22.340111+0000","scrub_duration":0.062175590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":195,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.867444+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:46:25.867444+0000","last_peered":"2024-09-13T14:46:25.867444+0000","last_clean":"2024-09-13T14:46:25.867444+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:46:25.867444+0000","last_undegraded":"2024-09-13T14:46:25.867444+0000","last_fullsized":"2024-09-13T14:46:25.867444+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":163,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.865851+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:46:25.865851+0000","last_peered":"2024-09-13T14:46:25.865851+0000","last_clean":"2024-09-13T14:46:25.865851+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:46:25.865851+0000","last_undegraded":"2024-09-13T14:46:25.865851+0000","last_fullsized":"2024-09-13T14:46:25.865851+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"216'9099","reported_seq":9266,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.072023+0000","last_change":"2024-09-13T14:45:44.488830+0000","last_active":"2024-09-13T14:46:37.072023+0000","last_peered":"2024-09-13T14:46:37.072023+0000","last_clean":"2024-09-13T14:46:37.072023+0000","last_became_active":"2024-09-13T14:45:44.445102+0000","last_became_peered":"2024-09-13T14:45:44.445102+0000","last_unstale":"2024-09-13T14:46:37.072023+0000","last_undegraded":"2024-09-13T14:46:37.072023+0000","last_fullsized":"2024-09-13T14:46:37.072023+0000","mapping_epoch":199,"log_start":"157'2211","ondisk_log_start":"157'2211","created":159,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3026","last_scrub_stamp":"2024-09-13T14:42:16.817418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:16.817418+0000","objects_scrubbed":1500,"log_size":6888,"log_dups_size":0,"ondisk_log_size":6888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:18.635831+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6080,"num_write_kb":146560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":267,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956796+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:46:31.956796+0000","last_peered":"2024-09-13T14:46:31.956796+0000","last_clean":"2024-09-13T14:46:31.956796+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:46:31.956796+0000","last_undegraded":"2024-09-13T14:46:31.956796+0000","last_fullsized":"2024-09-13T14:46:31.956796+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":253,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.956140+0000","last_change":"2024-09-13T14:37:08.750424+0000","last_active":"2024-09-13T14:46:31.956140+0000","last_peered":"2024-09-13T14:46:31.956140+0000","last_clean":"2024-09-13T14:46:31.956140+0000","last_became_active":"2024-09-13T14:37:08.749577+0000","last_became_peered":"2024-09-13T14:37:08.749577+0000","last_unstale":"2024-09-13T14:46:31.956140+0000","last_undegraded":"2024-09-13T14:46:31.956140+0000","last_fullsized":"2024-09-13T14:46:31.956140+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:16.435452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"216'9104","reported_seq":20718,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:37.057014+0000","last_change":"2024-09-13T14:46:04.656169+0000","last_active":"2024-09-13T14:46:37.057014+0000","last_peered":"2024-09-13T14:46:37.057014+0000","last_clean":"2024-09-13T14:46:37.057014+0000","last_became_active":"2024-09-13T14:46:04.637594+0000","last_became_peered":"2024-09-13T14:46:04.637594+0000","last_unstale":"2024-09-13T14:46:37.057014+0000","last_undegraded":"2024-09-13T14:46:37.057014+0000","last_fullsized":"2024-09-13T14:46:37.057014+0000","mapping_epoch":206,"log_start":"172'4045","ondisk_log_start":"172'4045","created":159,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'3124","last_scrub_stamp":"2024-09-13T14:42:17.909091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:41:40.658691+0000","last_clean_scrub_stamp":"2024-09-13T14:42:17.909091+0000","objects_scrubbed":1626,"log_size":5059,"log_dups_size":0,"ondisk_log_size":5059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:52.939922+0000","scrub_duration":450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5996,"num_write_kb":139840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3808,"num_bytes_recovered":249561088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":178,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.867411+0000","last_change":"2024-09-13T14:32:09.484255+0000","last_active":"2024-09-13T14:46:25.867411+0000","last_peered":"2024-09-13T14:46:25.867411+0000","last_clean":"2024-09-13T14:46:25.867411+0000","last_became_active":"2024-09-13T14:32:09.483903+0000","last_became_peered":"2024-09-13T14:32:09.483903+0000","last_unstale":"2024-09-13T14:46:25.867411+0000","last_undegraded":"2024-09-13T14:46:25.867411+0000","last_fullsized":"2024-09-13T14:46:25.867411+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:41.931522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":309,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-13T14:46:31.940553+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:46:31.940553+0000","last_peered":"2024-09-13T14:46:31.940553+0000","last_clean":"2024-09-13T14:46:31.940553+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:46:31.940553+0000","last_undegraded":"2024-09-13T14:46:31.940553+0000","last_fullsized":"2024-09-13T14:46:31.940553+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206527,"num_write_kb":5719361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50021,"num_bytes_recovered":2983395352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4825419776,"data_stored":4825415704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156930,"ondisk_log_size":156930,"up":78,"acting":74,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":182,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":808821,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365124,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2189992,"kb_used_data":1284768,"kb_used_omap":1,"kb_used_meta":905214,"kb_avail":91555160,"statfs":{"total":95995035648,"available":93752483840,"internally_reserved":0,"allocated":1315602432,"data_stored":1314452297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":926939480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561354,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":601436,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93143716,"statfs":{"total":95995035648,"available":95379165184,"internally_reserved":0,"allocated":3297280,"data_stored":2159232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790286,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2626828,"kb_used_data":1729924,"kb_used_omap":1,"kb_used_meta":896894,"kb_avail":91118324,"statfs":{"total":95995035648,"available":93305163776,"internally_reserved":0,"allocated":1771442176,"data_stored":1770315494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918419800},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986514,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905376,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":902718,"kb_avail":92839776,"statfs":{"total":95995035648,"available":95067930624,"internally_reserved":0,"allocated":2711552,"data_stored":1569236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924383576},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519669,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1487248,"kb_used_data":1056712,"kb_used_omap":4,"kb_used_meta":430523,"kb_avail":92257904,"statfs":{"total":95995035648,"available":94472093696,"internally_reserved":0,"allocated":1082073088,"data_stored":1080942438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":440856000},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304585,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":403984,"kb_used_data":2632,"kb_used_omap":6,"kb_used_meta":401337,"kb_avail":93341168,"statfs":{"total":95995035648,"available":95581356032,"internally_reserved":0,"allocated":2695168,"data_stored":1564962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":410969543},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827888,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1391888,"kb_used_data":660296,"kb_used_omap":5,"kb_used_meta":731578,"kb_avail":92353264,"statfs":{"total":95995035648,"available":94569742336,"internally_reserved":0,"allocated":676143104,"data_stored":675012865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":749136108},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285753,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546516,"kb_used_data":3212,"kb_used_omap":5,"kb_used_meta":543290,"kb_avail":93198636,"statfs":{"total":95995035648,"available":95435403264,"internally_reserved":0,"allocated":3289088,"data_stored":2154180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":556329019},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":666959872,"data_stored":666959872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1077411840,"data_stored":1077411840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1768751104,"data_stored":1768751104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1312296960,"data_stored":1312292888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:46:48.390 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:46:48.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 36 2024-09-13T14:46:48.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:48 smithi019 ceph-mon[91296]: 1.14 scrub starts 2024-09-13T14:46:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:48 smithi019 ceph-mon[91296]: 1.14 scrub ok 2024-09-13T14:46:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:48 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:46:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:48 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:48 smithi019 ceph-mon[91296]: osdmap e217: 8 total, 8 up, 8 in 2024-09-13T14:46:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:48 smithi019 ceph-mon[91296]: from='client.36087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:48.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:48.683 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:48.706 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:48.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:48.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:46:48.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:48.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:48.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:49.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:49.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:49.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:49.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:49.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:49.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:49.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:49 smithi003 ceph-mon[104007]: 1.6 scrub starts 2024-09-13T14:46:49.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:49 smithi003 ceph-mon[104007]: 1.6 scrub ok 2024-09-13T14:46:49.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:49 smithi003 ceph-mon[104007]: pgmap v722: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 795 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 220 op/s; 5084/78 objects degraded (6517.949%); 1.2 MiB/s, 247 objects/s recovering 2024-09-13T14:46:49.367 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:49 smithi003 ceph-mon[107278]: 1.6 scrub starts 2024-09-13T14:46:49.367 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:49 smithi003 ceph-mon[107278]: 1.6 scrub ok 2024-09-13T14:46:49.367 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:49 smithi003 ceph-mon[107278]: pgmap v722: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 795 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 220 op/s; 5084/78 objects degraded (6517.949%); 1.2 MiB/s, 247 objects/s recovering 2024-09-13T14:46:49.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:49.431 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:49 smithi019 ceph-mon[91296]: 1.6 scrub starts 2024-09-13T14:46:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:49 smithi019 ceph-mon[91296]: 1.6 scrub ok 2024-09-13T14:46:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:49 smithi019 ceph-mon[91296]: pgmap v722: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 795 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 220 op/s; 5084/78 objects degraded (6517.949%); 1.2 MiB/s, 247 objects/s recovering 2024-09-13T14:46:49.453 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:49.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:49.578 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:49.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.703 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:49.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:49.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:49.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:49.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:49.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:50.054 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:50.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:50.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:50.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:50.201 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:50.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:50.308 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:50 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T14:46:50.308 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:50 smithi003 ceph-mon[107278]: 1.b scrub ok 2024-09-13T14:46:50.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:50 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T14:46:50.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:50 smithi003 ceph-mon[104007]: 1.b scrub ok 2024-09-13T14:46:50.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:50.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:50.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:50 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T14:46:50.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:50 smithi019 ceph-mon[91296]: 1.b scrub ok 2024-09-13T14:46:50.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:46:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:46:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:50.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:50.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:50.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:50.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:50.700 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:50.747 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:50.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:50.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:50.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:50.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:51.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:51.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:51.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:51.197 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:51 smithi003 ceph-mon[107278]: 1.5 scrub starts 2024-09-13T14:46:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:51 smithi003 ceph-mon[107278]: 1.5 scrub ok 2024-09-13T14:46:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:51 smithi003 ceph-mon[107278]: pgmap v723: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 795 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 50 op/s; 5084/78 objects degraded (6517.949%); 1.2 MiB/s, 258 objects/s recovering 2024-09-13T14:46:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5084/78 objects degraded (6517.949%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:51.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:51 smithi003 ceph-mon[104007]: 1.5 scrub starts 2024-09-13T14:46:51.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:51 smithi003 ceph-mon[104007]: 1.5 scrub ok 2024-09-13T14:46:51.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:51 smithi003 ceph-mon[104007]: pgmap v723: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 795 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 50 op/s; 5084/78 objects degraded (6517.949%); 1.2 MiB/s, 258 objects/s recovering 2024-09-13T14:46:51.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5084/78 objects degraded (6517.949%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:51.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:51.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:51.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:51.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:46:51.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:51.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:51 smithi019 ceph-mon[91296]: 1.5 scrub starts 2024-09-13T14:46:51.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:51 smithi019 ceph-mon[91296]: 1.5 scrub ok 2024-09-13T14:46:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:51 smithi019 ceph-mon[91296]: pgmap v723: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 795 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 50 op/s; 5084/78 objects degraded (6517.949%); 1.2 MiB/s, 258 objects/s recovering 2024-09-13T14:46:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5084/78 objects degraded (6517.949%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:51.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.446 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:51.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:51.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:51.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:51.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:51.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:51.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:51.820 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:51.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:51.944 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:52.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:52.070 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:52.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:52.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:52.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:52.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:52.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:52.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:52.481 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:52.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:52.570 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:52.672 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:52.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:52.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:52.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:52.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:52.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:53.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 9 pg_num to 36 2024-09-13T14:46:53.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:53.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:53.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:53.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:53.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:53.319 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:53.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:53 smithi003 ceph-mon[107278]: pgmap v724: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 5084/78 objects degraded (6517.949%); 0 B/s, 239 objects/s recovering 2024-09-13T14:46:53.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/147208609' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"} : dispatch 2024-09-13T14:46:53.402 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:53 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"} : dispatch 2024-09-13T14:46:53.402 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:53 smithi003 ceph-mon[104007]: pgmap v724: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 5084/78 objects degraded (6517.949%); 0 B/s, 239 objects/s recovering 2024-09-13T14:46:53.402 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/147208609' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"} : dispatch 2024-09-13T14:46:53.402 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:53 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"} : dispatch 2024-09-13T14:46:53.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:53 smithi019 ceph-mon[91296]: pgmap v724: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 5084/78 objects degraded (6517.949%); 0 B/s, 239 objects/s recovering 2024-09-13T14:46:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/147208609' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"} : dispatch 2024-09-13T14:46:53.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:53 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"} : dispatch 2024-09-13T14:46:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:53.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:53.567 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:53.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:53.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:53.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:53.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:53.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:53.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:53.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:53.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:54.043 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:54.053 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-13T14:46:54.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-13T14:46:54.067 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:54.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.262 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:54.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:54.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.403 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:54.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:54.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:54 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"}]': finished 2024-09-13T14:46:54.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:54 smithi019 ceph-mon[91296]: osdmap e218: 8 total, 8 up, 8 in 2024-09-13T14:46:54.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-13T14:46:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:46:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:46:54] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:46:54.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:54 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"}]': finished 2024-09-13T14:46:54.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:54 smithi003 ceph-mon[104007]: osdmap e218: 8 total, 8 up, 8 in 2024-09-13T14:46:54.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-13T14:46:54.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:54 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"}]': finished 2024-09-13T14:46:54.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:54 smithi003 ceph-mon[107278]: osdmap e218: 8 total, 8 up, 8 in 2024-09-13T14:46:54.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-13T14:46:54.504 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:54.527 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:54.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:54.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:54.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:54.776 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:54.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:54.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:54.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:55.005 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:55.028 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:55.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:55.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:55.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:55.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:55.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:55.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:55.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:55.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[107278]: pgmap v726: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 5084/78 objects degraded (6517.949%); 0 B/s, 103 objects/s recovering 2024-09-13T14:46:55.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-13T14:46:55.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[107278]: osdmap e219: 8 total, 8 up, 8 in 2024-09-13T14:46:55.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:46:55.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:55.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:55.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[104007]: pgmap v726: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 5084/78 objects degraded (6517.949%); 0 B/s, 103 objects/s recovering 2024-09-13T14:46:55.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-13T14:46:55.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[104007]: osdmap e219: 8 total, 8 up, 8 in 2024-09-13T14:46:55.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:46:55.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:55.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:55 smithi019 ceph-mon[91296]: pgmap v726: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+remapped, 114 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 5084/78 objects degraded (6517.949%); 0 B/s, 103 objects/s recovering 2024-09-13T14:46:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-13T14:46:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:55 smithi019 ceph-mon[91296]: osdmap e219: 8 total, 8 up, 8 in 2024-09-13T14:46:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:46:55.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:55.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:55.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:55.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:55.660 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:55.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:55.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:55.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:55.890 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:55.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:46:55.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:46:55.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:46:55.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:46:55.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:46:55.908 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:55.908 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:55.908 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:55.909 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:55.909 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:55.914 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:55.935 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:46:55.935 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:55.953 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:46:55.953 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:55.970 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:46:55.971 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:56.016 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:56.040 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:56.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:56.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:56.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:56.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:56.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[107278]: osdmap e220: 8 total, 8 up, 8 in 2024-09-13T14:46:56.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:56.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:56.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[107278]: 9.f scrub starts 2024-09-13T14:46:56.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[107278]: 9.f scrub ok 2024-09-13T14:46:56.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[107278]: osdmap e221: 8 total, 8 up, 8 in 2024-09-13T14:46:56.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[107278]: 9.2 scrub starts 2024-09-13T14:46:56.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[107278]: 9.2 scrub ok 2024-09-13T14:46:56.381 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3864382812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:56.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[104007]: osdmap e220: 8 total, 8 up, 8 in 2024-09-13T14:46:56.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:56.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:56.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[104007]: 9.f scrub starts 2024-09-13T14:46:56.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[104007]: 9.f scrub ok 2024-09-13T14:46:56.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[104007]: osdmap e221: 8 total, 8 up, 8 in 2024-09-13T14:46:56.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[104007]: 9.2 scrub starts 2024-09-13T14:46:56.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[104007]: 9.2 scrub ok 2024-09-13T14:46:56.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3864382812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:56.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:56.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:56.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:56 smithi019 ceph-mon[91296]: osdmap e220: 8 total, 8 up, 8 in 2024-09-13T14:46:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:56 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:56 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:56 smithi019 ceph-mon[91296]: 9.f scrub starts 2024-09-13T14:46:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:56 smithi019 ceph-mon[91296]: 9.f scrub ok 2024-09-13T14:46:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:56 smithi019 ceph-mon[91296]: osdmap e221: 8 total, 8 up, 8 in 2024-09-13T14:46:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:56 smithi019 ceph-mon[91296]: 9.2 scrub starts 2024-09-13T14:46:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:56 smithi019 ceph-mon[91296]: 9.2 scrub ok 2024-09-13T14:46:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3864382812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:56.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:56.547 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:56.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:56.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:56.775 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:56.800 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:56.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:56.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:57.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:57.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:57.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:57.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:57.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:57.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:57.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:57 smithi019 ceph-mon[91296]: 9.b scrub starts 2024-09-13T14:46:57.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:57 smithi019 ceph-mon[91296]: 9.b scrub ok 2024-09-13T14:46:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:57 smithi019 ceph-mon[91296]: 9.1 scrub starts 2024-09-13T14:46:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:57 smithi019 ceph-mon[91296]: pgmap v730: 129 pgs: 3 peering, 3 active+undersized+remapped, 10 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 111 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2406/78 objects degraded (3084.615%) 2024-09-13T14:46:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:57 smithi019 ceph-mon[91296]: 9.1 scrub ok 2024-09-13T14:46:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:57 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2406/78 objects degraded (3084.615%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:57 smithi019 ceph-mon[91296]: osdmap e222: 8 total, 8 up, 8 in 2024-09-13T14:46:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:57 smithi019 ceph-mon[91296]: 9.22 scrub starts 2024-09-13T14:46:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:57 smithi019 ceph-mon[91296]: 9.22 scrub ok 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[107278]: 9.b scrub starts 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[107278]: 9.b scrub ok 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[107278]: 9.1 scrub starts 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[107278]: pgmap v730: 129 pgs: 3 peering, 3 active+undersized+remapped, 10 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 111 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2406/78 objects degraded (3084.615%) 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[107278]: 9.1 scrub ok 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2406/78 objects degraded (3084.615%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[107278]: osdmap e222: 8 total, 8 up, 8 in 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[107278]: 9.22 scrub starts 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[107278]: 9.22 scrub ok 2024-09-13T14:46:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[104007]: 9.b scrub starts 2024-09-13T14:46:57.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[104007]: 9.b scrub ok 2024-09-13T14:46:57.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[104007]: 9.1 scrub starts 2024-09-13T14:46:57.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[104007]: pgmap v730: 129 pgs: 3 peering, 3 active+undersized+remapped, 10 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 111 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2406/78 objects degraded (3084.615%) 2024-09-13T14:46:57.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[104007]: 9.1 scrub ok 2024-09-13T14:46:57.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2406/78 objects degraded (3084.615%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:57.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[104007]: osdmap e222: 8 total, 8 up, 8 in 2024-09-13T14:46:57.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[104007]: 9.22 scrub starts 2024-09-13T14:46:57.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:57 smithi003 ceph-mon[104007]: 9.22 scrub ok 2024-09-13T14:46:57.531 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:57.556 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:57.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:57.658 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:57.684 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:57.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:57.812 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:57.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:57.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:58.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:58.066 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:58.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:58.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[104007]: 9.1b scrub starts 2024-09-13T14:46:58.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[104007]: 9.1b scrub ok 2024-09-13T14:46:58.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[104007]: 9.c scrub starts 2024-09-13T14:46:58.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[104007]: 9.c scrub ok 2024-09-13T14:46:58.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[104007]: 9.0 scrub starts 2024-09-13T14:46:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[104007]: 9.0 scrub ok 2024-09-13T14:46:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[107278]: 9.1b scrub starts 2024-09-13T14:46:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[107278]: 9.1b scrub ok 2024-09-13T14:46:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[107278]: 9.c scrub starts 2024-09-13T14:46:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[107278]: 9.c scrub ok 2024-09-13T14:46:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[107278]: 9.0 scrub starts 2024-09-13T14:46:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[107278]: 9.0 scrub ok 2024-09-13T14:46:58.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:58.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:58.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:58.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:58.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:58.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:58.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:58 smithi019 ceph-mon[91296]: 9.1b scrub starts 2024-09-13T14:46:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:58 smithi019 ceph-mon[91296]: 9.1b scrub ok 2024-09-13T14:46:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:58 smithi019 ceph-mon[91296]: 9.c scrub starts 2024-09-13T14:46:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:58 smithi019 ceph-mon[91296]: 9.c scrub ok 2024-09-13T14:46:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:58 smithi019 ceph-mon[91296]: 9.0 scrub starts 2024-09-13T14:46:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:58 smithi019 ceph-mon[91296]: 9.0 scrub ok 2024-09-13T14:46:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:46:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:58.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:58.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:58.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:58.670 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:58.697 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:58.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:58.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:58.894 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:58.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:58.951 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:59.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:59.054 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:46:59.054 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:46:59.054 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-13T14:46:59.054 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:46:59.054 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:46:59.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-13T14:46:59.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:59.143 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:46:59.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:59.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:59.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:59.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:59 smithi019 ceph-mon[91296]: 9.21 scrub starts 2024-09-13T14:46:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:59 smithi019 ceph-mon[91296]: 9.21 scrub ok 2024-09-13T14:46:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:59 smithi019 ceph-mon[91296]: pgmap v732: 129 pgs: 3 peering, 3 active+undersized+remapped, 8 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2406/78 objects degraded (3084.615%) 2024-09-13T14:46:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:59 smithi019 ceph-mon[91296]: 9.20 scrub starts 2024-09-13T14:46:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:59 smithi019 ceph-mon[91296]: 9.20 scrub ok 2024-09-13T14:46:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4079357154' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:46:59 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:59.433 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:59.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[107278]: 9.21 scrub starts 2024-09-13T14:46:59.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[107278]: 9.21 scrub ok 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[107278]: pgmap v732: 129 pgs: 3 peering, 3 active+undersized+remapped, 8 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2406/78 objects degraded (3084.615%) 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[107278]: 9.20 scrub starts 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[107278]: 9.20 scrub ok 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4079357154' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[104007]: 9.21 scrub starts 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[104007]: 9.21 scrub ok 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[104007]: pgmap v732: 129 pgs: 3 peering, 3 active+undersized+remapped, 8 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 113 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 2406/78 objects degraded (3084.615%) 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[104007]: 9.20 scrub starts 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[104007]: 9.20 scrub ok 2024-09-13T14:46:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4079357154' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:59.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:46:59 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:59.461 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:59.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:59.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:59.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:46:59.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:59.717 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:59.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:59.845 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:46:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:59.923 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:46:59.924 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:46:59.924 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:46:59.924 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:46:59.924 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:46:59.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:46:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:00.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:00.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:00.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[107278]: 9.1c scrub starts 2024-09-13T14:47:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[107278]: 9.1c scrub ok 2024-09-13T14:47:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[107278]: osdmap e223: 8 total, 8 up, 8 in 2024-09-13T14:47:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[104007]: 9.1c scrub starts 2024-09-13T14:47:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[104007]: 9.1c scrub ok 2024-09-13T14:47:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:00 smithi003 ceph-mon[104007]: osdmap e223: 8 total, 8 up, 8 in 2024-09-13T14:47:00.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:00.230 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:00.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:00.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:00.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:47:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:47:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:00 smithi019 ceph-mon[91296]: 9.1c scrub starts 2024-09-13T14:47:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:00 smithi019 ceph-mon[91296]: 9.1c scrub ok 2024-09-13T14:47:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:00 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:00 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:00 smithi019 ceph-mon[91296]: osdmap e223: 8 total, 8 up, 8 in 2024-09-13T14:47:00.459 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:00.486 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:00.587 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:00.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:00.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:00.715 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:00.742 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:00.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:00.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:00.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:00.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:01.098 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:01.123 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:01.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:01.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:01.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:01.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:01.373 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:01 smithi019 ceph-mon[91296]: pgmap v734: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 peering, 3 active+undersized+remapped, 1 unknown, 1 active+recovering, 121 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 220 B/s rd, 0 op/s; 2703/78 objects degraded (3465.385%) 2024-09-13T14:47:01.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:01 smithi003 ceph-mon[104007]: pgmap v734: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 peering, 3 active+undersized+remapped, 1 unknown, 1 active+recovering, 121 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 220 B/s rd, 0 op/s; 2703/78 objects degraded (3465.385%) 2024-09-13T14:47:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:01 smithi003 ceph-mon[107278]: pgmap v734: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 peering, 3 active+undersized+remapped, 1 unknown, 1 active+recovering, 121 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 220 B/s rd, 0 op/s; 2703/78 objects degraded (3465.385%) 2024-09-13T14:47:01.475 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:01.498 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:01.600 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:01.622 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:01.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:01.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:01.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:01.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:02.000 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:02.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:02.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:02.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:02.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:02.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:02.377 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:02.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:02.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:02.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:02.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:02.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:02 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2703/78 objects degraded (3465.385%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:47:02.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:02 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2703/78 objects degraded (3465.385%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:47:02.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:02 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2703/78 objects degraded (3465.385%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:47:02.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:02.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:02.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:02.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:02.999 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:03.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:03.124 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:03.225 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:03.248 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:03.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:03 smithi019 ceph-mon[91296]: pgmap v735: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 126 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 167 B/s rd, 0 op/s; 2703/78 objects degraded (3465.385%) 2024-09-13T14:47:03.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:03.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:03.599 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:03.621 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:03.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:03 smithi003 ceph-mon[107278]: pgmap v735: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 126 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 167 B/s rd, 0 op/s; 2703/78 objects degraded (3465.385%) 2024-09-13T14:47:03.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:03 smithi003 ceph-mon[104007]: pgmap v735: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 126 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 167 B/s rd, 0 op/s; 2703/78 objects degraded (3465.385%) 2024-09-13T14:47:03.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:03.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:03.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:03.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:03.972 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:03.995 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:04.097 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:04.119 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:04.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:04.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:04.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:04.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:04.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:47:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:47:04] "GET /metrics HTTP/1.1" 200 36517 "" "Prometheus/2.51.0" 2024-09-13T14:47:04.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:04.495 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:04.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:04.620 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:04.722 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:04.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:04.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:04.868 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:04.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:04.970 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:04.992 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:05.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:05.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:05.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:05.243 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:05.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:05.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:05 smithi003 ceph-mon[107278]: pgmap v736: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 126 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 2703/78 objects degraded (3465.385%) 2024-09-13T14:47:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:05 smithi003 ceph-mon[107278]: osdmap e224: 8 total, 8 up, 8 in 2024-09-13T14:47:05.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:05 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:05 smithi003 ceph-mon[104007]: pgmap v736: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 126 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 2703/78 objects degraded (3465.385%) 2024-09-13T14:47:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:05 smithi003 ceph-mon[104007]: osdmap e224: 8 total, 8 up, 8 in 2024-09-13T14:47:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:05 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:05.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:05.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:05.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:05.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:05.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:05.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:05 smithi019 ceph-mon[91296]: pgmap v736: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 126 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 2703/78 objects degraded (3465.385%) 2024-09-13T14:47:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:05 smithi019 ceph-mon[91296]: osdmap e224: 8 total, 8 up, 8 in 2024-09-13T14:47:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:05 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:05.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:05.749 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:05.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:05.851 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:05.873 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:05.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:47:05.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:05.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:06.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:06.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:06.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:06.224 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:06.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:06.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:06.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:06.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:06.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[107278]: osdmap e225: 8 total, 8 up, 8 in 2024-09-13T14:47:06.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[107278]: pgmap v739: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 127 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 155 B/s rd, 0 op/s; 22 MiB/s, 957 objects/s recovering 2024-09-13T14:47:06.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:47:06.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-13T14:47:06.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:06.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[104007]: osdmap e225: 8 total, 8 up, 8 in 2024-09-13T14:47:06.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[104007]: pgmap v739: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 127 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 155 B/s rd, 0 op/s; 22 MiB/s, 957 objects/s recovering 2024-09-13T14:47:06.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:47:06.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-13T14:47:06.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:06.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:06.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:06.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:06.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:06 smithi019 ceph-mon[91296]: osdmap e225: 8 total, 8 up, 8 in 2024-09-13T14:47:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:06 smithi019 ceph-mon[91296]: pgmap v739: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 127 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 155 B/s rd, 0 op/s; 22 MiB/s, 957 objects/s recovering 2024-09-13T14:47:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:47:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-13T14:47:06.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:06.753 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:06.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:06.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:06.953 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:06.980 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:07.007 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:07.109 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:07.136 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:07.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:07.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:07 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2703/78 objects degraded (3465.385%), 2 pgs degraded) 2024-09-13T14:47:07.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:07 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:47:07.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:07 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-13T14:47:07.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:07 smithi003 ceph-mon[104007]: osdmap e226: 8 total, 8 up, 8 in 2024-09-13T14:47:07.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:07 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2703/78 objects degraded (3465.385%), 2 pgs degraded) 2024-09-13T14:47:07.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:07 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:47:07.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:07 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-13T14:47:07.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:07 smithi003 ceph-mon[107278]: osdmap e226: 8 total, 8 up, 8 in 2024-09-13T14:47:07.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:07.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:07.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:07.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:07.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:07.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:07.649 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:07.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:07 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2703/78 objects degraded (3465.385%), 2 pgs degraded) 2024-09-13T14:47:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:07 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:47:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:07 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-13T14:47:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:07 smithi019 ceph-mon[91296]: osdmap e226: 8 total, 8 up, 8 in 2024-09-13T14:47:07.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:07.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:07.875 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:07.902 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:07.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:08.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:08.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:08.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_4' removed 2024-09-13T14:47:08.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:08.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:08.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:08.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:08.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:08.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:08.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:08.671 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:08.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:08 smithi019 ceph-mon[91296]: osdmap e227: 8 total, 8 up, 8 in 2024-09-13T14:47:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:08 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3214611286' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:47:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:08 smithi019 ceph-mon[91296]: pgmap v742: 129 pgs: 8 remapped+peering, 1 active+undersized+remapped, 120 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 80 KiB/s, 308 objects/s recovering 2024-09-13T14:47:08.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:08 smithi003 ceph-mon[104007]: osdmap e227: 8 total, 8 up, 8 in 2024-09-13T14:47:08.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:08 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3214611286' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:47:08.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:08 smithi003 ceph-mon[104007]: pgmap v742: 129 pgs: 8 remapped+peering, 1 active+undersized+remapped, 120 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 80 KiB/s, 308 objects/s recovering 2024-09-13T14:47:08.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:08 smithi003 ceph-mon[107278]: osdmap e227: 8 total, 8 up, 8 in 2024-09-13T14:47:08.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:08 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3214611286' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:47:08.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:08 smithi003 ceph-mon[107278]: pgmap v742: 129 pgs: 8 remapped+peering, 1 active+undersized+remapped, 120 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 80 KiB/s, 308 objects/s recovering 2024-09-13T14:47:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:08.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:47:08.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:08.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:08.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:09.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:09.046 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:09.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:09.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:09.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:09.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:09.397 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:09.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:09.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3214611286' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:47:09.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:09 smithi003 ceph-mon[104007]: osdmap e228: 8 total, 8 up, 8 in 2024-09-13T14:47:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3214611286' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:47:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:09 smithi003 ceph-mon[107278]: osdmap e228: 8 total, 8 up, 8 in 2024-09-13T14:47:09.475 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:47:09.476 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:47:09.476 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-13T14:47:09.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-09-13T14:47:09.522 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:09.545 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:09.647 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:09.670 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:09.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3214611286' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:47:09.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:09 smithi019 ceph-mon[91296]: osdmap e228: 8 total, 8 up, 8 in 2024-09-13T14:47:09.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:09.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:09.897 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:09.922 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:10.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:10.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:10.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:10.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:10.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:10.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:10.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:10 smithi003 ceph-mon[107278]: pgmap v744: 93 pgs: 93 active+clean; 678 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 27 KiB/s rd, 10 op/s 2024-09-13T14:47:10.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:10 smithi003 ceph-mon[104007]: pgmap v744: 93 pgs: 93 active+clean; 678 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 27 KiB/s rd, 10 op/s 2024-09-13T14:47:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.434 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:10.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:10.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:10.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:10.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:47:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:47:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:10.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:10 smithi019 ceph-mon[91296]: pgmap v744: 93 pgs: 93 active+clean; 678 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 27 KiB/s rd, 10 op/s 2024-09-13T14:47:10.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:10.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:10.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:10.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:10.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:10.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:11.036 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:11.062 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:11.163 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:11.176 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:47:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:11.186 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:11.196 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:47:11.196 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:11.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:47:11.216 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:11.235 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:47:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:11.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:47:11.251 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:11.267 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:47:11.267 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:11.284 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:47:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:11.288 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:11.301 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:47:11.301 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:11.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:11.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:11.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/230979981' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:11.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/230979981' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/230979981' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:11.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:11.536 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:11.558 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:11.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:11.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:11.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:11.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:11.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:11.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:11.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:12.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:12.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:12.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:12.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:12.186 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:12.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:12.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:12.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:12.431 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:12.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:12 smithi003 ceph-mon[107278]: pgmap v745: 93 pgs: 93 active+clean; 678 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 20 KiB/s rd, 7 op/s 2024-09-13T14:47:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:12 smithi003 ceph-mon[104007]: pgmap v745: 93 pgs: 93 active+clean; 678 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 20 KiB/s rd, 7 op/s 2024-09-13T14:47:12.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:12.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:12.601 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:12 smithi019 ceph-mon[91296]: pgmap v745: 93 pgs: 93 active+clean; 678 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 20 KiB/s rd, 7 op/s 2024-09-13T14:47:12.658 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:12.680 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:12.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:12.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:12.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:12.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:13.034 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:13.056 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:13.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:13.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:13.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:13.292 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:13.292 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:13.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:13.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:13.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:13.529 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:13.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:13.656 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:13.661 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:47:13.661 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:47:13.661 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:47:13.682 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:13.784 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:13.810 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:13.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:13.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:13.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:13.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:14.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:14.064 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:14.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:47:14.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:14.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:14.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:14.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:14.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:14.425 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:47:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:47:14] "GET /metrics HTTP/1.1" 200 34176 "" "Prometheus/2.51.0" 2024-09-13T14:47:14.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:14.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:14.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:14.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:14.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:14.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:14.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:14.917 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:14 smithi003 ceph-mon[107278]: pgmap v746: 93 pgs: 93 active+clean; 678 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 16 KiB/s rd, 6 op/s 2024-09-13T14:47:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:14 smithi003 ceph-mon[104007]: pgmap v746: 93 pgs: 93 active+clean; 678 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 16 KiB/s rd, 6 op/s 2024-09-13T14:47:14.944 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:15.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:15.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:15.170 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:14 smithi019 ceph-mon[91296]: pgmap v746: 93 pgs: 93 active+clean; 678 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 16 KiB/s rd, 6 op/s 2024-09-13T14:47:15.196 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:15.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:15.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:15.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:15.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:15.546 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:15.599 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:15.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:15.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:15.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:15.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:15.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:15.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:16.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:16.101 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:16.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:16.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:16.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:16.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:16.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:16.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:16.478 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:16.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:16.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:16.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:16.730 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:16.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:16.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:16.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:16 smithi003 ceph-mon[107278]: pgmap v747: 93 pgs: 93 active+clean; 678 KiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 14 KiB/s rd, 5 op/s 2024-09-13T14:47:16.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:16 smithi003 ceph-mon[104007]: pgmap v747: 93 pgs: 93 active+clean; 678 KiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 14 KiB/s rd, 5 op/s 2024-09-13T14:47:16.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:16.985 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:17.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:17.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:16 smithi019 ceph-mon[91296]: pgmap v747: 93 pgs: 93 active+clean; 678 KiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 14 KiB/s rd, 5 op/s 2024-09-13T14:47:17.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:17.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:17.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:17.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:17.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:47:17.459 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:17.486 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:17.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:17.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:17.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:17.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:17.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:17.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:17.952 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:17.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:17.991 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:17.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:18.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:18.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:18.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:18.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:18.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:18.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:18.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:18.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:18.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:18.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:18.622 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:18.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:18.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:18.878 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:18.980 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:19.007 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_5' created 2024-09-13T14:47:19.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:19.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:19.233 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:19.261 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:19.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:19.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:19.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:19 smithi019 ceph-mon[91296]: pgmap v748: 93 pgs: 93 active+clean; 678 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 4 op/s 2024-09-13T14:47:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:19 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:19 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1350548619' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:47:19.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:19 smithi003 ceph-mon[107278]: pgmap v748: 93 pgs: 93 active+clean; 678 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 4 op/s 2024-09-13T14:47:19.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:19 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:19.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:19 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1350548619' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:47:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:19 smithi003 ceph-mon[104007]: pgmap v748: 93 pgs: 93 active+clean; 678 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 4 op/s 2024-09-13T14:47:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:19 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:19 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1350548619' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:47:19.452 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:47:19.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:47:19.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:19.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:19.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:19.646 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:19.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:19.775 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:19.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:19.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:20.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:20.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:20.126 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:20.152 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:20.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:20.282 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:20.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:20 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1350548619' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T14:47:20.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:20 smithi003 ceph-mon[107278]: osdmap e229: 8 total, 8 up, 8 in 2024-09-13T14:47:20.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:20 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1350548619' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T14:47:20.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:20 smithi003 ceph-mon[104007]: osdmap e229: 8 total, 8 up, 8 in 2024-09-13T14:47:20.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:20.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:20 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1350548619' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T14:47:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:20 smithi019 ceph-mon[91296]: osdmap e229: 8 total, 8 up, 8 in 2024-09-13T14:47:20.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:47:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:47:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:20.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:20.537 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:20.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:20.665 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:20.766 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:20.792 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:20.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:21.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:47:21.048 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:21.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:21.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:21.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:21.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:21 smithi003 ceph-mon[107278]: pgmap v750: 109 pgs: 1 creating+peering, 15 unknown, 93 active+clean; 678 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail 2024-09-13T14:47:21.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:21 smithi003 ceph-mon[107278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:47:21.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:21 smithi003 ceph-mon[107278]: osdmap e230: 8 total, 8 up, 8 in 2024-09-13T14:47:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1021870646' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:21 smithi003 ceph-mon[104007]: pgmap v750: 109 pgs: 1 creating+peering, 15 unknown, 93 active+clean; 678 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail 2024-09-13T14:47:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:21 smithi003 ceph-mon[104007]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:47:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:21 smithi003 ceph-mon[104007]: osdmap e230: 8 total, 8 up, 8 in 2024-09-13T14:47:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1021870646' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:21.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:21.303 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:21.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-13T14:47:21.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:21.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:21.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:21 smithi019 ceph-mon[91296]: pgmap v750: 109 pgs: 1 creating+peering, 15 unknown, 93 active+clean; 678 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail 2024-09-13T14:47:21.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:21 smithi019 ceph-mon[91296]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:47:21.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:21 smithi019 ceph-mon[91296]: osdmap e230: 8 total, 8 up, 8 in 2024-09-13T14:47:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1021870646' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:21.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:21.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:21.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:21.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:21.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:21.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:21.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:21.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.935 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:21.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:22.036 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:22.062 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:22.164 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:22.189 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:22.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:22.316 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:22.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:22 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:22.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1021870646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:22.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:22 smithi003 ceph-mon[107278]: osdmap e231: 8 total, 8 up, 8 in 2024-09-13T14:47:22.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:22 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:22.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1021870646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:22.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:22 smithi003 ceph-mon[104007]: osdmap e231: 8 total, 8 up, 8 in 2024-09-13T14:47:22.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:22 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1021870646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:22 smithi019 ceph-mon[91296]: osdmap e231: 8 total, 8 up, 8 in 2024-09-13T14:47:22.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:22.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:22.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:22.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:22.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:22.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:22.816 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:22.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.944 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:22.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:22.992 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:23.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:23.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:23.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:23.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:23.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:23.319 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:23.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:23 smithi003 ceph-mon[107278]: pgmap v753: 109 pgs: 3 creating+peering, 13 unknown, 93 active+clean; 678 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail 2024-09-13T14:47:23.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:23 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:23.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:23 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:23.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:23 smithi003 ceph-mon[104007]: pgmap v753: 109 pgs: 3 creating+peering, 13 unknown, 93 active+clean; 678 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail 2024-09-13T14:47:23.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:23 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:23.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:23 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:23.420 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:23.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:23 smithi019 ceph-mon[91296]: pgmap v753: 109 pgs: 3 creating+peering, 13 unknown, 93 active+clean; 678 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail 2024-09-13T14:47:23.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:23 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:23 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:23.445 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:23.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:23.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:23.671 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:23.694 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:23.796 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:23.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:23.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:23.947 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:24.035 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:47:24.036 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:24.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:24.053 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:47:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:24.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:47:24.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:24.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:24.088 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:47:24.089 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:24.105 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:47:24.105 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:24.120 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:47:24.120 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:24.138 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:47:24.138 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:24.154 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:47:24.154 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:24.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:24.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:24.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:24.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:24.343 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:47:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:47:24] "GET /metrics HTTP/1.1" 200 34176 "" "Prometheus/2.51.0" 2024-09-13T14:47:24.343 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:47:24.343 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:47:24.344 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:24.344 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:24 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3973651312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:24.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:47:24.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:47:24.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:24.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:24 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3973651312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:24.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:24.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:47:24.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:47:24.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:24.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:24 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3973651312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:24.450 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:24.552 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:24.579 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:24.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:24.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:47:24.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:24.806 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:24.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:24.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:24.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:24.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:25.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:25.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:25.212 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:25.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:25.310 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:25 smithi003 ceph-mon[107278]: pgmap v754: 109 pgs: 3 creating+peering, 10 unknown, 96 active+clean; 678 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail 2024-09-13T14:47:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:25 smithi003 ceph-mon[104007]: pgmap v754: 109 pgs: 3 creating+peering, 10 unknown, 96 active+clean; 678 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail 2024-09-13T14:47:25.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:25.338 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:25.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:25 smithi019 ceph-mon[91296]: pgmap v754: 109 pgs: 3 creating+peering, 10 unknown, 96 active+clean; 678 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail 2024-09-13T14:47:25.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:25.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:25.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:25.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:25.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:25.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:25.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:25.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:25.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:25.964 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:26.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:26.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:26.191 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:26.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:26.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:26.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:26.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:26.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:26.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:26.595 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:26.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:26.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:26.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:26.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:26.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:26.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:27.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:27.094 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:27 smithi003 ceph-mon[107278]: pgmap v755: 109 pgs: 109 active+clean; 678 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:27.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:27 smithi003 ceph-mon[104007]: pgmap v755: 109 pgs: 109 active+clean; 678 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:27.098 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:27.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:47:27.199 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:27.226 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:27.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:27.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:27 smithi019 ceph-mon[91296]: pgmap v755: 109 pgs: 109 active+clean; 678 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:27.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:27.480 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:27.509 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:27.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:27.608 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:27.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:27.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:27.834 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:27.861 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:27.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:27.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:28.059 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:47:28.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:28.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:28.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:28.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:28.337 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:28.363 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:28.383 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2416192816' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:47:28.383 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 7]} : dispatch 2024-09-13T14:47:28.383 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 0]} : dispatch 2024-09-13T14:47:28.383 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:28.383 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:28.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2416192816' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:47:28.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 7]} : dispatch 2024-09-13T14:47:28.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 0]} : dispatch 2024-09-13T14:47:28.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:28.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2416192816' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:47:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 7]} : dispatch 2024-09-13T14:47:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 0]} : dispatch 2024-09-13T14:47:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:28.465 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:28.492 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:28.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:28.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:28.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:28.718 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:28.745 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:28.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:28.874 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:28.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:28.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:28.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:29.099 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:29.125 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:29.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:29.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:29.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:29.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[104007]: pgmap v756: 109 pgs: 109 active+clean; 678 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:29.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2416192816' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:47:29.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 7]}]': finished 2024-09-13T14:47:29.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 0]}]': finished 2024-09-13T14:47:29.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[104007]: osdmap e232: 8 total, 8 up, 8 in 2024-09-13T14:47:29.379 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[107278]: pgmap v756: 109 pgs: 109 active+clean; 678 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:29.379 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2416192816' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:47:29.379 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 7]}]': finished 2024-09-13T14:47:29.379 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 0]}]': finished 2024-09-13T14:47:29.379 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:29 smithi003 ceph-mon[107278]: osdmap e232: 8 total, 8 up, 8 in 2024-09-13T14:47:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.380 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:29.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:29.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:29 smithi019 ceph-mon[91296]: pgmap v756: 109 pgs: 109 active+clean; 678 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:29 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2416192816' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:47:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 7]}]': finished 2024-09-13T14:47:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 0]}]': finished 2024-09-13T14:47:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:29 smithi019 ceph-mon[91296]: osdmap e232: 8 total, 8 up, 8 in 2024-09-13T14:47:29.482 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:29.508 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:29.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:29.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:29.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:29.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:29.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:29.888 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:29.989 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:30.016 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:30.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:30.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:30.207 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:47:30.207 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:47:30.207 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-13T14:47:30.207 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:47:30.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:30.273 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:30.375 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:30.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:30 smithi019 ceph-mon[91296]: osdmap e233: 8 total, 8 up, 8 in 2024-09-13T14:47:30.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:47:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:47:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:30.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:30 smithi003 ceph-mon[104007]: osdmap e233: 8 total, 8 up, 8 in 2024-09-13T14:47:30.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:30 smithi003 ceph-mon[107278]: osdmap e233: 8 total, 8 up, 8 in 2024-09-13T14:47:30.502 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:30.527 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:30.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:30.654 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:30.756 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:30.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:30.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:47:30.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:31.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:31.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:31.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:31.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:31.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:31.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:31.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:31.408 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:31 smithi003 ceph-mon[107278]: pgmap v759: 109 pgs: 1 peering, 108 active+clean; 678 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:31.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:31 smithi003 ceph-mon[104007]: pgmap v759: 109 pgs: 1 peering, 108 active+clean; 678 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:31.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:31 smithi019 ceph-mon[91296]: pgmap v759: 109 pgs: 1 peering, 108 active+clean; 678 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:31.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:31.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:31.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:31.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:31.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:31.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:31.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:31.781 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:31.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:31.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:32.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:32.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:32.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:32.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:32.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:32.283 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:32.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:47:32.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:32.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:32.510 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:32.537 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:32.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:32.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:32.667 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:32.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:32.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:32.894 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:33.023 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:33.049 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:33.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:33.178 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:33 smithi003 ceph-mon[107278]: pgmap v760: 109 pgs: 1 peering, 108 active+clean; 678 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:33.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:33.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:33 smithi003 ceph-mon[104007]: pgmap v760: 109 pgs: 1 peering, 108 active+clean; 678 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:33.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:33.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:33.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:33.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:33.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:33 smithi019 ceph-mon[91296]: pgmap v760: 109 pgs: 1 peering, 108 active+clean; 678 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:33.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.434 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:33.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:33.535 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:33.561 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:33.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:33.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:33.698 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:47:33.698 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:47:33.699 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:47:33.699 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:47:33.699 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:33.699 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:33.699 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:33.699 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:33.699 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:47:33.700 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:47:33.700 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:47:33.700 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:33.700 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:33.700 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:33.710 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:47:33.710 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:33.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:33.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:33.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:34.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:34.064 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:34.089 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:47:34.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:34.191 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:34.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:34.318 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:34.420 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:34.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:34 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/429113181' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:34 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/393208307' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:34.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:34 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:47:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:47:34] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-09-13T14:47:34.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:34 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/429113181' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:34 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/393208307' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:34 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:34 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/429113181' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:34 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/393208307' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:34 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:34.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.446 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:34.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:34.546 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:34.572 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:34.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:34.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:34.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:34.812 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:34.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:35.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:35.078 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:35.092 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-13T14:47:35.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[104007]: pgmap v761: 109 pgs: 1 peering, 108 active+clean; 678 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:47:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[104007]: osdmap e234: 8 total, 8 up, 8 in 2024-09-13T14:47:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3735663954' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:47:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[107278]: pgmap v761: 109 pgs: 1 peering, 108 active+clean; 678 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:47:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[107278]: osdmap e234: 8 total, 8 up, 8 in 2024-09-13T14:47:35.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:35 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3735663954' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:47:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:35.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:35.304 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:35.329 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:35.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:35.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:35 smithi019 ceph-mon[91296]: pgmap v761: 109 pgs: 1 peering, 108 active+clean; 678 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:35 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:35 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:47:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:35 smithi019 ceph-mon[91296]: osdmap e234: 8 total, 8 up, 8 in 2024-09-13T14:47:35.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:35 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3735663954' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:47:35.458 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:35.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:35.588 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:35.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:35.715 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:35.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:35.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:35.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:35.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:35.971 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:36.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:36.147 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:36.199 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:36.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:36 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3735663954' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:47:36.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:36 smithi003 ceph-mon[107278]: osdmap e235: 8 total, 8 up, 8 in 2024-09-13T14:47:36.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:36 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3735663954' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:47:36.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:36 smithi003 ceph-mon[104007]: osdmap e235: 8 total, 8 up, 8 in 2024-09-13T14:47:36.227 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:36.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:36.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:36.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:36 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3735663954' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:47:36.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:36 smithi019 ceph-mon[91296]: osdmap e235: 8 total, 8 up, 8 in 2024-09-13T14:47:36.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:36.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:47:36.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:36.583 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:36.610 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:36.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:36.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:36.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:36.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:36.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:36.863 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:36.965 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:36.992 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:37.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:37.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:37.218 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:37.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:37 smithi003 ceph-mon[107278]: pgmap v764: 109 pgs: 109 active+clean; 678 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:37.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:37 smithi003 ceph-mon[107278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:47:37.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:37 smithi003 ceph-mon[104007]: pgmap v764: 109 pgs: 109 active+clean; 678 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:37.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:37 smithi003 ceph-mon[104007]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:47:37.245 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:37.347 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:37.374 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:37 smithi019 ceph-mon[91296]: pgmap v764: 109 pgs: 109 active+clean; 678 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:37 smithi019 ceph-mon[91296]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:47:37.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:37.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:47:37.503 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:37.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:37.631 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:37.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:37.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:37.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:37.885 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:37.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:37.996 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-13T14:47:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:38.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:38.010 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:47:38.010 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:47:38.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T14:47:38.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:47:38.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_197468 2024-09-13T14:47:38.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:47:38.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:47:38.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:38.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:38.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:38.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:38.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:38.398 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:38.432 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:38.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:38.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:38.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:38.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:38.781 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:38.815 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:38.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:38.953 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:38.991 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:39.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3539 3523 220.151 220.188 0.0033932 0.00452327 2024-09-13T14:47:39.083 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:39.094 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:39.113 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:47:39.126 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:39.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:39 smithi003 ceph-mon[107278]: pgmap v765: 109 pgs: 109 active+clean; 678 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:39.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/172123530' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:47:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:39 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:47:39.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:39 smithi003 ceph-mon[104007]: pgmap v765: 109 pgs: 109 active+clean; 678 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:39.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/172123530' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:47:39.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:39 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:47:39.229 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:39.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.266 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:39.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:39.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:39.403 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:39.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:39.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:39 smithi019 ceph-mon[91296]: pgmap v765: 109 pgs: 109 active+clean; 678 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-09-13T14:47:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/172123530' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:47:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:39 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:47:39.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:39.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:39.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:39.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:39.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:39.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:39.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:40.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:40.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 7168 7152 223.469 226.812 0.00701304 0.00446544 2024-09-13T14:47:40.101 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:40.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:40.243 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:40.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.389 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:40.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:40.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:47:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:47:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:40 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:47:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:40 smithi019 ceph-mon[91296]: osdmap e236: 8 total, 8 up, 8 in 2024-09-13T14:47:40.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:40 smithi019 ceph-mon[91296]: osdmap e237: 8 total, 8 up, 8 in 2024-09-13T14:47:40.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:40 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:47:40.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:40 smithi003 ceph-mon[107278]: osdmap e236: 8 total, 8 up, 8 in 2024-09-13T14:47:40.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:40 smithi003 ceph-mon[107278]: osdmap e237: 8 total, 8 up, 8 in 2024-09-13T14:47:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:40 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:47:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:40 smithi003 ceph-mon[104007]: osdmap e236: 8 total, 8 up, 8 in 2024-09-13T14:47:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:40 smithi003 ceph-mon[104007]: osdmap e237: 8 total, 8 up, 8 in 2024-09-13T14:47:40.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:40.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:40.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:40.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:40.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:40.767 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:40.805 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:40.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.947 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:40.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:40.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:47:41.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:41.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 10400 10384 216.304 202 0.00453445 0.00461417 2024-09-13T14:47:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:41.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:41.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:41.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:41 smithi003 ceph-mon[107278]: pgmap v767: 109 pgs: 2 peering, 107 active+clean; 144 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 24 MiB/s wr, 382 op/s 2024-09-13T14:47:41.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:41 smithi003 ceph-mon[104007]: pgmap v767: 109 pgs: 2 peering, 107 active+clean; 144 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 24 MiB/s wr, 382 op/s 2024-09-13T14:47:41.238 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:41.340 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:41.377 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:41.406 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:41.406 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:41.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:41 smithi019 ceph-mon[91296]: pgmap v767: 109 pgs: 2 peering, 107 active+clean; 144 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 24 MiB/s wr, 382 op/s 2024-09-13T14:47:41.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:41.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.516 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:41.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:41.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:41.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:41.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:41.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:41.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:41.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:41.931 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:42.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:42.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:42.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 13136 13120 204.974 171 0.00475867 0.00487212 2024-09-13T14:47:42.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:42.205 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:42.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:42.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:42.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:42.485 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:42.587 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:42.622 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:42.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:42.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:42.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:43.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:43.038 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:43.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:43.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:47:43.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 16090 16074 200.901 184.625 0.00587954 0.00497084 2024-09-13T14:47:43.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:43.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:43.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:43 smithi003 ceph-mon[107278]: pgmap v769: 109 pgs: 4 peering, 105 active+clean; 194 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 29 MiB/s wr, 466 op/s 2024-09-13T14:47:43.301 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:43.301 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:43.301 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:43.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:43 smithi003 ceph-mon[104007]: pgmap v769: 109 pgs: 4 peering, 105 active+clean; 194 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 29 MiB/s wr, 466 op/s 2024-09-13T14:47:43.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:43.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:43.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:43.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:43.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:43 smithi019 ceph-mon[91296]: pgmap v769: 109 pgs: 4 peering, 105 active+clean; 194 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 29 MiB/s wr, 466 op/s 2024-09-13T14:47:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:43.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:43.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:43.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:43.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:43.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:43.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:43.763 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:43.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:43.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:43.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:44.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:44.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:44.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 15 18836 18821 196.03 171.688 0.00606418 0.00509454 2024-09-13T14:47:44.132 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:47:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:47:44] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:47:44.157 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:44.192 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:44.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:44.335 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:44.385 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4057255426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:44.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4057255426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:44.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4057255426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:44.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:44.495 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:47:44.495 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:47:44.495 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:44.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:44.618 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:44.709 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:44.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:44.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:44.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:44.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:44.911 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:45.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:45.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 21531 21515 192.077 168.375 0.00787396 0.00520168 2024-09-13T14:47:45.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:45.185 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:45.200 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:45.288 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:45.324 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:45.426 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:45 smithi019 ceph-mon[91296]: pgmap v770: 109 pgs: 4 peering, 105 active+clean; 377 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s 2024-09-13T14:47:45.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:45 smithi003 ceph-mon[104007]: pgmap v770: 109 pgs: 4 peering, 105 active+clean; 377 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s 2024-09-13T14:47:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:45 smithi003 ceph-mon[107278]: pgmap v770: 109 pgs: 4 peering, 105 active+clean; 377 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s 2024-09-13T14:47:45.457 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:45.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:45.598 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:45.700 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:45.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:45.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:45.877 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:45.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:46.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:46.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 24109 24093 188.206 161.125 0.0056163 0.00530734 2024-09-13T14:47:46.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:46.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:46.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:46.296 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:46.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:46.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:46.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:46.580 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:46.683 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:46.704 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:46.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:46.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:46.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:46.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:47.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 26085 26069 181.016 123.5 0.0103646 0.00551859 2024-09-13T14:47:47.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:47.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:47.174 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:47:47.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:47:47.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:47.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:47.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:47.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:47.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:47 smithi019 ceph-mon[91296]: pgmap v771: 109 pgs: 109 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 145 MiB/s wr, 2.31k op/s; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:47 smithi019 ceph-mon[91296]: 1.6 scrub starts 2024-09-13T14:47:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:47 smithi019 ceph-mon[91296]: 1.6 scrub ok 2024-09-13T14:47:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/455188602' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:47 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[107278]: pgmap v771: 109 pgs: 109 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 145 MiB/s wr, 2.31k op/s; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[107278]: 1.6 scrub starts 2024-09-13T14:47:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[107278]: 1.6 scrub ok 2024-09-13T14:47:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/455188602' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:47.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[104007]: pgmap v771: 109 pgs: 109 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 145 MiB/s wr, 2.31k op/s; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:47.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[104007]: 1.6 scrub starts 2024-09-13T14:47:47.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[104007]: 1.6 scrub ok 2024-09-13T14:47:47.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/455188602' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:47.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:47 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:47.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:47.541 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:47.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.671 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:47.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:47.772 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:47.808 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:47.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:48.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:48.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:48.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 27979 27963 174.75 118.375 0.00127915 0.00568976 2024-09-13T14:47:48.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:48.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:48.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:48.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:48.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:48 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:48.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:48 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:47:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:48 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:48 smithi003 ceph-mon[107278]: osdmap e238: 8 total, 8 up, 8 in 2024-09-13T14:47:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:48 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:48 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:47:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:48 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:48 smithi003 ceph-mon[104007]: osdmap e238: 8 total, 8 up, 8 in 2024-09-13T14:47:48.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:48.564 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:48.603 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:48.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:48 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:48 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:47:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:48 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:48 smithi019 ceph-mon[91296]: osdmap e238: 8 total, 8 up, 8 in 2024-09-13T14:47:48.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:48.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:48.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:48.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:48.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:49.021 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:49.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 29529 29513 167.669 96.875 0.00544824 0.00595862 2024-09-13T14:47:49.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:49.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:49.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:49.276 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:49.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:49.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:49.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:49 smithi003 ceph-mon[104007]: pgmap v773: 109 pgs: 109 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:49.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:49 smithi003 ceph-mon[107278]: pgmap v773: 109 pgs: 109 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:49.517 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:49.558 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:49.659 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:49.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:49 smithi019 ceph-mon[91296]: pgmap v773: 109 pgs: 109 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:49.684 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:49.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:49.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:49.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:49.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:50.067 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:50.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 31471 31455 163.811 121.375 0.0090288 0.00609907 2024-09-13T14:47:50.107 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:50.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:50.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:50.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:50.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:50.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:50.524 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:50.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:50 smithi003 ceph-mon[104007]: 1.f scrub starts 2024-09-13T14:47:50.575 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:50 smithi003 ceph-mon[104007]: 1.f scrub ok 2024-09-13T14:47:50.575 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:50 smithi003 ceph-mon[107278]: 1.f scrub starts 2024-09-13T14:47:50.575 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:50 smithi003 ceph-mon[107278]: 1.f scrub ok 2024-09-13T14:47:50.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:50.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:50.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:50 smithi019 ceph-mon[91296]: 1.f scrub starts 2024-09-13T14:47:50.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:50 smithi019 ceph-mon[91296]: 1.f scrub ok 2024-09-13T14:47:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:47:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:47:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:50.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:50.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:50.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:50.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:51.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:51.071 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:51.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 15 33500 33485 160.969 126.875 0.00477247 0.00620754 2024-09-13T14:47:51.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:51.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:51.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:51.352 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:51.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:51.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:51.594 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:51.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:51.640 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:51 smithi019 ceph-mon[91296]: pgmap v774: 109 pgs: 109 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 162 MiB/s wr, 2.59k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:47:51.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:51 smithi003 ceph-mon[107278]: pgmap v774: 109 pgs: 109 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 162 MiB/s wr, 2.59k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:47:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:51 smithi003 ceph-mon[104007]: pgmap v774: 109 pgs: 109 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 162 MiB/s wr, 2.59k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:47:51.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:51.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:51.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:51.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:52.011 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:52.049 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:52.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 35508 35492 158.43 125.438 0.0134954 0.00630638 2024-09-13T14:47:52.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:52.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:52.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:52.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:52.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:52.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:52.534 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:52 smithi003 ceph-mon[107278]: 1.7 scrub starts 2024-09-13T14:47:52.534 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:52 smithi003 ceph-mon[107278]: 1.7 scrub ok 2024-09-13T14:47:52.534 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:52 smithi003 ceph-mon[104007]: 1.7 scrub starts 2024-09-13T14:47:52.534 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:52 smithi003 ceph-mon[104007]: 1.7 scrub ok 2024-09-13T14:47:52.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:52.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:52.629 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:52 smithi019 ceph-mon[91296]: 1.7 scrub starts 2024-09-13T14:47:52.629 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:52 smithi019 ceph-mon[91296]: 1.7 scrub ok 2024-09-13T14:47:52.711 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:52.750 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:52.853 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.894 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:52.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:52.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:53.031 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:53.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 37649 37633 156.788 133.812 0.00516217 0.00637209 2024-09-13T14:47:53.134 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:53.170 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:53.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:53.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:53.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:53.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:53 smithi003 ceph-mon[104007]: pgmap v775: 109 pgs: 109 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:47:53.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:53 smithi003 ceph-mon[107278]: pgmap v775: 109 pgs: 109 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:47:53.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:53.447 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:53.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:53.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:53 smithi019 ceph-mon[91296]: pgmap v775: 109 pgs: 109 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:47:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:53.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:53.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:53.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:53.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:53.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:54.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:54.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 39680 39664 154.921 126.938 0.0154527 0.00644985 2024-09-13T14:47:54.112 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:54.134 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:54.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:54.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:54.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:54.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:54.385 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:54.396 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:47:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:47:54] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:47:54.422 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:54.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:54.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:54.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.707 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:54.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:54.808 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:54.842 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:54.915 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:47:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:47:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:47:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:47:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:36986 9.0 e157/152 9:0f008b62:::benchmark_data_smithi003_156985_object36985:head v 157'2382, mlcod=157'2380)", 2024-09-13T14:47:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:07.872995+0000", 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 347.03849688999998, 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16962896399999999, 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.872995+0000", 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.872995+0000", 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.872996+0000", 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7000000000000003e-07 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873018+0000", 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2504e-05 2024-09-13T14:47:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873029+0000", 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1243000000000001e-05 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873036+0000", 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.173e-06 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873089+0000", 2024-09-13T14:47:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3143999999999998e-05 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.873108+0000", 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9573e-05 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042505+0000", 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16939712600000001 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042624+0000", 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118331 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:36988 9.6 e157/152 9:669fab83:::benchmark_data_smithi003_156985_object36987:head v 157'2342, mlcod=157'2340)", 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:07.874028+0000", 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 347.03746342699998, 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17089444100000001, 2024-09-13T14:47:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874028+0000", 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874028+0000", 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874029+0000", 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8999999999999995e-07 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874080+0000", 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1335999999999998e-05 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874091+0000", 2024-09-13T14:47:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0913e-05 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874098+0000", 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5220000000000004e-06 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042652+0000", 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16855393799999999 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042683+0000", 2024-09-13T14:47:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1177999999999999e-05 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.044809+0000", 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021263950000000001 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.044923+0000", 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011336900000000001 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:36987 9.0 e157/152 9:0dc893eb:::benchmark_data_smithi003_156985_object36986:head v 157'2383, mlcod=157'2380)", 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:07.874717+0000", 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 347.03677465800001, 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17119854900000001, 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874717+0000", 2024-09-13T14:47:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874717+0000", 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874718+0000", 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.99e-07 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:07.874745+0000", 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7538999999999998e-05 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042433+0000", 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16768813399999999 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042462+0000", 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8983e-05 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042710+0000", 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024720500000000002 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.042728+0000", 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8365000000000001e-05 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.045743+0000", 2024-09-13T14:47:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003014961 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:08.045916+0000", 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017266300000000001 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41012 9.6 e157/152 9:63d12f41:::benchmark_data_smithi003_156985_object41011:head v 157'2584, mlcod=157'2582)", 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.198172+0000", 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 344.713319609, 2024-09-13T14:47:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.229939433, 2024-09-13T14:47:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198172+0000", 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198172+0000", 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198173+0000", 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1100000000000005e-07 2024-09-13T14:47:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198285+0000", 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111656 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198296+0000", 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1303e-05 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198302+0000", 2024-09-13T14:47:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5740000000000001e-06 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198371+0000", 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8409000000000002e-05 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.198389+0000", 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8077e-05 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.428071+0000", 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22968186900000001 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.428112+0000", 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0734e-05 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41014 9.0 e157/152 9:057a3b88:::benchmark_data_smithi003_156985_object41013:head v 157'2642, mlcod=157'2639)", 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.200366+0000", 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 344.71112541100001, 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.225644175, 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200366+0000", 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200366+0000", 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200367+0000", 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7000000000000003e-07 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200409+0000", 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1468999999999999e-05 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200420+0000", 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1521e-05 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200427+0000", 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5740000000000001e-06 2024-09-13T14:47:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200485+0000", 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8315e-05 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.200505+0000", 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0066999999999999e-05 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425972+0000", 2024-09-13T14:47:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.225466323 2024-09-13T14:47:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.426010+0000", 2024-09-13T14:47:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8936000000000001e-05 2024-09-13T14:47:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41016 9.6 e157/152 9:63031843:::benchmark_data_smithi003_156985_object41015:head v 157'2585, mlcod=157'2582)", 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.201276+0000", 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 344.71021538600002, 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22863520700000001, 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201276+0000", 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201276+0000", 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201277+0000", 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3099999999999996e-07 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201315+0000", 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7508000000000001e-05 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201338+0000", 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3682000000000001e-05 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201347+0000", 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3100000000000001e-06 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201424+0000", 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6872000000000005e-05 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.201442+0000", 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8805e-05 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.429846+0000", 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.228403949 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.429912+0000", 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5149999999999998e-05 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41023 9.2 e157/152 9:40cd60d0:::benchmark_data_smithi003_156985_object41022:head v 157'2589, mlcod=157'2586)", 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.203360+0000", 2024-09-13T14:47:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 344.70813182699999, 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22830435900000001, 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.203360+0000", 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.203360+0000", 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.203361+0000", 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5799999999999998e-07 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.203403+0000", 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2021000000000003e-05 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425885+0000", 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.222481715 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425897+0000", 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2736e-05 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425964+0000", 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6964999999999993e-05 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425986+0000", 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1676999999999999e-05 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.431557+0000", 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055714179999999999 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.431664+0000", 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106869 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:41025 9.6 e157/152 9:668eb8d4:::benchmark_data_smithi003_156985_object41024:head v 157'2586, mlcod=157'2582)", 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:42:10.204204+0000", 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 344.70728769900001, 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.225741672, 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.204204+0000", 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.204204+0000", 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.204205+0000", 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.133e-06 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.204253+0000", 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8340999999999997e-05 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425909+0000", 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22165542399999999 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.425919+0000", 2024-09-13T14:47:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0074999999999999e-05 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.427060+0000", 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00114062 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.427080+0000", 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0298000000000001e-05 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.429916+0000", 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002835859 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:42:10.429946+0000", 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9921999999999999e-05 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:215648 9.2 9:45569c25:::benchmark_data_smithi003_156985_object94732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-13T14:47:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.106715+0000", 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 111.80477623199999, 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18401425199999999, 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 215648 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.106715+0000", 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.106715+0000", 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.106717+0000", 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.671e-06 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.106722+0000", 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.215e-06 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.106735+0000", 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3053e-05 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.106743+0000", 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6809999999999998e-06 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.106806+0000", 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.266e-05 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.108980+0000", 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00217381 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.114872+0000", 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058927709999999998 2024-09-13T14:47:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.114968+0000", 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5551999999999996e-05 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.115094+0000", 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012651000000000001 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.121470+0000", 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063751110000000001 2024-09-13T14:47:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.286483+0000", 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.165013839 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.289730+0000", 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032468509999999998 2024-09-13T14:47:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.959 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.289787+0000", 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6613000000000002e-05 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.290730+0000", 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094291500000000001 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:215656 9.2 9:42fb7858:::benchmark_data_smithi003_156985_object94740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.113167+0000", 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 111.798325081, 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18778081499999999, 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:47:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 215656 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.113167+0000", 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.113167+0000", 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.113169+0000", 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1440000000000001e-06 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.113174+0000", 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7169999999999996e-06 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.113187+0000", 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.237e-05 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.113194+0000", 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4309999999999997e-06 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.118147+0000", 2024-09-13T14:47:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049524959999999998 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.118824+0000", 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00067697500000000002 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.286526+0000", 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.167702253 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.286648+0000", 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012235699999999999 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.286770+0000", 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121564 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.296264+0000", 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094944269999999997 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.296304+0000", 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9407999999999999e-05 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.298211+0000", 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001907726 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.298529+0000", 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031794400000000002 2024-09-13T14:47:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.300947+0000", 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024180030000000002 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:215659 9.10 e205/176 9:0c6b1b94:::benchmark_data_smithi003_156985_object94743:head v 205'8613, mlcod=205'8611)", 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.115525+0000", 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 111.795967143, 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16992805, 2024-09-13T14:47:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.115525+0000", 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.115525+0000", 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.115526+0000", 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.029e-06 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.115534+0000", 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5690000000000005e-06 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.115545+0000", 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0628000000000001e-05 2024-09-13T14:47:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.115552+0000", 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8179999999999998e-06 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.115600+0000", 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8801000000000003e-05 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.115619+0000", 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8893999999999999e-05 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.285399+0000", 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16977998799999999 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.285453+0000", 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3322999999999998e-05 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.45869.0:215648 9.2 e205/200 ondisk, result = 0)", 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.122669+0000", 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 111.788822235, 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17396265299999999, 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.122669+0000", 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.122669+0000", 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.122670+0000", 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9899999999999999e-07 2024-09-13T14:47:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.122675+0000", 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8160000000000001e-06 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.122686+0000", 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1379999999999999e-05 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.122693+0000", 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4439999999999996e-06 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.289714+0000", 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.167021011 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.289723+0000", 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9069999999999996e-06 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.296632+0000", 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069092959999999997 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:240193 9.2 9:45c407cc:::benchmark_data_smithi003_156985_object119277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:35.187078+0000", 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 79.724414034999995, 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20281595499999999, 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 240193 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.187078+0000", 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.187078+0000", 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.187080+0000", 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1900000000000002e-06 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.187086+0000", 2024-09-13T14:47:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8239999999999996e-06 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.187100+0000", 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4569000000000001e-05 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.187109+0000", 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3420000000000008e-06 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.187612+0000", 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050350099999999995 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.187740+0000", 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127949 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.187877+0000", 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013744000000000001 2024-09-13T14:47:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.193993+0000", 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061156439999999999 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.198691+0000", 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046981890000000002 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.389240+0000", 2024-09-13T14:47:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19054874399999999 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.389300+0000", 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0040000000000001e-05 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.389894+0000", 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00059352299999999999 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:27877 10.3 10:cfe350c3:::benchmark_data_smithi003_197468_object27876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:48.004684+0000", 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.906808034, 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28990059800000001, 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 27877 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004684+0000", 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004684+0000", 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004685+0000", 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2270000000000001e-06 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004706+0000", 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0746000000000001e-05 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004717+0000", 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1207e-05 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004724+0000", 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4259999999999997e-06 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004791+0000", 2024-09-13T14:47:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6811999999999997e-05 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004983+0000", 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000192098 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005103+0000", 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011979 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.007476+0000", 2024-09-13T14:47:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002372847 2024-09-13T14:47:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.008772+0000", 2024-09-13T14:47:54.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012964540000000001 2024-09-13T14:47:54.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294492+0000", 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28571956199999998 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294540+0000", 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8170999999999997e-05 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294584+0000", 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4258000000000002e-05 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:27878 10.3 10:c94d1851:::benchmark_data_smithi003_197468_object27877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:48.005297+0000", 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.9061949259999995, 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.289465476, 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 27878 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005297+0000", 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005297+0000", 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005298+0000", 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.111e-06 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005319+0000", 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1474000000000001e-05 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005330+0000", 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0824e-05 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005337+0000", 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2989999999999999e-06 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.006220+0000", 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00088279299999999999 2024-09-13T14:47:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.006329+0000", 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108799 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.006453+0000", 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012355600000000001 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.008786+0000", 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023338579999999999 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.010360+0000", 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015731569999999999 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294646+0000", 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.284286078 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294711+0000", 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5222000000000007e-05 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294762+0000", 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1304999999999998e-05 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:27879 10.3 10:ca6535d7:::benchmark_data_smithi003_197468_object27878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:48.005359+0000", 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.9061328189999998, 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28957026200000002, 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 27879 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005359+0000", 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005359+0000", 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005359+0000", 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7899999999999998e-07 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005387+0000", 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7073999999999999e-05 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005395+0000", 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4200000000000007e-06 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005401+0000", 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9649999999999996e-06 2024-09-13T14:47:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.007509+0000", 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021081400000000001 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.007692+0000", 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000183315 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:47:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.007820+0000", 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012717199999999999 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.008930+0000", 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011109340000000001 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.010476+0000", 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001545645 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294816+0000", 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28434024699999999 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294888+0000", 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1768999999999997e-05 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294929+0000", 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1001999999999998e-05 2024-09-13T14:47:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:27964 10.3 10:ca7aba1c:::benchmark_data_smithi003_197468_object27963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:48.051474+0000", 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.860018041, 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24511971399999999, 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 27964 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.051474+0000", 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.051474+0000", 2024-09-13T14:47:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.051475+0000", 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1429999999999999e-06 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.051514+0000", 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9067999999999999e-05 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.051527+0000", 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2908000000000001e-05 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.051532+0000", 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6949999999999997e-06 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.051612+0000", 2024-09-13T14:47:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9153999999999995e-05 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.051760+0000", 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014868800000000001 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.051930+0000", 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000170039 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.053003+0000", 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001072721 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.053448+0000", 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044537100000000002 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.295483+0000", 2024-09-13T14:47:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.242035001 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.296552+0000", 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010683870000000001 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.296593+0000", 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1539000000000003e-05 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:27976 10.3 10:c287e011:::benchmark_data_smithi003_197468_object27975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:48.063156+0000", 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.8483360910000002, 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.233836144, 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:47:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 27976 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.063156+0000", 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.063156+0000", 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.063157+0000", 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7349999999999999e-06 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.063217+0000", 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9531999999999999e-05 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.063232+0000", 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5376999999999999e-05 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.063240+0000", 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7910000000000005e-06 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.063309+0000", 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9037999999999999e-05 2024-09-13T14:47:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.063426+0000", 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011678799999999999 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.063549+0000", 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122965 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.064538+0000", 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00098953699999999997 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.064867+0000", 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032824800000000001 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.296793+0000", 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.231926359 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.296952+0000", 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015855700000000001 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.296992+0000", 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0216999999999998e-05 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:29525 10.5 10:aee7fb02:::benchmark_data_smithi003_197468_object29524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:49.073968+0000", 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.8375232710000002, 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.172717384, 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 29525 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.073968+0000", 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.073968+0000", 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.073970+0000", 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.398e-06 2024-09-13T14:47:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.073998+0000", 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8028e-05 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.074011+0000", 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2717999999999999e-05 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.074018+0000", 2024-09-13T14:47:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5000000000000002e-06 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.075487+0000", 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014692150000000001 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.075619+0000", 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013189000000000001 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:47:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.075789+0000", 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169779 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.077372+0000", 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001583154 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.078447+0000", 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010745900000000001 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.246569+0000", 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16812191700000001 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.246635+0000", 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6713999999999999e-05 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.246686+0000", 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0481e-05 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:29551 10.5 10:a871395d:::benchmark_data_smithi003_197468_object29550:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:49.085227+0000", 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.8262650499999999, 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16648759099999999, 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:47:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 29551 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.085227+0000", 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.085227+0000", 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.085228+0000", 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.16e-07 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.085255+0000", 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7191000000000001e-05 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:47:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.085263+0000", 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3559999999999993e-06 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.085267+0000", 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.211e-06 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.085325+0000", 2024-09-13T14:47:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7759e-05 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.085447+0000", 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012176300000000001 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.085568+0000", 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120764 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.086648+0000", 2024-09-13T14:47:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010807309999999999 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.087396+0000", 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00074780400000000003 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.251620+0000", 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16422412 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:47:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.251669+0000", 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8612999999999997e-05 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:49.251714+0000", 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5362999999999999e-05 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:47:55.032 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:47:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.034 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:55.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:55.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 41448 41432 152.308 110.5 0.00234802 0.00654408 2024-09-13T14:47:55.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:55.170 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:55.222 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:47:55.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:55.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:55.410 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:55.449 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:55.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:47:55.538 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:55 smithi003 ceph-mon[107278]: pgmap v776: 109 pgs: 109 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 158 MiB/s wr, 2.52k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:47:55.538 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2884667100' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:47:55.538 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:55 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:47:55.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:55 smithi003 ceph-mon[104007]: pgmap v776: 109 pgs: 109 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 158 MiB/s wr, 2.52k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:47:55.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2884667100' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:47:55.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:55 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:47:55.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:55.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:55.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:55 smithi019 ceph-mon[91296]: pgmap v776: 109 pgs: 109 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 158 MiB/s wr, 2.52k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:47:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2884667100' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:47:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:55 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:47:55.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:55.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:55.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:55.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:55.990 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:56.028 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:56.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 43247 43231 150.092 112.438 0.00691078 0.00665779 2024-09-13T14:47:56.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:56.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:56.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:56.292 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:56.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:56.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:56.534 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:56.573 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:56.675 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:56.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:56 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:47:56.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:56 smithi019 ceph-mon[91296]: osdmap e239: 8 total, 8 up, 8 in 2024-09-13T14:47:56.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:56 smithi019 ceph-mon[91296]: pgmap v778: 109 pgs: 109 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-09-13T14:47:56.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:56 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:47:56.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:56 smithi003 ceph-mon[107278]: osdmap e239: 8 total, 8 up, 8 in 2024-09-13T14:47:56.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:56 smithi003 ceph-mon[107278]: pgmap v778: 109 pgs: 109 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-09-13T14:47:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:56 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:47:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:56 smithi003 ceph-mon[104007]: osdmap e239: 8 total, 8 up, 8 in 2024-09-13T14:47:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:56 smithi003 ceph-mon[104007]: pgmap v778: 109 pgs: 109 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-09-13T14:47:56.710 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:56.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:56.849 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:56.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:56.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:56.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:57.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 44974 44958 147.873 107.938 0.00243902 0.00673607 2024-09-13T14:47:57.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:57.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:57.264 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:57.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:57.365 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.403 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:57.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:57.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.544 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:57.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:57.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:57.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:57.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:57.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:57.926 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:57.960 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:58.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:58.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:47:58.076724+0000 min lat: 0.00116002 max lat: 0.294117 avg lat: 0.00682674 2024-09-13T14:47:58.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:47:58.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 46861 46845 146.375 117.938 0.0103632 0.00682674 2024-09-13T14:47:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:58.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:58.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:58.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:58.381 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:58.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:58.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:58.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:58.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:47:58.660 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:58.762 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:58.798 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:58.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:58.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:58 smithi003 ceph-mon[107278]: pgmap v779: 109 pgs: 109 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-13T14:47:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:58 smithi003 ceph-mon[104007]: pgmap v779: 109 pgs: 109 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-13T14:47:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:58.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:59.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:59.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 48704 48688 144.89 115.188 0.00691677 0.00689657 2024-09-13T14:47:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:59.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:47:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:58 smithi019 ceph-mon[91296]: pgmap v779: 109 pgs: 109 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-13T14:47:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:47:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:59.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:59.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:59.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:59.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:47:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:59.478 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:59.517 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:59.615 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:59.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:59.646 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:59.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:59.787 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:59.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:59.922 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:47:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:59.923 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:00.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:00.030 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:47:59 smithi003 ceph-mon[104007]: 1.3 scrub starts 2024-09-13T14:48:00.030 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:47:59 smithi003 ceph-mon[107278]: 1.3 scrub starts 2024-09-13T14:48:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:00.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:00.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 50356 50340 142.997 103.25 0.00783856 0.00698798 2024-09-13T14:48:00.170 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:00.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:47:59 smithi019 ceph-mon[91296]: 1.3 scrub starts 2024-09-13T14:48:00.203 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:00.305 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:00.340 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:00.441 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:00.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:00.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:00.616 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:00.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:48:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:48:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:00.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:00.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:00.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:00.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:00.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:00 smithi003 ceph-mon[107278]: 1.3 scrub ok 2024-09-13T14:48:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:00 smithi003 ceph-mon[107278]: pgmap v780: 109 pgs: 109 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-13T14:48:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:00 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4291952043' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:00 smithi003 ceph-mon[104007]: 1.3 scrub ok 2024-09-13T14:48:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:00 smithi003 ceph-mon[104007]: pgmap v780: 109 pgs: 109 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-13T14:48:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:00 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4291952043' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:00.994 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:01.030 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:01.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 52039 52023 141.353 105.188 0.0067407 0.00706893 2024-09-13T14:48:01.132 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:01.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:00 smithi019 ceph-mon[91296]: 1.3 scrub ok 2024-09-13T14:48:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:00 smithi019 ceph-mon[91296]: pgmap v780: 109 pgs: 109 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-13T14:48:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:00 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4291952043' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:01.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:01.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:01.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:01.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:01.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:01.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:01.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:01.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:48:01.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:48:01.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:48:01.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:48:01.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:48:01.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:48:01.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:48:01.782 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:48:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:01.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:01.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:02.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:02.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 53802 53786 140.054 110.188 0.00442429 0.00713469 2024-09-13T14:48:02.103 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:02.140 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:02.242 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:02.277 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:02.379 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:02.418 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:02.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:02.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:02.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:02.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:02.801 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:02.838 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:02.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:02 smithi003 ceph-mon[107278]: pgmap v781: 109 pgs: 109 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:48:02.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:02 smithi003 ceph-mon[104007]: pgmap v781: 109 pgs: 109 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:48:02.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:02.974 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:03.036 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:03.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:03.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 55516 55500 138.721 107.125 0.0127844 0.00720337 2024-09-13T14:48:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:03.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:02 smithi019 ceph-mon[91296]: pgmap v781: 109 pgs: 109 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:48:03.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.260 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:03.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:03.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:03.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:03.501 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:03.539 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:03.641 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.679 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:03.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:03.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:03.816 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:03.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:03.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:04.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:04.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 57165 57149 137.349 103.062 0.00697638 0.0072751 2024-09-13T14:48:04.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:04.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:48:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:48:04] "GET /metrics HTTP/1.1" 200 36562 "" "Prometheus/2.51.0" 2024-09-13T14:48:04.196 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:04.229 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:04.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:04.331 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.374 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:04.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:04.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:04.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:04.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:48:04.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:04.653 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:04.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:04.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:04.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:04.825 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:48:04.897 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.938 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:04.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:05.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:05.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:05.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 58634 58618 135.663 91.8125 0.00830606 0.00736635 2024-09-13T14:48:05.177 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:05.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:04 smithi019 ceph-mon[91296]: pgmap v782: 109 pgs: 109 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T14:48:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:04 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/224763009' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:05.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:04 smithi003 ceph-mon[107278]: pgmap v782: 109 pgs: 109 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T14:48:05.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:04 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/224763009' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:04 smithi003 ceph-mon[104007]: pgmap v782: 109 pgs: 109 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T14:48:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:04 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/224763009' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:05.212 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:05.314 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:05.348 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:05.450 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:05.487 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:05.589 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.629 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:05.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:05.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:05.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:05.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:06.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:06.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 60225 60209 134.369 99.4375 0.00722974 0.00743714 2024-09-13T14:48:06.140 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:05 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/224763009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:05 smithi019 ceph-mon[91296]: osdmap e240: 8 total, 8 up, 8 in 2024-09-13T14:48:06.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:05 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:06.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/224763009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:06.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:05 smithi003 ceph-mon[107278]: osdmap e240: 8 total, 8 up, 8 in 2024-09-13T14:48:06.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:05 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:06.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/224763009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:06.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:05 smithi003 ceph-mon[104007]: osdmap e240: 8 total, 8 up, 8 in 2024-09-13T14:48:06.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:06.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:06.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:06.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:06.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:06.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:06.695 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:06.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:06.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:06.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:06.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:06.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:06.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:07.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 61623 61607 132.748 87.375 0.0110628 0.00752842 2024-09-13T14:48:07.098 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:07.136 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:07.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:07.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:06 smithi003 ceph-mon[107278]: pgmap v784: 109 pgs: 109 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:48:07.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:06 smithi003 ceph-mon[104007]: pgmap v784: 109 pgs: 109 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:48:07.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:06 smithi019 ceph-mon[91296]: pgmap v784: 109 pgs: 109 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:48:07.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:07.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:07.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:07.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:07.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:07.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:07.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:07.705 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:07.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:07.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:07.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:07.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:08.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 15 63019 63004 131.234 87.3125 0.0110022 0.00761489 2024-09-13T14:48:08.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:08.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:08.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:08.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:08.398 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:08.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:08.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:08.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:08.683 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:08.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:08.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:08.856 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:08.856 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:08.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:08.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:08 smithi003 ceph-mon[107278]: pgmap v785: 109 pgs: 109 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:48:08.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:08 smithi003 ceph-mon[104007]: pgmap v785: 109 pgs: 109 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:48:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:09.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:09.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 64361 64345 129.704 83.8125 0.0101182 0.00770509 2024-09-13T14:48:09.099 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:09.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:08 smithi019 ceph-mon[91296]: pgmap v785: 109 pgs: 109 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:48:09.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:09.238 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:09.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:09.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:09.486 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:09.520 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:09.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:09.662 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:09.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:09.802 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:09.870 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:48:09.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:09.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:48:09.938 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:10.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:48:10.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:10.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 65726 65710 128.316 85.3125 0.00311396 0.00775823 2024-09-13T14:48:10.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:10.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3781498560' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:48:10.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3781498560' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:48:10.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3781498560' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:48:10.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:10.331 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:10.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:10.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:10.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:10.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:48:10.571 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:10.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:10.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:10.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:48:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:48:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:10.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:10.743 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:10.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:10.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:10.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:11.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:11.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 65726 65710 124.428 0 - 0.00775823 2024-09-13T14:48:11.103 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:11.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:10 smithi003 ceph-mon[107278]: pgmap v786: 109 pgs: 109 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:48:11.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3781498560' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:48:11.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:10 smithi003 ceph-mon[107278]: osdmap e241: 8 total, 8 up, 8 in 2024-09-13T14:48:11.119 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:10 smithi003 ceph-mon[107278]: osdmap e242: 8 total, 8 up, 8 in 2024-09-13T14:48:11.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:10 smithi003 ceph-mon[104007]: pgmap v786: 109 pgs: 109 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:48:11.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3781498560' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:48:11.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:10 smithi003 ceph-mon[104007]: osdmap e241: 8 total, 8 up, 8 in 2024-09-13T14:48:11.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:10 smithi003 ceph-mon[104007]: osdmap e242: 8 total, 8 up, 8 in 2024-09-13T14:48:11.130 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:10 smithi019 ceph-mon[91296]: pgmap v786: 109 pgs: 109 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:48:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3781498560' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:48:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:10 smithi019 ceph-mon[91296]: osdmap e241: 8 total, 8 up, 8 in 2024-09-13T14:48:11.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:10 smithi019 ceph-mon[91296]: osdmap e242: 8 total, 8 up, 8 in 2024-09-13T14:48:11.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:11.259 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:11.345 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:11.345 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:11.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:11.386 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:11.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:11.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:11.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:11.645 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:11.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:11.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:11.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:11.880 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:11.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:11.998 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:12.025 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:12.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 65726 65710 120.769 0 - 0.00775823 2024-09-13T14:48:12.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:12.154 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:12.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:12.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:12.290 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:12.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:12.392 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:12.431 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:12.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:12.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:12.670 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.704 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:12.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:12.805 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.841 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:12.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:12.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:12 smithi019 ceph-mon[91296]: osdmap e243: 8 total, 8 up, 8 in 2024-09-13T14:48:12.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:12 smithi019 ceph-mon[91296]: pgmap v790: 109 pgs: 2 peering, 107 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T14:48:12.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:12 smithi003 ceph-mon[104007]: osdmap e243: 8 total, 8 up, 8 in 2024-09-13T14:48:12.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:12 smithi003 ceph-mon[104007]: pgmap v790: 109 pgs: 2 peering, 107 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T14:48:12.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:12 smithi003 ceph-mon[107278]: osdmap e243: 8 total, 8 up, 8 in 2024-09-13T14:48:12.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:12 smithi003 ceph-mon[107278]: pgmap v790: 109 pgs: 2 peering, 107 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T14:48:12.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:12.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:13.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:13.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 66625 66609 118.924 18.7292 0.00378295 0.00840003 2024-09-13T14:48:13.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:13.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:13.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:13.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:13.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:13.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:13.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:13.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:13.659 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:13.761 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:13.795 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:13.897 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:13.932 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:14.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:14.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 67954 67938 117.928 83.0625 0.00750022 0.00847347 2024-09-13T14:48:14.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:14.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:14.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:14.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:14.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:14.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:14.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:14.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:14.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:14.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:14.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:48:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:48:14] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:48:14.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:14.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:14.600 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:14.634 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:14.736 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:14.765 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:14.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:14.901 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:15.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:15.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:15.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:14 smithi003 ceph-mon[104007]: pgmap v791: 109 pgs: 1 active+recovering+undersized+remapped, 2 peering, 106 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 4152/194976 objects misplaced (2.129%); 277 KiB/s, 4 objects/s recovering 2024-09-13T14:48:15.081 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:14 smithi003 ceph-mon[107278]: pgmap v791: 109 pgs: 1 active+recovering+undersized+remapped, 2 peering, 106 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 4152/194976 objects misplaced (2.129%); 277 KiB/s, 4 objects/s recovering 2024-09-13T14:48:15.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 69234 69218 116.902 80 0.0190866 0.008549 2024-09-13T14:48:15.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:14 smithi019 ceph-mon[91296]: pgmap v791: 109 pgs: 1 active+recovering+undersized+remapped, 2 peering, 106 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 4152/194976 objects misplaced (2.129%); 277 KiB/s, 4 objects/s recovering 2024-09-13T14:48:15.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:15.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:15.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:15.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:15.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:15.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:15.501 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:48:15.501 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:15.539 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:48:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:15.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:15.575 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:48:15.575 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:15.618 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:48:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:15.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:15.661 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:48:15.661 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:15.689 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:48:15.690 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:15.727 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:48:15.727 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:15.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:15.759 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:48:15.759 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:15.761 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:15.863 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:15.899 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:15.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:16.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:16.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:16.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 15 70443 70428 115.816 75.625 0.0153128 0.00862872 2024-09-13T14:48:16.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1580364397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:16.100 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1580364397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:16.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:16.174 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1580364397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:16.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:16.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:16.421 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:16.457 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:16.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:16.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:16.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:16.697 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:16.736 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:16.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:16.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:16.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:17.073 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:17.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 71594 71578 114.689 71.875 0.0146521 0.00871067 2024-09-13T14:48:17.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:17.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:16 smithi019 ceph-mon[91296]: pgmap v792: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 746 op/s; 4152/203616 objects misplaced (2.039%); 277 KiB/s, 4 objects/s recovering 2024-09-13T14:48:17.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:16 smithi003 ceph-mon[107278]: pgmap v792: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 746 op/s; 4152/203616 objects misplaced (2.039%); 277 KiB/s, 4 objects/s recovering 2024-09-13T14:48:17.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:16 smithi003 ceph-mon[104007]: pgmap v792: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 746 op/s; 4152/203616 objects misplaced (2.039%); 277 KiB/s, 4 objects/s recovering 2024-09-13T14:48:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:17.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:17.347 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:17.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:17.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:17.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:17.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:17.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:17.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:17.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:17.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:18.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:18.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:18.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:48:18.081217+0000 min lat: 0.00116002 max lat: 2.37728 avg lat: 0.00878795 2024-09-13T14:48:18.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:48:18.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 72797 72781 113.702 75.1875 0.00505757 0.00878795 2024-09-13T14:48:18.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:18.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:18.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:18.331 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:18.433 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:18.467 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:18.568 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:18.603 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:18.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:18.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:18.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:18.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:18.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:19.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 74019 74003 112.791 76.375 0.0155961 0.00886024 2024-09-13T14:48:19.121 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:18 smithi003 ceph-mon[107278]: pgmap v793: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 36 MiB/s wr, 569 op/s; 4152/203616 objects misplaced (2.039%); 212 KiB/s, 3 objects/s recovering 2024-09-13T14:48:19.121 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:18 smithi003 ceph-mon[104007]: pgmap v793: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 36 MiB/s wr, 569 op/s; 4152/203616 objects misplaced (2.039%); 212 KiB/s, 3 objects/s recovering 2024-09-13T14:48:19.134 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:19.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:19.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:18 smithi019 ceph-mon[91296]: pgmap v793: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 36 MiB/s wr, 569 op/s; 4152/203616 objects misplaced (2.039%); 212 KiB/s, 3 objects/s recovering 2024-09-13T14:48:19.185 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:48:19.185 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:48:19.185 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:48:19.185 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:48:19.185 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:48:19.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:19.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:19.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:19.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:19.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:19.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:19.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:19.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:19.866 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:19.968 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:20.004 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:20.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 75231 75215 111.909 75.75 0.00431205 0.0089297 2024-09-13T14:48:20.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:20.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:20.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:20.286 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:20.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:20.426 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:20.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:20.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:48:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:20.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:20.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:20.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:48:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:48:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:20.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:20.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:20.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:48:20.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:21.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:21.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 76280 76264 110.831 65.5625 0.00868147 0.00901701 2024-09-13T14:48:21.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:20 smithi003 ceph-mon[104007]: pgmap v794: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s; 4066/219507 objects misplaced (1.852%); 788 KiB/s, 12 objects/s recovering 2024-09-13T14:48:21.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:20 smithi003 ceph-mon[107278]: pgmap v794: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s; 4066/219507 objects misplaced (1.852%); 788 KiB/s, 12 objects/s recovering 2024-09-13T14:48:21.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:21.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:21.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:20 smithi019 ceph-mon[91296]: pgmap v794: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s; 4066/219507 objects misplaced (1.852%); 788 KiB/s, 12 objects/s recovering 2024-09-13T14:48:21.273 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:21.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:21.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:21.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:21.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:21.498 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:21.511 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:21.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:21.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:21.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:21.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:21.842 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:48:21.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:21.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1994875400' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:48:21.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1994875400' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:48:21.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:22.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:22.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:22.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 77305 77289 109.768 64.0625 0.0537429 0.00910441 2024-09-13T14:48:22.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:22.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1994875400' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:48:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:22.314 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.352 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:22.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:22.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:22.485 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:22.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:22.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:22.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:22.767 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:22.843 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:48:22.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:23.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:23.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:23.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 78418 78402 108.875 69.5625 0.00874408 0.00917918 2024-09-13T14:48:23.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.173 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:23.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:23.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:22 smithi019 ceph-mon[91296]: pgmap v795: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 956 op/s; 4066/221880 objects misplaced (1.833%); 710 KiB/s, 11 objects/s recovering 2024-09-13T14:48:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1976858266' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1994875400' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:48:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:22 smithi019 ceph-mon[91296]: osdmap e244: 8 total, 8 up, 8 in 2024-09-13T14:48:23.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:22 smithi003 ceph-mon[107278]: pgmap v795: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 956 op/s; 4066/221880 objects misplaced (1.833%); 710 KiB/s, 11 objects/s recovering 2024-09-13T14:48:23.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1976858266' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:23.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1994875400' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:48:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:22 smithi003 ceph-mon[107278]: osdmap e244: 8 total, 8 up, 8 in 2024-09-13T14:48:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:22 smithi003 ceph-mon[104007]: pgmap v795: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 956 op/s; 4066/221880 objects misplaced (1.833%); 710 KiB/s, 11 objects/s recovering 2024-09-13T14:48:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1976858266' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1994875400' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:48:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:22 smithi003 ceph-mon[104007]: osdmap e244: 8 total, 8 up, 8 in 2024-09-13T14:48:23.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:23.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:23.409 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:23.443 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:23.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:23.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:23.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:23.710 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:23.811 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:23.849 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:23.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:23.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:24.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 79515 79499 107.998 68.5625 0.0126326 0.00925305 2024-09-13T14:48:24.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:24.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:23 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:23 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1976858266' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:23 smithi019 ceph-mon[91296]: osdmap e245: 8 total, 8 up, 8 in 2024-09-13T14:48:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:23 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:24.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:23 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:48:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1976858266' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[104007]: osdmap e245: 8 total, 8 up, 8 in 2024-09-13T14:48:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:48:24.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:48:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:48:24] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:48:24.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:24.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1976858266' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:24.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[107278]: osdmap e245: 8 total, 8 up, 8 in 2024-09-13T14:48:24.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:24.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:23 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:48:24.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:24.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:24.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:24.408 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:24.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:24.531 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:24.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:24.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:24.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:24.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:24.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:24.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:24.924 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:25.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:25.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:25.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 80261 80245 106.692 46.625 0.00469846 0.00936552 2024-09-13T14:48:25.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:25.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:25.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:25.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:25 smithi019 ceph-mon[91296]: pgmap v798: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 56 MiB/s wr, 888 op/s; 3972/224955 objects misplaced (1.766%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:48:25.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:25 smithi003 ceph-mon[107278]: pgmap v798: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 56 MiB/s wr, 888 op/s; 3972/224955 objects misplaced (1.766%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:48:25.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:25 smithi003 ceph-mon[104007]: pgmap v798: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 56 MiB/s wr, 888 op/s; 3972/224955 objects misplaced (1.766%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:48:25.437 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:25.583 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:25.629 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:25.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:25.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:25.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:26.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:26.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:26.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 15 81416 81401 105.975 72.25 0.00540668 0.0094294 2024-09-13T14:48:26.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:26.181 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:26.283 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:26.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:26.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:26.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:26.591 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:26.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:26.727 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:26.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:26.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:26.968 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:27.000 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:27.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 82353 82337 105.006 58.5 0.00668174 0.00951725 2024-09-13T14:48:27.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:27.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:27.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:27 smithi003 ceph-mon[107278]: pgmap v799: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s; 3972/238338 objects misplaced (1.667%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:48:27.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:27 smithi003 ceph-mon[104007]: pgmap v799: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s; 3972/238338 objects misplaced (1.667%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:48:27.242 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:27.272 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:27.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:27.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:27.408 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:27 smithi019 ceph-mon[91296]: pgmap v799: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s; 3972/238338 objects misplaced (1.667%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:48:27.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.549 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:27.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:27.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:27.683 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:27.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.817 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:27.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:27.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:28.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:28.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 83490 83474 104.327 71.0625 0.0133544 0.00957903 2024-09-13T14:48:28.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:28.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:28.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:28.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:28.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:48:28.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 6]} : dispatch 2024-09-13T14:48:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]} : dispatch 2024-09-13T14:48:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]} : dispatch 2024-09-13T14:48:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 6]} : dispatch 2024-09-13T14:48:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]} : dispatch 2024-09-13T14:48:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 5]} : dispatch 2024-09-13T14:48:28.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-13T14:48:28.398 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 7]} : dispatch 2024-09-13T14:48:28.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:28.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 6]} : dispatch 2024-09-13T14:48:28.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]} : dispatch 2024-09-13T14:48:28.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]} : dispatch 2024-09-13T14:48:28.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 6]} : dispatch 2024-09-13T14:48:28.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]} : dispatch 2024-09-13T14:48:28.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 5]} : dispatch 2024-09-13T14:48:28.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-13T14:48:28.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 7]} : dispatch 2024-09-13T14:48:28.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:28.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 6]} : dispatch 2024-09-13T14:48:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]} : dispatch 2024-09-13T14:48:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]} : dispatch 2024-09-13T14:48:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 6]} : dispatch 2024-09-13T14:48:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]} : dispatch 2024-09-13T14:48:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 5]} : dispatch 2024-09-13T14:48:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-13T14:48:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 7]} : dispatch 2024-09-13T14:48:28.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:28.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:28.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:28.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:28.704 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:28.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:28.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:28.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:28.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:48:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:28.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:29.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 84433 84417 103.436 58.9375 0.0105067 0.00965927 2024-09-13T14:48:29.103 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.140 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:29.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:29.184 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:48:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:29.220 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:48:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:29.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:29.250 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:48:29.251 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:29.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:29.284 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:48:29.285 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:29.317 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:48:29.317 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:29.350 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:48:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:29.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: pgmap v800: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 3972/238338 objects misplaced (1.667%); 752 KiB/s, 11 objects/s recovering 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 6]}]': finished 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]}]': finished 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]}]': finished 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 6]}]': finished 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 5]}]': finished 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 7]}]': finished 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-13T14:48:29.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[104007]: osdmap e246: 8 total, 8 up, 8 in 2024-09-13T14:48:29.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: pgmap v800: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 3972/238338 objects misplaced (1.667%); 752 KiB/s, 11 objects/s recovering 2024-09-13T14:48:29.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-13T14:48:29.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 6]}]': finished 2024-09-13T14:48:29.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]}]': finished 2024-09-13T14:48:29.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]}]': finished 2024-09-13T14:48:29.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 6]}]': finished 2024-09-13T14:48:29.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-13T14:48:29.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 5]}]': finished 2024-09-13T14:48:29.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-13T14:48:29.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 7]}]': finished 2024-09-13T14:48:29.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-13T14:48:29.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:29 smithi003 ceph-mon[107278]: osdmap e246: 8 total, 8 up, 8 in 2024-09-13T14:48:29.381 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:48:29.381 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:29.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:29.416 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:48:29.416 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:29.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:29.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: pgmap v800: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 3972/238338 objects misplaced (1.667%); 752 KiB/s, 11 objects/s recovering 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 6]}]': finished 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]}]': finished 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]}]': finished 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 6]}]': finished 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 5]}]': finished 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 7]}]': finished 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-13T14:48:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:29 smithi019 ceph-mon[91296]: osdmap e246: 8 total, 8 up, 8 in 2024-09-13T14:48:29.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:29.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:29.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:29.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.702 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:29.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:29.803 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:29.840 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:29.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:29.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:30.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:30.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 85267 85251 102.448 52.125 0.0163524 0.00975506 2024-09-13T14:48:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:30.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:30.214 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:30 smithi019 ceph-mon[91296]: osdmap e247: 8 total, 8 up, 8 in 2024-09-13T14:48:30.214 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:30 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/413931584' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:30.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:30.251 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:30.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:30.383 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:30.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:30.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:30 smithi003 ceph-mon[107278]: osdmap e247: 8 total, 8 up, 8 in 2024-09-13T14:48:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:30 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/413931584' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:30 smithi003 ceph-mon[104007]: osdmap e247: 8 total, 8 up, 8 in 2024-09-13T14:48:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:30 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/413931584' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:30.485 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:30.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:30.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:48:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:48:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:30.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:30.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:31.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:31.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:31.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 86116 86100 101.516 53.0625 0.0066826 0.00984521 2024-09-13T14:48:31.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:31.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:31.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:31.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:31.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:31 smithi019 ceph-mon[91296]: pgmap v803: 109 pgs: 1 active+recovering+undersized+remapped, 6 peering, 102 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s; 3831/251334 objects misplaced (1.524%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:48:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:31 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:48:31.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:31 smithi003 ceph-mon[107278]: pgmap v803: 109 pgs: 1 active+recovering+undersized+remapped, 6 peering, 102 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s; 3831/251334 objects misplaced (1.524%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:48:31.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:31 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:48:31.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:31 smithi003 ceph-mon[104007]: pgmap v803: 109 pgs: 1 active+recovering+undersized+remapped, 6 peering, 102 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s; 3831/251334 objects misplaced (1.524%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:48:31.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:31 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:48:31.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:31.484 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:31.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:31.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.622 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:31.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:31.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:31.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:31.854 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:31.854 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:31.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:31.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:32.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:32.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:32.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 87225 87209 100.92 69.3125 0.0156823 0.00990335 2024-09-13T14:48:32.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:32.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:32.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:32.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:32.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:32.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:32.599 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:32.632 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:32.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:48:32.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:32.784 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:32.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:32.918 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:33.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:33.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:33.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:33.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 88261 88245 100.262 64.75 0.00781936 0.00996805 2024-09-13T14:48:33.101 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:33.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:33.197 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:33.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:33 smithi003 ceph-mon[104007]: pgmap v804: 109 pgs: 1 active+recovering+undersized+remapped, 6 peering, 102 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 73 MiB/s wr, 1.18k op/s; 3831/253179 objects misplaced (1.513%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:33.303 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:33 smithi003 ceph-mon[107278]: pgmap v804: 109 pgs: 1 active+recovering+undersized+remapped, 6 peering, 102 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 73 MiB/s wr, 1.18k op/s; 3831/253179 objects misplaced (1.513%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:33.303 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:33.303 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:33.382 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:33.389 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:33.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:33.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:33.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:33.482 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:33 smithi019 ceph-mon[91296]: pgmap v804: 109 pgs: 1 active+recovering+undersized+remapped, 6 peering, 102 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 73 MiB/s wr, 1.18k op/s; 3831/253179 objects misplaced (1.513%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:33.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:33.483 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:33.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:33.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:33.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:33.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:33.851 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:33.886 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:33.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:34.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:34.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 89341 89325 99.6768 67.5 0.00781892 0.010023 2024-09-13T14:48:34.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:34.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:34.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:34.301 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:34.403 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:34 smithi003 ceph-mon[107278]: pgmap v805: 109 pgs: 1 active+recovering+undersized+remapped, 4 peering, 104 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 46 MiB/s wr, 735 op/s; 3687/255996 objects misplaced (1.440%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:48:34.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:34 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T14:48:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:34 smithi003 ceph-mon[104007]: pgmap v805: 109 pgs: 1 active+recovering+undersized+remapped, 4 peering, 104 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 46 MiB/s wr, 735 op/s; 3687/255996 objects misplaced (1.440%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:48:34.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:34 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T14:48:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:48:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:48:34] "GET /metrics HTTP/1.1" 200 36556 "" "Prometheus/2.51.0" 2024-09-13T14:48:34.435 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:34.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.574 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:34.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:34.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:34 smithi019 ceph-mon[91296]: pgmap v805: 109 pgs: 1 active+recovering+undersized+remapped, 4 peering, 104 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 46 MiB/s wr, 735 op/s; 3687/255996 objects misplaced (1.440%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:48:34.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:34 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T14:48:34.675 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:34.722 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:34.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:34.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:34.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:35.022 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:35.084 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 90461 90445 99.1561 70 0.0113191 0.0100795 2024-09-13T14:48:35.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:35.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:35.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.296 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:35.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:35.304 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:35.304 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:35.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:48:35.398 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:35.428 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:35.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:35.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:35.686 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:35.719 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:35.821 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:35.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:35.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:35.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:36.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:48:36.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 91420 91404 98.4759 59.9375 0.0199673 0.0101482 2024-09-13T14:48:36.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:36.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:36.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:36.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:36.391 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:36.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:36.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:36.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:36.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:36.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:36.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:36.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:36.860 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:36.962 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:36.999 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:37.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 92280 92264 97.7178 53.75 0.022877 0.0102279 2024-09-13T14:48:37.100 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:37.135 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:37.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:37.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:36 smithi019 ceph-mon[91296]: pgmap v806: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 3687/267918 objects misplaced (1.376%); 2.2 MiB/s, 1 keys/s, 35 objects/s recovering 2024-09-13T14:48:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:36 smithi003 ceph-mon[104007]: pgmap v806: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 3687/267918 objects misplaced (1.376%); 2.2 MiB/s, 1 keys/s, 35 objects/s recovering 2024-09-13T14:48:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:36 smithi003 ceph-mon[107278]: pgmap v806: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 3687/267918 objects misplaced (1.376%); 2.2 MiB/s, 1 keys/s, 35 objects/s recovering 2024-09-13T14:48:37.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:37.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:37.377 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:37.408 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:37.506 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:37.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:37.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:37.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:37.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:37.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:37.840 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:48:37.847 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:48:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:37.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:37.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3067178259' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:37.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:37 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:37.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2750224519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:48:37.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:37 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:48:37.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3067178259' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:37.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:37 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:37.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2750224519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:48:37.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:37 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:48:37.987 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:37.992 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3067178259' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:37.992 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:37 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:37.992 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2750224519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:48:37.992 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:37 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:48:38.009 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:38.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:48:38.086478+0000 min lat: 0.00116002 max lat: 2.37728 avg lat: 0.0102877 2024-09-13T14:48:38.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:48:38.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 92939 92923 96.7757 41.1875 0.013224 0.0102877 2024-09-13T14:48:38.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:38.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:38.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:38.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:38.291 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:38.292 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:38.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:38.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:38.507 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:38.609 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:38.632 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:38.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:38.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:38.885 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:38.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:38.987 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:39.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:39.081 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:38 smithi019 ceph-mon[91296]: pgmap v807: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 3687/267918 objects misplaced (1.376%); 1.8 MiB/s, 1 keys/s, 29 objects/s recovering 2024-09-13T14:48:39.081 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:38 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:39.081 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:38 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:39.081 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:38 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:48:39.081 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:38 smithi019 ceph-mon[91296]: osdmap e248: 8 total, 8 up, 8 in 2024-09-13T14:48:39.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 92939 92923 95.1894 0 - 0.0102877 2024-09-13T14:48:39.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:48:39.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:39.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:39.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[107278]: pgmap v807: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 3687/267918 objects misplaced (1.376%); 1.8 MiB/s, 1 keys/s, 29 objects/s recovering 2024-09-13T14:48:39.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:39.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:39.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:48:39.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[107278]: osdmap e248: 8 total, 8 up, 8 in 2024-09-13T14:48:39.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[104007]: pgmap v807: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 3687/267918 objects misplaced (1.376%); 1.8 MiB/s, 1 keys/s, 29 objects/s recovering 2024-09-13T14:48:39.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:39.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:39.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:48:39.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:38 smithi003 ceph-mon[104007]: osdmap e248: 8 total, 8 up, 8 in 2024-09-13T14:48:39.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:39.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:39.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:39.361 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:39.383 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:39.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:39.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:39.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:39.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:39.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:39.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:39.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:39 smithi003 ceph-mon[104007]: osdmap e249: 8 total, 8 up, 8 in 2024-09-13T14:48:39.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:39 smithi003 ceph-mon[104007]: osdmap e250: 8 total, 8 up, 8 in 2024-09-13T14:48:39.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:39 smithi003 ceph-mon[107278]: osdmap e249: 8 total, 8 up, 8 in 2024-09-13T14:48:39.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:39 smithi003 ceph-mon[107278]: osdmap e250: 8 total, 8 up, 8 in 2024-09-13T14:48:39.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:40.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:40.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 92963 92947 93.6784 0.75 0.00755365 0.0105512 2024-09-13T14:48:40.112 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.154 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:40.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:40.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:39 smithi019 ceph-mon[91296]: osdmap e249: 8 total, 8 up, 8 in 2024-09-13T14:48:40.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:39 smithi019 ceph-mon[91296]: osdmap e250: 8 total, 8 up, 8 in 2024-09-13T14:48:40.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:40.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:48:40.297 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:40.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:40.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:40.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:40.576 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:40.577 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:48:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:48:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:40.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:40.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:40.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:40.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:40.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:40.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:41.012 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:41.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 93852 93836 93.0734 55.5625 0.00686849 0.010739 2024-09-13T14:48:41.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:41.135 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:40 smithi019 ceph-mon[91296]: pgmap v810: 109 pgs: 1 active+recovering+undersized+remapped, 3 peering, 105 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 3513/277239 objects misplaced (1.267%); 2.5 MiB/s, 1 keys/s, 40 objects/s recovering 2024-09-13T14:48:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.157 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:41.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:41.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:48:41.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:40 smithi003 ceph-mon[104007]: pgmap v810: 109 pgs: 1 active+recovering+undersized+remapped, 3 peering, 105 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 3513/277239 objects misplaced (1.267%); 2.5 MiB/s, 1 keys/s, 40 objects/s recovering 2024-09-13T14:48:41.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:40 smithi003 ceph-mon[107278]: pgmap v810: 109 pgs: 1 active+recovering+undersized+remapped, 3 peering, 105 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 3513/277239 objects misplaced (1.267%); 2.5 MiB/s, 1 keys/s, 40 objects/s recovering 2024-09-13T14:48:41.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:41.299 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:41.299 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:41.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:41.389 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:41.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:41.439 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:41.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:41.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:41.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:41.714 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:41.816 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:41.852 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:41.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:41.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:42.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 94719 94703 92.4656 54.1875 0.022355 0.010808 2024-09-13T14:48:42.098 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:42.112 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:48:42.139 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:42.241 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:42.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:42.386 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:42.439 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:42.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.589 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:42.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:42.599 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:42.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.737 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:42.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:42.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:42.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:42.982 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:43.004 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:42 smithi019 ceph-mon[91296]: pgmap v812: 109 pgs: 1 active+recovering+undersized+remapped, 3 peering, 105 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 35 MiB/s wr, 564 op/s; 3513/278088 objects misplaced (1.263%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:48:43.004 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:43.031 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:43.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 95563 95547 91.8539 52.75 0.0068622 0.0108795 2024-09-13T14:48:43.088 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:43.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:43.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:42 smithi003 ceph-mon[104007]: pgmap v812: 109 pgs: 1 active+recovering+undersized+remapped, 3 peering, 105 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 35 MiB/s wr, 564 op/s; 3513/278088 objects misplaced (1.263%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:48:43.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:42 smithi003 ceph-mon[107278]: pgmap v812: 109 pgs: 1 active+recovering+undersized+remapped, 3 peering, 105 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 35 MiB/s wr, 564 op/s; 3513/278088 objects misplaced (1.263%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:48:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:43.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:43.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:43.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:43.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:43.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:43.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.607 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:43.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:43.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:43.977 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:44.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:44.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 96384 96368 91.2397 51.3125 0.00563433 0.0109535 2024-09-13T14:48:44.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:44.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:44.151 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:43 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 5759/280572 objects degraded (2.053%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:48:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:44.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:43 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 5759/280572 objects degraded (2.053%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:48:44.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:43 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 5759/280572 objects degraded (2.053%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:48:44.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:48:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:48:44] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:48:44.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:44.276 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:44.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:44.412 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:44.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:44.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:44.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:44.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:44.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:44.811 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:44.838 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:44.838 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:44.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:44.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:45.082 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:44 smithi019 ceph-mon[91296]: pgmap v813: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 peering, 105 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 44 MiB/s wr, 702 op/s; 5759/280572 objects degraded (2.053%); 3242/280572 objects misplaced (1.155%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:48:45.088 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 97134 97118 90.5775 46.875 0.00552832 0.0110345 2024-09-13T14:48:45.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:45.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:45.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:44 smithi003 ceph-mon[107278]: pgmap v813: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 peering, 105 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 44 MiB/s wr, 702 op/s; 5759/280572 objects degraded (2.053%); 3242/280572 objects misplaced (1.155%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:48:45.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:44 smithi003 ceph-mon[104007]: pgmap v813: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 peering, 105 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 44 MiB/s wr, 702 op/s; 5759/280572 objects degraded (2.053%); 3242/280572 objects misplaced (1.155%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:48:45.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:45.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:45.378 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:45.414 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:45.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:45.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.566 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:45.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:45.570 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:48:45.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:45.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:45.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:45.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:45.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:45.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:45.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:46.059 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:46.069 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:46.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 98068 98052 90.1039 58.375 0.0242233 0.0110927 2024-09-13T14:48:46.128 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:46.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:46.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:46.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:46.507 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:46.546 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:46.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:46.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:46.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:46.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:46.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.960 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:46.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:47.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:47.088 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 99028 99012 89.6676 60 0.0404178 0.0111468 2024-09-13T14:48:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:47.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:47.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:46 smithi019 ceph-mon[91296]: pgmap v814: 109 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 106 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 888 op/s; 11508/289041 objects degraded (3.981%); 3242/289041 objects misplaced (1.122%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:48:47.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:46 smithi003 ceph-mon[107278]: pgmap v814: 109 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 106 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 888 op/s; 11508/289041 objects degraded (3.981%); 3242/289041 objects misplaced (1.122%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:48:47.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:46 smithi003 ceph-mon[104007]: pgmap v814: 109 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 106 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 888 op/s; 11508/289041 objects degraded (3.981%); 3242/289041 objects misplaced (1.122%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:48:47.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:47.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:47.347 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:47.385 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:47.487 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:47.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:47.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:47.651 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:47.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:47.799 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:47.901 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.933 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:47.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:47.992 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:48:47.992 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:48:47.993 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:47.993 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:48.009 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:48:48.009 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:48.034 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:48.045 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:48:48.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.070 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:48.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:48.082 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:48:48.082 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:48.088 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 15 99877 99862 89.1456 53.125 0.0221993 0.0112123 2024-09-13T14:48:48.094 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:48.094 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:48.118 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:48:48.118 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:48.140 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3185101389' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:48.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3185101389' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:48.156 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:48:48.156 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:48.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:48.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3185101389' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:48.192 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:48:48.193 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:48.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:48.327 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:48.363 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:48.466 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:48.506 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:48.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:48.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:48.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:48.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:48.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:48.932 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:48.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:49.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:48:49.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:49.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:49.089 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 100796 100780 88.698 57.375 0.0255943 0.0112678 2024-09-13T14:48:49.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:48 smithi003 ceph-mon[107278]: pgmap v815: 109 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 106 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 35 MiB/s wr, 566 op/s; 11508/289041 objects degraded (3.981%); 3242/289041 objects misplaced (1.122%); 1.9 MiB/s, 31 objects/s recovering 2024-09-13T14:48:49.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:48 smithi003 ceph-mon[104007]: pgmap v815: 109 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 106 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 35 MiB/s wr, 566 op/s; 11508/289041 objects degraded (3.981%); 3242/289041 objects misplaced (1.122%); 1.9 MiB/s, 31 objects/s recovering 2024-09-13T14:48:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:48 smithi019 ceph-mon[91296]: pgmap v815: 109 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 106 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 35 MiB/s wr, 566 op/s; 11508/289041 objects degraded (3.981%); 3242/289041 objects misplaced (1.122%); 1.9 MiB/s, 31 objects/s recovering 2024-09-13T14:48:49.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:49.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:49.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:49.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:49.465 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:49.520 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:49.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:49.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:49.760 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.808 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:49.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:49.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:50.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:50.088 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 101669 101653 88.2238 54.5625 0.035275 0.0113279 2024-09-13T14:48:50.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:50.192 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:50.215 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:49 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11508/300711 objects degraded (3.827%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:50.229 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:50.331 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:50.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:50.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:49 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11508/300711 objects degraded (3.827%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:50.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:49 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11508/300711 objects degraded (3.827%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:50.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:50.510 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:50.612 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:50.643 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:48:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:48:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:50.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:50.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:50.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:50.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:50.916 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:51.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:51.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:51.088 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 102568 102552 87.7849 56.1875 0.0188711 0.0113866 2024-09-13T14:48:51.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:51 smithi003 ceph-mon[107278]: pgmap v816: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 49 MiB/s wr, 782 op/s; 11508/300711 objects degraded (3.827%); 3073/300711 objects misplaced (1.022%); 2.7 MiB/s, 44 objects/s recovering 2024-09-13T14:48:51.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:51 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:51.138 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:51 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:51.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:51 smithi003 ceph-mon[104007]: pgmap v816: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 49 MiB/s wr, 782 op/s; 11508/300711 objects degraded (3.827%); 3073/300711 objects misplaced (1.022%); 2.7 MiB/s, 44 objects/s recovering 2024-09-13T14:48:51.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:51 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:51.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:51 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:51.156 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:51.196 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:51.268 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:51 smithi019 ceph-mon[91296]: pgmap v816: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 49 MiB/s wr, 782 op/s; 11508/300711 objects degraded (3.827%); 3073/300711 objects misplaced (1.022%); 2.7 MiB/s, 44 objects/s recovering 2024-09-13T14:48:51.268 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:51 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:51.268 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:51 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:51.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:51.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:51.433 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:48:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T14:48:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T14:48:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:48:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191276 8.8 8:107ce2ea:::benchmark_data_smithi003_139172_object66483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.498528+0000", 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 580.92905141200004, 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0525131189999999, 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191276 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498528+0000", 2024-09-13T14:48:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498528+0000", 2024-09-13T14:48:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498527+0000", 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498532+0000", 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8600000000000001e-06 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498543+0000", 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1379999999999999e-05 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.498549+0000", 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6810000000000003e-06 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.508932+0000", 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010383593 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.509021+0000", 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8812000000000003e-05 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.509105+0000", 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3584999999999993e-05 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.522155+0000", 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.01304979 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.522384+0000", 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000229829 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536472+0000", 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014087288999999999 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536481+0000", 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8270000000000004e-06 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.550996+0000", 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0145151139999999 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551041+0000", 2024-09-13T14:48:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5269999999999999e-05 2024-09-13T14:48:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191280 8.8 8:1cc7168d:::benchmark_data_smithi003_139172_object66487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T14:48:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.499757+0000", 2024-09-13T14:48:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 580.92782183500003, 2024-09-13T14:48:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.051367782, 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191280 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499757+0000", 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499757+0000", 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499757+0000", 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499761+0000", 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5410000000000002e-06 2024-09-13T14:48:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499772+0000", 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0467e-05 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.499777+0000", 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5010000000000004e-06 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.515233+0000", 2024-09-13T14:48:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015456228000000001 2024-09-13T14:48:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.515683+0000", 2024-09-13T14:48:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00044969399999999999 2024-09-13T14:48:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:48:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.516086+0000", 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00040249099999999999 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.529283+0000", 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013197728000000001 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.529647+0000", 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00036357499999999999 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536551+0000", 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0069043840000000004 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536559+0000", 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551079+0000", 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0145203439999999 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551125+0000", 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5878999999999999e-05 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191287 8.8 8:12bf2cc4:::benchmark_data_smithi003_139172_object66494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.511850+0000", 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 580.91572876299995, 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0393236480000001, 2024-09-13T14:48:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191287 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511850+0000", 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511850+0000", 2024-09-13T14:48:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511849+0000", 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511854+0000", 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6979999999999997e-06 2024-09-13T14:48:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511869+0000", 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4596000000000001e-05 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.511875+0000", 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7830000000000004e-06 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.523525+0000", 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011650600000000001 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.523687+0000", 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016207799999999999 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.523832+0000", 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014468900000000001 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.529668+0000", 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0058362120000000003 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536588+0000", 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0069196329999999997 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.536594+0000", 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0750000000000003e-06 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551140+0000", 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0145458519999999 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551174+0000", 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4384999999999997e-05 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191307 8.8 8:187dddd2:::benchmark_data_smithi003_139172_object66514:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.539855+0000", 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 580.88772441399999, 2024-09-13T14:48:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0656769129999999, 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191307 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539855+0000", 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539855+0000", 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539854+0000", 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539858+0000", 2024-09-13T14:48:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4809999999999998e-06 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539893+0000", 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5342999999999997e-05 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539902+0000", 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2390000000000008e-06 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539985+0000", 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2859000000000001e-05 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.539994+0000", 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8310000000000007e-06 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551214+0000", 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0112197590000001 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551314+0000", 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000100556 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.551441+0000", 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012729399999999999 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.575082+0000", 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.023640372 2024-09-13T14:48:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605405+0000", 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.030322898000000001 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605441+0000", 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6359999999999997e-05 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605490+0000", 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8578999999999997e-05 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605532+0000", 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2021999999999998e-05 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.35454.0:191316 8.8 8:1339840c:::benchmark_data_smithi003_139172_object66523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:39:10.549840+0000", 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 580.87773915699995, 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.055850465, 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.35454", 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/225208000", 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 191316 2024-09-13T14:48:51.464 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.465 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:51.466 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.466 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.466 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.466 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549840+0000", 2024-09-13T14:48:51.466 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.466 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.466 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.466 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.466 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549840+0000", 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549839+0000", 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549844+0000", 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8980000000000004e-06 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.467 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549854+0000", 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.046e-05 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549860+0000", 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6289999999999998e-06 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549922+0000", 2024-09-13T14:48:51.468 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1762999999999998e-05 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:10.549931+0000", 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.4930000000000004e-06 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.556819+0000", 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.006887436 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.469 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.556938+0000", 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000119732 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.557056+0000", 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011714699999999999 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.575089+0000", 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.018033850000000001 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.470 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605570+0000", 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.030480854000000002 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605632+0000", 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2069999999999994e-05 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605663+0000", 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0624999999999999e-05 2024-09-13T14:48:51.471 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:39:11.605691+0000", 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7497000000000001e-05 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:120794 9.7 9:e7706e89:::benchmark_data_smithi003_156985_object120793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:18.841809+0000", 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 332.58577041199999, 2024-09-13T14:48:51.472 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4987142480000002, 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 120794 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841809+0000", 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.473 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841809+0000", 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841807+0000", 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.474 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841843+0000", 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.553e-05 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841859+0000", 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6364000000000001e-05 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.841868+0000", 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9709999999999993e-06 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.475 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842332+0000", 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00046374099999999999 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842456+0000", 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000124659 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842767+0000", 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00031060300000000001 2024-09-13T14:48:51.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.847411+0000", 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0046438859999999998 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.850363+0000", 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029518190000000001 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.872547+0000", 2024-09-13T14:48:51.477 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.022184261 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.872565+0000", 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7665000000000001e-05 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.854869+0000", 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98230381 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.478 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.854879+0000", 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0081000000000001e-05 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.329521+0000", 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.474642472 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for degraded object", 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.330222+0000", 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00070061600000000004 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.479 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.340464+0000", 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010241894 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.340523+0000", 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9441999999999998e-05 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.480 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:120798 9.17 9:e86e3aa7:::benchmark_data_smithi003_156985_object120797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:18.842605+0000", 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 332.58497429300002, 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5056814049999998, 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 120798 2024-09-13T14:48:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842605+0000", 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842605+0000", 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.482 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842604+0000", 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842633+0000", 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8603e-05 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842640+0000", 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4390000000000003e-06 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.842645+0000", 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2409999999999997e-06 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.844796+0000", 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00215161 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.844915+0000", 2024-09-13T14:48:51.484 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118263 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.845043+0000", 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012889600000000001 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.850380+0000", 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0053369899999999998 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.485 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.869291+0000", 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.018910756000000001 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.869300+0000", 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7430000000000008e-06 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.853609+0000", 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98430951 2024-09-13T14:48:51.486 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.853620+0000", 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0145e-05 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.334335+0000", 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4807156269999999 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for degraded object", 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.335040+0000", 2024-09-13T14:48:51.487 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00070517900000000001 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.348222+0000", 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013181877999999999 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.348286+0000", 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4036999999999997e-05 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:120805 9.17 9:ed42f0a7:::benchmark_data_smithi003_156985_object120804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:18.855612+0000", 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 332.57196709499999, 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4960159339999999, 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 120805 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.489 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855612+0000", 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855612+0000", 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.490 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855611+0000", 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855642+0000", 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0188999999999998e-05 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855650+0000", 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5199999999999997e-06 2024-09-13T14:48:51.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.855655+0000", 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3800000000000004e-06 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.856508+0000", 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00085295800000000001 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.856627+0000", 2024-09-13T14:48:51.492 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011941499999999999 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.856764+0000", 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013728899999999999 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.869327+0000", 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012562298 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.869332+0000", 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.339e-06 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.853647+0000", 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98431561999999995 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.853655+0000", 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1319999999999996e-06 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.494 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.335106+0000", 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4814514729999999 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for degraded object", 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.335718+0000", 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00061208699999999998 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.351572+0000", 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015854035999999998 2024-09-13T14:48:51.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.496 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.496 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.496 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.351628+0000", 2024-09-13T14:48:51.496 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5828999999999997e-05 2024-09-13T14:48:51.496 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.496 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.496 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.496 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.497 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.497 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214357 9.d 9:b77cc858:::benchmark_data_smithi003_156985_object93441:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:48:51.497 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.623932+0000", 2024-09-13T14:48:51.497 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 170.80364757000001, 2024-09-13T14:48:51.498 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0751467219999999, 2024-09-13T14:48:51.498 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.498 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.498 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.498 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.498 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.498 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214357 2024-09-13T14:48:51.498 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623932+0000", 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623932+0000", 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.499 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623930+0000", 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623937+0000", 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8650000000000003e-06 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.500 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623949+0000", 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2680999999999999e-05 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623955+0000", 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.032e-06 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.501 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.624003+0000", 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7698999999999999e-05 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.624010+0000", 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8909999999999998e-06 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.661403+0000", 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0373926769999999 2024-09-13T14:48:51.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.661489+0000", 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5983999999999995e-05 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.661566+0000", 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7749000000000004e-05 2024-09-13T14:48:51.503 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.671062+0000", 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0094950069999999998 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698569+0000", 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.027507851 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698627+0000", 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7685000000000001e-05 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698863+0000", 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00023623399999999999 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698931+0000", 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7773000000000003e-05 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699078+0000", 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000147312 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214375 9.d 9:be731116:::benchmark_data_smithi003_156985_object93459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.661092+0000", 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 170.766487165, 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.038199892, 2024-09-13T14:48:51.506 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214375 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661092+0000", 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.507 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661092+0000", 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661090+0000", 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661102+0000", 2024-09-13T14:48:51.508 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2103000000000001e-05 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661127+0000", 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5117e-05 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661138+0000", 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0953e-05 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.509 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661226+0000", 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7379000000000005e-05 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661239+0000", 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3434e-05 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.663711+0000", 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.002471949 2024-09-13T14:48:51.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.511 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.511 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.511 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.663792+0000", 2024-09-13T14:48:51.511 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0826000000000002e-05 2024-09-13T14:48:51.511 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.511 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.511 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:48:51.512 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.663877+0000", 2024-09-13T14:48:51.512 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5036000000000004e-05 2024-09-13T14:48:51.512 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.512 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.512 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675347+0000", 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011470173 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698670+0000", 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.023323282000000001 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698715+0000", 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5213000000000002e-05 2024-09-13T14:48:51.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.516 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.516 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.516 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699124+0000", 2024-09-13T14:48:51.516 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000408985 2024-09-13T14:48:51.516 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.516 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.517 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.517 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699157+0000", 2024-09-13T14:48:51.517 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2138000000000003e-05 2024-09-13T14:48:51.517 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.517 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699292+0000", 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013532799999999999 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214377 9.d 9:b52ded65:::benchmark_data_smithi003_156985_object93461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.664805+0000", 2024-09-13T14:48:51.519 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 170.762774417, 2024-09-13T14:48:51.519 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.034793217, 2024-09-13T14:48:51.519 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.519 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.519 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.520 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.520 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.520 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214377 2024-09-13T14:48:51.520 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.520 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.520 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.520 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.520 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664805+0000", 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664805+0000", 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664804+0000", 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664809+0000", 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.887e-06 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664819+0000", 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0817e-05 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664827+0000", 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0759999999999997e-06 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.522 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664891+0000", 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4422000000000001e-05 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664898+0000", 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8000000000000001e-06 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.666066+0000", 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.00116863 2024-09-13T14:48:51.523 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.666207+0000", 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014029099999999999 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.666345+0000", 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013788900000000001 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.524 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675357+0000", 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0090126630000000006 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698783+0000", 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.023425537999999999 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698825+0000", 2024-09-13T14:48:51.525 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1802999999999999e-05 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699411+0000", 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00058682999999999999 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699469+0000", 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7234999999999997e-05 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.526 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699598+0000", 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012936800000000001 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214394 9.d 9:b3e3fb24:::benchmark_data_smithi003_156985_object93478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.688931+0000", 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 170.73864845099999, 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.010993375, 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.527 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214394 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688931+0000", 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688931+0000", 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688929+0000", 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688936+0000", 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3320000000000003e-06 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688955+0000", 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.853e-05 2024-09-13T14:48:51.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688962+0000", 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4590000000000001e-06 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689081+0000", 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000119159 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.530 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689185+0000", 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010335 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.668846+0000", 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97966176900000002 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.669010+0000", 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016329499999999999 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.531 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.669126+0000", 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011607000000000001 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.679730+0000", 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010604165 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.532 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699355+0000", 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.019625423999999999 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699643+0000", 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000287638 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699741+0000", 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7619000000000002e-05 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.533 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699790+0000", 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9363000000000002e-05 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699924+0000", 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013407499999999999 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.534 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214396 9.d 9:bf725953:::benchmark_data_smithi003_156985_object93480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.689352+0000", 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 170.73822705500001, 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0107824670000001, 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214396 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.535 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689352+0000", 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689352+0000", 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.537 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689351+0000", 2024-09-13T14:48:51.537 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:51.537 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.537 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.537 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.537 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689357+0000", 2024-09-13T14:48:51.537 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3130000000000004e-06 2024-09-13T14:48:51.537 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689374+0000", 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6492999999999999e-05 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689380+0000", 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2149999999999998e-06 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.538 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689437+0000", 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7411999999999999e-05 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689445+0000", 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6920000000000002e-06 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.671112+0000", 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98166737599999998 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.671244+0000", 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013205399999999999 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.671336+0000", 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1335000000000006e-05 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.679738+0000", 2024-09-13T14:48:51.540 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0084018919999999993 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699678+0000", 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.019940684 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699710+0000", 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1491000000000001e-05 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.541 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699973+0000", 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00026296499999999999 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700009+0000", 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6374000000000001e-05 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700135+0000", 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000125384 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214400 9.d 9:b4303dac:::benchmark_data_smithi003_156985_object93484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.695445+0000", 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 170.73213450399999, 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0050245659999999, 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.544 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214400 2024-09-13T14:48:51.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.544 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.544 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.544 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.544 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695445+0000", 2024-09-13T14:48:51.544 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.544 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695445+0000", 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695444+0000", 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695450+0000", 2024-09-13T14:48:51.545 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9699999999999996e-06 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695461+0000", 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1472e-05 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695465+0000", 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.087e-06 2024-09-13T14:48:51.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.547 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.547 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.547 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695517+0000", 2024-09-13T14:48:51.547 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.189e-05 2024-09-13T14:48:51.547 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.547 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.547 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.547 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695526+0000", 2024-09-13T14:48:51.547 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.969e-06 2024-09-13T14:48:51.548 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.548 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.548 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.548 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.673704+0000", 2024-09-13T14:48:51.548 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97817781400000003 2024-09-13T14:48:51.549 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.549 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.549 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.549 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.673783+0000", 2024-09-13T14:48:51.549 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9720000000000002e-05 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.673857+0000", 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3239999999999997e-05 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.683953+0000", 2024-09-13T14:48:51.550 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010095807 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700189+0000", 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016236609999999999 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700235+0000", 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6057000000000003e-05 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.551 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700388+0000", 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015292100000000001 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700412+0000", 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4006e-05 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.552 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700469+0000", 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7154000000000002e-05 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214419 9.d 9:bc751d0e:::benchmark_data_smithi003_156985_object93503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.732552+0000", 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 170.695026872, 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96801581000000003, 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.553 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214419 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732552+0000", 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.554 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732552+0000", 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732551+0000", 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732558+0000", 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6279999999999999e-06 2024-09-13T14:48:51.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732570+0000", 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2157000000000001e-05 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732576+0000", 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9499999999999998e-06 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732623+0000", 2024-09-13T14:48:51.556 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7587999999999999e-05 2024-09-13T14:48:51.557 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.557 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.557 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.557 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732629+0000", 2024-09-13T14:48:51.557 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2160000000000003e-06 2024-09-13T14:48:51.557 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.557 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.557 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.557 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675405+0000", 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.94277538900000002 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675536+0000", 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013147199999999999 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675664+0000", 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012742799999999999 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.558 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.683960+0000", 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0082962709999999992 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700275+0000", 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016315273000000002 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700331+0000", 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5581999999999997e-05 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.559 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700490+0000", 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015882799999999999 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700512+0000", 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2340999999999998e-05 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700568+0000", 2024-09-13T14:48:51.560 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6063999999999997e-05 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214424 9.d 9:bfcad95c:::benchmark_data_smithi003_156985_object93508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.741966+0000", 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 170.685613289, 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.958713704, 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.561 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214424 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741966+0000", 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741966+0000", 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741964+0000", 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741973+0000", 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3589999999999997e-06 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741994+0000", 2024-09-13T14:48:51.563 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1138e-05 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.742002+0000", 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7829999999999999e-06 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.742058+0000", 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5889e-05 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.742066+0000", 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.807e-06 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.677279+0000", 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93521298600000002 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.677400+0000", 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000121215 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.677530+0000", 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012972400000000001 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.688181+0000", 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.01065105 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700297+0000", 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012115734 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700367+0000", 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0770000000000002e-05 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700604+0000", 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00023669699999999999 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700624+0000", 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0196000000000001e-05 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700680+0000", 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5381999999999999e-05 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.36168.0:27860 10.a e238/229 10:51e3daa3:::benchmark_data_smithi003_197468_object27859:head v 238'1740, mlcod=238'1738)", 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:47:48.002523+0000", 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 63.425056660000003, 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29010423400000002, 2024-09-13T14:48:51.568 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002523+0000", 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002523+0000", 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002521+0000", 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002546+0000", 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4780999999999999e-05 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002561+0000", 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5078e-05 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002568+0000", 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5640000000000002e-06 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002640+0000", 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1926000000000002e-05 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002677+0000", 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7039000000000002e-05 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292596+0000", 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28991923400000003 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292627+0000", 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0796000000000001e-05 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.572 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:27870 10.6 10:694fcd1d:::benchmark_data_smithi003_197468_object27869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:47:48.002696+0000", 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 63.424882713999999, 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29172562200000002, 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 27870 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.573 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002696+0000", 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002696+0000", 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002695+0000", 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002728+0000", 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3930000000000002e-05 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002743+0000", 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4092e-05 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002749+0000", 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2779999999999999e-06 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002799+0000", 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0587999999999997e-05 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002927+0000", 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012778099999999999 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:48:51.576 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.290644+0000", 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28771667299999998 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294288+0000", 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036443819999999998 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294297+0000", 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5660000000000007e-06 2024-09-13T14:48:51.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294345+0000", 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8038999999999999e-05 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294387+0000", 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2382999999999997e-05 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294422+0000", 2024-09-13T14:48:51.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4839999999999998e-05 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:27876 10.6 10:6f1b44f1:::benchmark_data_smithi003_197468_object27875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:47:48.004208+0000", 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 63.423371105999998, 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29044940699999999, 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:48:51.580 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 27876 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.004208+0000", 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.004208+0000", 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.004207+0000", 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.004253+0000", 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6487999999999999e-05 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.290612+0000", 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28635865599999999 2024-09-13T14:48:51.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.290626+0000", 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4279e-05 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292002+0000", 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001375874 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292154+0000", 2024-09-13T14:48:51.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015235499999999999 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292252+0000", 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7626000000000003e-05 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294492+0000", 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022405459999999999 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294514+0000", 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1203e-05 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294558+0000", 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4535e-05 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294621+0000", 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2793999999999995e-05 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.585 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294658+0000", 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6492999999999998e-05 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79806 10.6 10:67b6a0c3:::benchmark_data_smithi003_197468_object79805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.380081+0000", 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 27.047498100999999, 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29455382499999999, 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:51.586 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79806 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380081+0000", 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380081+0000", 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:48:51.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380079+0000", 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380113+0000", 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3945000000000001e-05 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380125+0000", 2024-09-13T14:48:51.588 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1814000000000001e-05 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380130+0000", 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4389999999999999e-06 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380349+0000", 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00021863699999999999 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:48:51.589 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380465+0000", 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000115486 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380589+0000", 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012487199999999999 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385248+0000", 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0046584390000000003 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385913+0000", 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00066478399999999997 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.674526+0000", 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28861310000000001 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.674586+0000", 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0374999999999997e-05 2024-09-13T14:48:51.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.674635+0000", 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8739999999999998e-05 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:48:51.592 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:48:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:51.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:51.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:48:51.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:51.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:51.859 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:51.896 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:51.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:52.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:52.090 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 103298 103282 87.2152 45.625 0.00556858 0.0114597 2024-09-13T14:48:52.149 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:52.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:52.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:52.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:52.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:48:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:48:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:52 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/538820329' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:48:52.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:48:52.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:48:52.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:52.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:52 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/538820329' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:48:52.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:48:52.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:48:52.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:52.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:52 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/538820329' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:48:52.464 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:52.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:52.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:52.663 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:48:52.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:52.747 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:48:52.753 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:48:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:52.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:52.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:52.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:53.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:53.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:53.088 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 104140 104124 86.754 52.625 0.0454336 0.0115213 2024-09-13T14:48:53.144 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:53.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:53.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:53 smithi019 ceph-mon[91296]: pgmap v817: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 44 MiB/s wr, 702 op/s; 11508/302391 objects degraded (3.806%); 3073/302391 objects misplaced (1.016%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:48:53.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/942204961' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:53.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:53 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:53.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:53 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:53.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/538820329' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:48:53.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:53 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:53.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:53 smithi019 ceph-mon[91296]: osdmap e251: 8 total, 8 up, 8 in 2024-09-13T14:48:53.233 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:53.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.374 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:53.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[107278]: pgmap v817: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 44 MiB/s wr, 702 op/s; 11508/302391 objects degraded (3.806%); 3073/302391 objects misplaced (1.016%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:48:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/942204961' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/538820329' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:48:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[107278]: osdmap e251: 8 total, 8 up, 8 in 2024-09-13T14:48:53.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[104007]: pgmap v817: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 44 MiB/s wr, 702 op/s; 11508/302391 objects degraded (3.806%); 3073/302391 objects misplaced (1.016%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:48:53.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/942204961' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:53.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:53.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:53.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/538820329' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:48:53.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:53.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:53 smithi003 ceph-mon[104007]: osdmap e251: 8 total, 8 up, 8 in 2024-09-13T14:48:53.476 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:53.518 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:53.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:53.654 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:53.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:53.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:53.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:48:53.897 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:53.929 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:54.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:54.089 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 105002 104986 86.321 53.875 0.00896726 0.0115783 2024-09-13T14:48:54.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:54.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:54.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:54.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:54.387 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:48:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:48:54] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:48:54.489 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:54.524 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:54.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:54.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:54.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:54.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:54.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:55.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:55.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 105817 105801 85.859 50.9375 0.0134432 0.0116413 2024-09-13T14:48:55.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:55.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:55 smithi003 ceph-mon[107278]: pgmap v819: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 45 MiB/s wr, 727 op/s; 11508/302391 objects degraded (3.806%); 3073/302391 objects misplaced (1.016%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:55.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:55 smithi003 ceph-mon[104007]: pgmap v819: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 45 MiB/s wr, 727 op/s; 11508/302391 objects degraded (3.806%); 3073/302391 objects misplaced (1.016%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:55.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:55.242 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:55 smithi019 ceph-mon[91296]: pgmap v819: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 45 MiB/s wr, 727 op/s; 11508/302391 objects degraded (3.806%); 3073/302391 objects misplaced (1.016%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:55.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:55.350 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:55.390 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:55.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:55.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:55.538 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:55.574 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:55.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:55.674 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:48:55.674 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.684 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:55.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:55.705 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:48:55.705 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:55.749 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:48:55.749 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:55.778 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:48:55.778 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:55.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:55.815 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:48:55.815 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:55.824 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:55.852 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:48:55.852 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:55.886 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:48:55.886 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:55.924 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:48:55.924 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:55.926 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:55.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:56.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:56.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 106677 106661 85.4473 53.75 0.0100116 0.0116968 2024-09-13T14:48:56.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:56.170 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/789010630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:56.170 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11508/302391 objects degraded (3.806%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:56.206 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:56.246 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:56.338 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:56.338 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:56.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:56.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/789010630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11508/302391 objects degraded (3.806%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/789010630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11508/302391 objects degraded (3.806%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:56.485 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:56.520 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:56.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:56.663 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:56.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:56.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:56.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:56.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:48:57.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:57.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:48:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:57.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 107580 107564 85.0801 56.4375 0.0109274 0.011748 2024-09-13T14:48:57.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:57.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:57.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:57.344 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:57 smithi003 ceph-mon[107278]: pgmap v820: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 867 op/s; 11508/315069 objects degraded (3.653%); 2893/315069 objects misplaced (0.918%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:48:57.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:57 smithi003 ceph-mon[104007]: pgmap v820: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 867 op/s; 11508/315069 objects degraded (3.653%); 2893/315069 objects misplaced (0.918%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:48:57.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:57.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:57 smithi019 ceph-mon[91296]: pgmap v820: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 867 op/s; 11508/315069 objects degraded (3.653%); 2893/315069 objects misplaced (0.918%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:48:57.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:57.515 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:57.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:57.525 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:57.617 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:57.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:57.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:57.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:57.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:58.069 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:58.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:48:58.091486+0000 min lat: 0.00116002 max lat: 2.37728 avg lat: 0.011803 2024-09-13T14:48:58.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:48:58.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 108437 108421 84.6861 53.5625 0.0124197 0.011803 2024-09-13T14:48:58.112 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:58.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:58.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:58.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:58.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:58.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:58.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:58.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:58.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:58.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.838 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:58.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:58.899 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:58 smithi003 ceph-mon[104007]: pgmap v821: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 867 op/s; 11508/315069 objects degraded (3.653%); 2893/315069 objects misplaced (0.918%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:48:58.899 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:58.899 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:48:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:58.899 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:58 smithi003 ceph-mon[107278]: pgmap v821: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 867 op/s; 11508/315069 objects degraded (3.653%); 2893/315069 objects misplaced (0.918%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:48:58.899 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:58.899 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:48:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:58.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:58.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:59.083 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:59.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 109296 109280 84.3034 53.6875 0.0150114 0.0118564 2024-09-13T14:48:59.126 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:58 smithi019 ceph-mon[91296]: pgmap v821: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 867 op/s; 11508/315069 objects degraded (3.653%); 2893/315069 objects misplaced (0.918%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:48:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:48:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:48:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:59.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:59.267 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:59.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:59.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:59.511 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:59.548 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:59.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:59.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:59.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:59.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:59.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:48:59.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:48:59.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:59.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:59.979 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:48:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:00.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:00.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 110191 110175 83.9569 55.9375 0.0344084 0.0119043 2024-09-13T14:49:00.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:00.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:00.264 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:00.366 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.402 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:00.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:00.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:00.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:00.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:00.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:49:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:49:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:00.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:00.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:00.827 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:00 smithi003 ceph-mon[107278]: pgmap v822: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 850 op/s; 11508/326250 objects degraded (3.527%); 2697/326250 objects misplaced (0.827%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:49:00.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:00 smithi003 ceph-mon[104007]: pgmap v822: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 850 op/s; 11508/326250 objects degraded (3.527%); 2697/326250 objects misplaced (0.827%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:49:00.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11508/326250 objects degraded (3.527%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:00.829 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:00.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:00.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:01.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:01.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 110970 110954 83.5319 48.6875 0.0146708 0.0119581 2024-09-13T14:49:01.094 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:01.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:00 smithi019 ceph-mon[91296]: pgmap v822: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 850 op/s; 11508/326250 objects degraded (3.527%); 2697/326250 objects misplaced (0.827%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:49:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11508/326250 objects degraded (3.527%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11508/326250 objects degraded (3.527%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:01.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:01.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:01.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:01.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:01.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:01.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:01.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:01.667 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:01.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:01.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:01.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:01.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:02.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:02.067 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:02.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 111736 111720 83.1074 47.875 0.00632731 0.0120251 2024-09-13T14:49:02.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:02.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:02.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:02.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:02.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:02.484 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:02.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:02.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:02.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:02.760 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:02.862 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:02.898 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:02.969 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:02.969 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:02.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:03.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:03.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 112602 112586 82.7664 54.125 0.0159605 0.012075 2024-09-13T14:49:03.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:03.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:03.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:02 smithi019 ceph-mon[91296]: pgmap v823: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s; 11508/327942 objects degraded (3.509%); 2697/327942 objects misplaced (0.822%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:49:03.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:02 smithi003 ceph-mon[107278]: pgmap v823: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s; 11508/327942 objects degraded (3.509%); 2697/327942 objects misplaced (0.822%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:49:03.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:02 smithi003 ceph-mon[104007]: pgmap v823: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s; 11508/327942 objects degraded (3.509%); 2697/327942 objects misplaced (0.822%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:49:03.277 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:03.314 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:03.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:03.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:03.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:03.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:03.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:03.743 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:03.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:03.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:03.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:04.016 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:04.093 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 113446 113430 82.4172 52.75 0.0103219 0.0121267 2024-09-13T14:49:04.100 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:49:04.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:04.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:04.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:04.394 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:04.424 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:49:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:49:04] "GET /metrics HTTP/1.1" 200 36554 "" "Prometheus/2.51.0" 2024-09-13T14:49:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:04.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:04.527 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:49:04.543 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:04.601 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:04.694 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:04.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:04.738 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:04.815 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T14:49:04.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:04 smithi019 ceph-mon[91296]: pgmap v824: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 48 MiB/s wr, 774 op/s; 11508/327942 objects degraded (3.509%); 2697/327942 objects misplaced (0.822%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:49:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:04 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1727523991' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:04 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:04.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:05.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:05.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:05.093 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 114265 114249 82.0582 51.1875 0.00533075 0.0121799 2024-09-13T14:49:05.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:05.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:05.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:04 smithi003 ceph-mon[107278]: pgmap v824: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 48 MiB/s wr, 774 op/s; 11508/327942 objects degraded (3.509%); 2697/327942 objects misplaced (0.822%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:49:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:04 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1727523991' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:04 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:04 smithi003 ceph-mon[104007]: pgmap v824: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 48 MiB/s wr, 774 op/s; 11508/327942 objects degraded (3.509%); 2697/327942 objects misplaced (0.822%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:49:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:04 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1727523991' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:04 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:05.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:05.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:05.395 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.434 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:05.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:05.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:05.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:05.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:05.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:05.815 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:49:05.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:05.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:49:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[107278]: osdmap e252: 8 total, 8 up, 8 in 2024-09-13T14:49:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11508/327942 objects degraded (3.509%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1297062142' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:05.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[107278]: osdmap e253: 8 total, 8 up, 8 in 2024-09-13T14:49:05.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:49:05.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[104007]: osdmap e252: 8 total, 8 up, 8 in 2024-09-13T14:49:05.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11508/327942 objects degraded (3.509%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:05.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1297062142' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:05.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:05.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:05.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:05.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:05 smithi003 ceph-mon[104007]: osdmap e253: 8 total, 8 up, 8 in 2024-09-13T14:49:05.956 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:06.002 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:06.093 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 115099 115083 81.7181 52.125 0.0229281 0.0122311 2024-09-13T14:49:06.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:06.139 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:06.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:06.175 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:05 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:49:06.176 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:05 smithi019 ceph-mon[91296]: osdmap e252: 8 total, 8 up, 8 in 2024-09-13T14:49:06.176 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11508/327942 objects degraded (3.509%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:06.176 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1297062142' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:06.176 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:05 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:06.176 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:05 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:06.176 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:05 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:06.176 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:05 smithi019 ceph-mon[91296]: osdmap e253: 8 total, 8 up, 8 in 2024-09-13T14:49:06.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:06.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:06.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:06.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:06.517 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.568 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:06.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:06.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:06.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:06.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:06.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:06.906 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:06.907 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:06.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:06.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:07.093 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 115976 115960 81.4158 54.8125 0.00618438 0.0122737 2024-09-13T14:49:07.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:07.100 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:06 smithi003 ceph-mon[107278]: pgmap v826: 109 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 845 op/s; 11508/340437 objects degraded (3.380%); 2504/340437 objects misplaced (0.736%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:49:07.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:06 smithi003 ceph-mon[104007]: pgmap v826: 109 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 845 op/s; 11508/340437 objects degraded (3.380%); 2504/340437 objects misplaced (0.736%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:49:07.148 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:49:07.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:49:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:07.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:07.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:06 smithi019 ceph-mon[91296]: pgmap v826: 109 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 845 op/s; 11508/340437 objects degraded (3.380%); 2504/340437 objects misplaced (0.736%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:49:07.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:07.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:07.422 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:07.455 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:07.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:07.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:07.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:07.601 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:49:07.697 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:07.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:07.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:07.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:08.031 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:49:08.093 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 90 2 116726 116724 81.0412 47.75 0.0254862 0.0123337 2024-09-13T14:49:08.131 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:08.153 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:08.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:08.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:08.382 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:08.404 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:08.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:08.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:08.572 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:08.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:08.661 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:08.763 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:08.785 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:08.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:08.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:09.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:09.034 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:09.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:08 smithi003 ceph-mon[107278]: pgmap v828: 109 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 37 MiB/s wr, 590 op/s; 11508/340437 objects degraded (3.380%); 2504/340437 objects misplaced (0.736%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:49:09.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:08 smithi003 ceph-mon[104007]: pgmap v828: 109 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 37 MiB/s wr, 590 op/s; 11508/340437 objects degraded (3.380%); 2504/340437 objects misplaced (0.736%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0316 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 116726 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 81.0312 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 44.9512 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 226.812 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1296 2024-09-13T14:49:09.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 719.23 2024-09-13T14:49:09.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3629 2024-09-13T14:49:09.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-09-13T14:49:09.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0123341 2024-09-13T14:49:09.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0395707 2024-09-13T14:49:09.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.37728 2024-09-13T14:49:09.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00116002 2024-09-13T14:49:09.101 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:49:09.136 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:09.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:09.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:08 smithi019 ceph-mon[91296]: pgmap v828: 109 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 37 MiB/s wr, 590 op/s; 11508/340437 objects degraded (3.380%); 2504/340437 objects misplaced (0.736%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:49:09.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:09.312 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:09.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:09.374 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:49:09.374 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:09.406 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:49:09.406 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:09.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:09.434 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:49:09.434 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:09.463 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:49:09.463 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:09.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:09.500 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:49:09.500 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:09.551 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:49:09.551 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:09.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:09.588 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:49:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:09.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:09.617 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:49:09.617 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:09.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:49:09.714 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:09.854 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:09.876 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:09.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/235790222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:09.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/235790222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:09.958 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:49:09.958 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:49:09.959 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T14:49:09.959 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:49:09.978 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.001 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:10.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:10.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:10.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:10.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/235790222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:10.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:10.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:10.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:10.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:10.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:10.502 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:10.542 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:10.644 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:10.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:49:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:49:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:10.685 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:10.787 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:10.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:10.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:10 smithi003 ceph-mon[104007]: pgmap v829: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.2 KiB/s rd, 56 MiB/s wr, 894 op/s; 8177/349197 objects degraded (2.342%); 5674/349197 objects misplaced (1.625%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:49:10.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:10 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8177/349197 objects degraded (2.342%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:10.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:10.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:11.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:11.100 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:10 smithi019 ceph-mon[91296]: pgmap v829: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.2 KiB/s rd, 56 MiB/s wr, 894 op/s; 8177/349197 objects degraded (2.342%); 5674/349197 objects misplaced (1.625%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:49:11.100 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:10 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8177/349197 objects degraded (2.342%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:11.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:11.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:49:11.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:10 smithi003 ceph-mon[107278]: pgmap v829: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.2 KiB/s rd, 56 MiB/s wr, 894 op/s; 8177/349197 objects degraded (2.342%); 5674/349197 objects misplaced (1.625%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:49:11.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:10 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8177/349197 objects degraded (2.342%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:11.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:11.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:11.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:11.360 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:11.462 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:11.488 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:11.517 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:49:11.590 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:11.630 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:11.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.772 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:11.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:11.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:11.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:12.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:12.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.079 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:12.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:12.181 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.221 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:12.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:12.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:12.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:12.456 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:12.489 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:12.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:12.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:12.744 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:12.775 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:12.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:12.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:13.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:13.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:13.085 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:12 smithi019 ceph-mon[91296]: pgmap v830: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.2 KiB/s rd, 58 MiB/s wr, 946 op/s; 8177/349917 objects degraded (2.337%); 5674/349917 objects misplaced (1.622%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:49:13.085 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:49:13.085 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:12 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:13.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:13.180 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:13.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:12 smithi003 ceph-mon[107278]: pgmap v830: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.2 KiB/s rd, 58 MiB/s wr, 946 op/s; 8177/349917 objects degraded (2.337%); 5674/349917 objects misplaced (1.622%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:49:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:49:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:12 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:13.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:12 smithi003 ceph-mon[104007]: pgmap v830: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.2 KiB/s rd, 58 MiB/s wr, 946 op/s; 8177/349917 objects degraded (2.337%); 5674/349917 objects misplaced (1.622%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:49:13.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:49:13.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:12 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:13.283 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.320 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:13.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:13.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:13.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:13.473 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:13.524 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:49:13.524 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:49:13.524 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:49:13.525 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:49:13.525 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:49:13.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:13.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:13.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:13.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:13.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:49:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:13.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:13.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:14.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:14.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:14.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:49:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:49:14] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:49:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:14.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:14.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:49:14.284 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:14.354 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:14.456 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.513 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:14.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:14.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:14.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:14.700 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:49:14.757 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:14.796 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:14.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:14.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:14 smithi003 ceph-mon[107278]: pgmap v831: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 3.8 KiB/s rd, 26 MiB/s wr, 437 op/s; 8177/349917 objects degraded (2.337%); 5674/349917 objects misplaced (1.622%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:49:14.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:14 smithi003 ceph-mon[107278]: from='client.36267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:14.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:14 smithi003 ceph-mon[104007]: pgmap v831: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 3.8 KiB/s rd, 26 MiB/s wr, 437 op/s; 8177/349917 objects degraded (2.337%); 5674/349917 objects misplaced (1.622%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:49:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:14 smithi003 ceph-mon[104007]: from='client.36267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:14.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:49:15.033 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:15.067 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:15.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:14 smithi019 ceph-mon[91296]: pgmap v831: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 3.8 KiB/s rd, 26 MiB/s wr, 437 op/s; 8177/349917 objects degraded (2.337%); 5674/349917 objects misplaced (1.622%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:49:15.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:14 smithi019 ceph-mon[91296]: from='client.36267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:15.219 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":831,"stamp":"2024-09-13T14:49:13.758643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7642972320,"num_objects":116639,"num_object_clones":0,"num_object_copies":349917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8177,"num_objects_misplaced":5674,"num_objects_unfound":0,"num_objects_dirty":116639,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":116910,"num_write_kb":7470570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1853,"num_bytes_recovered":121138737,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111084,"ondisk_log_size":111084,"up":327,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":28544328,"kb_used_data":22488200,"kb_used_omap":29,"kb_used_meta":6055906,"kb_avail":721416888,"statfs":{"total":767960285184,"available":738730893312,"internally_reserved":0,"allocated":23027916800,"data_stored":23017403912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6201248270},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":240216275,"num_objects":3665,"num_object_clones":0,"num_object_copies":10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3331,"num_objects_misplaced":3170,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":3898,"num_write_kb":241866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":10578109,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.946484"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":384,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.824752+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:49:05.824752+0000","last_peered":"2024-09-13T14:49:05.824752+0000","last_clean":"2024-09-13T14:49:05.824752+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:49:05.824752+0000","last_undegraded":"2024-09-13T14:49:05.824752+0000","last_fullsized":"2024-09-13T14:49:05.824752+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":225,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.827832+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:49:04.827832+0000","last_peered":"2024-09-13T14:49:04.827832+0000","last_clean":"2024-09-13T14:49:04.827832+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:49:04.827832+0000","last_undegraded":"2024-09-13T14:49:04.827832+0000","last_fullsized":"2024-09-13T14:49:04.827832+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":16,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815437+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:49:04.815437+0000","last_peered":"2024-09-13T14:49:04.815437+0000","last_clean":"2024-09-13T14:49:04.815437+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:49:04.815437+0000","last_undegraded":"2024-09-13T14:49:04.815437+0000","last_fullsized":"2024-09-13T14:49:04.815437+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":343,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.817620+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:49:05.817620+0000","last_peered":"2024-09-13T14:49:05.817620+0000","last_clean":"2024-09-13T14:49:05.817620+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:49:05.817620+0000","last_undegraded":"2024-09-13T14:49:05.817620+0000","last_fullsized":"2024-09-13T14:49:05.817620+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":199,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815437+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:49:04.815437+0000","last_peered":"2024-09-13T14:49:04.815437+0000","last_clean":"2024-09-13T14:49:04.815437+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:49:04.815437+0000","last_undegraded":"2024-09-13T14:49:04.815437+0000","last_fullsized":"2024-09-13T14:49:04.815437+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":232,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.829659+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:49:04.829659+0000","last_peered":"2024-09-13T14:49:04.829659+0000","last_clean":"2024-09-13T14:49:04.829659+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:49:04.829659+0000","last_undegraded":"2024-09-13T14:49:04.829659+0000","last_fullsized":"2024-09-13T14:49:04.829659+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":216,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815259+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:49:04.815259+0000","last_peered":"2024-09-13T14:49:04.815259+0000","last_clean":"2024-09-13T14:49:04.815259+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:49:04.815259+0000","last_undegraded":"2024-09-13T14:49:04.815259+0000","last_fullsized":"2024-09-13T14:49:04.815259+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":262,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.826016+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:49:04.826016+0000","last_peered":"2024-09-13T14:49:04.826016+0000","last_clean":"2024-09-13T14:49:04.826016+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:49:04.826016+0000","last_undegraded":"2024-09-13T14:49:04.826016+0000","last_fullsized":"2024-09-13T14:49:04.826016+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":372,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.824818+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:49:05.824818+0000","last_peered":"2024-09-13T14:49:05.824818+0000","last_clean":"2024-09-13T14:49:05.824818+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:49:05.824818+0000","last_undegraded":"2024-09-13T14:49:05.824818+0000","last_fullsized":"2024-09-13T14:49:05.824818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":367,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814362+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:49:05.814362+0000","last_peered":"2024-09-13T14:49:05.814362+0000","last_clean":"2024-09-13T14:49:05.814362+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:49:05.814362+0000","last_undegraded":"2024-09-13T14:49:05.814362+0000","last_fullsized":"2024-09-13T14:49:05.814362+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":234,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.829798+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:49:04.829798+0000","last_peered":"2024-09-13T14:49:04.829798+0000","last_clean":"2024-09-13T14:49:04.829798+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:49:04.829798+0000","last_undegraded":"2024-09-13T14:49:04.829798+0000","last_fullsized":"2024-09-13T14:49:04.829798+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":309,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.818628+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:49:05.818628+0000","last_peered":"2024-09-13T14:49:05.818628+0000","last_clean":"2024-09-13T14:49:05.818628+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:49:05.818628+0000","last_undegraded":"2024-09-13T14:49:05.818628+0000","last_fullsized":"2024-09-13T14:49:05.818628+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814548+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:49:05.814548+0000","last_peered":"2024-09-13T14:49:05.814548+0000","last_clean":"2024-09-13T14:49:05.814548+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:49:05.814548+0000","last_undegraded":"2024-09-13T14:49:05.814548+0000","last_fullsized":"2024-09-13T14:49:05.814548+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":402,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.848010+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:49:05.848010+0000","last_peered":"2024-09-13T14:49:05.848010+0000","last_clean":"2024-09-13T14:49:05.848010+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:49:05.848010+0000","last_undegraded":"2024-09-13T14:49:05.848010+0000","last_fullsized":"2024-09-13T14:49:05.848010+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":342,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.816147+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:49:05.816147+0000","last_peered":"2024-09-13T14:49:05.816147+0000","last_clean":"2024-09-13T14:49:05.816147+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:49:05.816147+0000","last_undegraded":"2024-09-13T14:49:05.816147+0000","last_fullsized":"2024-09-13T14:49:05.816147+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":215,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.820316+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:49:04.820316+0000","last_peered":"2024-09-13T14:49:04.820316+0000","last_clean":"2024-09-13T14:49:04.820316+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:49:04.820316+0000","last_undegraded":"2024-09-13T14:49:04.820316+0000","last_fullsized":"2024-09-13T14:49:04.820316+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":310,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815563+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:49:05.815563+0000","last_peered":"2024-09-13T14:49:05.815563+0000","last_clean":"2024-09-13T14:49:05.815563+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:49:05.815563+0000","last_undegraded":"2024-09-13T14:49:05.815563+0000","last_fullsized":"2024-09-13T14:49:05.815563+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":386,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.825031+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:49:05.825031+0000","last_peered":"2024-09-13T14:49:05.825031+0000","last_clean":"2024-09-13T14:49:05.825031+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:49:05.825031+0000","last_undegraded":"2024-09-13T14:49:05.825031+0000","last_fullsized":"2024-09-13T14:49:05.825031+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":315,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.828755+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:49:05.828755+0000","last_peered":"2024-09-13T14:49:05.828755+0000","last_clean":"2024-09-13T14:49:05.828755+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:49:05.828755+0000","last_undegraded":"2024-09-13T14:49:05.828755+0000","last_fullsized":"2024-09-13T14:49:05.828755+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815348+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:49:04.815348+0000","last_peered":"2024-09-13T14:49:04.815348+0000","last_clean":"2024-09-13T14:49:04.815348+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:49:04.815348+0000","last_undegraded":"2024-09-13T14:49:04.815348+0000","last_fullsized":"2024-09-13T14:49:04.815348+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":64,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814183+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:49:05.814183+0000","last_peered":"2024-09-13T14:49:05.814183+0000","last_clean":"2024-09-13T14:49:05.814183+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:49:05.814183+0000","last_undegraded":"2024-09-13T14:49:05.814183+0000","last_fullsized":"2024-09-13T14:49:05.814183+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.856566+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:49:05.856566+0000","last_peered":"2024-09-13T14:49:05.856566+0000","last_clean":"2024-09-13T14:49:05.856566+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:49:05.856566+0000","last_undegraded":"2024-09-13T14:49:05.856566+0000","last_fullsized":"2024-09-13T14:49:05.856566+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":154,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.818530+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:49:04.818530+0000","last_peered":"2024-09-13T14:49:04.818530+0000","last_clean":"2024-09-13T14:49:04.818530+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:49:04.818530+0000","last_undegraded":"2024-09-13T14:49:04.818530+0000","last_fullsized":"2024-09-13T14:49:04.818530+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":222,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.833771+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:49:04.833771+0000","last_peered":"2024-09-13T14:49:04.833771+0000","last_clean":"2024-09-13T14:49:04.833771+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:49:04.833771+0000","last_undegraded":"2024-09-13T14:49:04.833771+0000","last_fullsized":"2024-09-13T14:49:04.833771+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":215,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.827451+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:49:04.827451+0000","last_peered":"2024-09-13T14:49:04.827451+0000","last_clean":"2024-09-13T14:49:04.827451+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:49:04.827451+0000","last_undegraded":"2024-09-13T14:49:04.827451+0000","last_fullsized":"2024-09-13T14:49:04.827451+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"253'7234","reported_seq":7266,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.052943+0000","last_change":"2024-09-13T14:47:20.107455+0000","last_active":"2024-09-13T14:49:08.052943+0000","last_peered":"2024-09-13T14:49:08.052943+0000","last_clean":"2024-09-13T14:49:08.052943+0000","last_became_active":"2024-09-13T14:47:20.107142+0000","last_became_peered":"2024-09-13T14:47:20.107142+0000","last_unstale":"2024-09-13T14:49:08.052943+0000","last_undegraded":"2024-09-13T14:49:08.052943+0000","last_fullsized":"2024-09-13T14:49:08.052943+0000","mapping_epoch":229,"log_start":"237'500","ondisk_log_start":"237'500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6734,"log_dups_size":0,"ondisk_log_size":6734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:19.189519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474087424,"num_objects":7234,"num_object_clones":0,"num_object_copies":21702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":462976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":198,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.830234+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:49:04.830234+0000","last_peered":"2024-09-13T14:49:04.830234+0000","last_clean":"2024-09-13T14:49:04.830234+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:49:04.830234+0000","last_undegraded":"2024-09-13T14:49:04.830234+0000","last_fullsized":"2024-09-13T14:49:04.830234+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":248,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.833563+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:49:05.833563+0000","last_peered":"2024-09-13T14:49:05.833563+0000","last_clean":"2024-09-13T14:49:05.833563+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:49:05.833563+0000","last_undegraded":"2024-09-13T14:49:05.833563+0000","last_fullsized":"2024-09-13T14:49:05.833563+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":384,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.835321+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:49:05.835321+0000","last_peered":"2024-09-13T14:49:05.835321+0000","last_clean":"2024-09-13T14:49:05.835321+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:49:05.835321+0000","last_undegraded":"2024-09-13T14:49:05.835321+0000","last_fullsized":"2024-09-13T14:49:05.835321+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":72,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815591+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:49:05.815591+0000","last_peered":"2024-09-13T14:49:05.815591+0000","last_clean":"2024-09-13T14:49:05.815591+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:49:05.815591+0000","last_undegraded":"2024-09-13T14:49:05.815591+0000","last_fullsized":"2024-09-13T14:49:05.815591+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"253'7301","reported_seq":7333,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.054808+0000","last_change":"2024-09-13T14:47:20.054114+0000","last_active":"2024-09-13T14:49:08.054808+0000","last_peered":"2024-09-13T14:49:08.054808+0000","last_clean":"2024-09-13T14:49:08.054808+0000","last_became_active":"2024-09-13T14:47:20.053767+0000","last_became_peered":"2024-09-13T14:47:20.053767+0000","last_unstale":"2024-09-13T14:49:08.054808+0000","last_undegraded":"2024-09-13T14:49:08.054808+0000","last_fullsized":"2024-09-13T14:49:08.054808+0000","mapping_epoch":229,"log_start":"237'1000","ondisk_log_start":"237'1000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:48.035191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478478336,"num_objects":7301,"num_object_clones":0,"num_object_copies":21903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7301,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":353,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.860180+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:49:05.860180+0000","last_peered":"2024-09-13T14:49:05.860180+0000","last_clean":"2024-09-13T14:49:05.860180+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:49:05.860180+0000","last_undegraded":"2024-09-13T14:49:05.860180+0000","last_fullsized":"2024-09-13T14:49:05.860180+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.823570+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:49:04.823570+0000","last_peered":"2024-09-13T14:49:04.823570+0000","last_clean":"2024-09-13T14:49:04.823570+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:49:04.823570+0000","last_undegraded":"2024-09-13T14:49:04.823570+0000","last_fullsized":"2024-09-13T14:49:04.823570+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":341,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.826552+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:49:05.826552+0000","last_peered":"2024-09-13T14:49:05.826552+0000","last_clean":"2024-09-13T14:49:05.826552+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:49:05.826552+0000","last_undegraded":"2024-09-13T14:49:05.826552+0000","last_fullsized":"2024-09-13T14:49:05.826552+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":36,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.829163+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:49:04.829163+0000","last_peered":"2024-09-13T14:49:04.829163+0000","last_clean":"2024-09-13T14:49:04.829163+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:49:04.829163+0000","last_undegraded":"2024-09-13T14:49:04.829163+0000","last_fullsized":"2024-09-13T14:49:04.829163+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"253'7313","reported_seq":7345,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.027585+0000","last_change":"2024-09-13T14:47:20.052876+0000","last_active":"2024-09-13T14:49:08.027585+0000","last_peered":"2024-09-13T14:49:08.027585+0000","last_clean":"2024-09-13T14:49:08.027585+0000","last_became_active":"2024-09-13T14:47:20.052566+0000","last_became_peered":"2024-09-13T14:47:20.052566+0000","last_unstale":"2024-09-13T14:49:08.027585+0000","last_undegraded":"2024-09-13T14:49:08.027585+0000","last_fullsized":"2024-09-13T14:49:08.027585+0000","mapping_epoch":229,"log_start":"237'600","ondisk_log_start":"237'600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:56.317980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479264768,"num_objects":7313,"num_object_clones":0,"num_object_copies":21939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7313,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":370,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.825062+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:49:05.825062+0000","last_peered":"2024-09-13T14:49:05.825062+0000","last_clean":"2024-09-13T14:49:05.825062+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:49:05.825062+0000","last_undegraded":"2024-09-13T14:49:05.825062+0000","last_fullsized":"2024-09-13T14:49:05.825062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":228,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815333+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:49:04.815333+0000","last_peered":"2024-09-13T14:49:04.815333+0000","last_clean":"2024-09-13T14:49:04.815333+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:49:04.815333+0000","last_undegraded":"2024-09-13T14:49:04.815333+0000","last_fullsized":"2024-09-13T14:49:04.815333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":325,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814274+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:49:05.814274+0000","last_peered":"2024-09-13T14:49:05.814274+0000","last_clean":"2024-09-13T14:49:05.814274+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:49:05.814274+0000","last_undegraded":"2024-09-13T14:49:05.814274+0000","last_fullsized":"2024-09-13T14:49:05.814274+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":189,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.818796+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:49:04.818796+0000","last_peered":"2024-09-13T14:49:04.818796+0000","last_clean":"2024-09-13T14:49:04.818796+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:49:04.818796+0000","last_undegraded":"2024-09-13T14:49:04.818796+0000","last_fullsized":"2024-09-13T14:49:04.818796+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"253'7350","reported_seq":7382,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.067637+0000","last_change":"2024-09-13T14:47:20.052419+0000","last_active":"2024-09-13T14:49:08.067637+0000","last_peered":"2024-09-13T14:49:08.067637+0000","last_clean":"2024-09-13T14:49:08.067637+0000","last_became_active":"2024-09-13T14:47:20.051845+0000","last_became_peered":"2024-09-13T14:47:20.051845+0000","last_unstale":"2024-09-13T14:49:08.067637+0000","last_undegraded":"2024-09-13T14:49:08.067637+0000","last_fullsized":"2024-09-13T14:49:08.067637+0000","mapping_epoch":229,"log_start":"237'714","ondisk_log_start":"237'714","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6636,"log_dups_size":0,"ondisk_log_size":6636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.420332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481689600,"num_objects":7350,"num_object_clones":0,"num_object_copies":22050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7350,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815103+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:49:04.815103+0000","last_peered":"2024-09-13T14:49:04.815103+0000","last_clean":"2024-09-13T14:49:04.815103+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:49:04.815103+0000","last_undegraded":"2024-09-13T14:49:04.815103+0000","last_fullsized":"2024-09-13T14:49:04.815103+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":183,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.832386+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:49:05.832386+0000","last_peered":"2024-09-13T14:49:05.832386+0000","last_clean":"2024-09-13T14:49:05.832386+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:49:05.832386+0000","last_undegraded":"2024-09-13T14:49:05.832386+0000","last_fullsized":"2024-09-13T14:49:05.832386+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":231,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815522+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:49:04.815522+0000","last_peered":"2024-09-13T14:49:04.815522+0000","last_clean":"2024-09-13T14:49:04.815522+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:49:04.815522+0000","last_undegraded":"2024-09-13T14:49:04.815522+0000","last_fullsized":"2024-09-13T14:49:04.815522+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":209,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.839436+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:49:05.839436+0000","last_peered":"2024-09-13T14:49:05.839436+0000","last_clean":"2024-09-13T14:49:05.839436+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:49:05.839436+0000","last_undegraded":"2024-09-13T14:49:05.839436+0000","last_fullsized":"2024-09-13T14:49:05.839436+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"253'7212","reported_seq":7244,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:07.997209+0000","last_change":"2024-09-13T14:47:20.051919+0000","last_active":"2024-09-13T14:49:07.997209+0000","last_peered":"2024-09-13T14:49:07.997209+0000","last_clean":"2024-09-13T14:49:07.997209+0000","last_became_active":"2024-09-13T14:47:20.051626+0000","last_became_peered":"2024-09-13T14:47:20.051626+0000","last_unstale":"2024-09-13T14:49:07.997209+0000","last_undegraded":"2024-09-13T14:49:07.997209+0000","last_fullsized":"2024-09-13T14:49:07.997209+0000","mapping_epoch":229,"log_start":"237'500","ondisk_log_start":"237'500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6712,"log_dups_size":0,"ondisk_log_size":6712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:12.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472645632,"num_objects":7212,"num_object_clones":0,"num_object_copies":21636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7212,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":237,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814444+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:49:05.814444+0000","last_peered":"2024-09-13T14:49:05.814444+0000","last_clean":"2024-09-13T14:49:05.814444+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:49:05.814444+0000","last_undegraded":"2024-09-13T14:49:05.814444+0000","last_fullsized":"2024-09-13T14:49:05.814444+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":216,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.827534+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:49:04.827534+0000","last_peered":"2024-09-13T14:49:04.827534+0000","last_clean":"2024-09-13T14:49:04.827534+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:49:04.827534+0000","last_undegraded":"2024-09-13T14:49:04.827534+0000","last_fullsized":"2024-09-13T14:49:04.827534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":227,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.816596+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:49:04.816596+0000","last_peered":"2024-09-13T14:49:04.816596+0000","last_clean":"2024-09-13T14:49:04.816596+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:49:04.816596+0000","last_undegraded":"2024-09-13T14:49:04.816596+0000","last_fullsized":"2024-09-13T14:49:04.816596+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":36,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.816689+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:49:04.816689+0000","last_peered":"2024-09-13T14:49:04.816689+0000","last_clean":"2024-09-13T14:49:04.816689+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:49:04.816689+0000","last_undegraded":"2024-09-13T14:49:04.816689+0000","last_fullsized":"2024-09-13T14:49:04.816689+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"253'7294","reported_seq":7522,"reported_epoch":253,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:08.823237+0000","last_change":"2024-09-13T14:49:05.872778+0000","last_active":"2024-09-13T14:49:08.823237+0000","last_peered":"2024-09-13T14:49:08.823237+0000","last_clean":"2024-09-13T14:48:37.826901+0000","last_became_active":"2024-09-13T14:48:40.041761+0000","last_became_peered":"2024-09-13T14:48:40.041761+0000","last_unstale":"2024-09-13T14:49:08.823237+0000","last_undegraded":"2024-09-13T14:48:39.850843+0000","last_fullsized":"2024-09-13T14:48:39.825201+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7294,"log_dups_size":0,"ondisk_log_size":7294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478019584,"num_objects":7294,"num_object_clones":0,"num_object_copies":21882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":5674,"num_objects_unfound":0,"num_objects_dirty":7294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7294,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5505024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":5674},{"shards":"2,7","objects":1619}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":147,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.829182+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:49:04.829182+0000","last_peered":"2024-09-13T14:49:04.829182+0000","last_clean":"2024-09-13T14:49:04.829182+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:49:04.829182+0000","last_undegraded":"2024-09-13T14:49:04.829182+0000","last_fullsized":"2024-09-13T14:49:04.829182+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":177,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814150+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:49:05.814150+0000","last_peered":"2024-09-13T14:49:05.814150+0000","last_clean":"2024-09-13T14:49:05.814150+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:49:05.814150+0000","last_undegraded":"2024-09-13T14:49:05.814150+0000","last_fullsized":"2024-09-13T14:49:05.814150+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":368,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815176+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:49:05.815176+0000","last_peered":"2024-09-13T14:49:05.815176+0000","last_clean":"2024-09-13T14:49:05.815176+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:49:05.815176+0000","last_undegraded":"2024-09-13T14:49:05.815176+0000","last_fullsized":"2024-09-13T14:49:05.815176+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.834249+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:49:05.834249+0000","last_peered":"2024-09-13T14:49:05.834249+0000","last_clean":"2024-09-13T14:49:05.834249+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:49:05.834249+0000","last_undegraded":"2024-09-13T14:49:05.834249+0000","last_fullsized":"2024-09-13T14:49:05.834249+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"253'7407","reported_seq":7466,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:10.085752+0000","last_change":"2024-09-13T14:47:20.049747+0000","last_active":"2024-09-13T14:49:10.085752+0000","last_peered":"2024-09-13T14:49:10.085752+0000","last_clean":"2024-09-13T14:49:10.085752+0000","last_became_active":"2024-09-13T14:47:20.049448+0000","last_became_peered":"2024-09-13T14:47:20.049448+0000","last_unstale":"2024-09-13T14:49:10.085752+0000","last_undegraded":"2024-09-13T14:49:10.085752+0000","last_fullsized":"2024-09-13T14:49:10.085752+0000","mapping_epoch":229,"log_start":"236'400","ondisk_log_start":"236'400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7007,"log_dups_size":0,"ondisk_log_size":7007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:11.585800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479526912,"num_objects":7317,"num_object_clones":0,"num_object_copies":21951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":392,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.835205+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:49:05.835205+0000","last_peered":"2024-09-13T14:49:05.835205+0000","last_clean":"2024-09-13T14:49:05.835205+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:49:05.835205+0000","last_undegraded":"2024-09-13T14:49:05.835205+0000","last_fullsized":"2024-09-13T14:49:05.835205+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":401,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.825043+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:49:05.825043+0000","last_peered":"2024-09-13T14:49:05.825043+0000","last_clean":"2024-09-13T14:49:05.825043+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:49:05.825043+0000","last_undegraded":"2024-09-13T14:49:05.825043+0000","last_fullsized":"2024-09-13T14:49:05.825043+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":265,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.814063+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:49:04.814063+0000","last_peered":"2024-09-13T14:49:04.814063+0000","last_clean":"2024-09-13T14:49:04.814063+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:49:04.814063+0000","last_undegraded":"2024-09-13T14:49:04.814063+0000","last_fullsized":"2024-09-13T14:49:04.814063+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":189,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.819680+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:49:04.819680+0000","last_peered":"2024-09-13T14:49:04.819680+0000","last_clean":"2024-09-13T14:49:04.819680+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:49:04.819680+0000","last_undegraded":"2024-09-13T14:49:04.819680+0000","last_fullsized":"2024-09-13T14:49:04.819680+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"253'7323","reported_seq":7355,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.068246+0000","last_change":"2024-09-13T14:47:20.106371+0000","last_active":"2024-09-13T14:49:08.068246+0000","last_peered":"2024-09-13T14:49:08.068246+0000","last_clean":"2024-09-13T14:49:08.068246+0000","last_became_active":"2024-09-13T14:47:20.105934+0000","last_became_peered":"2024-09-13T14:47:20.105934+0000","last_unstale":"2024-09-13T14:49:08.068246+0000","last_undegraded":"2024-09-13T14:49:08.068246+0000","last_fullsized":"2024-09-13T14:49:08.068246+0000","mapping_epoch":229,"log_start":"237'600","ondisk_log_start":"237'600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6723,"log_dups_size":0,"ondisk_log_size":6723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:54.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479920128,"num_objects":7323,"num_object_clones":0,"num_object_copies":21969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7323,"num_write_kb":468672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":198,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.821027+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:49:04.821027+0000","last_peered":"2024-09-13T14:49:04.821027+0000","last_clean":"2024-09-13T14:49:04.821027+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:49:04.821027+0000","last_undegraded":"2024-09-13T14:49:04.821027+0000","last_fullsized":"2024-09-13T14:49:04.821027+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":300,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.816286+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:49:05.816286+0000","last_peered":"2024-09-13T14:49:05.816286+0000","last_clean":"2024-09-13T14:49:05.816286+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:49:05.816286+0000","last_undegraded":"2024-09-13T14:49:05.816286+0000","last_fullsized":"2024-09-13T14:49:05.816286+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":226,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.816842+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:49:04.816842+0000","last_peered":"2024-09-13T14:49:04.816842+0000","last_clean":"2024-09-13T14:49:04.816842+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:49:04.816842+0000","last_undegraded":"2024-09-13T14:49:04.816842+0000","last_fullsized":"2024-09-13T14:49:04.816842+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.859580+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:49:05.859580+0000","last_peered":"2024-09-13T14:49:05.859580+0000","last_clean":"2024-09-13T14:49:05.859580+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:49:05.859580+0000","last_undegraded":"2024-09-13T14:49:05.859580+0000","last_fullsized":"2024-09-13T14:49:05.859580+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"253'7221","reported_seq":7253,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.080973+0000","last_change":"2024-09-13T14:47:20.050773+0000","last_active":"2024-09-13T14:49:08.080973+0000","last_peered":"2024-09-13T14:49:08.080973+0000","last_clean":"2024-09-13T14:49:08.080973+0000","last_became_active":"2024-09-13T14:47:20.050243+0000","last_became_peered":"2024-09-13T14:47:20.050243+0000","last_unstale":"2024-09-13T14:49:08.080973+0000","last_undegraded":"2024-09-13T14:49:08.080973+0000","last_fullsized":"2024-09-13T14:49:08.080973+0000","mapping_epoch":229,"log_start":"237'900","ondisk_log_start":"237'900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:42.724078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473235456,"num_objects":7221,"num_object_clones":0,"num_object_copies":21663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.817066+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:49:05.817066+0000","last_peered":"2024-09-13T14:49:05.817066+0000","last_clean":"2024-09-13T14:49:05.817066+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:49:05.817066+0000","last_undegraded":"2024-09-13T14:49:05.817066+0000","last_fullsized":"2024-09-13T14:49:05.817066+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":311,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.816194+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:49:05.816194+0000","last_peered":"2024-09-13T14:49:05.816194+0000","last_clean":"2024-09-13T14:49:05.816194+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:49:05.816194+0000","last_undegraded":"2024-09-13T14:49:05.816194+0000","last_fullsized":"2024-09-13T14:49:05.816194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":36,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.814962+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:49:04.814962+0000","last_peered":"2024-09-13T14:49:04.814962+0000","last_clean":"2024-09-13T14:49:04.814962+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:49:04.814962+0000","last_undegraded":"2024-09-13T14:49:04.814962+0000","last_fullsized":"2024-09-13T14:49:04.814962+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"253'7223","reported_seq":7255,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:07.980048+0000","last_change":"2024-09-13T14:47:20.106351+0000","last_active":"2024-09-13T14:49:07.980048+0000","last_peered":"2024-09-13T14:49:07.980048+0000","last_clean":"2024-09-13T14:49:07.980048+0000","last_became_active":"2024-09-13T14:47:20.105946+0000","last_became_peered":"2024-09-13T14:47:20.105946+0000","last_unstale":"2024-09-13T14:49:07.980048+0000","last_undegraded":"2024-09-13T14:49:07.980048+0000","last_fullsized":"2024-09-13T14:49:07.980048+0000","mapping_epoch":229,"log_start":"237'500","ondisk_log_start":"237'500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6723,"log_dups_size":0,"ondisk_log_size":6723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:07.223637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473366528,"num_objects":7223,"num_object_clones":0,"num_object_copies":21669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":235,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.830481+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:49:04.830481+0000","last_peered":"2024-09-13T14:49:04.830481+0000","last_clean":"2024-09-13T14:49:04.830481+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:49:04.830481+0000","last_undegraded":"2024-09-13T14:49:04.830481+0000","last_fullsized":"2024-09-13T14:49:04.830481+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":216,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.830606+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:49:04.830606+0000","last_peered":"2024-09-13T14:49:04.830606+0000","last_clean":"2024-09-13T14:49:04.830606+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:49:04.830606+0000","last_undegraded":"2024-09-13T14:49:04.830606+0000","last_fullsized":"2024-09-13T14:49:04.830606+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":362,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.824631+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:49:05.824631+0000","last_peered":"2024-09-13T14:49:05.824631+0000","last_clean":"2024-09-13T14:49:05.824631+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:49:05.824631+0000","last_undegraded":"2024-09-13T14:49:05.824631+0000","last_fullsized":"2024-09-13T14:49:05.824631+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"253'7308","reported_seq":10876,"reported_epoch":253,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:08.046182+0000","last_change":"2024-09-13T14:49:05.871518+0000","last_active":"2024-09-13T14:49:08.046182+0000","last_peered":"2024-09-13T14:49:08.046182+0000","last_clean":"2024-09-13T14:48:09.848772+0000","last_became_active":"2024-09-13T14:48:12.232150+0000","last_became_peered":"2024-09-13T14:48:12.232150+0000","last_unstale":"2024-09-13T14:49:08.046182+0000","last_undegraded":"2024-09-13T14:49:05.858877+0000","last_fullsized":"2024-09-13T14:48:11.643512+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7308,"log_dups_size":0,"ondisk_log_size":7308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478937088,"num_objects":7308,"num_object_clones":0,"num_object_copies":21924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7308,"num_write_kb":467712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":114753536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":2427},{"shards":"2,5","objects":4881}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":232,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.832225+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:49:04.832225+0000","last_peered":"2024-09-13T14:49:04.832225+0000","last_clean":"2024-09-13T14:49:04.832225+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:49:04.832225+0000","last_undegraded":"2024-09-13T14:49:04.832225+0000","last_fullsized":"2024-09-13T14:49:04.832225+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":236,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.818533+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:49:04.818533+0000","last_peered":"2024-09-13T14:49:04.818533+0000","last_clean":"2024-09-13T14:49:04.818533+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:49:04.818533+0000","last_undegraded":"2024-09-13T14:49:04.818533+0000","last_fullsized":"2024-09-13T14:49:04.818533+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":18,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815555+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:49:04.815555+0000","last_peered":"2024-09-13T14:49:04.815555+0000","last_clean":"2024-09-13T14:49:04.815555+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:49:04.815555+0000","last_undegraded":"2024-09-13T14:49:04.815555+0000","last_fullsized":"2024-09-13T14:49:04.815555+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"253'7378","reported_seq":7410,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.047314+0000","last_change":"2024-09-13T14:47:20.106142+0000","last_active":"2024-09-13T14:49:08.047314+0000","last_peered":"2024-09-13T14:49:08.047314+0000","last_clean":"2024-09-13T14:49:08.047314+0000","last_became_active":"2024-09-13T14:47:20.105758+0000","last_became_peered":"2024-09-13T14:47:20.105758+0000","last_unstale":"2024-09-13T14:49:08.047314+0000","last_undegraded":"2024-09-13T14:49:08.047314+0000","last_fullsized":"2024-09-13T14:49:08.047314+0000","mapping_epoch":229,"log_start":"237'1100","ondisk_log_start":"237'1100","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:51.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483524608,"num_objects":7378,"num_object_clones":0,"num_object_copies":22134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":215,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.833606+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:49:04.833606+0000","last_peered":"2024-09-13T14:49:04.833606+0000","last_clean":"2024-09-13T14:49:04.833606+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:49:04.833606+0000","last_undegraded":"2024-09-13T14:49:04.833606+0000","last_fullsized":"2024-09-13T14:49:04.833606+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":341,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.823626+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:49:05.823626+0000","last_peered":"2024-09-13T14:49:05.823626+0000","last_clean":"2024-09-13T14:49:05.823626+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:49:05.823626+0000","last_undegraded":"2024-09-13T14:49:05.823626+0000","last_fullsized":"2024-09-13T14:49:05.823626+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":36,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.814172+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:49:04.814172+0000","last_peered":"2024-09-13T14:49:04.814172+0000","last_clean":"2024-09-13T14:49:04.814172+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:49:04.814172+0000","last_undegraded":"2024-09-13T14:49:04.814172+0000","last_fullsized":"2024-09-13T14:49:04.814172+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"253'7314","reported_seq":7373,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.072491+0000","last_change":"2024-09-13T14:47:20.107374+0000","last_active":"2024-09-13T14:49:08.072491+0000","last_peered":"2024-09-13T14:49:08.072491+0000","last_clean":"2024-09-13T14:49:08.072491+0000","last_became_active":"2024-09-13T14:47:20.107057+0000","last_became_peered":"2024-09-13T14:47:20.107057+0000","last_unstale":"2024-09-13T14:49:08.072491+0000","last_undegraded":"2024-09-13T14:49:08.072491+0000","last_fullsized":"2024-09-13T14:49:08.072491+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:05.814456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479330304,"num_objects":7314,"num_object_clones":0,"num_object_copies":21942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7314,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":380,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.824735+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:49:05.824735+0000","last_peered":"2024-09-13T14:49:05.824735+0000","last_clean":"2024-09-13T14:49:05.824735+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:49:05.824735+0000","last_undegraded":"2024-09-13T14:49:05.824735+0000","last_fullsized":"2024-09-13T14:49:05.824735+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":214,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815947+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:49:04.815947+0000","last_peered":"2024-09-13T14:49:04.815947+0000","last_clean":"2024-09-13T14:49:04.815947+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:49:04.815947+0000","last_undegraded":"2024-09-13T14:49:04.815947+0000","last_fullsized":"2024-09-13T14:49:04.815947+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":36,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.829419+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:49:04.829419+0000","last_peered":"2024-09-13T14:49:04.829419+0000","last_clean":"2024-09-13T14:49:04.829419+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:49:04.829419+0000","last_undegraded":"2024-09-13T14:49:04.829419+0000","last_fullsized":"2024-09-13T14:49:04.829419+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"253'7292","reported_seq":7351,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:10.120468+0000","last_change":"2024-09-13T14:47:20.108173+0000","last_active":"2024-09-13T14:49:10.120468+0000","last_peered":"2024-09-13T14:49:10.120468+0000","last_clean":"2024-09-13T14:49:10.120468+0000","last_became_active":"2024-09-13T14:47:20.107879+0000","last_became_peered":"2024-09-13T14:47:20.107879+0000","last_unstale":"2024-09-13T14:49:10.120468+0000","last_undegraded":"2024-09-13T14:49:10.120468+0000","last_fullsized":"2024-09-13T14:49:10.120468+0000","mapping_epoch":229,"log_start":"236'300","ondisk_log_start":"236'300","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6992,"log_dups_size":0,"ondisk_log_size":6992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:50.152706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472514560,"num_objects":7210,"num_object_clones":0,"num_object_copies":21630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7292,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":228,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.822115+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:49:05.822115+0000","last_peered":"2024-09-13T14:49:05.822115+0000","last_clean":"2024-09-13T14:49:05.822115+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:49:05.822115+0000","last_undegraded":"2024-09-13T14:49:05.822115+0000","last_fullsized":"2024-09-13T14:49:05.822115+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":229,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815254+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:49:04.815254+0000","last_peered":"2024-09-13T14:49:04.815254+0000","last_clean":"2024-09-13T14:49:04.815254+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:49:04.815254+0000","last_undegraded":"2024-09-13T14:49:04.815254+0000","last_fullsized":"2024-09-13T14:49:04.815254+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":189,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.820250+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:49:04.820250+0000","last_peered":"2024-09-13T14:49:04.820250+0000","last_clean":"2024-09-13T14:49:04.820250+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:49:04.820250+0000","last_undegraded":"2024-09-13T14:49:04.820250+0000","last_fullsized":"2024-09-13T14:49:04.820250+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"253'7245","reported_seq":7280,"reported_epoch":253,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:08.089901+0000","last_change":"2024-09-13T14:48:40.078958+0000","last_active":"2024-09-13T14:49:08.089901+0000","last_peered":"2024-09-13T14:49:08.089901+0000","last_clean":"2024-09-13T14:48:37.811111+0000","last_became_active":"2024-09-13T14:48:40.052181+0000","last_became_peered":"2024-09-13T14:48:40.052181+0000","last_unstale":"2024-09-13T14:49:08.089901+0000","last_undegraded":"2024-09-13T14:48:39.849823+0000","last_fullsized":"2024-09-13T14:48:39.826167+0000","mapping_epoch":249,"log_start":"237'500","ondisk_log_start":"237'500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:49.521146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474808320,"num_objects":7245,"num_object_clones":0,"num_object_copies":21735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7245,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":5749},{"shards":"3,6","objects":1496}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":242,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.830458+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:49:04.830458+0000","last_peered":"2024-09-13T14:49:04.830458+0000","last_clean":"2024-09-13T14:49:04.830458+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:49:04.830458+0000","last_undegraded":"2024-09-13T14:49:04.830458+0000","last_fullsized":"2024-09-13T14:49:04.830458+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":219,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815426+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:49:04.815426+0000","last_peered":"2024-09-13T14:49:04.815426+0000","last_clean":"2024-09-13T14:49:04.815426+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:49:04.815426+0000","last_undegraded":"2024-09-13T14:49:04.815426+0000","last_fullsized":"2024-09-13T14:49:04.815426+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":68,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815642+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:49:05.815642+0000","last_peered":"2024-09-13T14:49:05.815642+0000","last_clean":"2024-09-13T14:49:05.815642+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:49:05.815642+0000","last_undegraded":"2024-09-13T14:49:05.815642+0000","last_fullsized":"2024-09-13T14:49:05.815642+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"253'7424","reported_seq":7456,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:09.498332+0000","last_change":"2024-09-13T14:47:20.051824+0000","last_active":"2024-09-13T14:49:09.498332+0000","last_peered":"2024-09-13T14:49:09.498332+0000","last_clean":"2024-09-13T14:49:09.498332+0000","last_became_active":"2024-09-13T14:47:20.051487+0000","last_became_peered":"2024-09-13T14:47:20.051487+0000","last_unstale":"2024-09-13T14:49:09.498332+0000","last_undegraded":"2024-09-13T14:49:09.498332+0000","last_fullsized":"2024-09-13T14:49:09.498332+0000","mapping_epoch":229,"log_start":"236'300","ondisk_log_start":"236'300","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7124,"log_dups_size":0,"ondisk_log_size":7124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:11.533543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483000320,"num_objects":7370,"num_object_clones":0,"num_object_copies":22110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":24,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.820232+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:49:05.820232+0000","last_peered":"2024-09-13T14:49:05.820232+0000","last_clean":"2024-09-13T14:49:05.820232+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:49:05.820232+0000","last_undegraded":"2024-09-13T14:49:05.820232+0000","last_fullsized":"2024-09-13T14:49:05.820232+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.863693+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:49:05.863693+0000","last_peered":"2024-09-13T14:49:05.863693+0000","last_clean":"2024-09-13T14:49:05.863693+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:49:05.863693+0000","last_undegraded":"2024-09-13T14:49:05.863693+0000","last_fullsized":"2024-09-13T14:49:05.863693+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.826657+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:49:05.826657+0000","last_peered":"2024-09-13T14:49:05.826657+0000","last_clean":"2024-09-13T14:49:05.826657+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:49:05.826657+0000","last_undegraded":"2024-09-13T14:49:05.826657+0000","last_fullsized":"2024-09-13T14:49:05.826657+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":264,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.821982+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:49:04.821982+0000","last_peered":"2024-09-13T14:49:04.821982+0000","last_clean":"2024-09-13T14:49:04.821982+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:49:04.821982+0000","last_undegraded":"2024-09-13T14:49:04.821982+0000","last_fullsized":"2024-09-13T14:49:04.821982+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":350,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.846224+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:49:05.846224+0000","last_peered":"2024-09-13T14:49:05.846224+0000","last_clean":"2024-09-13T14:49:05.846224+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:49:05.846224+0000","last_undegraded":"2024-09-13T14:49:05.846224+0000","last_fullsized":"2024-09-13T14:49:05.846224+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":46,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815192+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:49:04.815192+0000","last_peered":"2024-09-13T14:49:04.815192+0000","last_clean":"2024-09-13T14:49:04.815192+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:49:04.815192+0000","last_undegraded":"2024-09-13T14:49:04.815192+0000","last_fullsized":"2024-09-13T14:49:04.815192+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":210,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.814252+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:49:04.814252+0000","last_peered":"2024-09-13T14:49:04.814252+0000","last_clean":"2024-09-13T14:49:04.814252+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:49:04.814252+0000","last_undegraded":"2024-09-13T14:49:04.814252+0000","last_fullsized":"2024-09-13T14:49:04.814252+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":394,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.848542+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:49:05.848542+0000","last_peered":"2024-09-13T14:49:05.848542+0000","last_clean":"2024-09-13T14:49:05.848542+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:49:05.848542+0000","last_undegraded":"2024-09-13T14:49:05.848542+0000","last_fullsized":"2024-09-13T14:49:05.848542+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":64,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.823451+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:49:05.823451+0000","last_peered":"2024-09-13T14:49:05.823451+0000","last_clean":"2024-09-13T14:49:05.823451+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:49:05.823451+0000","last_undegraded":"2024-09-13T14:49:05.823451+0000","last_fullsized":"2024-09-13T14:49:05.823451+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":400,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.859786+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:49:05.859786+0000","last_peered":"2024-09-13T14:49:05.859786+0000","last_clean":"2024-09-13T14:49:05.859786+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:49:05.859786+0000","last_undegraded":"2024-09-13T14:49:05.859786+0000","last_fullsized":"2024-09-13T14:49:05.859786+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.815346+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:49:04.815346+0000","last_peered":"2024-09-13T14:49:04.815346+0000","last_clean":"2024-09-13T14:49:04.815346+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:49:04.815346+0000","last_undegraded":"2024-09-13T14:49:04.815346+0000","last_fullsized":"2024-09-13T14:49:04.815346+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":36,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.816542+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:49:04.816542+0000","last_peered":"2024-09-13T14:49:04.816542+0000","last_clean":"2024-09-13T14:49:04.816542+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:49:04.816542+0000","last_undegraded":"2024-09-13T14:49:04.816542+0000","last_fullsized":"2024-09-13T14:49:04.816542+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":235,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:49:04.814300+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:49:04.814300+0000","last_peered":"2024-09-13T14:49:04.814300+0000","last_clean":"2024-09-13T14:49:04.814300+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:49:04.814300+0000","last_undegraded":"2024-09-13T14:49:04.814300+0000","last_fullsized":"2024-09-13T14:49:04.814300+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":59,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.832011+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:49:05.832011+0000","last_peered":"2024-09-13T14:49:05.832011+0000","last_clean":"2024-09-13T14:49:05.832011+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:49:05.832011+0000","last_undegraded":"2024-09-13T14:49:05.832011+0000","last_fullsized":"2024-09-13T14:49:05.832011+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":358,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.825161+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:49:05.825161+0000","last_peered":"2024-09-13T14:49:05.825161+0000","last_clean":"2024-09-13T14:49:05.825161+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:49:05.825161+0000","last_undegraded":"2024-09-13T14:49:05.825161+0000","last_fullsized":"2024-09-13T14:49:05.825161+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7642349568,"num_objects":116613,"num_object_clones":0,"num_object_copies":349839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8177,"num_objects_misplaced":5674,"num_objects_unfound":0,"num_objects_dirty":116613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116839,"num_write_kb":7470464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1835,"num_bytes_recovered":120258560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22955692032,"data_stored":22955687960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108925,"ondisk_log_size":108925,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365153,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2785156,"kb_used_data":1875772,"kb_used_omap":1,"kb_used_meta":909374,"kb_avail":90959996,"statfs":{"total":95995035648,"available":93143035904,"internally_reserved":0,"allocated":1920790528,"data_stored":1919467850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931199320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561383,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4157308,"kb_used_data":3255476,"kb_used_omap":1,"kb_used_meta":901822,"kb_avail":89587844,"statfs":{"total":95995035648,"available":91737952256,"internally_reserved":0,"allocated":3333607424,"data_stored":3332292761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923466072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790315,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2517492,"kb_used_data":1616428,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":91227660,"statfs":{"total":95995035648,"available":93417123840,"internally_reserved":0,"allocated":1655222272,"data_stored":1653911919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986543,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5281536,"kb_used_data":4378808,"kb_used_omap":1,"kb_used_meta":902718,"kb_avail":88463616,"statfs":{"total":95995035648,"available":90586742784,"internally_reserved":0,"allocated":4483899392,"data_stored":4482580971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924383576},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519699,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2643956,"kb_used_data":2127660,"kb_used_omap":4,"kb_used_meta":516219,"kb_avail":91101196,"statfs":{"total":95995035648,"available":93287624704,"internally_reserved":0,"allocated":2178723840,"data_stored":2177413497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":528608704},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304614,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3641528,"kb_used_data":3172080,"kb_used_omap":6,"kb_used_meta":469369,"kb_avail":90103624,"statfs":{"total":95995035648,"available":92266110976,"internally_reserved":0,"allocated":3248209920,"data_stored":3246895995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":480634311},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827918,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2177524,"kb_used_data":1383468,"kb_used_omap":5,"kb_used_meta":794042,"kb_avail":91567628,"statfs":{"total":95995035648,"available":93765251072,"internally_reserved":0,"allocated":1416671232,"data_stored":1415360316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":813099244},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285782,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5339828,"kb_used_data":4678508,"kb_used_omap":5,"kb_used_meta":661306,"kb_avail":88405324,"statfs":{"total":95995035648,"available":90527051776,"internally_reserved":0,"allocated":4790792192,"data_stored":4789480603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":677177403},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4764078080,"data_stored":4764074008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1410138112,"data_stored":1410138112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3248095232,"data_stored":3248095232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2168651776,"data_stored":2168651776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4480696320,"data_stored":4480696320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1636040704,"data_stored":1636040704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3332571136,"data_stored":3332571136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1915420672,"data_stored":1915420672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:49:15.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:49:15.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:15.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:15.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:15.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:15.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:15.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:15.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:15.813 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:15.916 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:15.949 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:15.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:16.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:16.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:16.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:16.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:15 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8177/349917 objects degraded (2.337%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:16.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:15 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8177/349917 objects degraded (2.337%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:16.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:15 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8177/349917 objects degraded (2.337%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:16.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:16.217 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:16.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:16.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:16.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:16.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:16.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:16.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:16.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:16.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:16.645 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:16.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:16.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:16.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:16.887 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:16.926 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:18282 9.e 9:7d5ed525:::benchmark_data_smithi003_156985_object18281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:41:58.603938+0000", 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 438.31751904100003, 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.250542296, 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 18282 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.603938+0000", 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.603938+0000", 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.603937+0000", 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604002+0000", 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5355e-05 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604017+0000", 2024-09-13T14:49:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4985e-05 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604023+0000", 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.161e-06 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604125+0000", 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101649 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604363+0000", 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000238772 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604578+0000", 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00021494299999999999 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.606902+0000", 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0023234969999999999 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.607164+0000", 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00026249200000000002 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.854323+0000", 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24715848200000001 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.854429+0000", 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106054 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.854480+0000", 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1465000000000003e-05 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:18285 9.e 9:74546ba7:::benchmark_data_smithi003_156985_object18284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:41:58.604780+0000", 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 438.31667680999999, 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25160389599999999, 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 18285 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604780+0000", 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604780+0000", 2024-09-13T14:49:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604779+0000", 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604854+0000", 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4584000000000003e-05 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604867+0000", 2024-09-13T14:49:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3027000000000001e-05 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.604873+0000", 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.02e-06 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.605694+0000", 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000821151 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.605797+0000", 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000103044 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.605942+0000", 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014505599999999999 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.607340+0000", 2024-09-13T14:49:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001398182 2024-09-13T14:49:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.608675+0000", 2024-09-13T14:49:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0013346810000000001 2024-09-13T14:49:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856283+0000", 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24760869199999999 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856341+0000", 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7840000000000002e-05 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856384+0000", 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2929000000000002e-05 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:18308 9.e 9:785346ca:::benchmark_data_smithi003_156985_object18307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:41:58.613764+0000", 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 438.30769295200002, 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24282445599999999, 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 18308 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.613764+0000", 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.613764+0000", 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.613762+0000", 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:49:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.613816+0000", 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3207000000000001e-05 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.613829+0000", 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3181e-05 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.613835+0000", 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8579999999999996e-06 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.613914+0000", 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8988999999999999e-05 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.614032+0000", 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118492 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.614174+0000", 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014204500000000001 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.615174+0000", 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00099961100000000008 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.615896+0000", 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00072256000000000002 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856459+0000", 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24056276800000001 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856551+0000", 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2158999999999996e-05 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856589+0000", 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7391000000000002e-05 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:18328 9.e 9:759d95eb:::benchmark_data_smithi003_156985_object18327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:41:58.623313+0000", 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 438.29814407700002, 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.23352804099999999, 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 18328 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.623313+0000", 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.623313+0000", 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.623312+0000", 2024-09-13T14:49:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.623364+0000", 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1629000000000003e-05 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.623377+0000", 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3137e-05 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.623382+0000", 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7919999999999998e-06 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.623432+0000", 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9296999999999999e-05 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.623568+0000", 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013608399999999999 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.623717+0000", 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014905699999999999 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.624504+0000", 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00078739599999999999 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.624749+0000", 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00024515499999999999 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856624+0000", 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.23187437 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856771+0000", 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014744399999999999 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856841+0000", 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9876000000000007e-05 2024-09-13T14:49:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:18331 9.e 9:7342a897:::benchmark_data_smithi003_156985_object18330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:41:58.627225+0000", 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 438.294232378, 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22979179999999999, 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 18331 2024-09-13T14:49:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.627225+0000", 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.627225+0000", 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.627224+0000", 2024-09-13T14:49:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.627284+0000", 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0109000000000003e-05 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.627293+0000", 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8749999999999995e-06 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.627298+0000", 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3229999999999999e-06 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.627333+0000", 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5290999999999998e-05 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.627415+0000", 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1494000000000003e-05 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:49:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.627497+0000", 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2634999999999997e-05 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.628126+0000", 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00062870999999999999 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.628430+0000", 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00030430799999999999 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856883+0000", 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22845262799999999 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.856980+0000", 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7264999999999998e-05 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:41:58.857017+0000", 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6446000000000003e-05 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:120826 9.d 9:b8f324f2:::benchmark_data_smithi003_156985_object120825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:18.899984+0000", 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 358.02147345499998, 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.39272128, 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 120826 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.899984+0000", 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.899984+0000", 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.899982+0000", 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.900025+0000", 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2503000000000003e-05 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.900037+0000", 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2221e-05 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.900043+0000", 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6749999999999999e-06 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.900101+0000", 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8396999999999997e-05 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:18.900114+0000", 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2652e-05 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.856330+0000", 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95621645899999996 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:19.856339+0000", 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6319999999999997e-06 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.213844+0000", 2024-09-13T14:49:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.357505368 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.213957+0000", 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000112639 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.214504+0000", 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00054692399999999996 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.224970+0000", 2024-09-13T14:49:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010466098 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.288925+0000", 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.063954895999999997 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.288982+0000", 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6855999999999999e-05 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.289023+0000", 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1158000000000001e-05 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.289093+0000", 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9874000000000002e-05 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:21.292705+0000", 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036122459999999999 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:183147 9.d 9:b0cef76e:::benchmark_data_smithi003_156985_object62231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:13.245281+0000", 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 243.67617621799999, 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.17058837900000001, 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183147 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.245281+0000", 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.245281+0000", 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.245280+0000", 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.245287+0000", 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9809999999999997e-06 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.245301+0000", 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4470999999999999e-05 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.245308+0000", 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.55e-06 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.245365+0000", 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7408000000000003e-05 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.245486+0000", 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000121242 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.245664+0000", 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000177286 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.248509+0000", 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028456200000000001 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.248672+0000", 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016315700000000001 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.248813+0000", 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014043899999999999 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.411621+0000", 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.162808392 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.411723+0000", 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101798 2024-09-13T14:49:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.415869+0000", 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0041464179999999998 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:183153 9.d 9:ba992324:::benchmark_data_smithi003_156985_object62237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:13.258316+0000", 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 243.66314133200001, 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.16372188400000001, 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183153 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.258316+0000", 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.258316+0000", 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.258315+0000", 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.258320+0000", 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8510000000000004e-06 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.258334+0000", 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.393e-05 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.258339+0000", 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9139999999999999e-06 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.258381+0000", 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1464000000000001e-05 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.258457+0000", 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6323999999999996e-05 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.258556+0000", 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.9401000000000006e-05 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.261210+0000", 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026532999999999999 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.261330+0000", 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120437 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.261907+0000", 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00057705799999999998 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.420614+0000", 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.15870715299999999 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.420648+0000", 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3352999999999997e-05 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:13.422038+0000", 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0013899959999999999 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:194172 9.3 9:c26644b1:::benchmark_data_smithi003_156985_object73256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T14:49:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:29.702112+0000", 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 227.21934505199999, 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.14174815399999999, 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 194172 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.702112+0000", 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.702112+0000", 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.702111+0000", 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.702118+0000", 2024-09-13T14:49:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5459999999999997e-06 2024-09-13T14:49:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.702131+0000", 2024-09-13T14:49:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3797e-05 2024-09-13T14:49:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.702138+0000", 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3820000000000001e-06 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.702192+0000", 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3797e-05 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.702306+0000", 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011403100000000001 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.702908+0000", 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00060236800000000004 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.706599+0000", 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036905879999999999 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.710267+0000", 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036683039999999998 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.843779+0000", 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13351207300000001 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.843825+0000", 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6381999999999997e-05 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:29.843860+0000", 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4940000000000001e-05 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:199807 9.e 9:7097fff2:::benchmark_data_smithi003_156985_object78891:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:38.226113+0000", 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 218.69534381, 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.17008457499999999, 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 199807 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.226113+0000", 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.226113+0000", 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.226113+0000", 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.226117+0000", 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2039999999999999e-06 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.226134+0000", 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7261e-05 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.226138+0000", 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2869999999999998e-06 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.228687+0000", 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0025490170000000002 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.228789+0000", 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101501 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.228899+0000", 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000109724 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.231634+0000", 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002735347 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.233743+0000", 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0021091980000000001 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.396119+0000", 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.16237626799999999 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.396164+0000", 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4932000000000003e-05 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.396198+0000", 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3562000000000002e-05 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:199815 9.e 9:71480e20:::benchmark_data_smithi003_156985_object78899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:45:38.234830+0000", 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 218.686627262, 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.17478075400000001, 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:16.994 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 199815 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.234830+0000", 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.234830+0000", 2024-09-13T14:49:16.995 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.234829+0000", 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.234835+0000", 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5379999999999999e-06 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.234848+0000", 2024-09-13T14:49:16.996 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2927e-05 2024-09-13T14:49:16.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.997 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:16.997 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.234854+0000", 2024-09-13T14:49:16.997 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6269999999999997e-06 2024-09-13T14:49:16.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.997 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.999 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.234906+0000", 2024-09-13T14:49:16.999 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2898000000000001e-05 2024-09-13T14:49:16.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:16.999 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:16.999 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:16.999 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.235050+0000", 2024-09-13T14:49:16.999 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014317799999999999 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.235148+0000", 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8109999999999994e-05 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.238301+0000", 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0031533899999999998 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.000 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.238326+0000", 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4584e-05 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.409420+0000", 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.17109429300000001 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.409577+0000", 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015713200000000001 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.001 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:45:38.409611+0000", 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3485999999999998e-05 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:240194 9.a 9:5533597d:::benchmark_data_smithi003_156985_object119278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:35.190849+0000", 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 161.730607967, 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.199780191, 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:17.002 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 240194 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.190849+0000", 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.190849+0000", 2024-09-13T14:49:17.003 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.190848+0000", 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.190855+0000", 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.99e-06 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.004 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.190867+0000", 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2213e-05 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.190871+0000", 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5260000000000004e-06 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.190928+0000", 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6440999999999997e-05 2024-09-13T14:49:17.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.191093+0000", 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016545199999999999 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.191257+0000", 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016358900000000001 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.006 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.198521+0000", 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0072638709999999999 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.199278+0000", 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00075733199999999999 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.389186+0000", 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.189907932 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.007 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.389240+0000", 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4148999999999999e-05 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.390629+0000", 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0013892539999999999 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.008 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:240200 9.e 9:7225e426:::benchmark_data_smithi003_156985_object119284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:35.198735+0000", 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 161.722722248, 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19391354299999999, 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 240200 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.009 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.198735+0000", 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.198735+0000", 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.198734+0000", 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:17.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.198740+0000", 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6109999999999999e-06 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.198754+0000", 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3658e-05 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.011 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.198760+0000", 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.19e-06 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.198808+0000", 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7822999999999999e-05 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.198986+0000", 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000178491 2024-09-13T14:49:17.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.199103+0000", 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011692 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.201273+0000", 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0021694069999999999 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.013 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.201792+0000", 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00051963599999999997 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.392561+0000", 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.190768615 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.392606+0000", 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4994000000000003e-05 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.392648+0000", 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2579e-05 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:240203 9.e 9:7a9fbd83:::benchmark_data_smithi003_156985_object119287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:35.202630+0000", 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 161.71882687999999, 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19133560199999999, 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:17.015 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 240203 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.202630+0000", 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:17.016 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.202630+0000", 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.202629+0000", 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.202635+0000", 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4640000000000003e-06 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.202647+0000", 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1970000000000001e-05 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.202654+0000", 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6429999999999998e-06 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.202702+0000", 2024-09-13T14:49:17.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8004000000000003e-05 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.202789+0000", 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6625999999999994e-05 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.202855+0000", 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6879999999999997e-05 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.019 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.204902+0000", 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0020465560000000002 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.205406+0000", 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00050421600000000004 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.393854+0000", 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.188447477 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.393931+0000", 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7395e-05 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:35.393966+0000", 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.481e-05 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.021 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:29532 10.c 10:3c85fbb1:::benchmark_data_smithi003_197468_object29531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:47:49.076778+0000", 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 87.844679619999994, 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.17388089200000001, 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 29532 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:17.022 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.076778+0000", 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.076778+0000", 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.076776+0000", 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:17.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.076840+0000", 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4115000000000005e-05 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.076856+0000", 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5322e-05 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.076862+0000", 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0979999999999999e-06 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.024 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.077372+0000", 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00051064000000000003 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.077499+0000", 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000126737 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.077635+0000", 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013601100000000001 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.025 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.079175+0000", 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0015403630000000001 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.081179+0000", 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002003459 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.250562+0000", 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.169382747 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.250614+0000", 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2274000000000001e-05 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:49.250658+0000", 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4610000000000001e-05 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:81453 10.c 10:3afdaa1e:::benchmark_data_smithi003_197468_object81452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:26.147925+0000", 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 50.773532230000001, 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.18169099499999999, 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 81453 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.147925+0000", 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.147925+0000", 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.147923+0000", 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.147991+0000", 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8677000000000007e-05 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.148016+0000", 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4434e-05 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.148025+0000", 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3989999999999993e-06 2024-09-13T14:49:17.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.148086+0000", 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0819999999999997e-05 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.148225+0000", 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013863499999999999 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.148379+0000", 2024-09-13T14:49:17.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000154637 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.153026+0000", 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0046471630000000002 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.153612+0000", 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00058589999999999998 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.032 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.329539+0000", 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.175926896 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.329574+0000", 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4292999999999998e-05 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:26.329616+0000", 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2354999999999997e-05 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.033 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:118405 10.c 10:35a6bf4a:::benchmark_data_smithi003_197468_object1676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:11.337433+0000", 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 5.5840244139999999, 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24404626400000001, 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 118405 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:17.034 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.337433+0000", 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.337433+0000", 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.337431+0000", 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.337440+0000", 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2359999999999993e-06 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.337457+0000", 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7178999999999999e-05 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.337465+0000", 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.157e-06 2024-09-13T14:49:17.036 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.337562+0000", 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7027999999999993e-05 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.337700+0000", 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013806699999999999 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.337843+0000", 2024-09-13T14:49:17.037 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014251199999999999 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.341524+0000", 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036806170000000002 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.342387+0000", 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00086310100000000002 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.581408+0000", 2024-09-13T14:49:17.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.239021702 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.581442+0000", 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3705999999999999e-05 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.581479+0000", 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6968000000000002e-05 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:118413 10.c 10:3130e482:::benchmark_data_smithi003_197468_object1684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:11.386387+0000", 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 5.5350699079999996, 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19517447099999999, 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 118413 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.040 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.386387+0000", 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.386387+0000", 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.386385+0000", 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.386396+0000", 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0866999999999999e-05 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.386419+0000", 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3354e-05 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.386428+0000", 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5490000000000006e-06 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.386485+0000", 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7006999999999998e-05 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.386750+0000", 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00026549700000000001 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.387160+0000", 2024-09-13T14:49:17.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00041002999999999998 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.407466+0000", 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.020305634999999999 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.407783+0000", 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00031742899999999999 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.581513+0000", 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.17372948599999999 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.581541+0000", 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7903999999999999e-05 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.581562+0000", 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0959e-05 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.045 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:118416 10.c 10:3702fc87:::benchmark_data_smithi003_197468_object1687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:11.387517+0000", 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 5.5339399939999998, 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.204410496, 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 118416 2024-09-13T14:49:17.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.387517+0000", 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.387517+0000", 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:17.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.387516+0000", 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.387522+0000", 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8749999999999997e-06 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.387535+0000", 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2554e-05 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:17.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.387539+0000", 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8239999999999999e-06 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.393763+0000", 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0062235850000000002 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.394056+0000", 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00029345400000000001 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:49:17.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.394423+0000", 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00036616200000000001 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.407974+0000", 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013551425000000001 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.408018+0000", 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3754000000000002e-05 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.591797+0000", 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.18377890099999999 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.591873+0000", 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.648e-05 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.591928+0000", 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4478999999999998e-05 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:118419 10.c 10:3e18bbc4:::benchmark_data_smithi003_197468_object1690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:11.388050+0000", 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 5.5334066870000003, 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.204481102, 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 118419 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:49:17.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.388050+0000", 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.388050+0000", 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.388050+0000", 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:49:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:17.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:17.054 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:16 smithi003 ceph-mon[107278]: pgmap v832: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 3.5 KiB/s rd, 20 MiB/s wr, 712 op/s; 8200/338775 objects degraded (2.420%); 5527/338775 objects misplaced (1.631%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:49:17.055 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:16 smithi003 ceph-mon[104007]: pgmap v832: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 3.5 KiB/s rd, 20 MiB/s wr, 712 op/s; 8200/338775 objects degraded (2.420%); 5527/338775 objects misplaced (1.631%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:49:17.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.388055+0000", 2024-09-13T14:49:17.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2440000000000001e-06 2024-09-13T14:49:17.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.055 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:49:17.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.388070+0000", 2024-09-13T14:49:17.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4693e-05 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.388076+0000", 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1469999999999998e-06 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.402151+0000", 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014075331 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.402411+0000", 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00025964899999999998 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.402895+0000", 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00048476500000000001 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.408042+0000", 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0051460589999999997 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.409173+0000", 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001131092 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.592455+0000", 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.18328230700000001 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.592494+0000", 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9453000000000003e-05 2024-09-13T14:49:17.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:11.592532+0000", 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7211e-05 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:49:17.059 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:49:17.156 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:17.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:16 smithi019 ceph-mon[91296]: pgmap v832: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 3.5 KiB/s rd, 20 MiB/s wr, 712 op/s; 8200/338775 objects degraded (2.420%); 5527/338775 objects misplaced (1.631%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:49:17.196 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:17.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:17.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:17.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:17.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.582 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:17.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:17.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:17.725 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:17.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:17.865 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:17.894 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:49:17.967 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:18.003 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:18.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:18.152 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:18.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:17 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4263523162' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:18.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:17 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:18.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:17 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4263523162' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:18.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:17 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:18.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:17 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4263523162' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:18.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:17 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:18.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:18.289 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:18.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:18.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:18.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:49:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.463 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:18.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:18.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:18.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:18.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:18.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:18.872 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:18.896 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:18.998 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.037 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:19.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:19.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:18 smithi019 ceph-mon[91296]: pgmap v833: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 596 op/s; 8200/338775 objects degraded (2.420%); 5527/338775 objects misplaced (1.631%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:49:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:18 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:18 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:18 smithi019 ceph-mon[91296]: osdmap e254: 8 total, 8 up, 8 in 2024-09-13T14:49:19.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:18 smithi003 ceph-mon[104007]: pgmap v833: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 596 op/s; 8200/338775 objects degraded (2.420%); 5527/338775 objects misplaced (1.631%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:49:19.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:18 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:19.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:18 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:19.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:18 smithi003 ceph-mon[104007]: osdmap e254: 8 total, 8 up, 8 in 2024-09-13T14:49:19.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:18 smithi003 ceph-mon[107278]: pgmap v833: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 596 op/s; 8200/338775 objects degraded (2.420%); 5527/338775 objects misplaced (1.631%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:49:19.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:18 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:19.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:18 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:19.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:18 smithi003 ceph-mon[107278]: osdmap e254: 8 total, 8 up, 8 in 2024-09-13T14:49:19.274 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:19.314 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:19.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:19.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:19.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:19.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:19.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:19.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:19.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.873 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:19.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:20.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:20.110 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:20.155 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:20.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.293 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.395 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:20.432 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:20.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:20.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:20.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:49:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:49:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:20.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:20.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:20.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:20.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:21.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:21.128 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:21.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:21.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:20 smithi003 ceph-mon[104007]: pgmap v835: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 2.0 MiB/s wr, 769 op/s; 8354/328077 objects degraded (2.546%); 5270/328077 objects misplaced (1.606%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:21.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:20 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8354/328077 objects degraded (2.546%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:21.174 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:20 smithi003 ceph-mon[107278]: pgmap v835: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 2.0 MiB/s wr, 769 op/s; 8354/328077 objects degraded (2.546%); 5270/328077 objects misplaced (1.606%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:21.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:20 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8354/328077 objects degraded (2.546%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:21.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:20 smithi019 ceph-mon[91296]: pgmap v835: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 2.0 MiB/s wr, 769 op/s; 8354/328077 objects degraded (2.546%); 5270/328077 objects misplaced (1.606%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:21.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:20 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8354/328077 objects degraded (2.546%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:21.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:21.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:21.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:21.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:21.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:21.661 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:21.699 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:21.802 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.841 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:21.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:21.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:21.985 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:21.995 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:49:21.996 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:21.997 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:49:21.997 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:22.047 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:49:22.048 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:22.083 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:49:22.083 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:22.087 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:22.114 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:49:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:22.116 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:22.145 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:49:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:22.186 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:49:22.186 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:22.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:22.223 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:49:22.224 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:22.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:22.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:49:22.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:22.433 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:49:22.438 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:49:22.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:22.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:22.577 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:22.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:22.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:22.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:22.857 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:22.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:22.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:22.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:23.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:23.123 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:22 smithi019 ceph-mon[91296]: pgmap v836: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 780 op/s; 8354/326502 objects degraded (2.559%); 5270/326502 objects misplaced (1.614%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:23.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1658527345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:23.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:22 smithi003 ceph-mon[104007]: pgmap v836: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 780 op/s; 8354/326502 objects degraded (2.559%); 5270/326502 objects misplaced (1.614%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1658527345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:22 smithi003 ceph-mon[107278]: pgmap v836: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 780 op/s; 8354/326502 objects degraded (2.559%); 5270/326502 objects misplaced (1.614%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1658527345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:23.225 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:23.259 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:23.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:23.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:23.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:23.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:23.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:23.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:23.646 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":836,"stamp":"2024-09-13T14:49:21.761794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7131398328,"num_objects":108834,"num_object_clones":0,"num_object_copies":326502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8354,"num_objects_misplaced":5270,"num_objects_unfound":0,"num_objects_dirty":108834,"num_whiteouts":0,"num_read":15,"num_read_kb":35,"num_write":124717,"num_write_kb":7470571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":136015409,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113091,"ondisk_log_size":113091,"up":327,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":333,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":27268284,"kb_used_data":21207920,"kb_used_omap":29,"kb_used_meta":6060066,"kb_avail":722692932,"statfs":{"total":767960285184,"available":740037562368,"internally_reserved":0,"allocated":21716910080,"data_stored":21706342494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6205508110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,5,12,5,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-511573992,"num_objects":-7805,"num_object_clones":0,"num_object_copies":-23415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":177,"num_objects_misplaced":-404,"num_objects_unfound":0,"num_objects_dirty":-7805,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7807,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":14876672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003907"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":386,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.905381+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:49:17.905381+0000","last_peered":"2024-09-13T14:49:17.905381+0000","last_clean":"2024-09-13T14:49:17.905381+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:49:17.905381+0000","last_undegraded":"2024-09-13T14:49:17.905381+0000","last_fullsized":"2024-09-13T14:49:17.905381+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":226,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.821770+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:49:05.821770+0000","last_peered":"2024-09-13T14:49:05.821770+0000","last_clean":"2024-09-13T14:49:05.821770+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:49:05.821770+0000","last_undegraded":"2024-09-13T14:49:05.821770+0000","last_fullsized":"2024-09-13T14:49:05.821770+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815152+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:49:05.815152+0000","last_peered":"2024-09-13T14:49:05.815152+0000","last_clean":"2024-09-13T14:49:05.815152+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:49:05.815152+0000","last_undegraded":"2024-09-13T14:49:05.815152+0000","last_fullsized":"2024-09-13T14:49:05.815152+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":343,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.817620+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:49:05.817620+0000","last_peered":"2024-09-13T14:49:05.817620+0000","last_clean":"2024-09-13T14:49:05.817620+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:49:05.817620+0000","last_undegraded":"2024-09-13T14:49:05.817620+0000","last_fullsized":"2024-09-13T14:49:05.817620+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":200,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814059+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:49:05.814059+0000","last_peered":"2024-09-13T14:49:05.814059+0000","last_clean":"2024-09-13T14:49:05.814059+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:49:05.814059+0000","last_undegraded":"2024-09-13T14:49:05.814059+0000","last_fullsized":"2024-09-13T14:49:05.814059+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":233,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.824251+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:49:05.824251+0000","last_peered":"2024-09-13T14:49:05.824251+0000","last_clean":"2024-09-13T14:49:05.824251+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:49:05.824251+0000","last_undegraded":"2024-09-13T14:49:05.824251+0000","last_fullsized":"2024-09-13T14:49:05.824251+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":217,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.816142+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:49:05.816142+0000","last_peered":"2024-09-13T14:49:05.816142+0000","last_clean":"2024-09-13T14:49:05.816142+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:49:05.816142+0000","last_undegraded":"2024-09-13T14:49:05.816142+0000","last_fullsized":"2024-09-13T14:49:05.816142+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":263,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.823291+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:49:05.823291+0000","last_peered":"2024-09-13T14:49:05.823291+0000","last_clean":"2024-09-13T14:49:05.823291+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:49:05.823291+0000","last_undegraded":"2024-09-13T14:49:05.823291+0000","last_fullsized":"2024-09-13T14:49:05.823291+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":374,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.903562+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:49:17.903562+0000","last_peered":"2024-09-13T14:49:17.903562+0000","last_clean":"2024-09-13T14:49:17.903562+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:49:17.903562+0000","last_undegraded":"2024-09-13T14:49:17.903562+0000","last_fullsized":"2024-09-13T14:49:17.903562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":369,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.911685+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:49:17.911685+0000","last_peered":"2024-09-13T14:49:17.911685+0000","last_clean":"2024-09-13T14:49:17.911685+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:49:17.911685+0000","last_undegraded":"2024-09-13T14:49:17.911685+0000","last_fullsized":"2024-09-13T14:49:17.911685+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":235,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.824393+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:49:05.824393+0000","last_peered":"2024-09-13T14:49:05.824393+0000","last_clean":"2024-09-13T14:49:05.824393+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:49:05.824393+0000","last_undegraded":"2024-09-13T14:49:05.824393+0000","last_fullsized":"2024-09-13T14:49:05.824393+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":309,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.818628+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:49:05.818628+0000","last_peered":"2024-09-13T14:49:05.818628+0000","last_clean":"2024-09-13T14:49:05.818628+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:49:05.818628+0000","last_undegraded":"2024-09-13T14:49:05.818628+0000","last_fullsized":"2024-09-13T14:49:05.818628+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.913925+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:49:17.913925+0000","last_peered":"2024-09-13T14:49:17.913925+0000","last_clean":"2024-09-13T14:49:17.913925+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:49:17.913925+0000","last_undegraded":"2024-09-13T14:49:17.913925+0000","last_fullsized":"2024-09-13T14:49:17.913925+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":404,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.904929+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:49:17.904929+0000","last_peered":"2024-09-13T14:49:17.904929+0000","last_clean":"2024-09-13T14:49:17.904929+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:49:17.904929+0000","last_undegraded":"2024-09-13T14:49:17.904929+0000","last_fullsized":"2024-09-13T14:49:17.904929+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":342,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.816147+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:49:05.816147+0000","last_peered":"2024-09-13T14:49:05.816147+0000","last_clean":"2024-09-13T14:49:05.816147+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:49:05.816147+0000","last_undegraded":"2024-09-13T14:49:05.816147+0000","last_fullsized":"2024-09-13T14:49:05.816147+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":216,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814297+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:49:05.814297+0000","last_peered":"2024-09-13T14:49:05.814297+0000","last_clean":"2024-09-13T14:49:05.814297+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:49:05.814297+0000","last_undegraded":"2024-09-13T14:49:05.814297+0000","last_fullsized":"2024-09-13T14:49:05.814297+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":310,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815563+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:49:05.815563+0000","last_peered":"2024-09-13T14:49:05.815563+0000","last_clean":"2024-09-13T14:49:05.815563+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:49:05.815563+0000","last_undegraded":"2024-09-13T14:49:05.815563+0000","last_fullsized":"2024-09-13T14:49:05.815563+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":388,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.905637+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:49:17.905637+0000","last_peered":"2024-09-13T14:49:17.905637+0000","last_clean":"2024-09-13T14:49:17.905637+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:49:17.905637+0000","last_undegraded":"2024-09-13T14:49:17.905637+0000","last_fullsized":"2024-09-13T14:49:17.905637+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":315,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.828755+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:49:05.828755+0000","last_peered":"2024-09-13T14:49:05.828755+0000","last_clean":"2024-09-13T14:49:05.828755+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:49:05.828755+0000","last_undegraded":"2024-09-13T14:49:05.828755+0000","last_fullsized":"2024-09-13T14:49:05.828755+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815069+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:49:05.815069+0000","last_peered":"2024-09-13T14:49:05.815069+0000","last_clean":"2024-09-13T14:49:05.815069+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:49:05.815069+0000","last_undegraded":"2024-09-13T14:49:05.815069+0000","last_fullsized":"2024-09-13T14:49:05.815069+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":66,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.911650+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:49:17.911650+0000","last_peered":"2024-09-13T14:49:17.911650+0000","last_clean":"2024-09-13T14:49:17.911650+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:49:17.911650+0000","last_undegraded":"2024-09-13T14:49:17.911650+0000","last_fullsized":"2024-09-13T14:49:17.911650+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.857008+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:49:05.857008+0000","last_peered":"2024-09-13T14:49:05.857008+0000","last_clean":"2024-09-13T14:49:05.857008+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:49:05.857008+0000","last_undegraded":"2024-09-13T14:49:05.857008+0000","last_fullsized":"2024-09-13T14:49:05.857008+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":155,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.813671+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:49:05.813671+0000","last_peered":"2024-09-13T14:49:05.813671+0000","last_clean":"2024-09-13T14:49:05.813671+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:49:05.813671+0000","last_undegraded":"2024-09-13T14:49:05.813671+0000","last_fullsized":"2024-09-13T14:49:05.813671+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":223,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.823326+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:49:05.823326+0000","last_peered":"2024-09-13T14:49:05.823326+0000","last_clean":"2024-09-13T14:49:05.823326+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:49:05.823326+0000","last_undegraded":"2024-09-13T14:49:05.823326+0000","last_fullsized":"2024-09-13T14:49:05.823326+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":216,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.823367+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:49:05.823367+0000","last_peered":"2024-09-13T14:49:05.823367+0000","last_clean":"2024-09-13T14:49:05.823367+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:49:05.823367+0000","last_undegraded":"2024-09-13T14:49:05.823367+0000","last_fullsized":"2024-09-13T14:49:05.823367+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"254'7698","reported_seq":7731,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:18.831618+0000","last_change":"2024-09-13T14:47:20.107455+0000","last_active":"2024-09-13T14:49:18.831618+0000","last_peered":"2024-09-13T14:49:18.831618+0000","last_clean":"2024-09-13T14:49:18.831618+0000","last_became_active":"2024-09-13T14:47:20.107142+0000","last_became_peered":"2024-09-13T14:47:20.107142+0000","last_unstale":"2024-09-13T14:49:18.831618+0000","last_undegraded":"2024-09-13T14:49:18.831618+0000","last_fullsized":"2024-09-13T14:49:18.831618+0000","mapping_epoch":229,"log_start":"237'1000","ondisk_log_start":"237'1000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6698,"log_dups_size":0,"ondisk_log_size":6698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:19.189519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443678720,"num_objects":6770,"num_object_clones":0,"num_object_copies":20310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7698,"num_write_kb":462976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":199,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.822011+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:49:05.822011+0000","last_peered":"2024-09-13T14:49:05.822011+0000","last_clean":"2024-09-13T14:49:05.822011+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:49:05.822011+0000","last_undegraded":"2024-09-13T14:49:05.822011+0000","last_fullsized":"2024-09-13T14:49:05.822011+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":249,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.834223+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:49:05.834223+0000","last_peered":"2024-09-13T14:49:05.834223+0000","last_clean":"2024-09-13T14:49:05.834223+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:49:05.834223+0000","last_undegraded":"2024-09-13T14:49:05.834223+0000","last_fullsized":"2024-09-13T14:49:05.834223+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":386,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.905111+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:49:17.905111+0000","last_peered":"2024-09-13T14:49:17.905111+0000","last_clean":"2024-09-13T14:49:17.905111+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:49:17.905111+0000","last_undegraded":"2024-09-13T14:49:17.905111+0000","last_fullsized":"2024-09-13T14:49:17.905111+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":72,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815591+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:49:05.815591+0000","last_peered":"2024-09-13T14:49:05.815591+0000","last_clean":"2024-09-13T14:49:05.815591+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:49:05.815591+0000","last_undegraded":"2024-09-13T14:49:05.815591+0000","last_fullsized":"2024-09-13T14:49:05.815591+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"254'7808","reported_seq":7841,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:19.020663+0000","last_change":"2024-09-13T14:47:20.054114+0000","last_active":"2024-09-13T14:49:19.020663+0000","last_peered":"2024-09-13T14:49:19.020663+0000","last_clean":"2024-09-13T14:49:19.020663+0000","last_became_active":"2024-09-13T14:47:20.053767+0000","last_became_peered":"2024-09-13T14:47:20.053767+0000","last_unstale":"2024-09-13T14:49:19.020663+0000","last_undegraded":"2024-09-13T14:49:19.020663+0000","last_fullsized":"2024-09-13T14:49:19.020663+0000","mapping_epoch":229,"log_start":"237'1500","ondisk_log_start":"237'1500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:48.035191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445251584,"num_objects":6794,"num_object_clones":0,"num_object_copies":20382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7808,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":353,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.860180+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:49:05.860180+0000","last_peered":"2024-09-13T14:49:05.860180+0000","last_clean":"2024-09-13T14:49:05.860180+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:49:05.860180+0000","last_undegraded":"2024-09-13T14:49:05.860180+0000","last_fullsized":"2024-09-13T14:49:05.860180+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814287+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:49:05.814287+0000","last_peered":"2024-09-13T14:49:05.814287+0000","last_clean":"2024-09-13T14:49:05.814287+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:49:05.814287+0000","last_undegraded":"2024-09-13T14:49:05.814287+0000","last_fullsized":"2024-09-13T14:49:05.814287+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":341,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.826552+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:49:05.826552+0000","last_peered":"2024-09-13T14:49:05.826552+0000","last_clean":"2024-09-13T14:49:05.826552+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:49:05.826552+0000","last_undegraded":"2024-09-13T14:49:05.826552+0000","last_fullsized":"2024-09-13T14:49:05.826552+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":37,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.824457+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:49:05.824457+0000","last_peered":"2024-09-13T14:49:05.824457+0000","last_clean":"2024-09-13T14:49:05.824457+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:49:05.824457+0000","last_undegraded":"2024-09-13T14:49:05.824457+0000","last_fullsized":"2024-09-13T14:49:05.824457+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"254'7775","reported_seq":7809,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:18.837562+0000","last_change":"2024-09-13T14:47:20.052876+0000","last_active":"2024-09-13T14:49:18.837562+0000","last_peered":"2024-09-13T14:49:18.837562+0000","last_clean":"2024-09-13T14:49:18.837562+0000","last_became_active":"2024-09-13T14:47:20.052566+0000","last_became_peered":"2024-09-13T14:47:20.052566+0000","last_unstale":"2024-09-13T14:49:18.837562+0000","last_undegraded":"2024-09-13T14:49:18.837562+0000","last_fullsized":"2024-09-13T14:49:18.837562+0000","mapping_epoch":229,"log_start":"237'1100","ondisk_log_start":"237'1100","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6675,"log_dups_size":0,"ondisk_log_size":6675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:56.317980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449052696,"num_objects":6853,"num_object_clones":0,"num_object_copies":20559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6853,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7775,"num_write_kb":468033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":372,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.903477+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:49:17.903477+0000","last_peered":"2024-09-13T14:49:17.903477+0000","last_clean":"2024-09-13T14:49:17.903477+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:49:17.903477+0000","last_undegraded":"2024-09-13T14:49:17.903477+0000","last_fullsized":"2024-09-13T14:49:17.903477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":229,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815330+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:49:05.815330+0000","last_peered":"2024-09-13T14:49:05.815330+0000","last_clean":"2024-09-13T14:49:05.815330+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:49:05.815330+0000","last_undegraded":"2024-09-13T14:49:05.815330+0000","last_fullsized":"2024-09-13T14:49:05.815330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":327,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.911576+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:49:17.911576+0000","last_peered":"2024-09-13T14:49:17.911576+0000","last_clean":"2024-09-13T14:49:17.911576+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:49:17.911576+0000","last_undegraded":"2024-09-13T14:49:17.911576+0000","last_fullsized":"2024-09-13T14:49:17.911576+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":190,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814022+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:49:05.814022+0000","last_peered":"2024-09-13T14:49:05.814022+0000","last_clean":"2024-09-13T14:49:05.814022+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:49:05.814022+0000","last_undegraded":"2024-09-13T14:49:05.814022+0000","last_fullsized":"2024-09-13T14:49:05.814022+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"254'7848","reported_seq":7881,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:18.949324+0000","last_change":"2024-09-13T14:47:20.052419+0000","last_active":"2024-09-13T14:49:18.949324+0000","last_peered":"2024-09-13T14:49:18.949324+0000","last_clean":"2024-09-13T14:49:18.949324+0000","last_became_active":"2024-09-13T14:47:20.051845+0000","last_became_peered":"2024-09-13T14:47:20.051845+0000","last_unstale":"2024-09-13T14:49:18.949324+0000","last_undegraded":"2024-09-13T14:49:18.949324+0000","last_fullsized":"2024-09-13T14:49:18.949324+0000","mapping_epoch":229,"log_start":"237'1014","ondisk_log_start":"237'1014","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6834,"log_dups_size":0,"ondisk_log_size":6834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.420332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449052672,"num_objects":6852,"num_object_clones":0,"num_object_copies":20556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814471+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:49:05.814471+0000","last_peered":"2024-09-13T14:49:05.814471+0000","last_clean":"2024-09-13T14:49:05.814471+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:49:05.814471+0000","last_undegraded":"2024-09-13T14:49:05.814471+0000","last_fullsized":"2024-09-13T14:49:05.814471+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":185,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.914301+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:49:17.914301+0000","last_peered":"2024-09-13T14:49:17.914301+0000","last_clean":"2024-09-13T14:49:17.914301+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:49:17.914301+0000","last_undegraded":"2024-09-13T14:49:17.914301+0000","last_fullsized":"2024-09-13T14:49:17.914301+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":232,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814758+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:49:05.814758+0000","last_peered":"2024-09-13T14:49:05.814758+0000","last_clean":"2024-09-13T14:49:05.814758+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:49:05.814758+0000","last_undegraded":"2024-09-13T14:49:05.814758+0000","last_fullsized":"2024-09-13T14:49:05.814758+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":210,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.839922+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:49:05.839922+0000","last_peered":"2024-09-13T14:49:05.839922+0000","last_clean":"2024-09-13T14:49:05.839922+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:49:05.839922+0000","last_undegraded":"2024-09-13T14:49:05.839922+0000","last_fullsized":"2024-09-13T14:49:05.839922+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"254'7730","reported_seq":7763,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:18.830103+0000","last_change":"2024-09-13T14:47:20.051919+0000","last_active":"2024-09-13T14:49:18.830103+0000","last_peered":"2024-09-13T14:49:18.830103+0000","last_clean":"2024-09-13T14:49:18.830103+0000","last_became_active":"2024-09-13T14:47:20.051626+0000","last_became_peered":"2024-09-13T14:47:20.051626+0000","last_unstale":"2024-09-13T14:49:18.830103+0000","last_undegraded":"2024-09-13T14:49:18.830103+0000","last_fullsized":"2024-09-13T14:49:18.830103+0000","mapping_epoch":229,"log_start":"237'1000","ondisk_log_start":"237'1000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6730,"log_dups_size":0,"ondisk_log_size":6730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:12.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438697984,"num_objects":6694,"num_object_clones":0,"num_object_copies":20082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7730,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":239,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.911761+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:49:17.911761+0000","last_peered":"2024-09-13T14:49:17.911761+0000","last_clean":"2024-09-13T14:49:17.911761+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:49:17.911761+0000","last_undegraded":"2024-09-13T14:49:17.911761+0000","last_fullsized":"2024-09-13T14:49:17.911761+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":217,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.823447+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:49:05.823447+0000","last_peered":"2024-09-13T14:49:05.823447+0000","last_clean":"2024-09-13T14:49:05.823447+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:49:05.823447+0000","last_undegraded":"2024-09-13T14:49:05.823447+0000","last_fullsized":"2024-09-13T14:49:05.823447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":228,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815069+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:49:05.815069+0000","last_peered":"2024-09-13T14:49:05.815069+0000","last_clean":"2024-09-13T14:49:05.815069+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:49:05.815069+0000","last_undegraded":"2024-09-13T14:49:05.815069+0000","last_fullsized":"2024-09-13T14:49:05.815069+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":37,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.833463+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:49:05.833463+0000","last_peered":"2024-09-13T14:49:05.833463+0000","last_clean":"2024-09-13T14:49:05.833463+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:49:05.833463+0000","last_undegraded":"2024-09-13T14:49:05.833463+0000","last_fullsized":"2024-09-13T14:49:05.833463+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"254'7781","reported_seq":8465,"reported_epoch":254,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:18.806912+0000","last_change":"2024-09-13T14:49:05.872778+0000","last_active":"2024-09-13T14:49:18.806912+0000","last_peered":"2024-09-13T14:49:18.806912+0000","last_clean":"2024-09-13T14:48:37.826901+0000","last_became_active":"2024-09-13T14:48:40.041761+0000","last_became_peered":"2024-09-13T14:48:40.041761+0000","last_unstale":"2024-09-13T14:49:18.806912+0000","last_undegraded":"2024-09-13T14:48:39.850843+0000","last_fullsized":"2024-09-13T14:48:39.825201+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7781,"log_dups_size":0,"ondisk_log_size":7781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446103552,"num_objects":6807,"num_object_clones":0,"num_object_copies":20421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":178,"num_objects_misplaced":5270,"num_objects_unfound":0,"num_objects_dirty":6807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7781,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":20381696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":5270},{"shards":"2,7","objects":1359}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":148,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815602+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:49:05.815602+0000","last_peered":"2024-09-13T14:49:05.815602+0000","last_clean":"2024-09-13T14:49:05.815602+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:49:05.815602+0000","last_undegraded":"2024-09-13T14:49:05.815602+0000","last_fullsized":"2024-09-13T14:49:05.815602+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":179,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.913668+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:49:17.913668+0000","last_peered":"2024-09-13T14:49:17.913668+0000","last_clean":"2024-09-13T14:49:17.913668+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:49:17.913668+0000","last_undegraded":"2024-09-13T14:49:17.913668+0000","last_fullsized":"2024-09-13T14:49:17.913668+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":370,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.922139+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:49:17.922139+0000","last_peered":"2024-09-13T14:49:17.922139+0000","last_clean":"2024-09-13T14:49:17.922139+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:49:17.922139+0000","last_undegraded":"2024-09-13T14:49:17.922139+0000","last_fullsized":"2024-09-13T14:49:17.922139+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":11,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.834595+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:49:05.834595+0000","last_peered":"2024-09-13T14:49:05.834595+0000","last_clean":"2024-09-13T14:49:05.834595+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:49:05.834595+0000","last_undegraded":"2024-09-13T14:49:05.834595+0000","last_fullsized":"2024-09-13T14:49:05.834595+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"254'7917","reported_seq":7977,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:20.138329+0000","last_change":"2024-09-13T14:47:20.049747+0000","last_active":"2024-09-13T14:49:20.138329+0000","last_peered":"2024-09-13T14:49:20.138329+0000","last_clean":"2024-09-13T14:49:20.138329+0000","last_became_active":"2024-09-13T14:47:20.049448+0000","last_became_peered":"2024-09-13T14:47:20.049448+0000","last_unstale":"2024-09-13T14:49:20.138329+0000","last_undegraded":"2024-09-13T14:49:20.138329+0000","last_fullsized":"2024-09-13T14:49:20.138329+0000","mapping_epoch":229,"log_start":"237'900","ondisk_log_start":"237'900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7017,"log_dups_size":0,"ondisk_log_size":7017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:11.585800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446103552,"num_objects":6807,"num_object_clones":0,"num_object_copies":20421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7917,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":394,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.904833+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:49:17.904833+0000","last_peered":"2024-09-13T14:49:17.904833+0000","last_clean":"2024-09-13T14:49:17.904833+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:49:17.904833+0000","last_undegraded":"2024-09-13T14:49:17.904833+0000","last_fullsized":"2024-09-13T14:49:17.904833+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":403,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.902125+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:49:17.902125+0000","last_peered":"2024-09-13T14:49:17.902125+0000","last_clean":"2024-09-13T14:49:17.902125+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:49:17.902125+0000","last_undegraded":"2024-09-13T14:49:17.902125+0000","last_fullsized":"2024-09-13T14:49:17.902125+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":266,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.812676+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:49:05.812676+0000","last_peered":"2024-09-13T14:49:05.812676+0000","last_clean":"2024-09-13T14:49:05.812676+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:49:05.812676+0000","last_undegraded":"2024-09-13T14:49:05.812676+0000","last_fullsized":"2024-09-13T14:49:05.812676+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":190,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.813469+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:49:05.813469+0000","last_peered":"2024-09-13T14:49:05.813469+0000","last_clean":"2024-09-13T14:49:05.813469+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:49:05.813469+0000","last_undegraded":"2024-09-13T14:49:05.813469+0000","last_fullsized":"2024-09-13T14:49:05.813469+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"254'7798","reported_seq":7831,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:18.811300+0000","last_change":"2024-09-13T14:47:20.106371+0000","last_active":"2024-09-13T14:49:18.811300+0000","last_peered":"2024-09-13T14:49:18.811300+0000","last_clean":"2024-09-13T14:49:18.811300+0000","last_became_active":"2024-09-13T14:47:20.105934+0000","last_became_peered":"2024-09-13T14:47:20.105934+0000","last_unstale":"2024-09-13T14:49:18.811300+0000","last_undegraded":"2024-09-13T14:49:18.811300+0000","last_fullsized":"2024-09-13T14:49:18.811300+0000","mapping_epoch":229,"log_start":"237'1100","ondisk_log_start":"237'1100","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6698,"log_dups_size":0,"ondisk_log_size":6698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:54.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448790528,"num_objects":6848,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":468672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":199,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.813913+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:49:05.813913+0000","last_peered":"2024-09-13T14:49:05.813913+0000","last_clean":"2024-09-13T14:49:05.813913+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:49:05.813913+0000","last_undegraded":"2024-09-13T14:49:05.813913+0000","last_fullsized":"2024-09-13T14:49:05.813913+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":300,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.816286+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:49:05.816286+0000","last_peered":"2024-09-13T14:49:05.816286+0000","last_clean":"2024-09-13T14:49:05.816286+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:49:05.816286+0000","last_undegraded":"2024-09-13T14:49:05.816286+0000","last_fullsized":"2024-09-13T14:49:05.816286+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":227,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815169+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:49:05.815169+0000","last_peered":"2024-09-13T14:49:05.815169+0000","last_clean":"2024-09-13T14:49:05.815169+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:49:05.815169+0000","last_undegraded":"2024-09-13T14:49:05.815169+0000","last_fullsized":"2024-09-13T14:49:05.815169+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":11,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.860177+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:49:05.860177+0000","last_peered":"2024-09-13T14:49:05.860177+0000","last_clean":"2024-09-13T14:49:05.860177+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:49:05.860177+0000","last_undegraded":"2024-09-13T14:49:05.860177+0000","last_fullsized":"2024-09-13T14:49:05.860177+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"254'7698","reported_seq":7731,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:19.038484+0000","last_change":"2024-09-13T14:47:20.050773+0000","last_active":"2024-09-13T14:49:19.038484+0000","last_peered":"2024-09-13T14:49:19.038484+0000","last_clean":"2024-09-13T14:49:19.038484+0000","last_became_active":"2024-09-13T14:47:20.050243+0000","last_became_peered":"2024-09-13T14:47:20.050243+0000","last_unstale":"2024-09-13T14:49:19.038484+0000","last_undegraded":"2024-09-13T14:49:19.038484+0000","last_fullsized":"2024-09-13T14:49:19.038484+0000","mapping_epoch":229,"log_start":"237'1400","ondisk_log_start":"237'1400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:42.724078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441974784,"num_objects":6744,"num_object_clones":0,"num_object_copies":20232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7698,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.911894+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:49:17.911894+0000","last_peered":"2024-09-13T14:49:17.911894+0000","last_clean":"2024-09-13T14:49:17.911894+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:49:17.911894+0000","last_undegraded":"2024-09-13T14:49:17.911894+0000","last_fullsized":"2024-09-13T14:49:17.911894+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":311,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.816194+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:49:05.816194+0000","last_peered":"2024-09-13T14:49:05.816194+0000","last_clean":"2024-09-13T14:49:05.816194+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:49:05.816194+0000","last_undegraded":"2024-09-13T14:49:05.816194+0000","last_fullsized":"2024-09-13T14:49:05.816194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":37,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814391+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:49:05.814391+0000","last_peered":"2024-09-13T14:49:05.814391+0000","last_clean":"2024-09-13T14:49:05.814391+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:49:05.814391+0000","last_undegraded":"2024-09-13T14:49:05.814391+0000","last_fullsized":"2024-09-13T14:49:05.814391+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"254'7722","reported_seq":7755,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:18.862910+0000","last_change":"2024-09-13T14:47:20.106351+0000","last_active":"2024-09-13T14:49:18.862910+0000","last_peered":"2024-09-13T14:49:18.862910+0000","last_clean":"2024-09-13T14:49:18.862910+0000","last_became_active":"2024-09-13T14:47:20.105946+0000","last_became_peered":"2024-09-13T14:47:20.105946+0000","last_unstale":"2024-09-13T14:49:18.862910+0000","last_undegraded":"2024-09-13T14:49:18.862910+0000","last_fullsized":"2024-09-13T14:49:18.862910+0000","mapping_epoch":229,"log_start":"237'1000","ondisk_log_start":"237'1000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6722,"log_dups_size":0,"ondisk_log_size":6722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:07.223637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440664064,"num_objects":6724,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":236,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.825223+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:49:05.825223+0000","last_peered":"2024-09-13T14:49:05.825223+0000","last_clean":"2024-09-13T14:49:05.825223+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:49:05.825223+0000","last_undegraded":"2024-09-13T14:49:05.825223+0000","last_fullsized":"2024-09-13T14:49:05.825223+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":217,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.822360+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:49:05.822360+0000","last_peered":"2024-09-13T14:49:05.822360+0000","last_clean":"2024-09-13T14:49:05.822360+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:49:05.822360+0000","last_undegraded":"2024-09-13T14:49:05.822360+0000","last_fullsized":"2024-09-13T14:49:05.822360+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":364,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.903282+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:49:17.903282+0000","last_peered":"2024-09-13T14:49:17.903282+0000","last_clean":"2024-09-13T14:49:17.903282+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:49:17.903282+0000","last_undegraded":"2024-09-13T14:49:17.903282+0000","last_fullsized":"2024-09-13T14:49:17.903282+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"254'7800","reported_seq":11370,"reported_epoch":254,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:18.788532+0000","last_change":"2024-09-13T14:49:05.871518+0000","last_active":"2024-09-13T14:49:18.788532+0000","last_peered":"2024-09-13T14:49:18.788532+0000","last_clean":"2024-09-13T14:48:09.848772+0000","last_became_active":"2024-09-13T14:48:12.232150+0000","last_became_peered":"2024-09-13T14:48:12.232150+0000","last_unstale":"2024-09-13T14:49:18.788532+0000","last_undegraded":"2024-09-13T14:49:05.858877+0000","last_fullsized":"2024-09-13T14:48:11.643512+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7800,"log_dups_size":0,"ondisk_log_size":7800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446693376,"num_objects":6816,"num_object_clones":0,"num_object_copies":20448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":467712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":114753536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":2427},{"shards":"2,5","objects":4389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":233,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.824598+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:49:05.824598+0000","last_peered":"2024-09-13T14:49:05.824598+0000","last_clean":"2024-09-13T14:49:05.824598+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:49:05.824598+0000","last_undegraded":"2024-09-13T14:49:05.824598+0000","last_fullsized":"2024-09-13T14:49:05.824598+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":237,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.813759+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:49:05.813759+0000","last_peered":"2024-09-13T14:49:05.813759+0000","last_clean":"2024-09-13T14:49:05.813759+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:49:05.813759+0000","last_undegraded":"2024-09-13T14:49:05.813759+0000","last_fullsized":"2024-09-13T14:49:05.813759+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":19,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814149+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:49:05.814149+0000","last_peered":"2024-09-13T14:49:05.814149+0000","last_clean":"2024-09-13T14:49:05.814149+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:49:05.814149+0000","last_undegraded":"2024-09-13T14:49:05.814149+0000","last_fullsized":"2024-09-13T14:49:05.814149+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"254'7876","reported_seq":7909,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:19.035893+0000","last_change":"2024-09-13T14:47:20.106142+0000","last_active":"2024-09-13T14:49:19.035893+0000","last_peered":"2024-09-13T14:49:19.035893+0000","last_clean":"2024-09-13T14:49:19.035893+0000","last_became_active":"2024-09-13T14:47:20.105758+0000","last_became_peered":"2024-09-13T14:47:20.105758+0000","last_unstale":"2024-09-13T14:49:19.035893+0000","last_undegraded":"2024-09-13T14:49:19.035893+0000","last_fullsized":"2024-09-13T14:49:19.035893+0000","mapping_epoch":229,"log_start":"237'1600","ondisk_log_start":"237'1600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6276,"log_dups_size":0,"ondisk_log_size":6276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:51.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450887680,"num_objects":6880,"num_object_clones":0,"num_object_copies":20640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7876,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":216,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.823161+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:49:05.823161+0000","last_peered":"2024-09-13T14:49:05.823161+0000","last_clean":"2024-09-13T14:49:05.823161+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:49:05.823161+0000","last_undegraded":"2024-09-13T14:49:05.823161+0000","last_fullsized":"2024-09-13T14:49:05.823161+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":341,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.823626+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:49:05.823626+0000","last_peered":"2024-09-13T14:49:05.823626+0000","last_clean":"2024-09-13T14:49:05.823626+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:49:05.823626+0000","last_undegraded":"2024-09-13T14:49:05.823626+0000","last_fullsized":"2024-09-13T14:49:05.823626+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":37,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.812847+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:49:05.812847+0000","last_peered":"2024-09-13T14:49:05.812847+0000","last_clean":"2024-09-13T14:49:05.812847+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:49:05.812847+0000","last_undegraded":"2024-09-13T14:49:05.812847+0000","last_fullsized":"2024-09-13T14:49:05.812847+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"254'7762","reported_seq":7823,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:18.853578+0000","last_change":"2024-09-13T14:47:20.107374+0000","last_active":"2024-09-13T14:49:18.853578+0000","last_peered":"2024-09-13T14:49:18.853578+0000","last_clean":"2024-09-13T14:49:18.853578+0000","last_became_active":"2024-09-13T14:47:20.107057+0000","last_became_peered":"2024-09-13T14:47:20.107057+0000","last_unstale":"2024-09-13T14:49:18.853578+0000","last_undegraded":"2024-09-13T14:49:18.853578+0000","last_fullsized":"2024-09-13T14:49:18.853578+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:05.814456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449970176,"num_objects":6866,"num_object_clones":0,"num_object_copies":20598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":382,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.903390+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:49:17.903390+0000","last_peered":"2024-09-13T14:49:17.903390+0000","last_clean":"2024-09-13T14:49:17.903390+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:49:17.903390+0000","last_undegraded":"2024-09-13T14:49:17.903390+0000","last_fullsized":"2024-09-13T14:49:17.903390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":215,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.816212+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:49:05.816212+0000","last_peered":"2024-09-13T14:49:05.816212+0000","last_clean":"2024-09-13T14:49:05.816212+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:49:05.816212+0000","last_undegraded":"2024-09-13T14:49:05.816212+0000","last_fullsized":"2024-09-13T14:49:05.816212+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":37,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.825037+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:49:05.825037+0000","last_peered":"2024-09-13T14:49:05.825037+0000","last_clean":"2024-09-13T14:49:05.825037+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:49:05.825037+0000","last_undegraded":"2024-09-13T14:49:05.825037+0000","last_fullsized":"2024-09-13T14:49:05.825037+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"254'7794","reported_seq":7854,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:20.146118+0000","last_change":"2024-09-13T14:47:20.108173+0000","last_active":"2024-09-13T14:49:20.146118+0000","last_peered":"2024-09-13T14:49:20.146118+0000","last_clean":"2024-09-13T14:49:20.146118+0000","last_became_active":"2024-09-13T14:47:20.107879+0000","last_became_peered":"2024-09-13T14:47:20.107879+0000","last_unstale":"2024-09-13T14:49:20.146118+0000","last_undegraded":"2024-09-13T14:49:20.146118+0000","last_fullsized":"2024-09-13T14:49:20.146118+0000","mapping_epoch":229,"log_start":"237'800","ondisk_log_start":"237'800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:50.152706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439615488,"num_objects":6708,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7794,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":229,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.827839+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:49:05.827839+0000","last_peered":"2024-09-13T14:49:05.827839+0000","last_clean":"2024-09-13T14:49:05.827839+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:49:05.827839+0000","last_undegraded":"2024-09-13T14:49:05.827839+0000","last_fullsized":"2024-09-13T14:49:05.827839+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":230,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815254+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:49:05.815254+0000","last_peered":"2024-09-13T14:49:05.815254+0000","last_clean":"2024-09-13T14:49:05.815254+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:49:05.815254+0000","last_undegraded":"2024-09-13T14:49:05.815254+0000","last_fullsized":"2024-09-13T14:49:05.815254+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":190,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814215+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:49:05.814215+0000","last_peered":"2024-09-13T14:49:05.814215+0000","last_clean":"2024-09-13T14:49:05.814215+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:49:05.814215+0000","last_undegraded":"2024-09-13T14:49:05.814215+0000","last_fullsized":"2024-09-13T14:49:05.814215+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"254'7703","reported_seq":7739,"reported_epoch":254,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:18.842499+0000","last_change":"2024-09-13T14:48:40.078958+0000","last_active":"2024-09-13T14:49:18.842499+0000","last_peered":"2024-09-13T14:49:18.842499+0000","last_clean":"2024-09-13T14:48:37.811111+0000","last_became_active":"2024-09-13T14:48:40.052181+0000","last_became_peered":"2024-09-13T14:48:40.052181+0000","last_unstale":"2024-09-13T14:49:18.842499+0000","last_undegraded":"2024-09-13T14:48:39.849823+0000","last_fullsized":"2024-09-13T14:48:39.826167+0000","mapping_epoch":249,"log_start":"237'1000","ondisk_log_start":"237'1000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6703,"log_dups_size":0,"ondisk_log_size":6703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:49.521146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444792832,"num_objects":6787,"num_object_clones":0,"num_object_copies":20361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7703,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":5291},{"shards":"3,6","objects":1038}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":243,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.822139+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:49:05.822139+0000","last_peered":"2024-09-13T14:49:05.822139+0000","last_clean":"2024-09-13T14:49:05.822139+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:49:05.822139+0000","last_undegraded":"2024-09-13T14:49:05.822139+0000","last_fullsized":"2024-09-13T14:49:05.822139+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":220,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.813561+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:49:05.813561+0000","last_peered":"2024-09-13T14:49:05.813561+0000","last_clean":"2024-09-13T14:49:05.813561+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:49:05.813561+0000","last_undegraded":"2024-09-13T14:49:05.813561+0000","last_fullsized":"2024-09-13T14:49:05.813561+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":68,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.815642+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:49:05.815642+0000","last_peered":"2024-09-13T14:49:05.815642+0000","last_clean":"2024-09-13T14:49:05.815642+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:49:05.815642+0000","last_undegraded":"2024-09-13T14:49:05.815642+0000","last_fullsized":"2024-09-13T14:49:05.815642+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"254'7936","reported_seq":7969,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:19.458978+0000","last_change":"2024-09-13T14:47:20.051824+0000","last_active":"2024-09-13T14:49:19.458978+0000","last_peered":"2024-09-13T14:49:19.458978+0000","last_clean":"2024-09-13T14:49:19.458978+0000","last_became_active":"2024-09-13T14:47:20.051487+0000","last_became_peered":"2024-09-13T14:47:20.051487+0000","last_unstale":"2024-09-13T14:49:19.458978+0000","last_undegraded":"2024-09-13T14:49:19.458978+0000","last_fullsized":"2024-09-13T14:49:19.458978+0000","mapping_epoch":229,"log_start":"236'300","ondisk_log_start":"236'300","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7636,"log_dups_size":0,"ondisk_log_size":7636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:11.533543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449445888,"num_objects":6858,"num_object_clones":0,"num_object_copies":20574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7936,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.820496+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:49:05.820496+0000","last_peered":"2024-09-13T14:49:05.820496+0000","last_clean":"2024-09-13T14:49:05.820496+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:49:05.820496+0000","last_undegraded":"2024-09-13T14:49:05.820496+0000","last_fullsized":"2024-09-13T14:49:05.820496+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.864135+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:49:05.864135+0000","last_peered":"2024-09-13T14:49:05.864135+0000","last_clean":"2024-09-13T14:49:05.864135+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:49:05.864135+0000","last_undegraded":"2024-09-13T14:49:05.864135+0000","last_fullsized":"2024-09-13T14:49:05.864135+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":11,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.827552+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:49:05.827552+0000","last_peered":"2024-09-13T14:49:05.827552+0000","last_clean":"2024-09-13T14:49:05.827552+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:49:05.827552+0000","last_undegraded":"2024-09-13T14:49:05.827552+0000","last_fullsized":"2024-09-13T14:49:05.827552+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":265,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.812785+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:49:05.812785+0000","last_peered":"2024-09-13T14:49:05.812785+0000","last_clean":"2024-09-13T14:49:05.812785+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:49:05.812785+0000","last_undegraded":"2024-09-13T14:49:05.812785+0000","last_fullsized":"2024-09-13T14:49:05.812785+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":350,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.846224+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:49:05.846224+0000","last_peered":"2024-09-13T14:49:05.846224+0000","last_clean":"2024-09-13T14:49:05.846224+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:49:05.846224+0000","last_undegraded":"2024-09-13T14:49:05.846224+0000","last_fullsized":"2024-09-13T14:49:05.846224+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":47,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814551+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:49:05.814551+0000","last_peered":"2024-09-13T14:49:05.814551+0000","last_clean":"2024-09-13T14:49:05.814551+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:49:05.814551+0000","last_undegraded":"2024-09-13T14:49:05.814551+0000","last_fullsized":"2024-09-13T14:49:05.814551+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":211,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.812972+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:49:05.812972+0000","last_peered":"2024-09-13T14:49:05.812972+0000","last_clean":"2024-09-13T14:49:05.812972+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:49:05.812972+0000","last_undegraded":"2024-09-13T14:49:05.812972+0000","last_fullsized":"2024-09-13T14:49:05.812972+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":396,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.905006+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:49:17.905006+0000","last_peered":"2024-09-13T14:49:17.905006+0000","last_clean":"2024-09-13T14:49:17.905006+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:49:17.905006+0000","last_undegraded":"2024-09-13T14:49:17.905006+0000","last_fullsized":"2024-09-13T14:49:17.905006+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":64,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.823451+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:49:05.823451+0000","last_peered":"2024-09-13T14:49:05.823451+0000","last_clean":"2024-09-13T14:49:05.823451+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:49:05.823451+0000","last_undegraded":"2024-09-13T14:49:05.823451+0000","last_fullsized":"2024-09-13T14:49:05.823451+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":402,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.901889+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:49:17.901889+0000","last_peered":"2024-09-13T14:49:17.901889+0000","last_clean":"2024-09-13T14:49:17.901889+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:49:17.901889+0000","last_undegraded":"2024-09-13T14:49:17.901889+0000","last_fullsized":"2024-09-13T14:49:17.901889+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.813956+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:49:05.813956+0000","last_peered":"2024-09-13T14:49:05.813956+0000","last_clean":"2024-09-13T14:49:05.813956+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:49:05.813956+0000","last_undegraded":"2024-09-13T14:49:05.813956+0000","last_fullsized":"2024-09-13T14:49:05.813956+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":37,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.814282+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:49:05.814282+0000","last_peered":"2024-09-13T14:49:05.814282+0000","last_clean":"2024-09-13T14:49:05.814282+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:49:05.814282+0000","last_undegraded":"2024-09-13T14:49:05.814282+0000","last_fullsized":"2024-09-13T14:49:05.814282+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":236,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:49:05.813164+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:49:05.813164+0000","last_peered":"2024-09-13T14:49:05.813164+0000","last_clean":"2024-09-13T14:49:05.813164+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:49:05.813164+0000","last_undegraded":"2024-09-13T14:49:05.813164+0000","last_fullsized":"2024-09-13T14:49:05.813164+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":61,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908617+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:49:17.908617+0000","last_peered":"2024-09-13T14:49:17.908617+0000","last_clean":"2024-09-13T14:49:17.908617+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:49:17.908617+0000","last_undegraded":"2024-09-13T14:49:17.908617+0000","last_fullsized":"2024-09-13T14:49:17.908617+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":360,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.902209+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:49:17.902209+0000","last_peered":"2024-09-13T14:49:17.902209+0000","last_clean":"2024-09-13T14:49:17.902209+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:49:17.902209+0000","last_undegraded":"2024-09-13T14:49:17.902209+0000","last_fullsized":"2024-09-13T14:49:17.902209+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7130775576,"num_objects":108808,"num_object_clones":0,"num_object_copies":326424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8354,"num_objects_misplaced":5270,"num_objects_unfound":0,"num_objects_dirty":108808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124646,"num_write_kb":7470465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":135135232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21584031744,"data_stored":21584019528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110932,"ondisk_log_size":110932,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365155,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2662740,"kb_used_data":1749196,"kb_used_omap":1,"kb_used_meta":913534,"kb_avail":91082412,"statfs":{"total":95995035648,"available":93268389888,"internally_reserved":0,"allocated":1791176704,"data_stored":1789846267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":935459160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561385,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3959956,"kb_used_data":3062280,"kb_used_omap":1,"kb_used_meta":897662,"kb_avail":89785196,"statfs":{"total":95995035648,"available":91940040704,"internally_reserved":0,"allocated":3135774720,"data_stored":3134448226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919206232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,8,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790317,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2412616,"kb_used_data":1503164,"kb_used_omap":1,"kb_used_meta":909374,"kb_avail":91332536,"statfs":{"total":95995035648,"available":93524516864,"internally_reserved":0,"allocated":1539239936,"data_stored":1537921824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931199320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986545,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5041172,"kb_used_data":4142604,"kb_used_omap":1,"kb_used_meta":898558,"kb_avail":88703980,"statfs":{"total":95995035648,"available":90832875520,"internally_reserved":0,"allocated":4242026496,"data_stored":4240696244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920123736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519700,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2567156,"kb_used_data":2046764,"kb_used_omap":4,"kb_used_meta":520379,"kb_avail":91177996,"statfs":{"total":95995035648,"available":93366267904,"internally_reserved":0,"allocated":2095886336,"data_stored":2094575993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":532868544},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304616,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3475340,"kb_used_data":3010112,"kb_used_omap":6,"kb_used_meta":465209,"kb_avail":90269812,"statfs":{"total":95995035648,"available":92436287488,"internally_reserved":0,"allocated":3082354688,"data_stored":3081033004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":476374471},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827919,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2124340,"kb_used_data":1330220,"kb_used_omap":5,"kb_used_meta":794042,"kb_avail":91620812,"statfs":{"total":95995035648,"available":93819711488,"internally_reserved":0,"allocated":1362145280,"data_stored":1360834364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":813099244},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285784,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5024964,"kb_used_data":4363580,"kb_used_omap":5,"kb_used_meta":661306,"kb_avail":88720188,"statfs":{"total":95995035648,"available":90849472512,"internally_reserved":0,"allocated":4468305920,"data_stored":4466986572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":677177403},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4429778944,"data_stored":4429774872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1354366976,"data_stored":1354366976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3060334592,"data_stored":3060334592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2089811968,"data_stored":2089811968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4222357504,"data_stored":4222353432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1528430592,"data_stored":1528430592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3111915520,"data_stored":3111911448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1787035648,"data_stored":1787035648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:49:23.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:49:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:23.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:23.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:23.802 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:23.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:23.940 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:24.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:24.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:24.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:23 smithi019 ceph-mon[91296]: from='client.56050 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:24.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:23 smithi003 ceph-mon[104007]: from='client.56050 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:24.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:49:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:49:24] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:49:24.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:23 smithi003 ceph-mon[107278]: from='client.56050 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:24.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:24.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:24.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:24.359 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:24.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:24.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:24.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:24.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:24.729 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:24.759 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:24.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:24.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:24.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:25.031 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:25.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:25.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:24 smithi019 ceph-mon[91296]: pgmap v837: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 780 op/s; 8354/326502 objects degraded (2.559%); 5270/326502 objects misplaced (1.614%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:24 smithi003 ceph-mon[107278]: pgmap v837: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 780 op/s; 8354/326502 objects degraded (2.559%); 5270/326502 objects misplaced (1.614%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:25.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:24 smithi003 ceph-mon[104007]: pgmap v837: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 780 op/s; 8354/326502 objects degraded (2.559%); 5270/326502 objects misplaced (1.614%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:25.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:25.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:25.336 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:25.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:25.425 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:25.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:25.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:25.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:25.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:25.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:25.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:25.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8354/326502 objects degraded (2.559%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:25.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8354/326502 objects degraded (2.559%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:26.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:26.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:26.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:26.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:26.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8354/326502 objects degraded (2.559%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:26.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:26.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:26.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:26.332 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:26.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:26.466 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:26.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:26.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:26.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:26.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:49:26.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:26.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:27.000 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:27.046 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:27.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:27.184 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:26 smithi019 ceph-mon[91296]: pgmap v838: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 8546/313236 objects degraded (2.728%); 4989/313236 objects misplaced (1.593%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:49:27.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:26 smithi003 ceph-mon[104007]: pgmap v838: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 8546/313236 objects degraded (2.728%); 4989/313236 objects misplaced (1.593%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:49:27.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:26 smithi003 ceph-mon[107278]: pgmap v838: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 8546/313236 objects degraded (2.728%); 4989/313236 objects misplaced (1.593%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:49:27.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:27.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:27.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:27.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:27.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:27.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:27.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:27.608 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:27.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:27.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:27.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:27.787 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:27.787 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:27.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:27.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:27.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:28.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:28.131 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.170 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:28.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:28.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:28.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:28.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.436 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:28.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:28.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:28.580 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:28.682 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.728 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:28.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.862 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:28.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:49:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:29.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:29.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:29.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:29.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:29.328 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:28 smithi003 ceph-mon[107278]: pgmap v839: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 8546/313236 objects degraded (2.728%); 4989/313236 objects misplaced (1.593%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:49:29.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:28 smithi003 ceph-mon[104007]: pgmap v839: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 8546/313236 objects degraded (2.728%); 4989/313236 objects misplaced (1.593%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:49:29.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:29.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:28 smithi019 ceph-mon[91296]: pgmap v839: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 8546/313236 objects degraded (2.728%); 4989/313236 objects misplaced (1.593%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:49:29.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:29.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:29.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:29.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:29.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:29.722 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:49:29.722 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8546/313236 objects degraded (2.728%), 3 pgs degraded, 1 pg undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1196,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":254,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238227,"num_in_osds":8,"osd_in_since":1726237182,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":106},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":109,"num_pools":5,"num_objects":104412,"data_bytes":6841598136,"bytes_used":26923450368,"bytes_avail":741036834816,"bytes_total":767960285184,"degraded_objects":8546,"degraded_total":313236,"degraded_ratio":0.027282943212146752,"misplaced_objects":4989,"misplaced_total":313236,"misplaced_ratio":0.015927288051181855,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":1257671,"recovering_keys_per_sec":0,"num_objects_recovered":192,"num_bytes_recovered":12582912,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":850},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.zmhiel","status":"up:active","gid":14646}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-13T14:43:29.603041+0000","services":{}},"progress_events":{"70b0e26c-ee48-4914-b6cc-9ce75e26a89c":{"message":"Global Recovery Event (60s)\n [===========================.] (remaining: 1s)","progress":0.97247707843780518,"add_to_ceph_s":true},"b67cb405-2ef0-4d69-99ef-7eee6a6810fd":{"message":"Upgrade to 19.3.0-4697-gd58f7306 (9m)\n [============................] (remaining: 11m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-13T14:49:29.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:29.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:29.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:29.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:30.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:30.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:29 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4070961809' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:49:30.080 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:29 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4070961809' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:49:30.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:30.172 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:30.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:30.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:29 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4070961809' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:49:30.273 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:30.319 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:30.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:30.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:30.527 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:49:30.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:49:30.547 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:30.556 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:30.587 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:49:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:49:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:30.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:30.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:30.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:30.993 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:31.031 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:31.045 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:30 smithi003 ceph-mon[104007]: pgmap v840: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8727/301251 objects degraded (2.897%); 4718/301251 objects misplaced (1.566%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:49:31.045 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8727/301251 objects degraded (2.897%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:31.045 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:30 smithi003 ceph-mon[107278]: pgmap v840: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8727/301251 objects degraded (2.897%); 4718/301251 objects misplaced (1.566%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:49:31.045 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8727/301251 objects degraded (2.897%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:31.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:31.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:31.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:31.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:31.280 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:31.309 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:31.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:31.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:30 smithi019 ceph-mon[91296]: pgmap v840: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8727/301251 objects degraded (2.897%); 4718/301251 objects misplaced (1.566%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:49:31.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8727/301251 objects degraded (2.897%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:31.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:31.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:31.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:31.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:31.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:31.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:31.891 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:31.973 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:49:31.993 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:32.021 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:32.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:32.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:32.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:32.299 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:32.336 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:32.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/494461867' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:32.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:31 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:32.341 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/494461867' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:32.341 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:31 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:32.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/494461867' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:31 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:32.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:32.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:32.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:32.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:32.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:32.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:32.824 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:32.865 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:32.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:32.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:33.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:33.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:33.154 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:33.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:33.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:32 smithi003 ceph-mon[107278]: pgmap v841: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 791 op/s; 8727/299565 objects degraded (2.913%); 4718/299565 objects misplaced (1.575%); 954 KiB/s, 14 objects/s recovering 2024-09-13T14:49:33.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:32 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:33.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:32 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:33.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:32 smithi003 ceph-mon[107278]: osdmap e255: 8 total, 8 up, 8 in 2024-09-13T14:49:33.285 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:32 smithi003 ceph-mon[104007]: pgmap v841: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 791 op/s; 8727/299565 objects degraded (2.913%); 4718/299565 objects misplaced (1.575%); 954 KiB/s, 14 objects/s recovering 2024-09-13T14:49:33.285 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:32 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:33.285 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:32 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:33.285 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:32 smithi003 ceph-mon[104007]: osdmap e255: 8 total, 8 up, 8 in 2024-09-13T14:49:33.290 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:33.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:33.427 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:33.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:32 smithi019 ceph-mon[91296]: pgmap v841: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 791 op/s; 8727/299565 objects degraded (2.913%); 4718/299565 objects misplaced (1.575%); 954 KiB/s, 14 objects/s recovering 2024-09-13T14:49:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:32 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:32 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:32 smithi019 ceph-mon[91296]: osdmap e255: 8 total, 8 up, 8 in 2024-09-13T14:49:33.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:33.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:33.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:33.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:33.709 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:33.811 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:33.850 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:33.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:33.984 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:34.086 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:34.128 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:34.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:34.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:34.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:34.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:49:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:49:34] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:49:34.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:34.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:34.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:34.687 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:34.789 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:34.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:34.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:35.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.101 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:35.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:35.169 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:34 smithi003 ceph-mon[107278]: pgmap v843: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 897 op/s; 8727/299565 objects degraded (2.913%); 4718/299565 objects misplaced (1.575%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:49:35.172 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:34 smithi003 ceph-mon[104007]: pgmap v843: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 897 op/s; 8727/299565 objects degraded (2.913%); 4718/299565 objects misplaced (1.575%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:49:35.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.233 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:35.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:35.334 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.373 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:35.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:35.390 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:49:35.390 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:49:35.390 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:49:35.391 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:35.391 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:35.391 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:35.391 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:35.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:49:35.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:49:35.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:49:35.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:49:35.391 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:49:35.392 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:35.392 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:35.392 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:35.392 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:34 smithi019 ceph-mon[91296]: pgmap v843: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 897 op/s; 8727/299565 objects degraded (2.913%); 4718/299565 objects misplaced (1.575%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:49:35.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:35.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:35.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:35.657 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:35.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:35.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:35.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:35.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:35.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:35.945 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:36.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:36.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:36.220 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:36.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:36.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:35 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1251739465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:36.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8727/299565 objects degraded (2.913%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:36.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:35 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1251739465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:36.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8727/299565 objects degraded (2.913%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:36.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:35 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1251739465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8727/299565 objects degraded (2.913%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:36.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:36.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:36.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:36.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:36.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:36.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:36.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:36.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:37.043 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:37.051 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:36 smithi003 ceph-mon[107278]: pgmap v844: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 8895/286422 objects degraded (3.106%); 4459/286422 objects misplaced (1.557%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:37.051 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:36 smithi003 ceph-mon[104007]: pgmap v844: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 8895/286422 objects degraded (3.106%); 4459/286422 objects misplaced (1.557%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:37.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:37.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:37.214 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:37.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:37.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:37.358 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:36 smithi019 ceph-mon[91296]: pgmap v844: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 8895/286422 objects degraded (3.106%); 4459/286422 objects misplaced (1.557%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:37.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:37.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:37.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:37.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:37.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:37.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:37.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:38.048 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:38.081 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:38.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:38.216 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:38.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:38.242 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:49:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:49:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:49:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:49:38.242 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:49:38.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:38.347 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:38.452 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:38.488 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:38.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:38.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:38.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:38.742 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:38.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:49:38.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:38.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:39.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:39.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:39.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:49:39.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:39.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:39.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:39.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:38 smithi003 ceph-mon[107278]: pgmap v845: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 8895/286422 objects degraded (3.106%); 4459/286422 objects misplaced (1.557%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:39.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:38 smithi003 ceph-mon[104007]: pgmap v845: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 8895/286422 objects degraded (3.106%); 4459/286422 objects misplaced (1.557%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:39.339 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:39.372 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:38 smithi019 ceph-mon[91296]: pgmap v845: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 8895/286422 objects degraded (3.106%); 4459/286422 objects misplaced (1.557%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:39.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:39.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:39.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:39.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:39.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:39.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:39.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:39.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:39.939 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:39.991 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:49:39.996 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:49:40.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:40.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:40.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:40.225 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:40.327 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:40.332 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:49:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:49:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:40.333 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:40 smithi019 ceph-mon[91296]: from='client.56065 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.360 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:40.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:40.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:40 smithi003 ceph-mon[107278]: from='client.56065 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:40 smithi003 ceph-mon[104007]: from='client.56065 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:40.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:40.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:40.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:40.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:40.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:40.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:40.866 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:40.898 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:41.000 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.048 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:41.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:41.149 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:41 smithi019 ceph-mon[91296]: pgmap v846: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 871 op/s; 9070/275088 objects degraded (3.297%); 4192/275088 objects misplaced (1.524%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9070/275088 objects degraded (3.297%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:41.187 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:41.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:41.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:41.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:41 smithi003 ceph-mon[107278]: pgmap v846: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 871 op/s; 9070/275088 objects degraded (3.297%); 4192/275088 objects misplaced (1.524%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9070/275088 objects degraded (3.297%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:41.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:41 smithi003 ceph-mon[104007]: pgmap v846: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 871 op/s; 9070/275088 objects degraded (3.297%); 4192/275088 objects misplaced (1.524%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:41.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9070/275088 objects degraded (3.297%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:41.447 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:41.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:41.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:41.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:41.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:41.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:49:41.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:41.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:41.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.009 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:42.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:42.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.295 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:42.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:42.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:42.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:42.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:42.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:42.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:42.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:42.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:42.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:42.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:43.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:43.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:43.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:43.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:43.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:43.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:43 smithi019 ceph-mon[91296]: pgmap v847: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 9070/273438 objects degraded (3.317%); 4192/273438 objects misplaced (1.533%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:43.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:43 smithi003 ceph-mon[107278]: pgmap v847: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 9070/273438 objects degraded (3.317%); 4192/273438 objects misplaced (1.533%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:43.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:43 smithi003 ceph-mon[104007]: pgmap v847: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 9070/273438 objects degraded (3.317%); 4192/273438 objects misplaced (1.533%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:43.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:43.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:43.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:43.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:43.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:43.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:43.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:44.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:44.086 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:44.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:44.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:44.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:44.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:44.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:44.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:49:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:49:44] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:49:44.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:44.514 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:44.615 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:44.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:44.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:44.796 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:44.898 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":845,"stamp":"2024-09-13T14:49:37.778658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6255837368,"num_objects":95474,"num_object_clones":0,"num_object_copies":286422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8895,"num_objects_misplaced":4459,"num_objects_unfound":0,"num_objects_dirty":95474,"num_whiteouts":0,"num_read":15,"num_read_kb":35,"num_write":138077,"num_write_kb":7470571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2350,"num_bytes_recovered":153710129,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115051,"ondisk_log_size":115051,"up":327,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":333,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24669984,"kb_used_data":18596560,"kb_used_omap":29,"kb_used_meta":6073186,"kb_avail":725291232,"statfs":{"total":767960285184,"available":742698221568,"internally_reserved":0,"allocated":19042877440,"data_stored":19032232264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6218942990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-585760768,"num_objects":-8938,"num_object_clones":0,"num_object_copies":-26814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":349,"num_objects_misplaced":-530,"num_objects_unfound":0,"num_objects_dirty":-8938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8938,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.013586"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":388,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979805+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:49:31.979805+0000","last_peered":"2024-09-13T14:49:31.979805+0000","last_clean":"2024-09-13T14:49:31.979805+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:49:31.979805+0000","last_undegraded":"2024-09-13T14:49:31.979805+0000","last_fullsized":"2024-09-13T14:49:31.979805+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":227,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.905829+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:49:17.905829+0000","last_peered":"2024-09-13T14:49:17.905829+0000","last_clean":"2024-09-13T14:49:17.905829+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:49:17.905829+0000","last_undegraded":"2024-09-13T14:49:17.905829+0000","last_fullsized":"2024-09-13T14:49:17.905829+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":18,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909220+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:49:17.909220+0000","last_peered":"2024-09-13T14:49:17.909220+0000","last_clean":"2024-09-13T14:49:17.909220+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:49:17.909220+0000","last_undegraded":"2024-09-13T14:49:17.909220+0000","last_fullsized":"2024-09-13T14:49:17.909220+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":347,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.962833+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:49:31.962833+0000","last_peered":"2024-09-13T14:49:31.962833+0000","last_clean":"2024-09-13T14:49:31.962833+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:49:31.962833+0000","last_undegraded":"2024-09-13T14:49:31.962833+0000","last_fullsized":"2024-09-13T14:49:31.962833+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":201,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908978+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:49:17.908978+0000","last_peered":"2024-09-13T14:49:17.908978+0000","last_clean":"2024-09-13T14:49:17.908978+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:49:17.908978+0000","last_undegraded":"2024-09-13T14:49:17.908978+0000","last_fullsized":"2024-09-13T14:49:17.908978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":234,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906441+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:49:17.906441+0000","last_peered":"2024-09-13T14:49:17.906441+0000","last_clean":"2024-09-13T14:49:17.906441+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:49:17.906441+0000","last_undegraded":"2024-09-13T14:49:17.906441+0000","last_fullsized":"2024-09-13T14:49:17.906441+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":218,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908986+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:49:17.908986+0000","last_peered":"2024-09-13T14:49:17.908986+0000","last_clean":"2024-09-13T14:49:17.908986+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:49:17.908986+0000","last_undegraded":"2024-09-13T14:49:17.908986+0000","last_fullsized":"2024-09-13T14:49:17.908986+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":264,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907279+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:49:17.907279+0000","last_peered":"2024-09-13T14:49:17.907279+0000","last_clean":"2024-09-13T14:49:17.907279+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:49:17.907279+0000","last_undegraded":"2024-09-13T14:49:17.907279+0000","last_fullsized":"2024-09-13T14:49:17.907279+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":376,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979502+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:49:31.979502+0000","last_peered":"2024-09-13T14:49:31.979502+0000","last_clean":"2024-09-13T14:49:31.979502+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:49:31.979502+0000","last_undegraded":"2024-09-13T14:49:31.979502+0000","last_fullsized":"2024-09-13T14:49:31.979502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":371,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.974194+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:49:31.974194+0000","last_peered":"2024-09-13T14:49:31.974194+0000","last_clean":"2024-09-13T14:49:31.974194+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:49:31.974194+0000","last_undegraded":"2024-09-13T14:49:31.974194+0000","last_fullsized":"2024-09-13T14:49:31.974194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":236,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906564+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:49:17.906564+0000","last_peered":"2024-09-13T14:49:17.906564+0000","last_clean":"2024-09-13T14:49:17.906564+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:49:17.906564+0000","last_undegraded":"2024-09-13T14:49:17.906564+0000","last_fullsized":"2024-09-13T14:49:17.906564+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":314,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.966303+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:49:31.966303+0000","last_peered":"2024-09-13T14:49:31.966303+0000","last_clean":"2024-09-13T14:49:31.966303+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:49:31.966303+0000","last_undegraded":"2024-09-13T14:49:31.966303+0000","last_fullsized":"2024-09-13T14:49:31.966303+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.981793+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:49:31.981793+0000","last_peered":"2024-09-13T14:49:31.981793+0000","last_clean":"2024-09-13T14:49:31.981793+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:49:31.981793+0000","last_undegraded":"2024-09-13T14:49:31.981793+0000","last_fullsized":"2024-09-13T14:49:31.981793+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":407,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983793+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:49:31.983793+0000","last_peered":"2024-09-13T14:49:31.983793+0000","last_clean":"2024-09-13T14:49:31.983793+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:49:31.983793+0000","last_undegraded":"2024-09-13T14:49:31.983793+0000","last_fullsized":"2024-09-13T14:49:31.983793+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":346,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963239+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:49:31.963239+0000","last_peered":"2024-09-13T14:49:31.963239+0000","last_clean":"2024-09-13T14:49:31.963239+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:49:31.963239+0000","last_undegraded":"2024-09-13T14:49:31.963239+0000","last_fullsized":"2024-09-13T14:49:31.963239+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":217,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.927273+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:49:17.927273+0000","last_peered":"2024-09-13T14:49:17.927273+0000","last_clean":"2024-09-13T14:49:17.927273+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:49:17.927273+0000","last_undegraded":"2024-09-13T14:49:17.927273+0000","last_fullsized":"2024-09-13T14:49:17.927273+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":314,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.962916+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:49:31.962916+0000","last_peered":"2024-09-13T14:49:31.962916+0000","last_clean":"2024-09-13T14:49:31.962916+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:49:31.962916+0000","last_undegraded":"2024-09-13T14:49:31.962916+0000","last_fullsized":"2024-09-13T14:49:31.962916+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":390,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983164+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:49:31.983164+0000","last_peered":"2024-09-13T14:49:31.983164+0000","last_clean":"2024-09-13T14:49:31.983164+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:49:31.983164+0000","last_undegraded":"2024-09-13T14:49:31.983164+0000","last_fullsized":"2024-09-13T14:49:31.983164+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":319,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963146+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:49:31.963146+0000","last_peered":"2024-09-13T14:49:31.963146+0000","last_clean":"2024-09-13T14:49:31.963146+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:49:31.963146+0000","last_undegraded":"2024-09-13T14:49:31.963146+0000","last_fullsized":"2024-09-13T14:49:31.963146+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909129+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:49:17.909129+0000","last_peered":"2024-09-13T14:49:17.909129+0000","last_clean":"2024-09-13T14:49:17.909129+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:49:17.909129+0000","last_undegraded":"2024-09-13T14:49:17.909129+0000","last_fullsized":"2024-09-13T14:49:17.909129+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":69,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.973959+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:49:31.973959+0000","last_peered":"2024-09-13T14:49:31.973959+0000","last_clean":"2024-09-13T14:49:31.973959+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:49:31.973959+0000","last_undegraded":"2024-09-13T14:49:31.973959+0000","last_fullsized":"2024-09-13T14:49:31.973959+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":18,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893668+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:49:17.893668+0000","last_peered":"2024-09-13T14:49:17.893668+0000","last_clean":"2024-09-13T14:49:17.893668+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:49:17.893668+0000","last_undegraded":"2024-09-13T14:49:17.893668+0000","last_fullsized":"2024-09-13T14:49:17.893668+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":156,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900455+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:49:17.900455+0000","last_peered":"2024-09-13T14:49:17.900455+0000","last_clean":"2024-09-13T14:49:17.900455+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:49:17.900455+0000","last_undegraded":"2024-09-13T14:49:17.900455+0000","last_fullsized":"2024-09-13T14:49:17.900455+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":224,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907731+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:49:17.907731+0000","last_peered":"2024-09-13T14:49:17.907731+0000","last_clean":"2024-09-13T14:49:17.907731+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:49:17.907731+0000","last_undegraded":"2024-09-13T14:49:17.907731+0000","last_fullsized":"2024-09-13T14:49:17.907731+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":217,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907339+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:49:17.907339+0000","last_peered":"2024-09-13T14:49:17.907339+0000","last_clean":"2024-09-13T14:49:17.907339+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:49:17.907339+0000","last_undegraded":"2024-09-13T14:49:17.907339+0000","last_fullsized":"2024-09-13T14:49:17.907339+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"255'8524","reported_seq":8558,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:33.825158+0000","last_change":"2024-09-13T14:47:20.107455+0000","last_active":"2024-09-13T14:49:33.825158+0000","last_peered":"2024-09-13T14:49:33.825158+0000","last_clean":"2024-09-13T14:49:33.825158+0000","last_became_active":"2024-09-13T14:47:20.107142+0000","last_became_peered":"2024-09-13T14:47:20.107142+0000","last_unstale":"2024-09-13T14:49:33.825158+0000","last_undegraded":"2024-09-13T14:49:33.825158+0000","last_fullsized":"2024-09-13T14:49:33.825158+0000","mapping_epoch":229,"log_start":"238'1800","ondisk_log_start":"238'1800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:19.189519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389545984,"num_objects":5944,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":462976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":200,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906070+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:49:17.906070+0000","last_peered":"2024-09-13T14:49:17.906070+0000","last_clean":"2024-09-13T14:49:17.906070+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:49:17.906070+0000","last_undegraded":"2024-09-13T14:49:17.906070+0000","last_fullsized":"2024-09-13T14:49:17.906070+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":250,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.927408+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:49:17.927408+0000","last_peered":"2024-09-13T14:49:17.927408+0000","last_clean":"2024-09-13T14:49:17.927408+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:49:17.927408+0000","last_undegraded":"2024-09-13T14:49:17.927408+0000","last_fullsized":"2024-09-13T14:49:17.927408+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":388,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.984010+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:49:31.984010+0000","last_peered":"2024-09-13T14:49:31.984010+0000","last_clean":"2024-09-13T14:49:31.984010+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:49:31.984010+0000","last_undegraded":"2024-09-13T14:49:31.984010+0000","last_fullsized":"2024-09-13T14:49:31.984010+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":76,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963621+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:49:31.963621+0000","last_peered":"2024-09-13T14:49:31.963621+0000","last_clean":"2024-09-13T14:49:31.963621+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:49:31.963621+0000","last_undegraded":"2024-09-13T14:49:31.963621+0000","last_fullsized":"2024-09-13T14:49:31.963621+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"255'8658","reported_seq":8692,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.033286+0000","last_change":"2024-09-13T14:47:20.054114+0000","last_active":"2024-09-13T14:49:34.033286+0000","last_peered":"2024-09-13T14:49:34.033286+0000","last_clean":"2024-09-13T14:49:34.033286+0000","last_became_active":"2024-09-13T14:47:20.053767+0000","last_became_peered":"2024-09-13T14:47:20.053767+0000","last_unstale":"2024-09-13T14:49:34.033286+0000","last_undegraded":"2024-09-13T14:49:34.033286+0000","last_fullsized":"2024-09-13T14:49:34.033286+0000","mapping_epoch":229,"log_start":"238'2400","ondisk_log_start":"238'2400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:48.035191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389545984,"num_objects":5944,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8658,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":357,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963022+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:49:31.963022+0000","last_peered":"2024-09-13T14:49:31.963022+0000","last_clean":"2024-09-13T14:49:31.963022+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:49:31.963022+0000","last_undegraded":"2024-09-13T14:49:31.963022+0000","last_fullsized":"2024-09-13T14:49:31.963022+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":91,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.899900+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:49:17.899900+0000","last_peered":"2024-09-13T14:49:17.899900+0000","last_clean":"2024-09-13T14:49:17.899900+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:49:17.899900+0000","last_undegraded":"2024-09-13T14:49:17.899900+0000","last_fullsized":"2024-09-13T14:49:17.899900+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":345,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.967020+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:49:31.967020+0000","last_peered":"2024-09-13T14:49:31.967020+0000","last_clean":"2024-09-13T14:49:31.967020+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:49:31.967020+0000","last_undegraded":"2024-09-13T14:49:31.967020+0000","last_fullsized":"2024-09-13T14:49:31.967020+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906883+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:49:17.906883+0000","last_peered":"2024-09-13T14:49:17.906883+0000","last_clean":"2024-09-13T14:49:17.906883+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:49:17.906883+0000","last_undegraded":"2024-09-13T14:49:17.906883+0000","last_fullsized":"2024-09-13T14:49:17.906883+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"255'8613","reported_seq":8648,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:33.874058+0000","last_change":"2024-09-13T14:47:20.052876+0000","last_active":"2024-09-13T14:49:33.874058+0000","last_peered":"2024-09-13T14:49:33.874058+0000","last_clean":"2024-09-13T14:49:33.874058+0000","last_became_active":"2024-09-13T14:47:20.052566+0000","last_became_peered":"2024-09-13T14:47:20.052566+0000","last_unstale":"2024-09-13T14:49:33.874058+0000","last_undegraded":"2024-09-13T14:49:33.874058+0000","last_fullsized":"2024-09-13T14:49:33.874058+0000","mapping_epoch":229,"log_start":"238'1900","ondisk_log_start":"238'1900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:56.317980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394133528,"num_objects":6015,"num_object_clones":0,"num_object_copies":18045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8613,"num_write_kb":468033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":374,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979425+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:49:31.979425+0000","last_peered":"2024-09-13T14:49:31.979425+0000","last_clean":"2024-09-13T14:49:31.979425+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:49:31.979425+0000","last_undegraded":"2024-09-13T14:49:31.979425+0000","last_fullsized":"2024-09-13T14:49:31.979425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":230,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894056+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:49:17.894056+0000","last_peered":"2024-09-13T14:49:17.894056+0000","last_clean":"2024-09-13T14:49:17.894056+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:49:17.894056+0000","last_undegraded":"2024-09-13T14:49:17.894056+0000","last_fullsized":"2024-09-13T14:49:17.894056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":329,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.974087+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:49:31.974087+0000","last_peered":"2024-09-13T14:49:31.974087+0000","last_clean":"2024-09-13T14:49:31.974087+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:49:31.974087+0000","last_undegraded":"2024-09-13T14:49:31.974087+0000","last_fullsized":"2024-09-13T14:49:31.974087+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":191,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.899663+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:49:17.899663+0000","last_peered":"2024-09-13T14:49:17.899663+0000","last_clean":"2024-09-13T14:49:17.899663+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:49:17.899663+0000","last_undegraded":"2024-09-13T14:49:17.899663+0000","last_fullsized":"2024-09-13T14:49:17.899663+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"255'8700","reported_seq":8733,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:33.963566+0000","last_change":"2024-09-13T14:47:20.052419+0000","last_active":"2024-09-13T14:49:33.963566+0000","last_peered":"2024-09-13T14:49:33.963566+0000","last_clean":"2024-09-13T14:49:33.963566+0000","last_became_active":"2024-09-13T14:47:20.051845+0000","last_became_peered":"2024-09-13T14:47:20.051845+0000","last_unstale":"2024-09-13T14:49:33.963566+0000","last_undegraded":"2024-09-13T14:49:33.963566+0000","last_fullsized":"2024-09-13T14:49:33.963566+0000","mapping_epoch":229,"log_start":"238'1814","ondisk_log_start":"238'1814","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.420332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216000,"num_objects":6000,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8700,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909136+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:49:17.909136+0000","last_peered":"2024-09-13T14:49:17.909136+0000","last_clean":"2024-09-13T14:49:17.909136+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:49:17.909136+0000","last_undegraded":"2024-09-13T14:49:17.909136+0000","last_fullsized":"2024-09-13T14:49:17.909136+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":187,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980494+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:49:31.980494+0000","last_peered":"2024-09-13T14:49:31.980494+0000","last_clean":"2024-09-13T14:49:31.980494+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:49:31.980494+0000","last_undegraded":"2024-09-13T14:49:31.980494+0000","last_fullsized":"2024-09-13T14:49:31.980494+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":233,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909062+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:49:17.909062+0000","last_peered":"2024-09-13T14:49:17.909062+0000","last_clean":"2024-09-13T14:49:17.909062+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:49:17.909062+0000","last_undegraded":"2024-09-13T14:49:17.909062+0000","last_fullsized":"2024-09-13T14:49:17.909062+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":211,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.927121+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:49:17.927121+0000","last_peered":"2024-09-13T14:49:17.927121+0000","last_clean":"2024-09-13T14:49:17.927121+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:49:17.927121+0000","last_undegraded":"2024-09-13T14:49:17.927121+0000","last_fullsized":"2024-09-13T14:49:17.927121+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"255'8544","reported_seq":8578,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:33.898211+0000","last_change":"2024-09-13T14:47:20.051919+0000","last_active":"2024-09-13T14:49:33.898211+0000","last_peered":"2024-09-13T14:49:33.898211+0000","last_clean":"2024-09-13T14:49:33.898211+0000","last_became_active":"2024-09-13T14:47:20.051626+0000","last_became_peered":"2024-09-13T14:47:20.051626+0000","last_unstale":"2024-09-13T14:49:33.898211+0000","last_undegraded":"2024-09-13T14:49:33.898211+0000","last_fullsized":"2024-09-13T14:49:33.898211+0000","mapping_epoch":229,"log_start":"238'1800","ondisk_log_start":"238'1800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:12.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385351680,"num_objects":5880,"num_object_clones":0,"num_object_copies":17640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8544,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":241,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.974271+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:49:31.974271+0000","last_peered":"2024-09-13T14:49:31.974271+0000","last_clean":"2024-09-13T14:49:31.974271+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:49:31.974271+0000","last_undegraded":"2024-09-13T14:49:31.974271+0000","last_fullsized":"2024-09-13T14:49:31.974271+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":218,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907402+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:49:17.907402+0000","last_peered":"2024-09-13T14:49:17.907402+0000","last_clean":"2024-09-13T14:49:17.907402+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:49:17.907402+0000","last_undegraded":"2024-09-13T14:49:17.907402+0000","last_fullsized":"2024-09-13T14:49:17.907402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":229,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909454+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:49:17.909454+0000","last_peered":"2024-09-13T14:49:17.909454+0000","last_clean":"2024-09-13T14:49:17.909454+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:49:17.909454+0000","last_undegraded":"2024-09-13T14:49:17.909454+0000","last_fullsized":"2024-09-13T14:49:17.909454+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908930+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:49:17.908930+0000","last_peered":"2024-09-13T14:49:17.908930+0000","last_clean":"2024-09-13T14:49:17.908930+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:49:17.908930+0000","last_undegraded":"2024-09-13T14:49:17.908930+0000","last_fullsized":"2024-09-13T14:49:17.908930+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"255'8593","reported_seq":9820,"reported_epoch":255,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:33.897657+0000","last_change":"2024-09-13T14:49:05.872778+0000","last_active":"2024-09-13T14:49:33.897657+0000","last_peered":"2024-09-13T14:49:33.897657+0000","last_clean":"2024-09-13T14:48:37.826901+0000","last_became_active":"2024-09-13T14:48:40.041761+0000","last_became_peered":"2024-09-13T14:48:40.041761+0000","last_unstale":"2024-09-13T14:49:33.897657+0000","last_undegraded":"2024-09-13T14:48:39.850843+0000","last_fullsized":"2024-09-13T14:48:39.825201+0000","mapping_epoch":249,"log_start":"236'400","ondisk_log_start":"236'400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8193,"log_dups_size":0,"ondisk_log_size":8193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392888320,"num_objects":5995,"num_object_clones":0,"num_object_copies":17985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":719,"num_objects_misplaced":4459,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8593,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":581,"num_bytes_recovered":38076416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":4459},{"shards":"2,7","objects":817}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":149,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.910300+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:49:17.910300+0000","last_peered":"2024-09-13T14:49:17.910300+0000","last_clean":"2024-09-13T14:49:17.910300+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:49:17.910300+0000","last_undegraded":"2024-09-13T14:49:17.910300+0000","last_fullsized":"2024-09-13T14:49:17.910300+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":181,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.971494+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:49:31.971494+0000","last_peered":"2024-09-13T14:49:31.971494+0000","last_clean":"2024-09-13T14:49:31.971494+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:49:31.971494+0000","last_undegraded":"2024-09-13T14:49:31.971494+0000","last_fullsized":"2024-09-13T14:49:31.971494+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":372,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.986429+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:49:31.986429+0000","last_peered":"2024-09-13T14:49:31.986429+0000","last_clean":"2024-09-13T14:49:31.986429+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:49:31.986429+0000","last_undegraded":"2024-09-13T14:49:31.986429+0000","last_fullsized":"2024-09-13T14:49:31.986429+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893457+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:49:17.893457+0000","last_peered":"2024-09-13T14:49:17.893457+0000","last_clean":"2024-09-13T14:49:17.893457+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:49:17.893457+0000","last_undegraded":"2024-09-13T14:49:17.893457+0000","last_fullsized":"2024-09-13T14:49:17.893457+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"255'8743","reported_seq":8807,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.168649+0000","last_change":"2024-09-13T14:47:20.049747+0000","last_active":"2024-09-13T14:49:35.168649+0000","last_peered":"2024-09-13T14:49:35.168649+0000","last_clean":"2024-09-13T14:49:35.168649+0000","last_became_active":"2024-09-13T14:47:20.049448+0000","last_became_peered":"2024-09-13T14:47:20.049448+0000","last_unstale":"2024-09-13T14:49:35.168649+0000","last_undegraded":"2024-09-13T14:49:35.168649+0000","last_fullsized":"2024-09-13T14:49:35.168649+0000","mapping_epoch":229,"log_start":"238'1700","ondisk_log_start":"238'1700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:11.585800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391970816,"num_objects":5981,"num_object_clones":0,"num_object_copies":17943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8743,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":396,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983706+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:49:31.983706+0000","last_peered":"2024-09-13T14:49:31.983706+0000","last_clean":"2024-09-13T14:49:31.983706+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:49:31.983706+0000","last_undegraded":"2024-09-13T14:49:31.983706+0000","last_fullsized":"2024-09-13T14:49:31.983706+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":405,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980571+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:49:31.980571+0000","last_peered":"2024-09-13T14:49:31.980571+0000","last_clean":"2024-09-13T14:49:31.980571+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:49:31.980571+0000","last_undegraded":"2024-09-13T14:49:31.980571+0000","last_fullsized":"2024-09-13T14:49:31.980571+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":267,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894213+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:49:17.894213+0000","last_peered":"2024-09-13T14:49:17.894213+0000","last_clean":"2024-09-13T14:49:17.894213+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:49:17.894213+0000","last_undegraded":"2024-09-13T14:49:17.894213+0000","last_fullsized":"2024-09-13T14:49:17.894213+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":191,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900789+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:49:17.900789+0000","last_peered":"2024-09-13T14:49:17.900789+0000","last_clean":"2024-09-13T14:49:17.900789+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:49:17.900789+0000","last_undegraded":"2024-09-13T14:49:17.900789+0000","last_fullsized":"2024-09-13T14:49:17.900789+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"255'8641","reported_seq":8675,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:33.874524+0000","last_change":"2024-09-13T14:47:20.106371+0000","last_active":"2024-09-13T14:49:33.874524+0000","last_peered":"2024-09-13T14:49:33.874524+0000","last_clean":"2024-09-13T14:49:33.874524+0000","last_became_active":"2024-09-13T14:47:20.105934+0000","last_became_peered":"2024-09-13T14:47:20.105934+0000","last_unstale":"2024-09-13T14:49:33.874524+0000","last_undegraded":"2024-09-13T14:49:33.874524+0000","last_fullsized":"2024-09-13T14:49:33.874524+0000","mapping_epoch":229,"log_start":"238'1900","ondisk_log_start":"238'1900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6741,"log_dups_size":0,"ondisk_log_size":6741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:54.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393543680,"num_objects":6005,"num_object_clones":0,"num_object_copies":18015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8641,"num_write_kb":468672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":200,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.902644+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:49:17.902644+0000","last_peered":"2024-09-13T14:49:17.902644+0000","last_clean":"2024-09-13T14:49:17.902644+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:49:17.902644+0000","last_undegraded":"2024-09-13T14:49:17.902644+0000","last_fullsized":"2024-09-13T14:49:17.902644+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":305,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.962693+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:49:31.962693+0000","last_peered":"2024-09-13T14:49:31.962693+0000","last_clean":"2024-09-13T14:49:31.962693+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:49:31.962693+0000","last_undegraded":"2024-09-13T14:49:31.962693+0000","last_fullsized":"2024-09-13T14:49:31.962693+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":228,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909335+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:49:17.909335+0000","last_peered":"2024-09-13T14:49:17.909335+0000","last_clean":"2024-09-13T14:49:17.909335+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:49:17.909335+0000","last_undegraded":"2024-09-13T14:49:17.909335+0000","last_fullsized":"2024-09-13T14:49:17.909335+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893850+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:49:17.893850+0000","last_peered":"2024-09-13T14:49:17.893850+0000","last_clean":"2024-09-13T14:49:17.893850+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:49:17.893850+0000","last_undegraded":"2024-09-13T14:49:17.893850+0000","last_fullsized":"2024-09-13T14:49:17.893850+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"255'8512","reported_seq":8546,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:33.943184+0000","last_change":"2024-09-13T14:47:20.050773+0000","last_active":"2024-09-13T14:49:33.943184+0000","last_peered":"2024-09-13T14:49:33.943184+0000","last_clean":"2024-09-13T14:49:33.943184+0000","last_became_active":"2024-09-13T14:47:20.050243+0000","last_became_peered":"2024-09-13T14:47:20.050243+0000","last_unstale":"2024-09-13T14:49:33.943184+0000","last_undegraded":"2024-09-13T14:49:33.943184+0000","last_fullsized":"2024-09-13T14:49:33.943184+0000","mapping_epoch":229,"log_start":"238'2200","ondisk_log_start":"238'2200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:42.724078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388628480,"num_objects":5930,"num_object_clones":0,"num_object_copies":17790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8512,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.971578+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:49:31.971578+0000","last_peered":"2024-09-13T14:49:31.971578+0000","last_clean":"2024-09-13T14:49:31.971578+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:49:31.971578+0000","last_undegraded":"2024-09-13T14:49:31.971578+0000","last_fullsized":"2024-09-13T14:49:31.971578+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":315,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963329+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:49:31.963329+0000","last_peered":"2024-09-13T14:49:31.963329+0000","last_clean":"2024-09-13T14:49:31.963329+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:49:31.963329+0000","last_undegraded":"2024-09-13T14:49:31.963329+0000","last_fullsized":"2024-09-13T14:49:31.963329+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909424+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:49:17.909424+0000","last_peered":"2024-09-13T14:49:17.909424+0000","last_clean":"2024-09-13T14:49:17.909424+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:49:17.909424+0000","last_undegraded":"2024-09-13T14:49:17.909424+0000","last_fullsized":"2024-09-13T14:49:17.909424+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"255'8513","reported_seq":8547,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:33.867780+0000","last_change":"2024-09-13T14:47:20.106351+0000","last_active":"2024-09-13T14:49:33.867780+0000","last_peered":"2024-09-13T14:49:33.867780+0000","last_clean":"2024-09-13T14:49:33.867780+0000","last_became_active":"2024-09-13T14:47:20.105946+0000","last_became_peered":"2024-09-13T14:47:20.105946+0000","last_unstale":"2024-09-13T14:49:33.867780+0000","last_undegraded":"2024-09-13T14:49:33.867780+0000","last_fullsized":"2024-09-13T14:49:33.867780+0000","mapping_epoch":229,"log_start":"238'1800","ondisk_log_start":"238'1800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:07.223637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388825088,"num_objects":5933,"num_object_clones":0,"num_object_copies":17799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8513,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":237,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908259+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:49:17.908259+0000","last_peered":"2024-09-13T14:49:17.908259+0000","last_clean":"2024-09-13T14:49:17.908259+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:49:17.908259+0000","last_undegraded":"2024-09-13T14:49:17.908259+0000","last_fullsized":"2024-09-13T14:49:17.908259+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":218,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906187+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:49:17.906187+0000","last_peered":"2024-09-13T14:49:17.906187+0000","last_clean":"2024-09-13T14:49:17.906187+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:49:17.906187+0000","last_undegraded":"2024-09-13T14:49:17.906187+0000","last_fullsized":"2024-09-13T14:49:17.906187+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":366,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979246+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:49:31.979246+0000","last_peered":"2024-09-13T14:49:31.979246+0000","last_clean":"2024-09-13T14:49:31.979246+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:49:31.979246+0000","last_undegraded":"2024-09-13T14:49:31.979246+0000","last_fullsized":"2024-09-13T14:49:31.979246+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"255'8632","reported_seq":12205,"reported_epoch":255,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:33.895378+0000","last_change":"2024-09-13T14:49:05.871518+0000","last_active":"2024-09-13T14:49:33.895378+0000","last_peered":"2024-09-13T14:49:33.895378+0000","last_clean":"2024-09-13T14:48:09.848772+0000","last_became_active":"2024-09-13T14:48:12.232150+0000","last_became_peered":"2024-09-13T14:48:12.232150+0000","last_unstale":"2024-09-13T14:49:33.895378+0000","last_undegraded":"2024-09-13T14:49:05.858877+0000","last_fullsized":"2024-09-13T14:48:11.643512+0000","mapping_epoch":242,"log_start":"237'500","ondisk_log_start":"237'500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8132,"log_dups_size":0,"ondisk_log_size":8132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392167424,"num_objects":5984,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8632,"num_write_kb":467712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":114753536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":2427},{"shards":"2,5","objects":3557}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":234,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907684+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:49:17.907684+0000","last_peered":"2024-09-13T14:49:17.907684+0000","last_clean":"2024-09-13T14:49:17.907684+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:49:17.907684+0000","last_undegraded":"2024-09-13T14:49:17.907684+0000","last_fullsized":"2024-09-13T14:49:17.907684+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":238,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900518+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:49:17.900518+0000","last_peered":"2024-09-13T14:49:17.900518+0000","last_clean":"2024-09-13T14:49:17.900518+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:49:17.900518+0000","last_undegraded":"2024-09-13T14:49:17.900518+0000","last_fullsized":"2024-09-13T14:49:17.900518+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908661+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:49:17.908661+0000","last_peered":"2024-09-13T14:49:17.908661+0000","last_clean":"2024-09-13T14:49:17.908661+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:49:17.908661+0000","last_undegraded":"2024-09-13T14:49:17.908661+0000","last_fullsized":"2024-09-13T14:49:17.908661+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"255'8706","reported_seq":8740,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.001269+0000","last_change":"2024-09-13T14:47:20.106142+0000","last_active":"2024-09-13T14:49:34.001269+0000","last_peered":"2024-09-13T14:49:34.001269+0000","last_clean":"2024-09-13T14:49:34.001269+0000","last_became_active":"2024-09-13T14:47:20.105758+0000","last_became_peered":"2024-09-13T14:47:20.105758+0000","last_unstale":"2024-09-13T14:49:34.001269+0000","last_undegraded":"2024-09-13T14:49:34.001269+0000","last_fullsized":"2024-09-13T14:49:34.001269+0000","mapping_epoch":229,"log_start":"238'2400","ondisk_log_start":"238'2400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6306,"log_dups_size":0,"ondisk_log_size":6306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:51.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396492800,"num_objects":6050,"num_object_clones":0,"num_object_copies":18150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8706,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":217,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907605+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:49:17.907605+0000","last_peered":"2024-09-13T14:49:17.907605+0000","last_clean":"2024-09-13T14:49:17.907605+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:49:17.907605+0000","last_undegraded":"2024-09-13T14:49:17.907605+0000","last_fullsized":"2024-09-13T14:49:17.907605+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":345,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.966805+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:49:31.966805+0000","last_peered":"2024-09-13T14:49:31.966805+0000","last_clean":"2024-09-13T14:49:31.966805+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:49:31.966805+0000","last_undegraded":"2024-09-13T14:49:31.966805+0000","last_fullsized":"2024-09-13T14:49:31.966805+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894548+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:49:17.894548+0000","last_peered":"2024-09-13T14:49:17.894548+0000","last_clean":"2024-09-13T14:49:17.894548+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:49:17.894548+0000","last_undegraded":"2024-09-13T14:49:17.894548+0000","last_fullsized":"2024-09-13T14:49:17.894548+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"255'8624","reported_seq":8688,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:33.897433+0000","last_change":"2024-09-13T14:47:20.107374+0000","last_active":"2024-09-13T14:49:33.897433+0000","last_peered":"2024-09-13T14:49:33.897433+0000","last_clean":"2024-09-13T14:49:33.897433+0000","last_became_active":"2024-09-13T14:47:20.107057+0000","last_became_peered":"2024-09-13T14:47:20.107057+0000","last_unstale":"2024-09-13T14:49:33.897433+0000","last_undegraded":"2024-09-13T14:49:33.897433+0000","last_fullsized":"2024-09-13T14:49:33.897433+0000","mapping_epoch":229,"log_start":"237'500","ondisk_log_start":"237'500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8124,"log_dups_size":0,"ondisk_log_size":8124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:05.814456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393478144,"num_objects":6004,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8624,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":385,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979344+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:49:31.979344+0000","last_peered":"2024-09-13T14:49:31.979344+0000","last_clean":"2024-09-13T14:49:31.979344+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:49:31.979344+0000","last_undegraded":"2024-09-13T14:49:31.979344+0000","last_fullsized":"2024-09-13T14:49:31.979344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":216,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909078+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:49:17.909078+0000","last_peered":"2024-09-13T14:49:17.909078+0000","last_clean":"2024-09-13T14:49:17.909078+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:49:17.909078+0000","last_undegraded":"2024-09-13T14:49:17.909078+0000","last_fullsized":"2024-09-13T14:49:17.909078+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907945+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:49:17.907945+0000","last_peered":"2024-09-13T14:49:17.907945+0000","last_clean":"2024-09-13T14:49:17.907945+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:49:17.907945+0000","last_undegraded":"2024-09-13T14:49:17.907945+0000","last_fullsized":"2024-09-13T14:49:17.907945+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"255'8638","reported_seq":8702,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.203420+0000","last_change":"2024-09-13T14:47:20.108173+0000","last_active":"2024-09-13T14:49:35.203420+0000","last_peered":"2024-09-13T14:49:35.203420+0000","last_clean":"2024-09-13T14:49:35.203420+0000","last_became_active":"2024-09-13T14:47:20.107879+0000","last_became_peered":"2024-09-13T14:47:20.107879+0000","last_unstale":"2024-09-13T14:49:35.203420+0000","last_undegraded":"2024-09-13T14:49:35.203420+0000","last_fullsized":"2024-09-13T14:49:35.203420+0000","mapping_epoch":229,"log_start":"237'1600","ondisk_log_start":"237'1600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7038,"log_dups_size":0,"ondisk_log_size":7038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:50.152706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384303104,"num_objects":5864,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8638,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":230,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900396+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:49:17.900396+0000","last_peered":"2024-09-13T14:49:17.900396+0000","last_clean":"2024-09-13T14:49:17.900396+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:49:17.900396+0000","last_undegraded":"2024-09-13T14:49:17.900396+0000","last_fullsized":"2024-09-13T14:49:17.900396+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":231,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893980+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:49:17.893980+0000","last_peered":"2024-09-13T14:49:17.893980+0000","last_clean":"2024-09-13T14:49:17.893980+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:49:17.893980+0000","last_undegraded":"2024-09-13T14:49:17.893980+0000","last_fullsized":"2024-09-13T14:49:17.893980+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":191,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.927203+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:49:17.927203+0000","last_peered":"2024-09-13T14:49:17.927203+0000","last_clean":"2024-09-13T14:49:17.927203+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:49:17.927203+0000","last_undegraded":"2024-09-13T14:49:17.927203+0000","last_fullsized":"2024-09-13T14:49:17.927203+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"255'8564","reported_seq":8600,"reported_epoch":255,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:33.892234+0000","last_change":"2024-09-13T14:48:40.078958+0000","last_active":"2024-09-13T14:49:33.892234+0000","last_peered":"2024-09-13T14:49:33.892234+0000","last_clean":"2024-09-13T14:48:37.811111+0000","last_became_active":"2024-09-13T14:48:40.052181+0000","last_became_peered":"2024-09-13T14:48:40.052181+0000","last_unstale":"2024-09-13T14:49:33.892234+0000","last_undegraded":"2024-09-13T14:48:39.849823+0000","last_fullsized":"2024-09-13T14:48:39.826167+0000","mapping_epoch":249,"log_start":"238'1800","ondisk_log_start":"238'1800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6764,"log_dups_size":0,"ondisk_log_size":6764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:49.521146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388366336,"num_objects":5926,"num_object_clones":0,"num_object_copies":17778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8564,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":4430},{"shards":"3,6","objects":177}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":244,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906118+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:49:17.906118+0000","last_peered":"2024-09-13T14:49:17.906118+0000","last_clean":"2024-09-13T14:49:17.906118+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:49:17.906118+0000","last_undegraded":"2024-09-13T14:49:17.906118+0000","last_fullsized":"2024-09-13T14:49:17.906118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":221,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894377+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:49:17.894377+0000","last_peered":"2024-09-13T14:49:17.894377+0000","last_clean":"2024-09-13T14:49:17.894377+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:49:17.894377+0000","last_undegraded":"2024-09-13T14:49:17.894377+0000","last_fullsized":"2024-09-13T14:49:17.894377+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":73,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.964672+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:49:31.964672+0000","last_peered":"2024-09-13T14:49:31.964672+0000","last_clean":"2024-09-13T14:49:31.964672+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:49:31.964672+0000","last_undegraded":"2024-09-13T14:49:31.964672+0000","last_fullsized":"2024-09-13T14:49:31.964672+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"255'8801","reported_seq":8834,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.569140+0000","last_change":"2024-09-13T14:47:20.051824+0000","last_active":"2024-09-13T14:49:34.569140+0000","last_peered":"2024-09-13T14:49:34.569140+0000","last_clean":"2024-09-13T14:49:34.569140+0000","last_became_active":"2024-09-13T14:47:20.051487+0000","last_became_peered":"2024-09-13T14:47:20.051487+0000","last_unstale":"2024-09-13T14:49:34.569140+0000","last_undegraded":"2024-09-13T14:49:34.569140+0000","last_fullsized":"2024-09-13T14:49:34.569140+0000","mapping_epoch":229,"log_start":"237'600","ondisk_log_start":"237'600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:11.533543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392757248,"num_objects":5993,"num_object_clones":0,"num_object_copies":17979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8801,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908691+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:49:17.908691+0000","last_peered":"2024-09-13T14:49:17.908691+0000","last_clean":"2024-09-13T14:49:17.908691+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:49:17.908691+0000","last_undegraded":"2024-09-13T14:49:17.908691+0000","last_fullsized":"2024-09-13T14:49:17.908691+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894562+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:49:17.894562+0000","last_peered":"2024-09-13T14:49:17.894562+0000","last_clean":"2024-09-13T14:49:17.894562+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:49:17.894562+0000","last_undegraded":"2024-09-13T14:49:17.894562+0000","last_fullsized":"2024-09-13T14:49:17.894562+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893775+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:49:17.893775+0000","last_peered":"2024-09-13T14:49:17.893775+0000","last_clean":"2024-09-13T14:49:17.893775+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:49:17.893775+0000","last_undegraded":"2024-09-13T14:49:17.893775+0000","last_fullsized":"2024-09-13T14:49:17.893775+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":266,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900128+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:49:17.900128+0000","last_peered":"2024-09-13T14:49:17.900128+0000","last_clean":"2024-09-13T14:49:17.900128+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:49:17.900128+0000","last_undegraded":"2024-09-13T14:49:17.900128+0000","last_fullsized":"2024-09-13T14:49:17.900128+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":354,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.966873+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:49:31.966873+0000","last_peered":"2024-09-13T14:49:31.966873+0000","last_clean":"2024-09-13T14:49:31.966873+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:49:31.966873+0000","last_undegraded":"2024-09-13T14:49:31.966873+0000","last_fullsized":"2024-09-13T14:49:31.966873+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":48,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909215+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:49:17.909215+0000","last_peered":"2024-09-13T14:49:17.909215+0000","last_clean":"2024-09-13T14:49:17.909215+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:49:17.909215+0000","last_undegraded":"2024-09-13T14:49:17.909215+0000","last_fullsized":"2024-09-13T14:49:17.909215+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":212,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894657+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:49:17.894657+0000","last_peered":"2024-09-13T14:49:17.894657+0000","last_clean":"2024-09-13T14:49:17.894657+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:49:17.894657+0000","last_undegraded":"2024-09-13T14:49:17.894657+0000","last_fullsized":"2024-09-13T14:49:17.894657+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":398,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983921+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:49:31.983921+0000","last_peered":"2024-09-13T14:49:31.983921+0000","last_clean":"2024-09-13T14:49:31.983921+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:49:31.983921+0000","last_undegraded":"2024-09-13T14:49:31.983921+0000","last_fullsized":"2024-09-13T14:49:31.983921+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":69,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.966556+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:49:31.966556+0000","last_peered":"2024-09-13T14:49:31.966556+0000","last_clean":"2024-09-13T14:49:31.966556+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:49:31.966556+0000","last_undegraded":"2024-09-13T14:49:31.966556+0000","last_fullsized":"2024-09-13T14:49:31.966556+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":404,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983937+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:49:31.983937+0000","last_peered":"2024-09-13T14:49:31.983937+0000","last_clean":"2024-09-13T14:49:31.983937+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:49:31.983937+0000","last_undegraded":"2024-09-13T14:49:31.983937+0000","last_fullsized":"2024-09-13T14:49:31.983937+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908881+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:49:17.908881+0000","last_peered":"2024-09-13T14:49:17.908881+0000","last_clean":"2024-09-13T14:49:17.908881+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:49:17.908881+0000","last_undegraded":"2024-09-13T14:49:17.908881+0000","last_fullsized":"2024-09-13T14:49:17.908881+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908750+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:49:17.908750+0000","last_peered":"2024-09-13T14:49:17.908750+0000","last_clean":"2024-09-13T14:49:17.908750+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:49:17.908750+0000","last_undegraded":"2024-09-13T14:49:17.908750+0000","last_fullsized":"2024-09-13T14:49:17.908750+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":237,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894404+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:49:17.894404+0000","last_peered":"2024-09-13T14:49:17.894404+0000","last_clean":"2024-09-13T14:49:17.894404+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:49:17.894404+0000","last_undegraded":"2024-09-13T14:49:17.894404+0000","last_fullsized":"2024-09-13T14:49:17.894404+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":63,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980206+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:49:31.980206+0000","last_peered":"2024-09-13T14:49:31.980206+0000","last_clean":"2024-09-13T14:49:31.980206+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:49:31.980206+0000","last_undegraded":"2024-09-13T14:49:31.980206+0000","last_fullsized":"2024-09-13T14:49:31.980206+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":362,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980676+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:49:31.980676+0000","last_peered":"2024-09-13T14:49:31.980676+0000","last_clean":"2024-09-13T14:49:31.980676+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:49:31.980676+0000","last_undegraded":"2024-09-13T14:49:31.980676+0000","last_fullsized":"2024-09-13T14:49:31.980676+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6255214616,"num_objects":95448,"num_object_clones":0,"num_object_copies":286344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8895,"num_objects_misplaced":4459,"num_objects_unfound":0,"num_objects_dirty":95448,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138006,"num_write_kb":7470465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2332,"num_bytes_recovered":152829952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18923270144,"data_stored":18923257928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112892,"ondisk_log_size":112892,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365158,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2459624,"kb_used_data":1550236,"kb_used_omap":1,"kb_used_meta":909374,"kb_avail":91285528,"statfs":{"total":95995035648,"available":93476380672,"internally_reserved":0,"allocated":1587441664,"data_stored":1586103468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931199320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561388,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3592292,"kb_used_data":2690456,"kb_used_omap":1,"kb_used_meta":901822,"kb_avail":90152860,"statfs":{"total":95995035648,"available":92316528640,"internally_reserved":0,"allocated":2755026944,"data_stored":2753692691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923466072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790320,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2253912,"kb_used_data":1344460,"kb_used_omap":1,"kb_used_meta":909374,"kb_avail":91491240,"statfs":{"total":95995035648,"available":93687029760,"internally_reserved":0,"allocated":1376727040,"data_stored":1375401169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931199320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986548,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4607268,"kb_used_data":3717020,"kb_used_omap":1,"kb_used_meta":890238,"kb_avail":89137884,"statfs":{"total":95995035648,"available":91277193216,"internally_reserved":0,"allocated":3806228480,"data_stored":3804890469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911604056},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519704,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2212052,"kb_used_data":1687500,"kb_used_omap":4,"kb_used_meta":524539,"kb_avail":91533100,"statfs":{"total":95995035648,"available":93729894400,"internally_reserved":0,"allocated":1728000000,"data_stored":1726674139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":537128384},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304619,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3171228,"kb_used_data":2688720,"kb_used_omap":6,"kb_used_meta":482489,"kb_avail":90573924,"statfs":{"total":95995035648,"available":92747698176,"internally_reserved":0,"allocated":2753249280,"data_stored":2751919837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":494069191},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827923,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1917780,"kb_used_data":1123724,"kb_used_omap":5,"kb_used_meta":794042,"kb_avail":91827372,"statfs":{"total":95995035648,"available":94031228928,"internally_reserved":0,"allocated":1150693376,"data_stored":1149366942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":813099244},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285787,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4455828,"kb_used_data":3794444,"kb_used_omap":5,"kb_used_meta":661306,"kb_avail":89289324,"statfs":{"total":95995035648,"available":91432267776,"internally_reserved":0,"allocated":3885510656,"data_stored":3884183549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":677177403},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3878883328,"data_stored":3878879256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1142489088,"data_stored":1142489088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2731671552,"data_stored":2731671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1723138048,"data_stored":1723138048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3783266304,"data_stored":3783262232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1363673088,"data_stored":1363673088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2729775104,"data_stored":2729771032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1570373632,"data_stored":1570373632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:49:44.960 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:44.961 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:44.962 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:44.969 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:44.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:45.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:45.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:45 smithi003 ceph-mon[104007]: pgmap v848: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 9070/273438 objects degraded (3.317%); 4192/273438 objects misplaced (1.533%); 991 KiB/s, 15 objects/s recovering 2024-09-13T14:49:45.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:45 smithi003 ceph-mon[107278]: pgmap v848: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 9070/273438 objects degraded (3.317%); 4192/273438 objects misplaced (1.533%); 991 KiB/s, 15 objects/s recovering 2024-09-13T14:49:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:45.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:45.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:45.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:45.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:45.418 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:45.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:45 smithi019 ceph-mon[91296]: pgmap v848: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 9070/273438 objects degraded (3.317%); 4192/273438 objects misplaced (1.533%); 991 KiB/s, 15 objects/s recovering 2024-09-13T14:49:45.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:45.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:45.662 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:45.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.701 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:45.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:45.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:45.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:45.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:45.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:46.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:46.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9070/273438 objects degraded (3.317%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9070/273438 objects degraded (3.317%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:46.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:46.245 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:46.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:46.347 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.389 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:46.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:46.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9070/273438 objects degraded (3.317%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:46.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:46.526 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:46.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:46.672 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:46.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:46.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:46.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:46.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:46.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:47.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:47.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:47.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:47.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:47.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:47.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:47 smithi019 ceph-mon[91296]: pgmap v849: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 9254/260775 objects degraded (3.549%); 3915/260775 objects misplaced (1.501%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:47.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:47 smithi003 ceph-mon[107278]: pgmap v849: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 9254/260775 objects degraded (3.549%); 3915/260775 objects misplaced (1.501%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:47 smithi003 ceph-mon[104007]: pgmap v849: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 9254/260775 objects degraded (3.549%); 3915/260775 objects misplaced (1.501%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:49:47.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:47.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:47.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:47.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:47.705 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:47.741 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:47.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:47.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:47.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:47.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:49:47.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:48.012 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:48.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:48.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:48.288 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:48.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:48.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:48.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:48.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:48.660 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.701 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:48.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:48.802 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:48.842 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:48.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:49.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.204 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:49.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:49.305 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:49.339 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:49.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:49 smithi003 ceph-mon[107278]: pgmap v850: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 711 op/s; 9254/260775 objects degraded (3.549%); 3915/260775 objects misplaced (1.501%); 986 KiB/s, 15 objects/s recovering 2024-09-13T14:49:49.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:49 smithi003 ceph-mon[104007]: pgmap v850: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 711 op/s; 9254/260775 objects degraded (3.549%); 3915/260775 objects misplaced (1.501%); 986 KiB/s, 15 objects/s recovering 2024-09-13T14:49:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:49 smithi019 ceph-mon[91296]: pgmap v850: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 711 op/s; 9254/260775 objects degraded (3.549%); 3915/260775 objects misplaced (1.501%); 986 KiB/s, 15 objects/s recovering 2024-09-13T14:49:49.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:49.481 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:49.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:49.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:49.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:49.763 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:49.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:50.014 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:50.050 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:50.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:50.190 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:50.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120809 9.15 9:ad532cb5:::benchmark_data_smithi003_156985_object120808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.868760+0000", 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 391.36053349600002, 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4990805589999998, 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.300 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120809 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868760+0000", 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868760+0000", 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868762+0000", 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.012e-06 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868793+0000", 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1041000000000003e-05 2024-09-13T14:49:50.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868810+0000", 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7215999999999999e-05 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.868820+0000", 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0104e-05 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.871590+0000", 2024-09-13T14:49:50.304 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027697830000000001 2024-09-13T14:49:50.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.871718+0000", 2024-09-13T14:49:50.304 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127603 2024-09-13T14:49:50.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.871855+0000", 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013778600000000001 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.886738+0000", 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014882948999999999 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.886755+0000", 2024-09-13T14:49:50.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6398000000000001e-05 2024-09-13T14:49:50.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.857773+0000", 2024-09-13T14:49:50.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97101844299999995 2024-09-13T14:49:50.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.857785+0000", 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2305e-05 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.301380+0000", 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4435946200000001 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.302932+0000", 2024-09-13T14:49:50.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001552166 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.367740+0000", 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.064808097999999995 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.367840+0000", 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100035 2024-09-13T14:49:50.308 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120816 9.8 9:11d2346f:::benchmark_data_smithi003_156985_object120815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.887707+0000", 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 391.34158606400001, 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6405986719999999, 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.309 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120816 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887707+0000", 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887707+0000", 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887709+0000", 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.483e-06 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887734+0000", 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.561e-05 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887750+0000", 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5675000000000002e-05 2024-09-13T14:49:50.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887758+0000", 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2730000000000002e-06 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.888218+0000", 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00046011899999999999 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.888229+0000", 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0944000000000001e-05 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856495+0000", 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96826583099999997 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856507+0000", 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1977999999999999e-05 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517113+0000", 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.660605506 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517220+0000", 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107624 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517354+0000", 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013387300000000001 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.521270+0000", 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039158910000000003 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.521574+0000", 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030395699999999999 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.525399+0000", 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038248459999999998 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.525464+0000", 2024-09-13T14:49:50.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5418000000000003e-05 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.528306+0000", 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028416439999999999 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120817 9.18 9:1a90c8b1:::benchmark_data_smithi003_156985_object120816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.887776+0000", 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 391.34151766799999, 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6247730489999999, 2024-09-13T14:49:50.316 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120817 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887776+0000", 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887776+0000", 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887776+0000", 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8899999999999999e-07 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887799+0000", 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2682000000000001e-05 2024-09-13T14:49:50.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887807+0000", 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2290000000000003e-06 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.887814+0000", 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3319999999999997e-06 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.888257+0000", 2024-09-13T14:49:50.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000443218 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.888265+0000", 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3799999999999994e-06 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856535+0000", 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96826975900000001 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856544+0000", 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7420000000000003e-06 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.504430+0000", 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6478863850000001 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.504557+0000", 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127195 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:49:50.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.504670+0000", 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011225700000000001 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.510528+0000", 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058583569999999998 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.510581+0000", 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2958999999999999e-05 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.510680+0000", 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8884999999999999e-05 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.510757+0000", 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7373999999999995e-05 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.512549+0000", 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017916060000000001 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.323 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120822 9.18 9:1e2c2cd9:::benchmark_data_smithi003_156985_object120821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.895452+0000", 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 391.33384104499999, 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.623319607, 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120822 2024-09-13T14:49:50.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895452+0000", 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895452+0000", 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895454+0000", 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.409e-06 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895477+0000", 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2963999999999999e-05 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895490+0000", 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3627e-05 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895499+0000", 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3599999999999996e-06 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895564+0000", 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5571e-05 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.895576+0000", 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2102e-05 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856568+0000", 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96099208400000002 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.856576+0000", 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8879999999999997e-06 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.508342+0000", 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6517653860000001 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.508521+0000", 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000179218 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.508709+0000", 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000188443 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.512606+0000", 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003896254 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.512658+0000", 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2846000000000002e-05 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517156+0000", 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044978889999999997 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.517196+0000", 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9301999999999997e-05 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.518772+0000", 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001576264 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120825 9.15 9:aa3ed837:::benchmark_data_smithi003_156985_object120824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:49:50.331 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.899645+0000", 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 391.32964870000001, 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.477297343, 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120825 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899645+0000", 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899645+0000", 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899647+0000", 2024-09-13T14:49:50.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9379999999999999e-06 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899671+0000", 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4493000000000001e-05 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899686+0000", 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5183e-05 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899695+0000", 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6419999999999996e-06 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899772+0000", 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7247000000000001e-05 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.899785+0000", 2024-09-13T14:49:50.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3063e-05 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.857816+0000", 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95803042299999996 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.858198+0000", 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00038188000000000002 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.367951+0000", 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5097530529999998 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.368107+0000", 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156074 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.368287+0000", 2024-09-13T14:49:50.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000180416 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.372843+0000", 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045564170000000001 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.373071+0000", 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000227963 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.374047+0000", 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00097598800000000001 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.374110+0000", 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2889e-05 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:21.376942+0000", 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028316740000000002 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.339 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:136803 9.8 9:11e973c6:::benchmark_data_smithi003_156985_object15887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:58.441150+0000", 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 351.78814365900001, 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.53725493300000005, 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136803 2024-09-13T14:49:50.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441150+0000", 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441150+0000", 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441150+0000", 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.16e-07 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441155+0000", 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0249999999999996e-06 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441165+0000", 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0662000000000001e-05 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.441173+0000", 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2330000000000006e-06 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.461177+0000", 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020003204 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.546296+0000", 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.085119427999999997 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.956515+0000", 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.41021918899999998 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.956747+0000", 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000231771 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.957329+0000", 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058164000000000002 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.963091+0000", 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057619639999999996 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.972590+0000", 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094997450000000004 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.975825+0000", 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032349420000000002 2024-09-13T14:49:50.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.975889+0000", 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3830000000000004e-05 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:58.978405+0000", 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025154840000000001 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:215903 9.15 9:add94de9:::benchmark_data_smithi003_156985_object94987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.677703+0000", 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 226.55159014700001, 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97887381799999995, 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 215903 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677703+0000", 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677703+0000", 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677704+0000", 2024-09-13T14:49:50.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.265e-06 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677709+0000", 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9609999999999996e-06 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677725+0000", 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5999999999999999e-05 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677733+0000", 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6399999999999997e-06 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677802+0000", 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9051e-05 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.677814+0000", 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.208e-05 2024-09-13T14:49:50.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.653049+0000", 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97523451400000005 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.653161+0000", 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112428 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:50.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.653586+0000", 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042491799999999999 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656399+0000", 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028127040000000001 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656414+0000", 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5299999999999999e-05 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656477+0000", 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2767000000000004e-05 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656534+0000", 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6839999999999998e-05 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656577+0000", 2024-09-13T14:49:50.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3350000000000003e-05 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:215952 9.15 9:a81f33b3:::benchmark_data_smithi003_156985_object95036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.737954+0000", 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 226.49133926100001, 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.932049456, 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 215952 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737954+0000", 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737954+0000", 2024-09-13T14:49:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737956+0000", 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7379999999999999e-06 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737961+0000", 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3009999999999997e-06 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737972+0000", 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0822000000000001e-05 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.737979+0000", 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9460000000000001e-06 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.748544+0000", 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010564726999999999 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.748552+0000", 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1820000000000006e-06 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656616+0000", 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90806452999999998 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.656726+0000", 2024-09-13T14:49:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010941400000000001 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.657068+0000", 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000342274 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663538+0000", 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064696969999999999 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663609+0000", 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1557000000000006e-05 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.669908+0000", 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062991239999999997 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.669960+0000", 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1600000000000001e-05 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670004+0000", 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3544000000000002e-05 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216095 9.15 9:ae47eb7d:::benchmark_data_smithi003_156985_object95179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.897037+0000", 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 226.33225678599999, 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.77305581400000001, 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.361 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216095 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897037+0000", 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897037+0000", 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897038+0000", 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3400000000000001e-06 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897043+0000", 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8189999999999999e-06 2024-09-13T14:49:50.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897054+0000", 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1031e-05 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897060+0000", 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4989999999999999e-06 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897107+0000", 2024-09-13T14:49:50.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6891000000000002e-05 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.897115+0000", 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7689999999999996e-06 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.660201+0000", 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.76308632899999995 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.660327+0000", 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012528999999999999 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.660801+0000", 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000474184 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663648+0000", 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028476769999999998 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.666408+0000", 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002759863 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670013+0000", 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036042719999999999 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670055+0000", 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2967999999999999e-05 2024-09-13T14:49:50.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670092+0000", 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6881999999999997e-05 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216126 9.15 9:aeb4fb92:::benchmark_data_smithi003_156985_object95210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.937787+0000", 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 226.29150663999999, 2024-09-13T14:49:50.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.73238416799999995, 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216126 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937787+0000", 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937787+0000", 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937788+0000", 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.581e-06 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937794+0000", 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2229999999999998e-06 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937807+0000", 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3628999999999999e-05 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937814+0000", 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1300000000000003e-06 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937879+0000", 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5115000000000003e-05 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.937888+0000", 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8929999999999994e-06 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663676+0000", 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.72578801999999998 2024-09-13T14:49:50.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.663788+0000", 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111539 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.664178+0000", 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039035099999999999 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.669828+0000", 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056497170000000003 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.669895+0000", 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6657999999999998e-05 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670102+0000", 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020704999999999999 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670138+0000", 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6746999999999999e-05 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670171+0000", 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2515000000000003e-05 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216152 9.15 9:aaebaddc:::benchmark_data_smithi003_156985_object95236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:03.973029+0000", 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 226.25626470200001, 2024-09-13T14:49:50.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.70077610199999996, 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216152 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973029+0000", 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973029+0000", 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973030+0000", 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1400000000000001e-06 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973035+0000", 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1059999999999999e-06 2024-09-13T14:49:50.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973046+0000", 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1299e-05 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973053+0000", 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7700000000000004e-06 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973111+0000", 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8301000000000003e-05 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:03.973120+0000", 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1300000000000007e-06 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.666444+0000", 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.69332316500000002 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.666570+0000", 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012622499999999999 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.666968+0000", 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039803899999999999 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670221+0000", 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032530189999999998 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670280+0000", 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9024000000000003e-05 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673518+0000", 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032381599999999999 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673759+0000", 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000241308 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673805+0000", 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5416e-05 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216195 9.15 9:a896c16a:::benchmark_data_smithi003_156985_object95279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:04.016195+0000", 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 226.21309801000001, 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.66127026700000002, 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.382 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216195 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016195+0000", 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016195+0000", 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016197+0000", 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8279999999999999e-06 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016205+0000", 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8369999999999997e-06 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016227+0000", 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1956000000000001e-05 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016238+0000", 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.062e-05 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016319+0000", 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1804000000000005e-05 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.016331+0000", 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1819e-05 2024-09-13T14:49:50.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670325+0000", 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.65399382699999997 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670436+0000", 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111094 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.670811+0000", 2024-09-13T14:49:50.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000374741 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677294+0000", 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064832329999999997 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677352+0000", 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7731e-05 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677363+0000", 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1659e-05 2024-09-13T14:49:50.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677397+0000", 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3451000000000002e-05 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677466+0000", 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8666999999999998e-05 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216200 9.15 9:af738736:::benchmark_data_smithi003_156985_object95284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:04.025460+0000", 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 226.20383311000001, 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.65968305100000002, 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216200 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025460+0000", 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025460+0000", 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025462+0000", 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8619999999999999e-06 2024-09-13T14:49:50.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025469+0000", 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0720000000000002e-06 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025487+0000", 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7937000000000001e-05 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025499+0000", 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2038000000000001e-05 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025585+0000", 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5599000000000005e-05 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.025597+0000", 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2665e-05 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673839+0000", 2024-09-13T14:49:50.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.64824168299999996 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.673947+0000", 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107537 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.674396+0000", 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044932300000000001 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.684929+0000", 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01053345 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.684986+0000", 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6527999999999998e-05 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685046+0000", 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9769000000000003e-05 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685101+0000", 2024-09-13T14:49:50.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4866e-05 2024-09-13T14:49:50.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685143+0000", 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2722000000000002e-05 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:216286 9.15 9:ad9baf3d:::benchmark_data_smithi003_156985_object95370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:04.132237+0000", 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 226.09705611199999, 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.55320481799999999, 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.396 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.397 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.397 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.397 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:49:50.397 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:49:50.397 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216286 2024-09-13T14:49:50.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.397 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132237+0000", 2024-09-13T14:49:50.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132237+0000", 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132238+0000", 2024-09-13T14:49:50.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3499999999999996e-07 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132240+0000", 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3539999999999998e-06 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132246+0000", 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1689999999999997e-06 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132251+0000", 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7269999999999998e-06 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132412+0000", 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016087499999999999 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.132422+0000", 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7589999999999992e-06 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677499+0000", 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.54507772099999996 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.677613+0000", 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114028 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.678090+0000", 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047650500000000001 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685159+0000", 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070695580000000001 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685199+0000", 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0033000000000001e-05 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685359+0000", 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015973499999999999 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685408+0000", 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8856000000000001e-05 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:04.685442+0000", 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3963e-05 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.404 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:79800 10.1 e245/229 10:8fcf9e35:::benchmark_data_smithi003_197468_object79799:head v 245'4934, mlcod=245'4932)", 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:48:24.375369+0000", 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 85.853923867999995, 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.30018646599999999, 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.375369+0000", 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.375369+0000", 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.375371+0000", 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1030000000000001e-06 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.375423+0000", 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2605999999999997e-05 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.375433+0000", 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9219999999999992e-06 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.375439+0000", 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0159999999999997e-06 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.375494+0000", 2024-09-13T14:49:50.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5328999999999998e-05 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.375513+0000", 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8851999999999998e-05 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.675471+0000", 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.29995809600000001 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.675556+0000", 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4542000000000005e-05 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:79805 10.8 10:18c284a0:::benchmark_data_smithi003_197468_object79804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:48:24.377980+0000", 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 85.851313239000007, 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.30062833799999999, 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.409 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 79805 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.377980+0000", 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.377980+0000", 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.377982+0000", 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.421e-06 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.378005+0000", 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3275999999999999e-05 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.378017+0000", 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2106000000000001e-05 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.378024+0000", 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2629999999999997e-06 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.379613+0000", 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015891880000000001 2024-09-13T14:49:50.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.379757+0000", 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000143203 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.379941+0000", 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000184592 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.674933+0000", 2024-09-13T14:49:50.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.29499218300000002 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.675063+0000", 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013005100000000001 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.678499+0000", 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034354630000000001 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.678560+0000", 2024-09-13T14:49:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1421999999999996e-05 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:24.678608+0000", 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8170000000000001e-05 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149393 10.4 10:29621cc7:::benchmark_data_smithi003_197468_object32664:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.307435+0000", 2024-09-13T14:49:50.415 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.921857868, 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33918270499999997, 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149393 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.307435+0000", 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.307435+0000", 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.307437+0000", 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.268e-06 2024-09-13T14:49:50.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.307441+0000", 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.138e-06 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.307451+0000", 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0504e-05 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.307458+0000", 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8460000000000002e-06 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.307782+0000", 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032379199999999999 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.307917+0000", 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000134719 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308065+0000", 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014855399999999999 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.319583+0000", 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011517846999999999 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.319624+0000", 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0371999999999999e-05 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646514+0000", 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32689057700000002 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646571+0000", 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7240000000000001e-05 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646618+0000", 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6848000000000003e-05 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149396 10.4 10:22a31fdc:::benchmark_data_smithi003_197468_object32667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.309336+0000", 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.9199577670000001, 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33742976099999999, 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149396 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309336+0000", 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309336+0000", 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309337+0000", 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.195e-06 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309341+0000", 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4689999999999999e-06 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309352+0000", 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1141999999999999e-05 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309360+0000", 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4479999999999997e-06 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.311804+0000", 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024446730000000001 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.311917+0000", 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112763 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.312024+0000", 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010710800000000001 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.319642+0000", 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076171629999999997 2024-09-13T14:49:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322830+0000", 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031883879999999999 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646678+0000", 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32384833499999999 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646728+0000", 2024-09-13T14:49:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9768000000000002e-05 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646765+0000", 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7308999999999998e-05 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149401 10.4 10:21db60e8:::benchmark_data_smithi003_197468_object32672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.315770+0000", 2024-09-13T14:49:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.913523412, 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33646247600000001, 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149401 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.315770+0000", 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.315770+0000", 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.315771+0000", 2024-09-13T14:49:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.068e-06 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.315776+0000", 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3120000000000001e-06 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.315789+0000", 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.253e-05 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.315797+0000", 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.568e-06 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.315946+0000", 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014904100000000001 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316078+0000", 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013137199999999999 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316200+0000", 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121773 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.319707+0000", 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035069820000000001 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322882+0000", 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031752629999999998 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.652111+0000", 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.329229145 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.652174+0000", 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2977000000000004e-05 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.652232+0000", 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8445e-05 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149410 10.4 10:2596f49b:::benchmark_data_smithi003_197468_object32681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.332216+0000", 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.8970777060000001, 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32045011499999998, 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:49:50.435 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149410 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.332216+0000", 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.332216+0000", 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.332217+0000", 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5859999999999999e-06 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:49:50.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.332223+0000", 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.93e-06 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.332239+0000", 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5866999999999999e-05 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.332247+0000", 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6199999999999999e-06 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.332299+0000", 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2042000000000001e-05 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.332390+0000", 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1545000000000005e-05 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.332472+0000", 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1892999999999997e-05 2024-09-13T14:49:50.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.335935+0000", 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034630849999999999 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.336627+0000", 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069205199999999997 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.652498+0000", 2024-09-13T14:49:50.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.31587036699999999 2024-09-13T14:49:50.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:49:50.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.652622+0000", 2024-09-13T14:49:50.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000124232 2024-09-13T14:49:50.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:49:50.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:49:50.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:49:50.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:50.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.652666+0000", 2024-09-13T14:49:50.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3896000000000001e-05 2024-09-13T14:49:50.442 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.442 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.442 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.442 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:49:50.442 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:49:50.442 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:49:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:50.459 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:49:50.459 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:49:50.459 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:49:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:50.460 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:50.460 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:50.489 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:49:50.489 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:50.524 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:49:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:50.543 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:50.553 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:49:50.553 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.578 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:50.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:50.587 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:49:50.587 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:50.621 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:49:50.621 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:50.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:50.683 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:49:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:49:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:50.719 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:50.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:50.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:50.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:50.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:51.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:51.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:51.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:51.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:51.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:51.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.423 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:51.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:51 smithi019 ceph-mon[91296]: pgmap v851: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9359/249585 objects degraded (3.750%); 3686/249585 objects misplaced (1.477%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:49:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:51 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2481396094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9359/249585 objects degraded (3.750%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:51 smithi003 ceph-mon[107278]: pgmap v851: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9359/249585 objects degraded (3.750%); 3686/249585 objects misplaced (1.477%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:49:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:51 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2481396094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9359/249585 objects degraded (3.750%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:51.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:51 smithi003 ceph-mon[104007]: pgmap v851: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9359/249585 objects degraded (3.750%); 3686/249585 objects misplaced (1.477%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:49:51.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:51 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2481396094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:51.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9359/249585 objects degraded (3.750%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:51.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:49:51.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:51.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:51.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:51.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:51.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:51.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:52.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:52.047 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:52.149 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:52.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:49:52.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:49:52.189 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:52.209 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:52.291 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:52.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:52.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:49:52.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:52.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:52.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:52.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:52.636 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:52.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:52.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:52.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:52.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:53.004 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:53.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:53 smithi003 ceph-mon[104007]: pgmap v852: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 9359/248007 objects degraded (3.774%); 3686/248007 objects misplaced (1.486%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:53.061 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:53 smithi003 ceph-mon[107278]: pgmap v852: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 9359/248007 objects degraded (3.774%); 3686/248007 objects misplaced (1.486%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:53.062 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:53.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:53.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:53.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:53.352 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:53 smithi019 ceph-mon[91296]: pgmap v852: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 9359/248007 objects degraded (3.774%); 3686/248007 objects misplaced (1.486%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:53.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:53.590 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:53.625 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:53.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:53.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:53.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:53.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:53.998 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:54.045 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:54.146 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:54.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:49:54.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:54.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:54.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:49:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:49:54] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:49:54.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:54.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:54.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:54.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:54.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:54.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:54.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.931 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:54.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:55.032 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:55.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.074 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:55.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:55.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:55.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:55.279 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:55 smithi003 ceph-mon[107278]: pgmap v853: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 9359/248007 objects degraded (3.774%); 3686/248007 objects misplaced (1.486%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:55.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:55 smithi003 ceph-mon[104007]: pgmap v853: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 9359/248007 objects degraded (3.774%); 3686/248007 objects misplaced (1.486%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:55.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:55.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:55 smithi019 ceph-mon[91296]: pgmap v853: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 9359/248007 objects degraded (3.774%); 3686/248007 objects misplaced (1.486%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:49:55.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:55.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:55.592 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:55.630 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:55.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:55.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:55.866 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:55.893 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:55.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:56.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.037 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:56.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:56.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:56.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:56.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:56.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:56.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:56.416 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9359/248007 objects degraded (3.774%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9359/248007 objects degraded (3.774%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9359/248007 objects degraded (3.774%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:56.456 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:56.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:56.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:56.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:56.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.750 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:56.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:56.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:56.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:56.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:57.021 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:49:57.022 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:49:57.030 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:57.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:57.131 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:57.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:57.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:57.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:57.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:57 smithi019 ceph-mon[91296]: pgmap v854: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 9533/234984 objects degraded (4.057%); 3409/234984 objects misplaced (1.451%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:49:57.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:57 smithi003 ceph-mon[104007]: pgmap v854: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 9533/234984 objects degraded (4.057%); 3409/234984 objects misplaced (1.451%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:49:57.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:57 smithi003 ceph-mon[107278]: pgmap v854: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 9533/234984 objects degraded (4.057%); 3409/234984 objects misplaced (1.451%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:49:57.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:57.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:57.570 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:57.603 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:57.705 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.751 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:57.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:57.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:57.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:57.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:58.032 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:58.051 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":854,"stamp":"2024-09-13T14:49:55.784913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5132157112,"num_objects":78328,"num_object_clones":0,"num_object_copies":234984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9533,"num_objects_misplaced":3409,"num_objects_unfound":0,"num_objects_dirty":78328,"num_whiteouts":0,"num_read":15,"num_read_kb":35,"num_write":155223,"num_write_kb":7470571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2762,"num_bytes_recovered":180710961,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114897,"ondisk_log_size":114897,"up":327,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":333,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21721680,"kb_used_data":15616144,"kb_used_omap":29,"kb_used_meta":6105186,"kb_avail":728239536,"statfs":{"total":767960285184,"available":745717284864,"internally_reserved":0,"allocated":15990931456,"data_stored":15980286280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6251710990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-840040448,"num_objects":-12818,"num_object_clones":0,"num_object_copies":-38454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":463,"num_objects_misplaced":-783,"num_objects_unfound":0,"num_objects_dirty":-12818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12818,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004442"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":388,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979805+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:49:31.979805+0000","last_peered":"2024-09-13T14:49:31.979805+0000","last_clean":"2024-09-13T14:49:31.979805+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:49:31.979805+0000","last_undegraded":"2024-09-13T14:49:31.979805+0000","last_fullsized":"2024-09-13T14:49:31.979805+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":227,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.905829+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:49:17.905829+0000","last_peered":"2024-09-13T14:49:17.905829+0000","last_clean":"2024-09-13T14:49:17.905829+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:49:17.905829+0000","last_undegraded":"2024-09-13T14:49:17.905829+0000","last_fullsized":"2024-09-13T14:49:17.905829+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":18,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909220+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:49:17.909220+0000","last_peered":"2024-09-13T14:49:17.909220+0000","last_clean":"2024-09-13T14:49:17.909220+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:49:17.909220+0000","last_undegraded":"2024-09-13T14:49:17.909220+0000","last_fullsized":"2024-09-13T14:49:17.909220+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":347,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.962833+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:49:31.962833+0000","last_peered":"2024-09-13T14:49:31.962833+0000","last_clean":"2024-09-13T14:49:31.962833+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:49:31.962833+0000","last_undegraded":"2024-09-13T14:49:31.962833+0000","last_fullsized":"2024-09-13T14:49:31.962833+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":201,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908978+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:49:17.908978+0000","last_peered":"2024-09-13T14:49:17.908978+0000","last_clean":"2024-09-13T14:49:17.908978+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:49:17.908978+0000","last_undegraded":"2024-09-13T14:49:17.908978+0000","last_fullsized":"2024-09-13T14:49:17.908978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":234,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906441+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:49:17.906441+0000","last_peered":"2024-09-13T14:49:17.906441+0000","last_clean":"2024-09-13T14:49:17.906441+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:49:17.906441+0000","last_undegraded":"2024-09-13T14:49:17.906441+0000","last_fullsized":"2024-09-13T14:49:17.906441+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":218,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908986+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:49:17.908986+0000","last_peered":"2024-09-13T14:49:17.908986+0000","last_clean":"2024-09-13T14:49:17.908986+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:49:17.908986+0000","last_undegraded":"2024-09-13T14:49:17.908986+0000","last_fullsized":"2024-09-13T14:49:17.908986+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":264,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907279+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:49:17.907279+0000","last_peered":"2024-09-13T14:49:17.907279+0000","last_clean":"2024-09-13T14:49:17.907279+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:49:17.907279+0000","last_undegraded":"2024-09-13T14:49:17.907279+0000","last_fullsized":"2024-09-13T14:49:17.907279+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":376,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979502+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:49:31.979502+0000","last_peered":"2024-09-13T14:49:31.979502+0000","last_clean":"2024-09-13T14:49:31.979502+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:49:31.979502+0000","last_undegraded":"2024-09-13T14:49:31.979502+0000","last_fullsized":"2024-09-13T14:49:31.979502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":371,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.974194+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:49:31.974194+0000","last_peered":"2024-09-13T14:49:31.974194+0000","last_clean":"2024-09-13T14:49:31.974194+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:49:31.974194+0000","last_undegraded":"2024-09-13T14:49:31.974194+0000","last_fullsized":"2024-09-13T14:49:31.974194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":236,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906564+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:49:17.906564+0000","last_peered":"2024-09-13T14:49:17.906564+0000","last_clean":"2024-09-13T14:49:17.906564+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:49:17.906564+0000","last_undegraded":"2024-09-13T14:49:17.906564+0000","last_fullsized":"2024-09-13T14:49:17.906564+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":314,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.966303+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:49:31.966303+0000","last_peered":"2024-09-13T14:49:31.966303+0000","last_clean":"2024-09-13T14:49:31.966303+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:49:31.966303+0000","last_undegraded":"2024-09-13T14:49:31.966303+0000","last_fullsized":"2024-09-13T14:49:31.966303+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.981793+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:49:31.981793+0000","last_peered":"2024-09-13T14:49:31.981793+0000","last_clean":"2024-09-13T14:49:31.981793+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:49:31.981793+0000","last_undegraded":"2024-09-13T14:49:31.981793+0000","last_fullsized":"2024-09-13T14:49:31.981793+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":407,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983793+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:49:31.983793+0000","last_peered":"2024-09-13T14:49:31.983793+0000","last_clean":"2024-09-13T14:49:31.983793+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:49:31.983793+0000","last_undegraded":"2024-09-13T14:49:31.983793+0000","last_fullsized":"2024-09-13T14:49:31.983793+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":346,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963239+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:49:31.963239+0000","last_peered":"2024-09-13T14:49:31.963239+0000","last_clean":"2024-09-13T14:49:31.963239+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:49:31.963239+0000","last_undegraded":"2024-09-13T14:49:31.963239+0000","last_fullsized":"2024-09-13T14:49:31.963239+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":217,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.927273+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:49:17.927273+0000","last_peered":"2024-09-13T14:49:17.927273+0000","last_clean":"2024-09-13T14:49:17.927273+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:49:17.927273+0000","last_undegraded":"2024-09-13T14:49:17.927273+0000","last_fullsized":"2024-09-13T14:49:17.927273+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":314,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.962916+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:49:31.962916+0000","last_peered":"2024-09-13T14:49:31.962916+0000","last_clean":"2024-09-13T14:49:31.962916+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:49:31.962916+0000","last_undegraded":"2024-09-13T14:49:31.962916+0000","last_fullsized":"2024-09-13T14:49:31.962916+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":390,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983164+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:49:31.983164+0000","last_peered":"2024-09-13T14:49:31.983164+0000","last_clean":"2024-09-13T14:49:31.983164+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:49:31.983164+0000","last_undegraded":"2024-09-13T14:49:31.983164+0000","last_fullsized":"2024-09-13T14:49:31.983164+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":319,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963146+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:49:31.963146+0000","last_peered":"2024-09-13T14:49:31.963146+0000","last_clean":"2024-09-13T14:49:31.963146+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:49:31.963146+0000","last_undegraded":"2024-09-13T14:49:31.963146+0000","last_fullsized":"2024-09-13T14:49:31.963146+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909129+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:49:17.909129+0000","last_peered":"2024-09-13T14:49:17.909129+0000","last_clean":"2024-09-13T14:49:17.909129+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:49:17.909129+0000","last_undegraded":"2024-09-13T14:49:17.909129+0000","last_fullsized":"2024-09-13T14:49:17.909129+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":69,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.973959+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:49:31.973959+0000","last_peered":"2024-09-13T14:49:31.973959+0000","last_clean":"2024-09-13T14:49:31.973959+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:49:31.973959+0000","last_undegraded":"2024-09-13T14:49:31.973959+0000","last_fullsized":"2024-09-13T14:49:31.973959+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":18,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893668+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:49:17.893668+0000","last_peered":"2024-09-13T14:49:17.893668+0000","last_clean":"2024-09-13T14:49:17.893668+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:49:17.893668+0000","last_undegraded":"2024-09-13T14:49:17.893668+0000","last_fullsized":"2024-09-13T14:49:17.893668+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":156,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900455+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:49:17.900455+0000","last_peered":"2024-09-13T14:49:17.900455+0000","last_clean":"2024-09-13T14:49:17.900455+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:49:17.900455+0000","last_undegraded":"2024-09-13T14:49:17.900455+0000","last_fullsized":"2024-09-13T14:49:17.900455+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":224,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907731+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:49:17.907731+0000","last_peered":"2024-09-13T14:49:17.907731+0000","last_clean":"2024-09-13T14:49:17.907731+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:49:17.907731+0000","last_undegraded":"2024-09-13T14:49:17.907731+0000","last_fullsized":"2024-09-13T14:49:17.907731+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":217,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907339+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:49:17.907339+0000","last_peered":"2024-09-13T14:49:17.907339+0000","last_clean":"2024-09-13T14:49:17.907339+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:49:17.907339+0000","last_undegraded":"2024-09-13T14:49:17.907339+0000","last_fullsized":"2024-09-13T14:49:17.907339+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"255'9562","reported_seq":9596,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.896768+0000","last_change":"2024-09-13T14:47:20.107455+0000","last_active":"2024-09-13T14:49:53.896768+0000","last_peered":"2024-09-13T14:49:53.896768+0000","last_clean":"2024-09-13T14:49:53.896768+0000","last_became_active":"2024-09-13T14:47:20.107142+0000","last_became_peered":"2024-09-13T14:47:20.107142+0000","last_unstale":"2024-09-13T14:49:53.896768+0000","last_undegraded":"2024-09-13T14:49:53.896768+0000","last_fullsized":"2024-09-13T14:49:53.896768+0000","mapping_epoch":229,"log_start":"239'2800","ondisk_log_start":"239'2800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6762,"log_dups_size":0,"ondisk_log_size":6762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:19.189519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321519616,"num_objects":4906,"num_object_clones":0,"num_object_copies":14718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":462976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":200,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906070+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:49:17.906070+0000","last_peered":"2024-09-13T14:49:17.906070+0000","last_clean":"2024-09-13T14:49:17.906070+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:49:17.906070+0000","last_undegraded":"2024-09-13T14:49:17.906070+0000","last_fullsized":"2024-09-13T14:49:17.906070+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":250,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.927408+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:49:17.927408+0000","last_peered":"2024-09-13T14:49:17.927408+0000","last_clean":"2024-09-13T14:49:17.927408+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:49:17.927408+0000","last_undegraded":"2024-09-13T14:49:17.927408+0000","last_fullsized":"2024-09-13T14:49:17.927408+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":388,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.984010+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:49:31.984010+0000","last_peered":"2024-09-13T14:49:31.984010+0000","last_clean":"2024-09-13T14:49:31.984010+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:49:31.984010+0000","last_undegraded":"2024-09-13T14:49:31.984010+0000","last_fullsized":"2024-09-13T14:49:31.984010+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":76,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963621+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:49:31.963621+0000","last_peered":"2024-09-13T14:49:31.963621+0000","last_clean":"2024-09-13T14:49:31.963621+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:49:31.963621+0000","last_undegraded":"2024-09-13T14:49:31.963621+0000","last_fullsized":"2024-09-13T14:49:31.963621+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"255'9699","reported_seq":9733,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:54.066892+0000","last_change":"2024-09-13T14:47:20.054114+0000","last_active":"2024-09-13T14:49:54.066892+0000","last_peered":"2024-09-13T14:49:54.066892+0000","last_clean":"2024-09-13T14:49:54.066892+0000","last_became_active":"2024-09-13T14:47:20.053767+0000","last_became_peered":"2024-09-13T14:47:20.053767+0000","last_unstale":"2024-09-13T14:49:54.066892+0000","last_undegraded":"2024-09-13T14:49:54.066892+0000","last_fullsized":"2024-09-13T14:49:54.066892+0000","mapping_epoch":229,"log_start":"239'3400","ondisk_log_start":"239'3400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6299,"log_dups_size":0,"ondisk_log_size":6299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:48.035191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321323008,"num_objects":4903,"num_object_clones":0,"num_object_copies":14709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":357,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963022+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:49:31.963022+0000","last_peered":"2024-09-13T14:49:31.963022+0000","last_clean":"2024-09-13T14:49:31.963022+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:49:31.963022+0000","last_undegraded":"2024-09-13T14:49:31.963022+0000","last_fullsized":"2024-09-13T14:49:31.963022+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":91,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.899900+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:49:17.899900+0000","last_peered":"2024-09-13T14:49:17.899900+0000","last_clean":"2024-09-13T14:49:17.899900+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:49:17.899900+0000","last_undegraded":"2024-09-13T14:49:17.899900+0000","last_fullsized":"2024-09-13T14:49:17.899900+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":345,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.967020+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:49:31.967020+0000","last_peered":"2024-09-13T14:49:31.967020+0000","last_clean":"2024-09-13T14:49:31.967020+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:49:31.967020+0000","last_undegraded":"2024-09-13T14:49:31.967020+0000","last_fullsized":"2024-09-13T14:49:31.967020+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906883+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:49:17.906883+0000","last_peered":"2024-09-13T14:49:17.906883+0000","last_clean":"2024-09-13T14:49:17.906883+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:49:17.906883+0000","last_undegraded":"2024-09-13T14:49:17.906883+0000","last_fullsized":"2024-09-13T14:49:17.906883+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"255'9676","reported_seq":9711,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.924239+0000","last_change":"2024-09-13T14:47:20.052876+0000","last_active":"2024-09-13T14:49:53.924239+0000","last_peered":"2024-09-13T14:49:53.924239+0000","last_clean":"2024-09-13T14:49:53.924239+0000","last_became_active":"2024-09-13T14:47:20.052566+0000","last_became_peered":"2024-09-13T14:47:20.052566+0000","last_unstale":"2024-09-13T14:49:53.924239+0000","last_undegraded":"2024-09-13T14:49:53.924239+0000","last_fullsized":"2024-09-13T14:49:53.924239+0000","mapping_epoch":229,"log_start":"239'3000","ondisk_log_start":"239'3000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6676,"log_dups_size":0,"ondisk_log_size":6676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:56.317980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324468760,"num_objects":4952,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9676,"num_write_kb":468033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":374,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979425+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:49:31.979425+0000","last_peered":"2024-09-13T14:49:31.979425+0000","last_clean":"2024-09-13T14:49:31.979425+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:49:31.979425+0000","last_undegraded":"2024-09-13T14:49:31.979425+0000","last_fullsized":"2024-09-13T14:49:31.979425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":230,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894056+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:49:17.894056+0000","last_peered":"2024-09-13T14:49:17.894056+0000","last_clean":"2024-09-13T14:49:17.894056+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:49:17.894056+0000","last_undegraded":"2024-09-13T14:49:17.894056+0000","last_fullsized":"2024-09-13T14:49:17.894056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":330,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.974139+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:49:31.974139+0000","last_peered":"2024-09-13T14:49:31.974139+0000","last_clean":"2024-09-13T14:49:31.974139+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:49:31.974139+0000","last_undegraded":"2024-09-13T14:49:31.974139+0000","last_fullsized":"2024-09-13T14:49:31.974139+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":191,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.899663+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:49:17.899663+0000","last_peered":"2024-09-13T14:49:17.899663+0000","last_clean":"2024-09-13T14:49:17.899663+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:49:17.899663+0000","last_undegraded":"2024-09-13T14:49:17.899663+0000","last_fullsized":"2024-09-13T14:49:17.899663+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"255'9819","reported_seq":9848,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:54.013439+0000","last_change":"2024-09-13T14:47:20.052419+0000","last_active":"2024-09-13T14:49:54.013439+0000","last_peered":"2024-09-13T14:49:54.013439+0000","last_clean":"2024-09-13T14:49:54.013439+0000","last_became_active":"2024-09-13T14:47:20.051845+0000","last_became_peered":"2024-09-13T14:47:20.051845+0000","last_unstale":"2024-09-13T14:49:54.013439+0000","last_undegraded":"2024-09-13T14:49:54.013439+0000","last_fullsized":"2024-09-13T14:49:54.013439+0000","mapping_epoch":229,"log_start":"239'2914","ondisk_log_start":"239'2914","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6905,"log_dups_size":0,"ondisk_log_size":6905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.420332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319881216,"num_objects":4881,"num_object_clones":0,"num_object_copies":14643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9819,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909136+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:49:17.909136+0000","last_peered":"2024-09-13T14:49:17.909136+0000","last_clean":"2024-09-13T14:49:17.909136+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:49:17.909136+0000","last_undegraded":"2024-09-13T14:49:17.909136+0000","last_fullsized":"2024-09-13T14:49:17.909136+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":188,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980527+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:49:31.980527+0000","last_peered":"2024-09-13T14:49:31.980527+0000","last_clean":"2024-09-13T14:49:31.980527+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:49:31.980527+0000","last_undegraded":"2024-09-13T14:49:31.980527+0000","last_fullsized":"2024-09-13T14:49:31.980527+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":233,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909062+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:49:17.909062+0000","last_peered":"2024-09-13T14:49:17.909062+0000","last_clean":"2024-09-13T14:49:17.909062+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:49:17.909062+0000","last_undegraded":"2024-09-13T14:49:17.909062+0000","last_fullsized":"2024-09-13T14:49:17.909062+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":211,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.927121+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:49:17.927121+0000","last_peered":"2024-09-13T14:49:17.927121+0000","last_clean":"2024-09-13T14:49:17.927121+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:49:17.927121+0000","last_undegraded":"2024-09-13T14:49:17.927121+0000","last_fullsized":"2024-09-13T14:49:17.927121+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"255'9596","reported_seq":9630,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.919000+0000","last_change":"2024-09-13T14:47:20.051919+0000","last_active":"2024-09-13T14:49:53.919000+0000","last_peered":"2024-09-13T14:49:53.919000+0000","last_clean":"2024-09-13T14:49:53.919000+0000","last_became_active":"2024-09-13T14:47:20.051626+0000","last_became_peered":"2024-09-13T14:47:20.051626+0000","last_unstale":"2024-09-13T14:49:53.919000+0000","last_undegraded":"2024-09-13T14:49:53.919000+0000","last_fullsized":"2024-09-13T14:49:53.919000+0000","mapping_epoch":229,"log_start":"239'2900","ondisk_log_start":"239'2900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:12.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316407808,"num_objects":4828,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9596,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":242,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.974312+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:49:31.974312+0000","last_peered":"2024-09-13T14:49:31.974312+0000","last_clean":"2024-09-13T14:49:31.974312+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:49:31.974312+0000","last_undegraded":"2024-09-13T14:49:31.974312+0000","last_fullsized":"2024-09-13T14:49:31.974312+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":218,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907402+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:49:17.907402+0000","last_peered":"2024-09-13T14:49:17.907402+0000","last_clean":"2024-09-13T14:49:17.907402+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:49:17.907402+0000","last_undegraded":"2024-09-13T14:49:17.907402+0000","last_fullsized":"2024-09-13T14:49:17.907402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":229,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909454+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:49:17.909454+0000","last_peered":"2024-09-13T14:49:17.909454+0000","last_clean":"2024-09-13T14:49:17.909454+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:49:17.909454+0000","last_undegraded":"2024-09-13T14:49:17.909454+0000","last_fullsized":"2024-09-13T14:49:17.909454+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908930+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:49:17.908930+0000","last_peered":"2024-09-13T14:49:17.908930+0000","last_clean":"2024-09-13T14:49:17.908930+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:49:17.908930+0000","last_undegraded":"2024-09-13T14:49:17.908930+0000","last_fullsized":"2024-09-13T14:49:17.908930+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"255'9640","reported_seq":11691,"reported_epoch":255,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:54.045075+0000","last_change":"2024-09-13T14:49:05.872778+0000","last_active":"2024-09-13T14:49:54.045075+0000","last_peered":"2024-09-13T14:49:54.045075+0000","last_clean":"2024-09-13T14:48:37.826901+0000","last_became_active":"2024-09-13T14:48:40.041761+0000","last_became_peered":"2024-09-13T14:48:40.041761+0000","last_unstale":"2024-09-13T14:49:54.045075+0000","last_undegraded":"2024-09-13T14:48:39.850843+0000","last_fullsized":"2024-09-13T14:48:39.825201+0000","mapping_epoch":249,"log_start":"237'1500","ondisk_log_start":"237'1500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8140,"log_dups_size":0,"ondisk_log_size":8140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324272128,"num_objects":4948,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":3409,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9640,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":65077248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":3409},{"shards":"2,7","objects":182}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":149,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.910300+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:49:17.910300+0000","last_peered":"2024-09-13T14:49:17.910300+0000","last_clean":"2024-09-13T14:49:17.910300+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:49:17.910300+0000","last_undegraded":"2024-09-13T14:49:17.910300+0000","last_fullsized":"2024-09-13T14:49:17.910300+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":182,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.971543+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:49:31.971543+0000","last_peered":"2024-09-13T14:49:31.971543+0000","last_clean":"2024-09-13T14:49:31.971543+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:49:31.971543+0000","last_undegraded":"2024-09-13T14:49:31.971543+0000","last_fullsized":"2024-09-13T14:49:31.971543+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":372,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.986429+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:49:31.986429+0000","last_peered":"2024-09-13T14:49:31.986429+0000","last_clean":"2024-09-13T14:49:31.986429+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:49:31.986429+0000","last_undegraded":"2024-09-13T14:49:31.986429+0000","last_fullsized":"2024-09-13T14:49:31.986429+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893457+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:49:17.893457+0000","last_peered":"2024-09-13T14:49:17.893457+0000","last_clean":"2024-09-13T14:49:17.893457+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:49:17.893457+0000","last_undegraded":"2024-09-13T14:49:17.893457+0000","last_fullsized":"2024-09-13T14:49:17.893457+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"255'9849","reported_seq":9913,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:55.242977+0000","last_change":"2024-09-13T14:47:20.049747+0000","last_active":"2024-09-13T14:49:55.242977+0000","last_peered":"2024-09-13T14:49:55.242977+0000","last_clean":"2024-09-13T14:49:55.242977+0000","last_became_active":"2024-09-13T14:47:20.049448+0000","last_became_peered":"2024-09-13T14:47:20.049448+0000","last_unstale":"2024-09-13T14:49:55.242977+0000","last_undegraded":"2024-09-13T14:49:55.242977+0000","last_fullsized":"2024-09-13T14:49:55.242977+0000","mapping_epoch":229,"log_start":"239'2800","ondisk_log_start":"239'2800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7049,"log_dups_size":0,"ondisk_log_size":7049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:11.585800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319488000,"num_objects":4875,"num_object_clones":0,"num_object_copies":14625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9849,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":396,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983706+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:49:31.983706+0000","last_peered":"2024-09-13T14:49:31.983706+0000","last_clean":"2024-09-13T14:49:31.983706+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:49:31.983706+0000","last_undegraded":"2024-09-13T14:49:31.983706+0000","last_fullsized":"2024-09-13T14:49:31.983706+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":405,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980571+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:49:31.980571+0000","last_peered":"2024-09-13T14:49:31.980571+0000","last_clean":"2024-09-13T14:49:31.980571+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:49:31.980571+0000","last_undegraded":"2024-09-13T14:49:31.980571+0000","last_fullsized":"2024-09-13T14:49:31.980571+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":267,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894213+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:49:17.894213+0000","last_peered":"2024-09-13T14:49:17.894213+0000","last_clean":"2024-09-13T14:49:17.894213+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:49:17.894213+0000","last_undegraded":"2024-09-13T14:49:17.894213+0000","last_fullsized":"2024-09-13T14:49:17.894213+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":191,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900789+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:49:17.900789+0000","last_peered":"2024-09-13T14:49:17.900789+0000","last_clean":"2024-09-13T14:49:17.900789+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:49:17.900789+0000","last_undegraded":"2024-09-13T14:49:17.900789+0000","last_fullsized":"2024-09-13T14:49:17.900789+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"255'9680","reported_seq":9714,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.900202+0000","last_change":"2024-09-13T14:47:20.106371+0000","last_active":"2024-09-13T14:49:53.900202+0000","last_peered":"2024-09-13T14:49:53.900202+0000","last_clean":"2024-09-13T14:49:53.900202+0000","last_became_active":"2024-09-13T14:47:20.105934+0000","last_became_peered":"2024-09-13T14:47:20.105934+0000","last_unstale":"2024-09-13T14:49:53.900202+0000","last_undegraded":"2024-09-13T14:49:53.900202+0000","last_fullsized":"2024-09-13T14:49:53.900202+0000","mapping_epoch":229,"log_start":"239'3000","ondisk_log_start":"239'3000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6680,"log_dups_size":0,"ondisk_log_size":6680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:54.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325451776,"num_objects":4966,"num_object_clones":0,"num_object_copies":14898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9680,"num_write_kb":468672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":200,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.902644+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:49:17.902644+0000","last_peered":"2024-09-13T14:49:17.902644+0000","last_clean":"2024-09-13T14:49:17.902644+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:49:17.902644+0000","last_undegraded":"2024-09-13T14:49:17.902644+0000","last_fullsized":"2024-09-13T14:49:17.902644+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":305,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.962693+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:49:31.962693+0000","last_peered":"2024-09-13T14:49:31.962693+0000","last_clean":"2024-09-13T14:49:31.962693+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:49:31.962693+0000","last_undegraded":"2024-09-13T14:49:31.962693+0000","last_fullsized":"2024-09-13T14:49:31.962693+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":228,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909335+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:49:17.909335+0000","last_peered":"2024-09-13T14:49:17.909335+0000","last_clean":"2024-09-13T14:49:17.909335+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:49:17.909335+0000","last_undegraded":"2024-09-13T14:49:17.909335+0000","last_fullsized":"2024-09-13T14:49:17.909335+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893850+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:49:17.893850+0000","last_peered":"2024-09-13T14:49:17.893850+0000","last_clean":"2024-09-13T14:49:17.893850+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:49:17.893850+0000","last_undegraded":"2024-09-13T14:49:17.893850+0000","last_fullsized":"2024-09-13T14:49:17.893850+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"255'9609","reported_seq":9643,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:54.028446+0000","last_change":"2024-09-13T14:47:20.050773+0000","last_active":"2024-09-13T14:49:54.028446+0000","last_peered":"2024-09-13T14:49:54.028446+0000","last_clean":"2024-09-13T14:49:54.028446+0000","last_became_active":"2024-09-13T14:47:20.050243+0000","last_became_peered":"2024-09-13T14:47:20.050243+0000","last_unstale":"2024-09-13T14:49:54.028446+0000","last_undegraded":"2024-09-13T14:49:54.028446+0000","last_fullsized":"2024-09-13T14:49:54.028446+0000","mapping_epoch":229,"log_start":"239'3300","ondisk_log_start":"239'3300","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6309,"log_dups_size":0,"ondisk_log_size":6309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:42.724078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316735488,"num_objects":4833,"num_object_clones":0,"num_object_copies":14499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9609,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.971578+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:49:31.971578+0000","last_peered":"2024-09-13T14:49:31.971578+0000","last_clean":"2024-09-13T14:49:31.971578+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:49:31.971578+0000","last_undegraded":"2024-09-13T14:49:31.971578+0000","last_fullsized":"2024-09-13T14:49:31.971578+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":315,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.963329+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:49:31.963329+0000","last_peered":"2024-09-13T14:49:31.963329+0000","last_clean":"2024-09-13T14:49:31.963329+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:49:31.963329+0000","last_undegraded":"2024-09-13T14:49:31.963329+0000","last_fullsized":"2024-09-13T14:49:31.963329+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909424+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:49:17.909424+0000","last_peered":"2024-09-13T14:49:17.909424+0000","last_clean":"2024-09-13T14:49:17.909424+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:49:17.909424+0000","last_undegraded":"2024-09-13T14:49:17.909424+0000","last_fullsized":"2024-09-13T14:49:17.909424+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"255'9584","reported_seq":9618,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.899438+0000","last_change":"2024-09-13T14:47:20.106351+0000","last_active":"2024-09-13T14:49:53.899438+0000","last_peered":"2024-09-13T14:49:53.899438+0000","last_clean":"2024-09-13T14:49:53.899438+0000","last_became_active":"2024-09-13T14:47:20.105946+0000","last_became_peered":"2024-09-13T14:47:20.105946+0000","last_unstale":"2024-09-13T14:49:53.899438+0000","last_undegraded":"2024-09-13T14:49:53.899438+0000","last_fullsized":"2024-09-13T14:49:53.899438+0000","mapping_epoch":229,"log_start":"239'2900","ondisk_log_start":"239'2900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:07.223637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318636032,"num_objects":4862,"num_object_clones":0,"num_object_copies":14586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9584,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":237,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908259+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:49:17.908259+0000","last_peered":"2024-09-13T14:49:17.908259+0000","last_clean":"2024-09-13T14:49:17.908259+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:49:17.908259+0000","last_undegraded":"2024-09-13T14:49:17.908259+0000","last_fullsized":"2024-09-13T14:49:17.908259+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":218,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906187+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:49:17.906187+0000","last_peered":"2024-09-13T14:49:17.906187+0000","last_clean":"2024-09-13T14:49:17.906187+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:49:17.906187+0000","last_undegraded":"2024-09-13T14:49:17.906187+0000","last_fullsized":"2024-09-13T14:49:17.906187+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":366,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979246+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:49:31.979246+0000","last_peered":"2024-09-13T14:49:31.979246+0000","last_clean":"2024-09-13T14:49:31.979246+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:49:31.979246+0000","last_undegraded":"2024-09-13T14:49:31.979246+0000","last_fullsized":"2024-09-13T14:49:31.979246+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"255'9787","reported_seq":13358,"reported_epoch":255,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:54.045120+0000","last_change":"2024-09-13T14:49:05.871518+0000","last_active":"2024-09-13T14:49:54.045120+0000","last_peered":"2024-09-13T14:49:54.045120+0000","last_clean":"2024-09-13T14:48:09.848772+0000","last_became_active":"2024-09-13T14:48:12.232150+0000","last_became_peered":"2024-09-13T14:48:12.232150+0000","last_unstale":"2024-09-13T14:49:54.045120+0000","last_undegraded":"2024-09-13T14:49:05.858877+0000","last_fullsized":"2024-09-13T14:48:11.643512+0000","mapping_epoch":242,"log_start":"237'1600","ondisk_log_start":"237'1600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8187,"log_dups_size":0,"ondisk_log_size":8187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316473344,"num_objects":4829,"num_object_clones":0,"num_object_copies":14487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9787,"num_write_kb":467712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":114753536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":1699},{"shards":"2,5","objects":2402}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":234,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907684+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:49:17.907684+0000","last_peered":"2024-09-13T14:49:17.907684+0000","last_clean":"2024-09-13T14:49:17.907684+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:49:17.907684+0000","last_undegraded":"2024-09-13T14:49:17.907684+0000","last_fullsized":"2024-09-13T14:49:17.907684+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":238,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900518+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:49:17.900518+0000","last_peered":"2024-09-13T14:49:17.900518+0000","last_clean":"2024-09-13T14:49:17.900518+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:49:17.900518+0000","last_undegraded":"2024-09-13T14:49:17.900518+0000","last_fullsized":"2024-09-13T14:49:17.900518+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908661+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:49:17.908661+0000","last_peered":"2024-09-13T14:49:17.908661+0000","last_clean":"2024-09-13T14:49:17.908661+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:49:17.908661+0000","last_undegraded":"2024-09-13T14:49:17.908661+0000","last_fullsized":"2024-09-13T14:49:17.908661+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"255'9785","reported_seq":9819,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:54.056577+0000","last_change":"2024-09-13T14:47:20.106142+0000","last_active":"2024-09-13T14:49:54.056577+0000","last_peered":"2024-09-13T14:49:54.056577+0000","last_clean":"2024-09-13T14:49:54.056577+0000","last_became_active":"2024-09-13T14:47:20.105758+0000","last_became_peered":"2024-09-13T14:47:20.105758+0000","last_unstale":"2024-09-13T14:49:54.056577+0000","last_undegraded":"2024-09-13T14:49:54.056577+0000","last_fullsized":"2024-09-13T14:49:54.056577+0000","mapping_epoch":229,"log_start":"239'3500","ondisk_log_start":"239'3500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6285,"log_dups_size":0,"ondisk_log_size":6285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:51.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325779456,"num_objects":4971,"num_object_clones":0,"num_object_copies":14913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9785,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":217,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907605+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:49:17.907605+0000","last_peered":"2024-09-13T14:49:17.907605+0000","last_clean":"2024-09-13T14:49:17.907605+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:49:17.907605+0000","last_undegraded":"2024-09-13T14:49:17.907605+0000","last_fullsized":"2024-09-13T14:49:17.907605+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":345,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.966805+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:49:31.966805+0000","last_peered":"2024-09-13T14:49:31.966805+0000","last_clean":"2024-09-13T14:49:31.966805+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:49:31.966805+0000","last_undegraded":"2024-09-13T14:49:31.966805+0000","last_fullsized":"2024-09-13T14:49:31.966805+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894548+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:49:17.894548+0000","last_peered":"2024-09-13T14:49:17.894548+0000","last_clean":"2024-09-13T14:49:17.894548+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:49:17.894548+0000","last_undegraded":"2024-09-13T14:49:17.894548+0000","last_fullsized":"2024-09-13T14:49:17.894548+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"255'9683","reported_seq":9747,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.958777+0000","last_change":"2024-09-13T14:47:20.107374+0000","last_active":"2024-09-13T14:49:53.958777+0000","last_peered":"2024-09-13T14:49:53.958777+0000","last_clean":"2024-09-13T14:49:53.958777+0000","last_became_active":"2024-09-13T14:47:20.107057+0000","last_became_peered":"2024-09-13T14:47:20.107057+0000","last_unstale":"2024-09-13T14:49:53.958777+0000","last_undegraded":"2024-09-13T14:49:53.958777+0000","last_fullsized":"2024-09-13T14:49:53.958777+0000","mapping_epoch":229,"log_start":"237'1500","ondisk_log_start":"237'1500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8183,"log_dups_size":0,"ondisk_log_size":8183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:05.814456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324075520,"num_objects":4945,"num_object_clones":0,"num_object_copies":14835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":385,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979344+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:49:31.979344+0000","last_peered":"2024-09-13T14:49:31.979344+0000","last_clean":"2024-09-13T14:49:31.979344+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:49:31.979344+0000","last_undegraded":"2024-09-13T14:49:31.979344+0000","last_fullsized":"2024-09-13T14:49:31.979344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":216,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909078+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:49:17.909078+0000","last_peered":"2024-09-13T14:49:17.909078+0000","last_clean":"2024-09-13T14:49:17.909078+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:49:17.909078+0000","last_undegraded":"2024-09-13T14:49:17.909078+0000","last_fullsized":"2024-09-13T14:49:17.909078+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.907945+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:49:17.907945+0000","last_peered":"2024-09-13T14:49:17.907945+0000","last_clean":"2024-09-13T14:49:17.907945+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:49:17.907945+0000","last_undegraded":"2024-09-13T14:49:17.907945+0000","last_fullsized":"2024-09-13T14:49:17.907945+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"255'9694","reported_seq":9758,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:55.184453+0000","last_change":"2024-09-13T14:47:20.108173+0000","last_active":"2024-09-13T14:49:55.184453+0000","last_peered":"2024-09-13T14:49:55.184453+0000","last_clean":"2024-09-13T14:49:55.184453+0000","last_became_active":"2024-09-13T14:47:20.107879+0000","last_became_peered":"2024-09-13T14:47:20.107879+0000","last_unstale":"2024-09-13T14:49:55.184453+0000","last_undegraded":"2024-09-13T14:49:55.184453+0000","last_fullsized":"2024-09-13T14:49:55.184453+0000","mapping_epoch":229,"log_start":"239'2700","ondisk_log_start":"239'2700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:50.152706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315097088,"num_objects":4808,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9694,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":230,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900396+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:49:17.900396+0000","last_peered":"2024-09-13T14:49:17.900396+0000","last_clean":"2024-09-13T14:49:17.900396+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:49:17.900396+0000","last_undegraded":"2024-09-13T14:49:17.900396+0000","last_fullsized":"2024-09-13T14:49:17.900396+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":231,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893980+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:49:17.893980+0000","last_peered":"2024-09-13T14:49:17.893980+0000","last_clean":"2024-09-13T14:49:17.893980+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:49:17.893980+0000","last_undegraded":"2024-09-13T14:49:17.893980+0000","last_fullsized":"2024-09-13T14:49:17.893980+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":191,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.927203+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:49:17.927203+0000","last_peered":"2024-09-13T14:49:17.927203+0000","last_clean":"2024-09-13T14:49:17.927203+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:49:17.927203+0000","last_undegraded":"2024-09-13T14:49:17.927203+0000","last_fullsized":"2024-09-13T14:49:17.927203+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"255'9553","reported_seq":9590,"reported_epoch":255,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:53.877198+0000","last_change":"2024-09-13T14:48:40.078958+0000","last_active":"2024-09-13T14:49:53.877198+0000","last_peered":"2024-09-13T14:49:53.877198+0000","last_clean":"2024-09-13T14:48:37.811111+0000","last_became_active":"2024-09-13T14:48:40.052181+0000","last_became_peered":"2024-09-13T14:48:40.052181+0000","last_unstale":"2024-09-13T14:49:53.877198+0000","last_undegraded":"2024-09-13T14:48:39.849823+0000","last_fullsized":"2024-09-13T14:48:39.826167+0000","mapping_epoch":249,"log_start":"239'2800","ondisk_log_start":"239'2800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6753,"log_dups_size":0,"ondisk_log_size":6753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:49.521146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323551232,"num_objects":4937,"num_object_clones":0,"num_object_copies":14811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9553,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":3441},{"shards":"3,6","objects":-812}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":244,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.906118+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:49:17.906118+0000","last_peered":"2024-09-13T14:49:17.906118+0000","last_clean":"2024-09-13T14:49:17.906118+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:49:17.906118+0000","last_undegraded":"2024-09-13T14:49:17.906118+0000","last_fullsized":"2024-09-13T14:49:17.906118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":221,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894377+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:49:17.894377+0000","last_peered":"2024-09-13T14:49:17.894377+0000","last_clean":"2024-09-13T14:49:17.894377+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:49:17.894377+0000","last_undegraded":"2024-09-13T14:49:17.894377+0000","last_fullsized":"2024-09-13T14:49:17.894377+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":73,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.964672+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:49:31.964672+0000","last_peered":"2024-09-13T14:49:31.964672+0000","last_clean":"2024-09-13T14:49:31.964672+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:49:31.964672+0000","last_undegraded":"2024-09-13T14:49:31.964672+0000","last_fullsized":"2024-09-13T14:49:31.964672+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"255'9936","reported_seq":9970,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:54.598230+0000","last_change":"2024-09-13T14:47:20.051824+0000","last_active":"2024-09-13T14:49:54.598230+0000","last_peered":"2024-09-13T14:49:54.598230+0000","last_clean":"2024-09-13T14:49:54.598230+0000","last_became_active":"2024-09-13T14:47:20.051487+0000","last_became_peered":"2024-09-13T14:47:20.051487+0000","last_unstale":"2024-09-13T14:49:54.598230+0000","last_undegraded":"2024-09-13T14:49:54.598230+0000","last_fullsized":"2024-09-13T14:49:54.598230+0000","mapping_epoch":229,"log_start":"238'1800","ondisk_log_start":"238'1800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:11.533543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318373888,"num_objects":4858,"num_object_clones":0,"num_object_copies":14574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9936,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908691+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:49:17.908691+0000","last_peered":"2024-09-13T14:49:17.908691+0000","last_clean":"2024-09-13T14:49:17.908691+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:49:17.908691+0000","last_undegraded":"2024-09-13T14:49:17.908691+0000","last_fullsized":"2024-09-13T14:49:17.908691+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894562+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:49:17.894562+0000","last_peered":"2024-09-13T14:49:17.894562+0000","last_clean":"2024-09-13T14:49:17.894562+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:49:17.894562+0000","last_undegraded":"2024-09-13T14:49:17.894562+0000","last_fullsized":"2024-09-13T14:49:17.894562+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.893775+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:49:17.893775+0000","last_peered":"2024-09-13T14:49:17.893775+0000","last_clean":"2024-09-13T14:49:17.893775+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:49:17.893775+0000","last_undegraded":"2024-09-13T14:49:17.893775+0000","last_fullsized":"2024-09-13T14:49:17.893775+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":266,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.900128+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:49:17.900128+0000","last_peered":"2024-09-13T14:49:17.900128+0000","last_clean":"2024-09-13T14:49:17.900128+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:49:17.900128+0000","last_undegraded":"2024-09-13T14:49:17.900128+0000","last_fullsized":"2024-09-13T14:49:17.900128+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":354,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.966873+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:49:31.966873+0000","last_peered":"2024-09-13T14:49:31.966873+0000","last_clean":"2024-09-13T14:49:31.966873+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:49:31.966873+0000","last_undegraded":"2024-09-13T14:49:31.966873+0000","last_fullsized":"2024-09-13T14:49:31.966873+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":48,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.909215+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:49:17.909215+0000","last_peered":"2024-09-13T14:49:17.909215+0000","last_clean":"2024-09-13T14:49:17.909215+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:49:17.909215+0000","last_undegraded":"2024-09-13T14:49:17.909215+0000","last_fullsized":"2024-09-13T14:49:17.909215+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":212,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894657+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:49:17.894657+0000","last_peered":"2024-09-13T14:49:17.894657+0000","last_clean":"2024-09-13T14:49:17.894657+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:49:17.894657+0000","last_undegraded":"2024-09-13T14:49:17.894657+0000","last_fullsized":"2024-09-13T14:49:17.894657+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":398,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983921+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:49:31.983921+0000","last_peered":"2024-09-13T14:49:31.983921+0000","last_clean":"2024-09-13T14:49:31.983921+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:49:31.983921+0000","last_undegraded":"2024-09-13T14:49:31.983921+0000","last_fullsized":"2024-09-13T14:49:31.983921+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":69,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.966556+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:49:31.966556+0000","last_peered":"2024-09-13T14:49:31.966556+0000","last_clean":"2024-09-13T14:49:31.966556+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:49:31.966556+0000","last_undegraded":"2024-09-13T14:49:31.966556+0000","last_fullsized":"2024-09-13T14:49:31.966556+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":404,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983937+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:49:31.983937+0000","last_peered":"2024-09-13T14:49:31.983937+0000","last_clean":"2024-09-13T14:49:31.983937+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:49:31.983937+0000","last_undegraded":"2024-09-13T14:49:31.983937+0000","last_fullsized":"2024-09-13T14:49:31.983937+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908881+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:49:17.908881+0000","last_peered":"2024-09-13T14:49:17.908881+0000","last_clean":"2024-09-13T14:49:17.908881+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:49:17.908881+0000","last_undegraded":"2024-09-13T14:49:17.908881+0000","last_fullsized":"2024-09-13T14:49:17.908881+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.908750+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:49:17.908750+0000","last_peered":"2024-09-13T14:49:17.908750+0000","last_clean":"2024-09-13T14:49:17.908750+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:49:17.908750+0000","last_undegraded":"2024-09-13T14:49:17.908750+0000","last_fullsized":"2024-09-13T14:49:17.908750+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":237,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:49:17.894404+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:49:17.894404+0000","last_peered":"2024-09-13T14:49:17.894404+0000","last_clean":"2024-09-13T14:49:17.894404+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:49:17.894404+0000","last_undegraded":"2024-09-13T14:49:17.894404+0000","last_fullsized":"2024-09-13T14:49:17.894404+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":64,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980247+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:49:31.980247+0000","last_peered":"2024-09-13T14:49:31.980247+0000","last_clean":"2024-09-13T14:49:31.980247+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:49:31.980247+0000","last_undegraded":"2024-09-13T14:49:31.980247+0000","last_fullsized":"2024-09-13T14:49:31.980247+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":362,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980676+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:49:31.980676+0000","last_peered":"2024-09-13T14:49:31.980676+0000","last_clean":"2024-09-13T14:49:31.980676+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:49:31.980676+0000","last_undegraded":"2024-09-13T14:49:31.980676+0000","last_fullsized":"2024-09-13T14:49:31.980676+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5131534360,"num_objects":78302,"num_object_clones":0,"num_object_copies":234906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9533,"num_objects_misplaced":3409,"num_objects_unfound":0,"num_objects_dirty":78302,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155152,"num_write_kb":7470465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2744,"num_bytes_recovered":179830784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15876304896,"data_stored":15876292680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112738,"ondisk_log_size":112738,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365162,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2173732,"kb_used_data":1260188,"kb_used_omap":1,"kb_used_meta":913534,"kb_avail":91571420,"statfs":{"total":95995035648,"available":93769134080,"internally_reserved":0,"allocated":1290432512,"data_stored":1289094316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":935459160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561392,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3119264,"kb_used_data":2225752,"kb_used_omap":1,"kb_used_meta":893502,"kb_avail":90625888,"statfs":{"total":95995035648,"available":92800909312,"internally_reserved":0,"allocated":2279170048,"data_stored":2277835795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914946392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790324,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2031252,"kb_used_data":1130188,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":91713900,"statfs":{"total":95995035648,"available":93915033600,"internally_reserved":0,"allocated":1157312512,"data_stored":1155986641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986552,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4056932,"kb_used_data":3166556,"kb_used_omap":1,"kb_used_meta":890238,"kb_avail":89688220,"statfs":{"total":95995035648,"available":91840737280,"internally_reserved":0,"allocated":3242553344,"data_stored":3241215333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911604056},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519707,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2004372,"kb_used_data":1479756,"kb_used_omap":4,"kb_used_meta":524539,"kb_avail":91740780,"statfs":{"total":95995035648,"available":93942558720,"internally_reserved":0,"allocated":1515270144,"data_stored":1513944283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":537128384},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304623,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2767896,"kb_used_data":2277072,"kb_used_omap":6,"kb_used_meta":490809,"kb_avail":90977256,"statfs":{"total":95995035648,"available":93160710144,"internally_reserved":0,"allocated":2331721728,"data_stored":2330392285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":502588871},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827926,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1759508,"kb_used_data":973708,"kb_used_omap":5,"kb_used_meta":785722,"kb_avail":91985644,"statfs":{"total":95995035648,"available":94193299456,"internally_reserved":0,"allocated":997076992,"data_stored":995750558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":804579564},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285791,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3808724,"kb_used_data":3102924,"kb_used_omap":5,"kb_used_meta":705786,"kb_avail":89936428,"statfs":{"total":95995035648,"available":92094902272,"internally_reserved":0,"allocated":3177394176,"data_stored":3176067069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":722724923},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3172077568,"data_stored":3172073496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":988217344,"data_stored":988217344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2307653632,"data_stored":2307653632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1510539264,"data_stored":1510539264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3215921152,"data_stored":3215917080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1145307136,"data_stored":1145307136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2251624448,"data_stored":2251620376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1284964352,"data_stored":1284964352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:49:58.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:49:58.134 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:58.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:58 smithi019 ceph-mon[91296]: from='client.56071 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:58.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:58.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:58.301 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:58.310 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:58 smithi003 ceph-mon[107278]: from='client.56071 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:58.310 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:58.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:58 smithi003 ceph-mon[104007]: from='client.56071 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:58.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:58.403 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.439 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:58.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:58.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:58.582 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:58.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:58.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:58.731 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:58.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:58.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:58.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:59.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:59.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:59.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:59.273 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:59.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:59.411 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:49:59 smithi019 ceph-mon[91296]: pgmap v855: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 9533/234984 objects degraded (4.057%); 3409/234984 objects misplaced (1.451%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:49:59.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:49:59 smithi003 ceph-mon[107278]: pgmap v855: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 9533/234984 objects degraded (4.057%); 3409/234984 objects misplaced (1.451%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:49:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:49:59 smithi003 ceph-mon[104007]: pgmap v855: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 9533/234984 objects degraded (4.057%); 3409/234984 objects misplaced (1.451%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:49:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.463 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:49:59.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:59.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.600 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:59.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:59.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:59.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:59.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:59.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:49:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:59.978 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:00.023 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:00.125 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:00.160 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:00.263 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:00.312 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:00.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:00.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:50:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:50:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:00 smithi019 ceph-mon[91296]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 9533/234984 objects degraded (4.057%), 3 pgs degraded, 3 pgs undersized 2024-09-13T14:50:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:00 smithi019 ceph-mon[91296]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T14:50:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:00 smithi019 ceph-mon[91296]: [WRN] PG_DEGRADED: Degraded data redundancy: 9533/234984 objects degraded (4.057%), 3 pgs degraded, 3 pgs undersized 2024-09-13T14:50:00.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:00 smithi019 ceph-mon[91296]: pg 10.3 is stuck undersized for 106s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,5] 2024-09-13T14:50:00.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:00 smithi019 ceph-mon[91296]: pg 10.7 is stuck undersized for 77s, current state active+recovery_wait+undersized+degraded+remapped, last acting [6,3] 2024-09-13T14:50:00.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:00 smithi019 ceph-mon[91296]: pg 10.9 is stuck undersized for 77s, current state active+recovering+undersized+degraded+remapped, last acting [2,7] 2024-09-13T14:50:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[107278]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 9533/234984 objects degraded (4.057%), 3 pgs degraded, 3 pgs undersized 2024-09-13T14:50:00.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[107278]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T14:50:00.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[107278]: [WRN] PG_DEGRADED: Degraded data redundancy: 9533/234984 objects degraded (4.057%), 3 pgs degraded, 3 pgs undersized 2024-09-13T14:50:00.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[107278]: pg 10.3 is stuck undersized for 106s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,5] 2024-09-13T14:50:00.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[107278]: pg 10.7 is stuck undersized for 77s, current state active+recovery_wait+undersized+degraded+remapped, last acting [6,3] 2024-09-13T14:50:00.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[107278]: pg 10.9 is stuck undersized for 77s, current state active+recovering+undersized+degraded+remapped, last acting [2,7] 2024-09-13T14:50:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[104007]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 9533/234984 objects degraded (4.057%), 3 pgs degraded, 3 pgs undersized 2024-09-13T14:50:00.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[104007]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T14:50:00.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[104007]: [WRN] PG_DEGRADED: Degraded data redundancy: 9533/234984 objects degraded (4.057%), 3 pgs degraded, 3 pgs undersized 2024-09-13T14:50:00.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[104007]: pg 10.3 is stuck undersized for 106s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,5] 2024-09-13T14:50:00.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[104007]: pg 10.7 is stuck undersized for 77s, current state active+recovery_wait+undersized+degraded+remapped, last acting [6,3] 2024-09-13T14:50:00.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:00 smithi003 ceph-mon[104007]: pg 10.9 is stuck undersized for 77s, current state active+recovering+undersized+degraded+remapped, last acting [2,7] 2024-09-13T14:50:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:00.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:00.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:00.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:00.699 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:00.745 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:00.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:00.849 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.888 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:00.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:00.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:01.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:01.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:01.261 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:01.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:01.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:01.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:01 smithi003 ceph-mon[107278]: pgmap v856: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 9723/223689 objects degraded (4.347%); 3122/223689 objects misplaced (1.396%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:01.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:01.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:01.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:01 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9723/223689 objects degraded (4.347%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:01 smithi003 ceph-mon[104007]: pgmap v856: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 9723/223689 objects degraded (4.347%); 3122/223689 objects misplaced (1.396%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:01 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9723/223689 objects degraded (4.347%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:01.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:01 smithi019 ceph-mon[91296]: pgmap v856: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 9723/223689 objects degraded (4.347%); 3122/223689 objects misplaced (1.396%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:01.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:01.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:01.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:01 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9723/223689 objects degraded (4.347%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:01.436 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:01.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:01.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:01.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:01.645 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:01.646 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:01.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:01.710 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:01.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:01.847 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:01.949 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:01.982 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:01.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:02.084 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:02.114 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:02.216 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:02.238 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:02.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:02.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:02.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:02.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:02.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:02.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:02.650 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:02.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.788 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:02.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:02.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:02.924 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:03.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:50:03.026 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:03.055 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:03.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:03.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:03.260 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:03 smithi003 ceph-mon[104007]: pgmap v857: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 763 op/s; 9723/222066 objects degraded (4.378%); 3122/222066 objects misplaced (1.406%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:50:03.266 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:03 smithi003 ceph-mon[107278]: pgmap v857: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 763 op/s; 9723/222066 objects degraded (4.378%); 3122/222066 objects misplaced (1.406%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:50:03.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:03.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:03 smithi019 ceph-mon[91296]: pgmap v857: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 763 op/s; 9723/222066 objects degraded (4.378%); 3122/222066 objects misplaced (1.406%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:50:03.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.522 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:03.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:03.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:03.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:03.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:03.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.811 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:03.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:03.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:03.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:04.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:04.089 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:04.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:04.177 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:04.184 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:04.191 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:04.217 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:04.270 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:04.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:04.335 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:50:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:50:04] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T14:50:04.368 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:04.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:04.502 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:04.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:04.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:04.752 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:04.787 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:04.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:04.917 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:05.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:05.051 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:05.146 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:50:05.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:05.191 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:05.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:05.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:05.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:05 smithi019 ceph-mon[91296]: pgmap v858: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 9723/222066 objects degraded (4.378%); 3122/222066 objects misplaced (1.406%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:50:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3977098377' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:05 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:05.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:05 smithi003 ceph-mon[107278]: pgmap v858: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 9723/222066 objects degraded (4.378%); 3122/222066 objects misplaced (1.406%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:50:05.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3977098377' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:05.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:05 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:05 smithi003 ceph-mon[104007]: pgmap v858: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 9723/222066 objects degraded (4.378%); 3122/222066 objects misplaced (1.406%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:50:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3977098377' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:05 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:05.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:05.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:05.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:05.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:05.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:05.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:05.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:05.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:06.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:06.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:06.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:06.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:06.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:06.343 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:06 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:06.344 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:06 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:06.344 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:06 smithi003 ceph-mon[107278]: osdmap e256: 8 total, 8 up, 8 in 2024-09-13T14:50:06.344 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9723/222066 objects degraded (4.378%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:06.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:06 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:06.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:06 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:06.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:06 smithi003 ceph-mon[104007]: osdmap e256: 8 total, 8 up, 8 in 2024-09-13T14:50:06.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9723/222066 objects degraded (4.378%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:06.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:06 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:06 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:06 smithi019 ceph-mon[91296]: osdmap e256: 8 total, 8 up, 8 in 2024-09-13T14:50:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9723/222066 objects degraded (4.378%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:06.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:06.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:06.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:06.694 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:06.723 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:06.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:06.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:06.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:07.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:07.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.142 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:07.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:07.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:07.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:07.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:07.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:07.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:07 smithi019 ceph-mon[91296]: pgmap v860: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 9872/208986 objects degraded (4.724%); 2865/208986 objects misplaced (1.371%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:50:07.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:07 smithi003 ceph-mon[107278]: pgmap v860: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 9872/208986 objects degraded (4.724%); 2865/208986 objects misplaced (1.371%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:50:07.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:07 smithi003 ceph-mon[104007]: pgmap v860: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 9872/208986 objects degraded (4.724%); 2865/208986 objects misplaced (1.371%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:50:07.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:07.562 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:07.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:07.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:07.804 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:07.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:07.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:07.972 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:08.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:08.107 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:08.209 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:08.244 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:08.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:08.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:08.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:08.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:08.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:08.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:08.660 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:08.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:08.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:08.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:08.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:09.049 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:09.088 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:09.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:09.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:09.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:09.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:09.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:09.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:09.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:09 smithi003 ceph-mon[104007]: pgmap v861: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 9872/208986 objects degraded (4.724%); 2865/208986 objects misplaced (1.371%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:50:09.364 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:09 smithi003 ceph-mon[107278]: pgmap v861: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 9872/208986 objects degraded (4.724%); 2865/208986 objects misplaced (1.371%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:50:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:09 smithi019 ceph-mon[91296]: pgmap v861: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 9872/208986 objects degraded (4.724%); 2865/208986 objects misplaced (1.371%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:50:09.463 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:09.497 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:09.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:09.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:50:09.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:09.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:09.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:09.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:09.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:09.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:09.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:09.956 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:10.058 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.092 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:10.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:10.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:10.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:10.339 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:10.369 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:50:10.369 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9872/208986 objects degraded (4.724%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1236,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":256,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238227,"num_in_osds":8,"osd_in_since":1726237182,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":106},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":109,"num_pools":5,"num_objects":69662,"data_bytes":4564222136,"bytes_used":20565757952,"bytes_avail":747394527232,"bytes_total":767960285184,"degraded_objects":9872,"degraded_total":208986,"degraded_ratio":0.047237614002851869,"misplaced_objects":2865,"misplaced_total":208986,"misplaced_ratio":0.01370905228101404,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":1343123,"recovering_keys_per_sec":0,"num_objects_recovered":205,"num_bytes_recovered":13434880,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":866},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.zmhiel","status":"up:active","gid":14646}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-13T14:43:29.603041+0000","services":{}},"progress_events":{"70b0e26c-ee48-4914-b6cc-9ce75e26a89c":{"message":"Global Recovery Event (100s)\n [===========================.] (remaining: 2s)","progress":0.97247707843780518,"add_to_ceph_s":true},"b67cb405-2ef0-4d69-99ef-7eee6a6810fd":{"message":"Upgrade to 19.3.0-4697-gd58f7306 (9m)\n [============................] (remaining: 11m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-13T14:50:10.378 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:10.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:50:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:50:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1217101460' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:50:10.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1217101460' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:50:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1217101460' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:50:10.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:10.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:10.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:10.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:10.747 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.784 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:10.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:10.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:10.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:11.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:11.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:11.177 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:11.227 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:11.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:11.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:11 smithi003 ceph-mon[104007]: pgmap v862: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 10035/197721 objects degraded (5.075%); 2596/197721 objects misplaced (1.313%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:11.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10035/197721 objects degraded (5.075%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:11.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1708195495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:11.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:11 smithi003 ceph-mon[107278]: pgmap v862: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 10035/197721 objects degraded (5.075%); 2596/197721 objects misplaced (1.313%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:11.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10035/197721 objects degraded (5.075%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:11.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1708195495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:11.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:11.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:11 smithi019 ceph-mon[91296]: pgmap v862: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 10035/197721 objects degraded (5.075%); 2596/197721 objects misplaced (1.313%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10035/197721 objects degraded (5.075%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1708195495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:11.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:11.511 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:11.613 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:50:11.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:50:11.613 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:11.746 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:11.748 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:50:11.748 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:50:11.748 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:50:11.749 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:50:11.749 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:50:11.749 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:50:11.749 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:50:11.749 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:50:11.749 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:11.749 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:11.749 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:11.750 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:11.750 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:11.750 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:11.750 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:11.750 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:11.784 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:11.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:11.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.926 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:11.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:12.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:12.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:12.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:12.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:12.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:12.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:12.466 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:12.500 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:12.601 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:12.634 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:12.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:12.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:12.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:12.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:13.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:13.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:13.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:13.080 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:50:13.080 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:50:13.080 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:50:13.080 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:50:13.081 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:50:13.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:13.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:13.356 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:13.359 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.360 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:13.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:13 smithi019 ceph-mon[91296]: pgmap v863: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 10035/196086 objects degraded (5.118%); 2596/196086 objects misplaced (1.324%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:13.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:13.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:13 smithi003 ceph-mon[104007]: pgmap v863: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 10035/196086 objects degraded (5.118%); 2596/196086 objects misplaced (1.324%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:13 smithi003 ceph-mon[107278]: pgmap v863: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 10035/196086 objects degraded (5.118%); 2596/196086 objects misplaced (1.324%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:13.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:13.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:13.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:13.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:13.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.799 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:13.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:13.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:50:13.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:13.934 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:14.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:14.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:14.191 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.226 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:14.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:14.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:14.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:14.390 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:50:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:50:14] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T14:50:14.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:14.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:14.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:50:14.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:14.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:14.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:14.810 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:14.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:14.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:14.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:15.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:15.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:15.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:15 smithi003 ceph-mon[104007]: pgmap v864: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 10035/196086 objects degraded (5.118%); 2596/196086 objects misplaced (1.324%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:15.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:15.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:15.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:15.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.360 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:15.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:15 smithi019 ceph-mon[91296]: pgmap v864: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 10035/196086 objects degraded (5.118%); 2596/196086 objects misplaced (1.324%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:15.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:15 smithi003 ceph-mon[107278]: pgmap v864: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 10035/196086 objects degraded (5.118%); 2596/196086 objects misplaced (1.324%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:15.462 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:15.500 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:15.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:15.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:15.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:15.736 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:15.780 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:15.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:15.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:16.017 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:16.061 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:16.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:16.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:16.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:16.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:16.437 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:16.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10035/196086 objects degraded (5.118%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:16.444 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10035/196086 objects degraded (5.118%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:16.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:16.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:16.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10035/196086 objects degraded (5.118%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:16.699 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:16.736 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:16.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:16.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:17.014 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:17.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:17.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:17.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:17.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:17.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:17.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:17.428 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:17.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:17 smithi003 ceph-mon[104007]: pgmap v865: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 10203/183246 objects degraded (5.568%); 2317/183246 objects misplaced (1.264%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:50:17.492 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:17 smithi003 ceph-mon[107278]: pgmap v865: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 10203/183246 objects degraded (5.568%); 2317/183246 objects misplaced (1.264%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:50:17.530 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:17.577 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:17.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:17.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:17 smithi019 ceph-mon[91296]: pgmap v865: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 10203/183246 objects degraded (5.568%); 2317/183246 objects misplaced (1.264%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:50:17.712 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:17.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:17.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:17.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:18.084 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.123 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:18.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:18.224 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:18.259 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:18.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:18.361 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:18.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:18.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:18.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.533 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:18.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:18.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.675 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:18.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:18.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:18.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:18.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:19.057 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:19.088 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:19.190 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:19.222 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:19.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:19.363 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:19.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:19 smithi003 ceph-mon[104007]: pgmap v866: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 10203/183246 objects degraded (5.568%); 2317/183246 objects misplaced (1.264%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:50:19.382 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:19 smithi003 ceph-mon[107278]: pgmap v866: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 10203/183246 objects degraded (5.568%); 2317/183246 objects misplaced (1.264%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:50:19.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:19.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:19.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:19.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:19 smithi019 ceph-mon[91296]: pgmap v866: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 10203/183246 objects degraded (5.568%); 2317/183246 objects misplaced (1.264%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:50:19.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:19.780 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:19.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:19.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.964 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:19.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:20.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:20.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:20.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:20.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.262 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:20.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:20.266 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:50:20.271 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:50:20.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:20.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:20.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:20.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:20.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:20.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.674 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:20.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:50:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:50:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:20.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:20.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:20.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.959 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:20.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:21.032 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:21.032 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:21.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:21.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:21.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:50:21.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:21.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:21.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:21.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:21.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[107278]: pgmap v867: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 10340/172233 objects degraded (6.003%); 2050/172233 objects misplaced (1.190%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:50:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[107278]: from='client.46346 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10340/172233 objects degraded (6.003%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1464437763' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[104007]: pgmap v867: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 10340/172233 objects degraded (6.003%); 2050/172233 objects misplaced (1.190%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:50:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[104007]: from='client.46346 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10340/172233 objects degraded (6.003%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1464437763' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:21.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:21 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:21.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:21.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:21.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:21.622 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":867,"stamp":"2024-09-13T14:50:19.794047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3761340600,"num_objects":57411,"num_object_clones":0,"num_object_copies":172233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10340,"num_objects_misplaced":2050,"num_objects_unfound":0,"num_objects_dirty":57411,"num_whiteouts":0,"num_read":15,"num_read_kb":35,"num_write":176140,"num_write_kb":7470571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3314,"num_bytes_recovered":216886833,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114814,"ondisk_log_size":114814,"up":327,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":333,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18016080,"kb_used_data":11923280,"kb_used_omap":29,"kb_used_meta":6092706,"kb_avail":731945136,"statfs":{"total":767960285184,"available":749511819264,"internally_reserved":0,"allocated":12209438720,"data_stored":12198731472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6238931470},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,3,6,11,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-802881536,"num_objects":-12251,"num_object_clones":0,"num_object_copies":-36753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":468,"num_objects_misplaced":-815,"num_objects_unfound":0,"num_objects_dirty":-12251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12251,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":22740992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":391,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.153236+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:50:05.153236+0000","last_peered":"2024-09-13T14:50:05.153236+0000","last_clean":"2024-09-13T14:50:05.153236+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:50:05.153236+0000","last_undegraded":"2024-09-13T14:50:05.153236+0000","last_fullsized":"2024-09-13T14:50:05.153236+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":228,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.975881+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:49:31.975881+0000","last_peered":"2024-09-13T14:49:31.975881+0000","last_clean":"2024-09-13T14:49:31.975881+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:49:31.975881+0000","last_undegraded":"2024-09-13T14:49:31.975881+0000","last_fullsized":"2024-09-13T14:49:31.975881+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":19,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.983305+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:49:31.983305+0000","last_peered":"2024-09-13T14:49:31.983305+0000","last_clean":"2024-09-13T14:49:31.983305+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:49:31.983305+0000","last_undegraded":"2024-09-13T14:49:31.983305+0000","last_fullsized":"2024-09-13T14:49:31.983305+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":349,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149619+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:50:05.149619+0000","last_peered":"2024-09-13T14:50:05.149619+0000","last_clean":"2024-09-13T14:50:05.149619+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:50:05.149619+0000","last_undegraded":"2024-09-13T14:50:05.149619+0000","last_fullsized":"2024-09-13T14:50:05.149619+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":202,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.981016+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:49:31.981016+0000","last_peered":"2024-09-13T14:49:31.981016+0000","last_clean":"2024-09-13T14:49:31.981016+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:49:31.981016+0000","last_undegraded":"2024-09-13T14:49:31.981016+0000","last_fullsized":"2024-09-13T14:49:31.981016+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":235,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.978104+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:49:31.978104+0000","last_peered":"2024-09-13T14:49:31.978104+0000","last_clean":"2024-09-13T14:49:31.978104+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:49:31.978104+0000","last_undegraded":"2024-09-13T14:49:31.978104+0000","last_fullsized":"2024-09-13T14:49:31.978104+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":219,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980869+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:49:31.980869+0000","last_peered":"2024-09-13T14:49:31.980869+0000","last_clean":"2024-09-13T14:49:31.980869+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:49:31.980869+0000","last_undegraded":"2024-09-13T14:49:31.980869+0000","last_fullsized":"2024-09-13T14:49:31.980869+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":265,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.976260+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:49:31.976260+0000","last_peered":"2024-09-13T14:49:31.976260+0000","last_clean":"2024-09-13T14:49:31.976260+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:49:31.976260+0000","last_undegraded":"2024-09-13T14:49:31.976260+0000","last_fullsized":"2024-09-13T14:49:31.976260+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":378,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.154068+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:50:05.154068+0000","last_peered":"2024-09-13T14:50:05.154068+0000","last_clean":"2024-09-13T14:50:05.154068+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:50:05.154068+0000","last_undegraded":"2024-09-13T14:50:05.154068+0000","last_fullsized":"2024-09-13T14:50:05.154068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":373,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.161118+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:50:05.161118+0000","last_peered":"2024-09-13T14:50:05.161118+0000","last_clean":"2024-09-13T14:50:05.161118+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:50:05.161118+0000","last_undegraded":"2024-09-13T14:50:05.161118+0000","last_fullsized":"2024-09-13T14:50:05.161118+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":237,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.978234+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:49:31.978234+0000","last_peered":"2024-09-13T14:49:31.978234+0000","last_clean":"2024-09-13T14:49:31.978234+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:49:31.978234+0000","last_undegraded":"2024-09-13T14:49:31.978234+0000","last_fullsized":"2024-09-13T14:49:31.978234+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":316,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149366+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:50:05.149366+0000","last_peered":"2024-09-13T14:50:05.149366+0000","last_clean":"2024-09-13T14:50:05.149366+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:50:05.149366+0000","last_undegraded":"2024-09-13T14:50:05.149366+0000","last_fullsized":"2024-09-13T14:50:05.149366+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.170983+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:50:05.170983+0000","last_peered":"2024-09-13T14:50:05.170983+0000","last_clean":"2024-09-13T14:50:05.170983+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:50:05.170983+0000","last_undegraded":"2024-09-13T14:50:05.170983+0000","last_fullsized":"2024-09-13T14:50:05.170983+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":409,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.157033+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:50:05.157033+0000","last_peered":"2024-09-13T14:50:05.157033+0000","last_clean":"2024-09-13T14:50:05.157033+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:50:05.157033+0000","last_undegraded":"2024-09-13T14:50:05.157033+0000","last_fullsized":"2024-09-13T14:50:05.157033+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":348,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149911+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:50:05.149911+0000","last_peered":"2024-09-13T14:50:05.149911+0000","last_clean":"2024-09-13T14:50:05.149911+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:50:05.149911+0000","last_undegraded":"2024-09-13T14:50:05.149911+0000","last_fullsized":"2024-09-13T14:50:05.149911+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":218,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.971526+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:49:31.971526+0000","last_peered":"2024-09-13T14:49:31.971526+0000","last_clean":"2024-09-13T14:49:31.971526+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:49:31.971526+0000","last_undegraded":"2024-09-13T14:49:31.971526+0000","last_fullsized":"2024-09-13T14:49:31.971526+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":316,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.145818+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:50:05.145818+0000","last_peered":"2024-09-13T14:50:05.145818+0000","last_clean":"2024-09-13T14:50:05.145818+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:50:05.145818+0000","last_undegraded":"2024-09-13T14:50:05.145818+0000","last_fullsized":"2024-09-13T14:50:05.145818+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":392,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.153623+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:50:05.153623+0000","last_peered":"2024-09-13T14:50:05.153623+0000","last_clean":"2024-09-13T14:50:05.153623+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:50:05.153623+0000","last_undegraded":"2024-09-13T14:50:05.153623+0000","last_fullsized":"2024-09-13T14:50:05.153623+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":321,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149226+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:50:05.149226+0000","last_peered":"2024-09-13T14:50:05.149226+0000","last_clean":"2024-09-13T14:50:05.149226+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:50:05.149226+0000","last_undegraded":"2024-09-13T14:50:05.149226+0000","last_fullsized":"2024-09-13T14:50:05.149226+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.982072+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:49:31.982072+0000","last_peered":"2024-09-13T14:49:31.982072+0000","last_clean":"2024-09-13T14:49:31.982072+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:49:31.982072+0000","last_undegraded":"2024-09-13T14:49:31.982072+0000","last_fullsized":"2024-09-13T14:49:31.982072+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":71,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.160927+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:50:05.160927+0000","last_peered":"2024-09-13T14:50:05.160927+0000","last_clean":"2024-09-13T14:50:05.160927+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:50:05.160927+0000","last_undegraded":"2024-09-13T14:50:05.160927+0000","last_fullsized":"2024-09-13T14:50:05.160927+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":19,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979983+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:49:31.979983+0000","last_peered":"2024-09-13T14:49:31.979983+0000","last_clean":"2024-09-13T14:49:31.979983+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:49:31.979983+0000","last_undegraded":"2024-09-13T14:49:31.979983+0000","last_fullsized":"2024-09-13T14:49:31.979983+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":157,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.968085+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:49:31.968085+0000","last_peered":"2024-09-13T14:49:31.968085+0000","last_clean":"2024-09-13T14:49:31.968085+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:49:31.968085+0000","last_undegraded":"2024-09-13T14:49:31.968085+0000","last_fullsized":"2024-09-13T14:49:31.968085+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":225,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.976398+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:49:31.976398+0000","last_peered":"2024-09-13T14:49:31.976398+0000","last_clean":"2024-09-13T14:49:31.976398+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:49:31.976398+0000","last_undegraded":"2024-09-13T14:49:31.976398+0000","last_fullsized":"2024-09-13T14:49:31.976398+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":218,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.976328+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:49:31.976328+0000","last_peered":"2024-09-13T14:49:31.976328+0000","last_clean":"2024-09-13T14:49:31.976328+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:49:31.976328+0000","last_undegraded":"2024-09-13T14:49:31.976328+0000","last_fullsized":"2024-09-13T14:49:31.976328+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"256'10896","reported_seq":10931,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:18.901139+0000","last_change":"2024-09-13T14:47:20.107455+0000","last_active":"2024-09-13T14:50:18.901139+0000","last_peered":"2024-09-13T14:50:18.901139+0000","last_clean":"2024-09-13T14:50:18.901139+0000","last_became_active":"2024-09-13T14:47:20.107142+0000","last_became_peered":"2024-09-13T14:47:20.107142+0000","last_unstale":"2024-09-13T14:50:18.901139+0000","last_undegraded":"2024-09-13T14:50:18.901139+0000","last_fullsized":"2024-09-13T14:50:18.901139+0000","mapping_epoch":229,"log_start":"243'4200","ondisk_log_start":"243'4200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:19.189519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234094592,"num_objects":3572,"num_object_clones":0,"num_object_copies":10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":462976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":201,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.976436+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:49:31.976436+0000","last_peered":"2024-09-13T14:49:31.976436+0000","last_clean":"2024-09-13T14:49:31.976436+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:49:31.976436+0000","last_undegraded":"2024-09-13T14:49:31.976436+0000","last_fullsized":"2024-09-13T14:49:31.976436+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":251,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.971698+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:49:31.971698+0000","last_peered":"2024-09-13T14:49:31.971698+0000","last_clean":"2024-09-13T14:49:31.971698+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:49:31.971698+0000","last_undegraded":"2024-09-13T14:49:31.971698+0000","last_fullsized":"2024-09-13T14:49:31.971698+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":390,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.157182+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:50:05.157182+0000","last_peered":"2024-09-13T14:50:05.157182+0000","last_clean":"2024-09-13T14:50:05.157182+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:50:05.157182+0000","last_undegraded":"2024-09-13T14:50:05.157182+0000","last_fullsized":"2024-09-13T14:50:05.157182+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":78,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149369+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:50:05.149369+0000","last_peered":"2024-09-13T14:50:05.149369+0000","last_clean":"2024-09-13T14:50:05.149369+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:50:05.149369+0000","last_undegraded":"2024-09-13T14:50:05.149369+0000","last_fullsized":"2024-09-13T14:50:05.149369+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"256'10979","reported_seq":11014,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:18.995186+0000","last_change":"2024-09-13T14:47:20.054114+0000","last_active":"2024-09-13T14:50:18.995186+0000","last_peered":"2024-09-13T14:50:18.995186+0000","last_clean":"2024-09-13T14:50:18.995186+0000","last_became_active":"2024-09-13T14:47:20.053767+0000","last_became_peered":"2024-09-13T14:47:20.053767+0000","last_unstale":"2024-09-13T14:50:18.995186+0000","last_undegraded":"2024-09-13T14:50:18.995186+0000","last_fullsized":"2024-09-13T14:50:18.995186+0000","mapping_epoch":229,"log_start":"243'4700","ondisk_log_start":"243'4700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:48.035191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237436928,"num_objects":3623,"num_object_clones":0,"num_object_copies":10869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10979,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":359,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149795+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:50:05.149795+0000","last_peered":"2024-09-13T14:50:05.149795+0000","last_clean":"2024-09-13T14:50:05.149795+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:50:05.149795+0000","last_undegraded":"2024-09-13T14:50:05.149795+0000","last_fullsized":"2024-09-13T14:50:05.149795+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.968567+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:49:31.968567+0000","last_peered":"2024-09-13T14:49:31.968567+0000","last_clean":"2024-09-13T14:49:31.968567+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:49:31.968567+0000","last_undegraded":"2024-09-13T14:49:31.968567+0000","last_fullsized":"2024-09-13T14:49:31.968567+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":348,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.154688+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:50:05.154688+0000","last_peered":"2024-09-13T14:50:05.154688+0000","last_clean":"2024-09-13T14:50:05.154688+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:50:05.154688+0000","last_undegraded":"2024-09-13T14:50:05.154688+0000","last_fullsized":"2024-09-13T14:50:05.154688+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":39,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.976663+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:49:31.976663+0000","last_peered":"2024-09-13T14:49:31.976663+0000","last_clean":"2024-09-13T14:49:31.976663+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:49:31.976663+0000","last_undegraded":"2024-09-13T14:49:31.976663+0000","last_fullsized":"2024-09-13T14:49:31.976663+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"256'11105","reported_seq":11141,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:18.928816+0000","last_change":"2024-09-13T14:47:20.052876+0000","last_active":"2024-09-13T14:50:18.928816+0000","last_peered":"2024-09-13T14:50:18.928816+0000","last_clean":"2024-09-13T14:50:18.928816+0000","last_became_active":"2024-09-13T14:47:20.052566+0000","last_became_peered":"2024-09-13T14:47:20.052566+0000","last_unstale":"2024-09-13T14:50:18.928816+0000","last_undegraded":"2024-09-13T14:50:18.928816+0000","last_fullsized":"2024-09-13T14:50:18.928816+0000","mapping_epoch":229,"log_start":"243'4400","ondisk_log_start":"243'4400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:56.317980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230817816,"num_objects":3523,"num_object_clones":0,"num_object_copies":10569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11105,"num_write_kb":468033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":377,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.154022+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:50:05.154022+0000","last_peered":"2024-09-13T14:50:05.154022+0000","last_clean":"2024-09-13T14:50:05.154022+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:50:05.154022+0000","last_undegraded":"2024-09-13T14:50:05.154022+0000","last_fullsized":"2024-09-13T14:50:05.154022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":231,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980312+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:49:31.980312+0000","last_peered":"2024-09-13T14:49:31.980312+0000","last_clean":"2024-09-13T14:49:31.980312+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:49:31.980312+0000","last_undegraded":"2024-09-13T14:49:31.980312+0000","last_fullsized":"2024-09-13T14:49:31.980312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":332,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.161033+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:50:05.161033+0000","last_peered":"2024-09-13T14:50:05.161033+0000","last_clean":"2024-09-13T14:50:05.161033+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:50:05.161033+0000","last_undegraded":"2024-09-13T14:50:05.161033+0000","last_fullsized":"2024-09-13T14:50:05.161033+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":192,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.968354+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:49:31.968354+0000","last_peered":"2024-09-13T14:49:31.968354+0000","last_clean":"2024-09-13T14:49:31.968354+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:49:31.968354+0000","last_undegraded":"2024-09-13T14:49:31.968354+0000","last_fullsized":"2024-09-13T14:49:31.968354+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"256'11153","reported_seq":11185,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.013357+0000","last_change":"2024-09-13T14:47:20.052419+0000","last_active":"2024-09-13T14:50:19.013357+0000","last_peered":"2024-09-13T14:50:19.013357+0000","last_clean":"2024-09-13T14:50:19.013357+0000","last_became_active":"2024-09-13T14:47:20.051845+0000","last_became_peered":"2024-09-13T14:47:20.051845+0000","last_unstale":"2024-09-13T14:50:19.013357+0000","last_undegraded":"2024-09-13T14:50:19.013357+0000","last_fullsized":"2024-09-13T14:50:19.013357+0000","mapping_epoch":229,"log_start":"243'4314","ondisk_log_start":"243'4314","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6839,"log_dups_size":0,"ondisk_log_size":6839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.420332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232456192,"num_objects":3547,"num_object_clones":0,"num_object_copies":10641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11153,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979715+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:49:31.979715+0000","last_peered":"2024-09-13T14:49:31.979715+0000","last_clean":"2024-09-13T14:49:31.979715+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:49:31.979715+0000","last_undegraded":"2024-09-13T14:49:31.979715+0000","last_fullsized":"2024-09-13T14:49:31.979715+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.154896+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:50:05.154896+0000","last_peered":"2024-09-13T14:50:05.154896+0000","last_clean":"2024-09-13T14:50:05.154896+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:50:05.154896+0000","last_undegraded":"2024-09-13T14:50:05.154896+0000","last_fullsized":"2024-09-13T14:50:05.154896+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":234,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980960+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:49:31.980960+0000","last_peered":"2024-09-13T14:49:31.980960+0000","last_clean":"2024-09-13T14:49:31.980960+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:49:31.980960+0000","last_undegraded":"2024-09-13T14:49:31.980960+0000","last_fullsized":"2024-09-13T14:49:31.980960+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":212,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.970796+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:49:31.970796+0000","last_peered":"2024-09-13T14:49:31.970796+0000","last_clean":"2024-09-13T14:49:31.970796+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:49:31.970796+0000","last_undegraded":"2024-09-13T14:49:31.970796+0000","last_fullsized":"2024-09-13T14:49:31.970796+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"256'10902","reported_seq":10937,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:18.931608+0000","last_change":"2024-09-13T14:47:20.051919+0000","last_active":"2024-09-13T14:50:18.931608+0000","last_peered":"2024-09-13T14:50:18.931608+0000","last_clean":"2024-09-13T14:50:18.931608+0000","last_became_active":"2024-09-13T14:47:20.051626+0000","last_became_peered":"2024-09-13T14:47:20.051626+0000","last_unstale":"2024-09-13T14:50:18.931608+0000","last_undegraded":"2024-09-13T14:50:18.931608+0000","last_fullsized":"2024-09-13T14:50:18.931608+0000","mapping_epoch":229,"log_start":"243'4200","ondisk_log_start":"243'4200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6702,"log_dups_size":0,"ondisk_log_size":6702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:12.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230817792,"num_objects":3522,"num_object_clones":0,"num_object_copies":10566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":244,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.161189+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:50:05.161189+0000","last_peered":"2024-09-13T14:50:05.161189+0000","last_clean":"2024-09-13T14:50:05.161189+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:50:05.161189+0000","last_undegraded":"2024-09-13T14:50:05.161189+0000","last_fullsized":"2024-09-13T14:50:05.161189+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":219,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.976401+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:49:31.976401+0000","last_peered":"2024-09-13T14:49:31.976401+0000","last_clean":"2024-09-13T14:49:31.976401+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:49:31.976401+0000","last_undegraded":"2024-09-13T14:49:31.976401+0000","last_fullsized":"2024-09-13T14:49:31.976401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":230,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980008+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:49:31.980008+0000","last_peered":"2024-09-13T14:49:31.980008+0000","last_clean":"2024-09-13T14:49:31.980008+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:49:31.980008+0000","last_undegraded":"2024-09-13T14:49:31.980008+0000","last_fullsized":"2024-09-13T14:49:31.980008+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":39,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.978980+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:49:31.978980+0000","last_peered":"2024-09-13T14:49:31.978980+0000","last_clean":"2024-09-13T14:49:31.978980+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:49:31.978980+0000","last_undegraded":"2024-09-13T14:49:31.978980+0000","last_fullsized":"2024-09-13T14:49:31.978980+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"256'11002","reported_seq":14159,"reported_epoch":256,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:19.209621+0000","last_change":"2024-09-13T14:49:05.872778+0000","last_active":"2024-09-13T14:50:19.209621+0000","last_peered":"2024-09-13T14:50:19.209621+0000","last_clean":"2024-09-13T14:48:37.826901+0000","last_became_active":"2024-09-13T14:48:40.041761+0000","last_became_peered":"2024-09-13T14:48:40.041761+0000","last_unstale":"2024-09-13T14:50:19.209621+0000","last_undegraded":"2024-09-13T14:48:39.850843+0000","last_fullsized":"2024-09-13T14:48:39.825201+0000","mapping_epoch":249,"log_start":"239'2800","ondisk_log_start":"239'2800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235012096,"num_objects":3586,"num_object_clones":0,"num_object_copies":10758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":2050,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1545,"num_bytes_recovered":101253120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":2050},{"shards":"2,7","objects":-628}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":150,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.981349+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:49:31.981349+0000","last_peered":"2024-09-13T14:49:31.981349+0000","last_clean":"2024-09-13T14:49:31.981349+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:49:31.981349+0000","last_undegraded":"2024-09-13T14:49:31.981349+0000","last_fullsized":"2024-09-13T14:49:31.981349+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":184,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.164433+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:50:05.164433+0000","last_peered":"2024-09-13T14:50:05.164433+0000","last_clean":"2024-09-13T14:50:05.164433+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:50:05.164433+0000","last_undegraded":"2024-09-13T14:50:05.164433+0000","last_fullsized":"2024-09-13T14:50:05.164433+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":375,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.171343+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:50:05.171343+0000","last_peered":"2024-09-13T14:50:05.171343+0000","last_clean":"2024-09-13T14:50:05.171343+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:50:05.171343+0000","last_undegraded":"2024-09-13T14:50:05.171343+0000","last_fullsized":"2024-09-13T14:50:05.171343+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.978860+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:49:31.978860+0000","last_peered":"2024-09-13T14:49:31.978860+0000","last_clean":"2024-09-13T14:49:31.978860+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:49:31.978860+0000","last_undegraded":"2024-09-13T14:49:31.978860+0000","last_fullsized":"2024-09-13T14:49:31.978860+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"256'10947","reported_seq":11012,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.335038+0000","last_change":"2024-09-13T14:47:20.049747+0000","last_active":"2024-09-13T14:50:15.335038+0000","last_peered":"2024-09-13T14:50:15.335038+0000","last_clean":"2024-09-13T14:50:15.335038+0000","last_became_active":"2024-09-13T14:47:20.049448+0000","last_became_peered":"2024-09-13T14:47:20.049448+0000","last_unstale":"2024-09-13T14:50:15.335038+0000","last_undegraded":"2024-09-13T14:50:15.335038+0000","last_fullsized":"2024-09-13T14:50:15.335038+0000","mapping_epoch":229,"log_start":"240'3900","ondisk_log_start":"240'3900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7047,"log_dups_size":0,"ondisk_log_size":7047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:11.585800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247529472,"num_objects":3777,"num_object_clones":0,"num_object_copies":11331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10947,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":398,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.156945+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:50:05.156945+0000","last_peered":"2024-09-13T14:50:05.156945+0000","last_clean":"2024-09-13T14:50:05.156945+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:50:05.156945+0000","last_undegraded":"2024-09-13T14:50:05.156945+0000","last_fullsized":"2024-09-13T14:50:05.156945+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":408,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.154422+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:50:05.154422+0000","last_peered":"2024-09-13T14:50:05.154422+0000","last_clean":"2024-09-13T14:50:05.154422+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:50:05.154422+0000","last_undegraded":"2024-09-13T14:50:05.154422+0000","last_fullsized":"2024-09-13T14:50:05.154422+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":268,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979985+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:49:31.979985+0000","last_peered":"2024-09-13T14:49:31.979985+0000","last_clean":"2024-09-13T14:49:31.979985+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:49:31.979985+0000","last_undegraded":"2024-09-13T14:49:31.979985+0000","last_fullsized":"2024-09-13T14:49:31.979985+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":192,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.968179+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:49:31.968179+0000","last_peered":"2024-09-13T14:49:31.968179+0000","last_clean":"2024-09-13T14:49:31.968179+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:49:31.968179+0000","last_undegraded":"2024-09-13T14:49:31.968179+0000","last_fullsized":"2024-09-13T14:49:31.968179+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"256'11059","reported_seq":11094,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:18.911256+0000","last_change":"2024-09-13T14:47:20.106371+0000","last_active":"2024-09-13T14:50:18.911256+0000","last_peered":"2024-09-13T14:50:18.911256+0000","last_clean":"2024-09-13T14:50:18.911256+0000","last_became_active":"2024-09-13T14:47:20.105934+0000","last_became_peered":"2024-09-13T14:47:20.105934+0000","last_unstale":"2024-09-13T14:50:18.911256+0000","last_undegraded":"2024-09-13T14:50:18.911256+0000","last_fullsized":"2024-09-13T14:50:18.911256+0000","mapping_epoch":229,"log_start":"243'4300","ondisk_log_start":"243'4300","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6759,"log_dups_size":0,"ondisk_log_size":6759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:54.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235077632,"num_objects":3587,"num_object_clones":0,"num_object_copies":10761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11059,"num_write_kb":468672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":201,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.969737+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:49:31.969737+0000","last_peered":"2024-09-13T14:49:31.969737+0000","last_clean":"2024-09-13T14:49:31.969737+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:49:31.969737+0000","last_undegraded":"2024-09-13T14:49:31.969737+0000","last_fullsized":"2024-09-13T14:49:31.969737+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":307,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149040+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:50:05.149040+0000","last_peered":"2024-09-13T14:50:05.149040+0000","last_clean":"2024-09-13T14:50:05.149040+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:50:05.149040+0000","last_undegraded":"2024-09-13T14:50:05.149040+0000","last_fullsized":"2024-09-13T14:50:05.149040+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":229,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.981626+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:49:31.981626+0000","last_peered":"2024-09-13T14:49:31.981626+0000","last_clean":"2024-09-13T14:49:31.981626+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:49:31.981626+0000","last_undegraded":"2024-09-13T14:49:31.981626+0000","last_fullsized":"2024-09-13T14:49:31.981626+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.978239+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:49:31.978239+0000","last_peered":"2024-09-13T14:49:31.978239+0000","last_clean":"2024-09-13T14:49:31.978239+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:49:31.978239+0000","last_undegraded":"2024-09-13T14:49:31.978239+0000","last_fullsized":"2024-09-13T14:49:31.978239+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"256'10948","reported_seq":10981,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.069979+0000","last_change":"2024-09-13T14:47:20.050773+0000","last_active":"2024-09-13T14:50:19.069979+0000","last_peered":"2024-09-13T14:50:19.069979+0000","last_clean":"2024-09-13T14:50:19.069979+0000","last_became_active":"2024-09-13T14:47:20.050243+0000","last_became_peered":"2024-09-13T14:47:20.050243+0000","last_unstale":"2024-09-13T14:50:19.069979+0000","last_undegraded":"2024-09-13T14:50:19.069979+0000","last_fullsized":"2024-09-13T14:50:19.069979+0000","mapping_epoch":229,"log_start":"243'4600","ondisk_log_start":"243'4600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:42.724078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228982784,"num_objects":3494,"num_object_clones":0,"num_object_copies":10482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":30,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.162476+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:50:05.162476+0000","last_peered":"2024-09-13T14:50:05.162476+0000","last_clean":"2024-09-13T14:50:05.162476+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:50:05.162476+0000","last_undegraded":"2024-09-13T14:50:05.162476+0000","last_fullsized":"2024-09-13T14:50:05.162476+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":318,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.150069+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:50:05.150069+0000","last_peered":"2024-09-13T14:50:05.150069+0000","last_clean":"2024-09-13T14:50:05.150069+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:50:05.150069+0000","last_undegraded":"2024-09-13T14:50:05.150069+0000","last_fullsized":"2024-09-13T14:50:05.150069+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":39,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979645+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:49:31.979645+0000","last_peered":"2024-09-13T14:49:31.979645+0000","last_clean":"2024-09-13T14:49:31.979645+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:49:31.979645+0000","last_undegraded":"2024-09-13T14:49:31.979645+0000","last_fullsized":"2024-09-13T14:49:31.979645+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"256'10883","reported_seq":10918,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:18.892333+0000","last_change":"2024-09-13T14:47:20.106351+0000","last_active":"2024-09-13T14:50:18.892333+0000","last_peered":"2024-09-13T14:50:18.892333+0000","last_clean":"2024-09-13T14:50:18.892333+0000","last_became_active":"2024-09-13T14:47:20.105946+0000","last_became_peered":"2024-09-13T14:47:20.105946+0000","last_unstale":"2024-09-13T14:50:18.892333+0000","last_undegraded":"2024-09-13T14:50:18.892333+0000","last_fullsized":"2024-09-13T14:50:18.892333+0000","mapping_epoch":229,"log_start":"243'4200","ondisk_log_start":"243'4200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6683,"log_dups_size":0,"ondisk_log_size":6683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:07.223637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233504768,"num_objects":3563,"num_object_clones":0,"num_object_copies":10689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10883,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":238,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.977397+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:49:31.977397+0000","last_peered":"2024-09-13T14:49:31.977397+0000","last_clean":"2024-09-13T14:49:31.977397+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:49:31.977397+0000","last_undegraded":"2024-09-13T14:49:31.977397+0000","last_fullsized":"2024-09-13T14:49:31.977397+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":219,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.976557+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:49:31.976557+0000","last_peered":"2024-09-13T14:49:31.976557+0000","last_clean":"2024-09-13T14:49:31.976557+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:49:31.976557+0000","last_undegraded":"2024-09-13T14:49:31.976557+0000","last_fullsized":"2024-09-13T14:49:31.976557+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":368,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.153804+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:50:05.153804+0000","last_peered":"2024-09-13T14:50:05.153804+0000","last_clean":"2024-09-13T14:50:05.153804+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:50:05.153804+0000","last_undegraded":"2024-09-13T14:50:05.153804+0000","last_fullsized":"2024-09-13T14:50:05.153804+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"256'11116","reported_seq":14691,"reported_epoch":256,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:19.216399+0000","last_change":"2024-09-13T14:49:05.871518+0000","last_active":"2024-09-13T14:50:19.216399+0000","last_peered":"2024-09-13T14:50:19.216399+0000","last_clean":"2024-09-13T14:48:09.848772+0000","last_became_active":"2024-09-13T14:48:12.232150+0000","last_became_peered":"2024-09-13T14:48:12.232150+0000","last_unstale":"2024-09-13T14:50:19.216399+0000","last_undegraded":"2024-09-13T14:49:05.858877+0000","last_fullsized":"2024-09-13T14:48:11.643512+0000","mapping_epoch":242,"log_start":"239'3000","ondisk_log_start":"239'3000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8116,"log_dups_size":0,"ondisk_log_size":8116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229376000,"num_objects":3500,"num_object_clones":0,"num_object_copies":10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11116,"num_write_kb":467712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":114753536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":370},{"shards":"2,5","objects":1073}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":235,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.977018+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:49:31.977018+0000","last_peered":"2024-09-13T14:49:31.977018+0000","last_clean":"2024-09-13T14:49:31.977018+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:49:31.977018+0000","last_undegraded":"2024-09-13T14:49:31.977018+0000","last_fullsized":"2024-09-13T14:49:31.977018+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":239,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.968155+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:49:31.968155+0000","last_peered":"2024-09-13T14:49:31.968155+0000","last_clean":"2024-09-13T14:49:31.968155+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:49:31.968155+0000","last_undegraded":"2024-09-13T14:49:31.968155+0000","last_fullsized":"2024-09-13T14:49:31.968155+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":21,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.978721+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:49:31.978721+0000","last_peered":"2024-09-13T14:49:31.978721+0000","last_clean":"2024-09-13T14:49:31.978721+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:49:31.978721+0000","last_undegraded":"2024-09-13T14:49:31.978721+0000","last_fullsized":"2024-09-13T14:49:31.978721+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"256'11152","reported_seq":11187,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.114126+0000","last_change":"2024-09-13T14:47:20.106142+0000","last_active":"2024-09-13T14:50:19.114126+0000","last_peered":"2024-09-13T14:50:19.114126+0000","last_clean":"2024-09-13T14:50:19.114126+0000","last_became_active":"2024-09-13T14:47:20.105758+0000","last_became_peered":"2024-09-13T14:47:20.105758+0000","last_unstale":"2024-09-13T14:50:19.114126+0000","last_undegraded":"2024-09-13T14:50:19.114126+0000","last_fullsized":"2024-09-13T14:50:19.114126+0000","mapping_epoch":229,"log_start":"244'4900","ondisk_log_start":"244'4900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6252,"log_dups_size":0,"ondisk_log_size":6252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:51.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236191744,"num_objects":3604,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11152,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":218,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.976149+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:49:31.976149+0000","last_peered":"2024-09-13T14:49:31.976149+0000","last_clean":"2024-09-13T14:49:31.976149+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:49:31.976149+0000","last_undegraded":"2024-09-13T14:49:31.976149+0000","last_fullsized":"2024-09-13T14:49:31.976149+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":347,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149447+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:50:05.149447+0000","last_peered":"2024-09-13T14:50:05.149447+0000","last_clean":"2024-09-13T14:50:05.149447+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:50:05.149447+0000","last_undegraded":"2024-09-13T14:50:05.149447+0000","last_fullsized":"2024-09-13T14:50:05.149447+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":39,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980185+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:49:31.980185+0000","last_peered":"2024-09-13T14:49:31.980185+0000","last_clean":"2024-09-13T14:49:31.980185+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:49:31.980185+0000","last_undegraded":"2024-09-13T14:49:31.980185+0000","last_fullsized":"2024-09-13T14:49:31.980185+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"256'11026","reported_seq":11092,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.201684+0000","last_change":"2024-09-13T14:47:20.107374+0000","last_active":"2024-09-13T14:50:19.201684+0000","last_peered":"2024-09-13T14:50:19.201684+0000","last_clean":"2024-09-13T14:50:19.201684+0000","last_became_active":"2024-09-13T14:47:20.107057+0000","last_became_peered":"2024-09-13T14:47:20.107057+0000","last_unstale":"2024-09-13T14:50:19.201684+0000","last_undegraded":"2024-09-13T14:50:19.201684+0000","last_fullsized":"2024-09-13T14:50:19.201684+0000","mapping_epoch":229,"log_start":"239'2900","ondisk_log_start":"239'2900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8126,"log_dups_size":0,"ondisk_log_size":8126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:05.814456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236060672,"num_objects":3602,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":387,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.153908+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:50:05.153908+0000","last_peered":"2024-09-13T14:50:05.153908+0000","last_clean":"2024-09-13T14:50:05.153908+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:50:05.153908+0000","last_undegraded":"2024-09-13T14:50:05.153908+0000","last_fullsized":"2024-09-13T14:50:05.153908+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":217,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980961+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:49:31.980961+0000","last_peered":"2024-09-13T14:49:31.980961+0000","last_clean":"2024-09-13T14:49:31.980961+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:49:31.980961+0000","last_undegraded":"2024-09-13T14:49:31.980961+0000","last_fullsized":"2024-09-13T14:49:31.980961+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":39,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.977192+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:49:31.977192+0000","last_peered":"2024-09-13T14:49:31.977192+0000","last_clean":"2024-09-13T14:49:31.977192+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:49:31.977192+0000","last_undegraded":"2024-09-13T14:49:31.977192+0000","last_fullsized":"2024-09-13T14:49:31.977192+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"256'10794","reported_seq":10860,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.329887+0000","last_change":"2024-09-13T14:47:20.108173+0000","last_active":"2024-09-13T14:50:15.329887+0000","last_peered":"2024-09-13T14:50:15.329887+0000","last_clean":"2024-09-13T14:50:15.329887+0000","last_became_active":"2024-09-13T14:47:20.107879+0000","last_became_peered":"2024-09-13T14:47:20.107879+0000","last_unstale":"2024-09-13T14:50:15.329887+0000","last_undegraded":"2024-09-13T14:50:15.329887+0000","last_fullsized":"2024-09-13T14:50:15.329887+0000","mapping_epoch":229,"log_start":"240'3800","ondisk_log_start":"240'3800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:50.152706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243007488,"num_objects":3708,"num_object_clones":0,"num_object_copies":11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10794,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":231,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.968016+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:49:31.968016+0000","last_peered":"2024-09-13T14:49:31.968016+0000","last_clean":"2024-09-13T14:49:31.968016+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:49:31.968016+0000","last_undegraded":"2024-09-13T14:49:31.968016+0000","last_fullsized":"2024-09-13T14:49:31.968016+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":232,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980242+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:49:31.980242+0000","last_peered":"2024-09-13T14:49:31.980242+0000","last_clean":"2024-09-13T14:49:31.980242+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:49:31.980242+0000","last_undegraded":"2024-09-13T14:49:31.980242+0000","last_fullsized":"2024-09-13T14:49:31.980242+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":192,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.971221+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:49:31.971221+0000","last_peered":"2024-09-13T14:49:31.971221+0000","last_clean":"2024-09-13T14:49:31.971221+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:49:31.971221+0000","last_undegraded":"2024-09-13T14:49:31.971221+0000","last_fullsized":"2024-09-13T14:49:31.971221+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"256'10845","reported_seq":10882,"reported_epoch":256,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:18.883768+0000","last_change":"2024-09-13T14:48:40.078958+0000","last_active":"2024-09-13T14:50:18.883768+0000","last_peered":"2024-09-13T14:50:18.883768+0000","last_clean":"2024-09-13T14:48:37.811111+0000","last_became_active":"2024-09-13T14:48:40.052181+0000","last_became_peered":"2024-09-13T14:48:40.052181+0000","last_unstale":"2024-09-13T14:50:18.883768+0000","last_undegraded":"2024-09-13T14:48:39.849823+0000","last_fullsized":"2024-09-13T14:48:39.826167+0000","mapping_epoch":249,"log_start":"243'4100","ondisk_log_start":"243'4100","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:49.521146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238878720,"num_objects":3645,"num_object_clones":0,"num_object_copies":10935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10845,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":2149},{"shards":"3,6","objects":-2104}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":245,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.976498+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:49:31.976498+0000","last_peered":"2024-09-13T14:49:31.976498+0000","last_clean":"2024-09-13T14:49:31.976498+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:49:31.976498+0000","last_undegraded":"2024-09-13T14:49:31.976498+0000","last_fullsized":"2024-09-13T14:49:31.976498+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":222,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979377+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:49:31.979377+0000","last_peered":"2024-09-13T14:49:31.979377+0000","last_clean":"2024-09-13T14:49:31.979377+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:49:31.979377+0000","last_undegraded":"2024-09-13T14:49:31.979377+0000","last_fullsized":"2024-09-13T14:49:31.979377+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":75,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149477+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:50:05.149477+0000","last_peered":"2024-09-13T14:50:05.149477+0000","last_clean":"2024-09-13T14:50:05.149477+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:50:05.149477+0000","last_undegraded":"2024-09-13T14:50:05.149477+0000","last_fullsized":"2024-09-13T14:50:05.149477+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"256'11262","reported_seq":11297,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.683435+0000","last_change":"2024-09-13T14:47:20.051824+0000","last_active":"2024-09-13T14:50:19.683435+0000","last_peered":"2024-09-13T14:50:19.683435+0000","last_clean":"2024-09-13T14:50:19.683435+0000","last_became_active":"2024-09-13T14:47:20.051487+0000","last_became_peered":"2024-09-13T14:47:20.051487+0000","last_unstale":"2024-09-13T14:50:19.683435+0000","last_undegraded":"2024-09-13T14:50:19.683435+0000","last_fullsized":"2024-09-13T14:50:19.683435+0000","mapping_epoch":229,"log_start":"239'3100","ondisk_log_start":"239'3100","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8162,"log_dups_size":0,"ondisk_log_size":8162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:11.533543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231473152,"num_objects":3532,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11262,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.978792+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:49:31.978792+0000","last_peered":"2024-09-13T14:49:31.978792+0000","last_clean":"2024-09-13T14:49:31.978792+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:49:31.978792+0000","last_undegraded":"2024-09-13T14:49:31.978792+0000","last_fullsized":"2024-09-13T14:49:31.978792+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979600+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:49:31.979600+0000","last_peered":"2024-09-13T14:49:31.979600+0000","last_clean":"2024-09-13T14:49:31.979600+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:49:31.979600+0000","last_undegraded":"2024-09-13T14:49:31.979600+0000","last_fullsized":"2024-09-13T14:49:31.979600+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980065+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:49:31.980065+0000","last_peered":"2024-09-13T14:49:31.980065+0000","last_clean":"2024-09-13T14:49:31.980065+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:49:31.980065+0000","last_undegraded":"2024-09-13T14:49:31.980065+0000","last_fullsized":"2024-09-13T14:49:31.980065+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":267,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.967383+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:49:31.967383+0000","last_peered":"2024-09-13T14:49:31.967383+0000","last_clean":"2024-09-13T14:49:31.967383+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:49:31.967383+0000","last_undegraded":"2024-09-13T14:49:31.967383+0000","last_fullsized":"2024-09-13T14:49:31.967383+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":356,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.154730+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:50:05.154730+0000","last_peered":"2024-09-13T14:50:05.154730+0000","last_clean":"2024-09-13T14:50:05.154730+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:50:05.154730+0000","last_undegraded":"2024-09-13T14:50:05.154730+0000","last_fullsized":"2024-09-13T14:50:05.154730+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":49,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979788+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:49:31.979788+0000","last_peered":"2024-09-13T14:49:31.979788+0000","last_clean":"2024-09-13T14:49:31.979788+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:49:31.979788+0000","last_undegraded":"2024-09-13T14:49:31.979788+0000","last_fullsized":"2024-09-13T14:49:31.979788+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":213,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980310+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:49:31.980310+0000","last_peered":"2024-09-13T14:49:31.980310+0000","last_clean":"2024-09-13T14:49:31.980310+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:49:31.980310+0000","last_undegraded":"2024-09-13T14:49:31.980310+0000","last_fullsized":"2024-09-13T14:49:31.980310+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":400,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.157106+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:50:05.157106+0000","last_peered":"2024-09-13T14:50:05.157106+0000","last_clean":"2024-09-13T14:50:05.157106+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:50:05.157106+0000","last_undegraded":"2024-09-13T14:50:05.157106+0000","last_fullsized":"2024-09-13T14:50:05.157106+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":71,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149256+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:50:05.149256+0000","last_peered":"2024-09-13T14:50:05.149256+0000","last_clean":"2024-09-13T14:50:05.149256+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:50:05.149256+0000","last_undegraded":"2024-09-13T14:50:05.149256+0000","last_fullsized":"2024-09-13T14:50:05.149256+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":407,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.154406+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:50:05.154406+0000","last_peered":"2024-09-13T14:50:05.154406+0000","last_clean":"2024-09-13T14:50:05.154406+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:50:05.154406+0000","last_undegraded":"2024-09-13T14:50:05.154406+0000","last_fullsized":"2024-09-13T14:50:05.154406+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.980843+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:49:31.980843+0000","last_peered":"2024-09-13T14:49:31.980843+0000","last_clean":"2024-09-13T14:49:31.980843+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:49:31.980843+0000","last_undegraded":"2024-09-13T14:49:31.980843+0000","last_fullsized":"2024-09-13T14:49:31.980843+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":39,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.978863+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:49:31.978863+0000","last_peered":"2024-09-13T14:49:31.978863+0000","last_clean":"2024-09-13T14:49:31.978863+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:49:31.978863+0000","last_undegraded":"2024-09-13T14:49:31.978863+0000","last_fullsized":"2024-09-13T14:49:31.978863+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":238,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:49:31.979056+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:49:31.979056+0000","last_peered":"2024-09-13T14:49:31.979056+0000","last_clean":"2024-09-13T14:49:31.979056+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:49:31.979056+0000","last_undegraded":"2024-09-13T14:49:31.979056+0000","last_fullsized":"2024-09-13T14:49:31.979056+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":66,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.154584+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:50:05.154584+0000","last_peered":"2024-09-13T14:50:05.154584+0000","last_clean":"2024-09-13T14:50:05.154584+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:50:05.154584+0000","last_undegraded":"2024-09-13T14:50:05.154584+0000","last_fullsized":"2024-09-13T14:50:05.154584+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":364,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.155012+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:50:05.155012+0000","last_peered":"2024-09-13T14:50:05.155012+0000","last_clean":"2024-09-13T14:50:05.155012+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:50:05.155012+0000","last_undegraded":"2024-09-13T14:50:05.155012+0000","last_fullsized":"2024-09-13T14:50:05.155012+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3760717848,"num_objects":57385,"num_object_clones":0,"num_object_copies":172155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10340,"num_objects_misplaced":2050,"num_objects_unfound":0,"num_objects_dirty":57385,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176069,"num_write_kb":7470465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3296,"num_bytes_recovered":216006656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12099334144,"data_stored":12099321928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112655,"ondisk_log_size":112655,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365167,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1820084,"kb_used_data":919020,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":91925068,"statfs":{"total":95995035648,"available":94131269632,"internally_reserved":0,"allocated":941076480,"data_stored":939730525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561397,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2526704,"kb_used_data":1629032,"kb_used_omap":1,"kb_used_meta":897662,"kb_avail":91218448,"statfs":{"total":95995035648,"available":93407690752,"internally_reserved":0,"allocated":1668128768,"data_stored":1666786756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919206232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790329,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1787748,"kb_used_data":878364,"kb_used_omap":1,"kb_used_meta":909374,"kb_avail":91957404,"statfs":{"total":95995035648,"available":94164381696,"internally_reserved":0,"allocated":899444736,"data_stored":898111106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931199320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986557,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3379956,"kb_used_data":2485548,"kb_used_omap":1,"kb_used_meta":894398,"kb_avail":90365196,"statfs":{"total":95995035648,"available":92533960704,"internally_reserved":0,"allocated":2545201152,"data_stored":2543855382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915863896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519712,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1654244,"kb_used_data":1138012,"kb_used_omap":4,"kb_used_meta":516219,"kb_avail":92090908,"statfs":{"total":95995035648,"available":94301089792,"internally_reserved":0,"allocated":1165324288,"data_stored":1163990668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":528608704},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304628,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2238888,"kb_used_data":1748064,"kb_used_omap":6,"kb_used_meta":490809,"kb_avail":91506264,"statfs":{"total":95995035648,"available":93702414336,"internally_reserved":0,"allocated":1790017536,"data_stored":1788680334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":502588871},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827931,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501348,"kb_used_data":715612,"kb_used_omap":5,"kb_used_meta":785722,"kb_avail":92243804,"statfs":{"total":95995035648,"available":94457655296,"internally_reserved":0,"allocated":732786688,"data_stored":731452495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":804579564},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285795,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3107108,"kb_used_data":2409628,"kb_used_omap":5,"kb_used_meta":697466,"kb_avail":90638044,"statfs":{"total":95995035648,"available":92813357056,"internally_reserved":0,"allocated":2467459072,"data_stored":2466124206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":714205243},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2460487680,"data_stored":2460483608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":723976192,"data_stored":723976192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1782710272,"data_stored":1782710272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1161363456,"data_stored":1161363456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2511474688,"data_stored":2511470616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":886046720,"data_stored":886046720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1640566784,"data_stored":1640562712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":932708352,"data_stored":932708352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:50:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:21.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:21 smithi019 ceph-mon[91296]: pgmap v867: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 10340/172233 objects degraded (6.003%); 2050/172233 objects misplaced (1.190%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:50:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:21 smithi019 ceph-mon[91296]: from='client.46346 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10340/172233 objects degraded (6.003%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1464437763' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:21 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:21.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:21.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:21.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:21.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:22.053 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.093 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:22.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:22.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:22.232 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:22.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:22.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:22.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:22.398 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:22 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:22.398 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:22 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:50:22.398 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:22 smithi003 ceph-mon[107278]: osdmap e257: 8 total, 8 up, 8 in 2024-09-13T14:50:22.398 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:22 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:22.398 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:22 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:50:22.398 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:22 smithi003 ceph-mon[104007]: osdmap e257: 8 total, 8 up, 8 in 2024-09-13T14:50:22.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:22.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:22.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:22.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.659 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:22.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:22 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:22 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:50:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:22 smithi019 ceph-mon[91296]: osdmap e257: 8 total, 8 up, 8 in 2024-09-13T14:50:22.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:22.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:22.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:22.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.936 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:22.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:23.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.074 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:23.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:23.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:23.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:23.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:23.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:23.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:23.499 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:23.601 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:23.644 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:23.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:23.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:23 smithi019 ceph-mon[91296]: pgmap v869: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 10340/170634 objects degraded (6.060%); 2050/170634 objects misplaced (1.201%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:50:23.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:23 smithi003 ceph-mon[107278]: pgmap v869: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 10340/170634 objects degraded (6.060%); 2050/170634 objects misplaced (1.201%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:50:23.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:23 smithi003 ceph-mon[104007]: pgmap v869: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 10340/170634 objects degraded (6.060%); 2050/170634 objects misplaced (1.201%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:50:23.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:23.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:23.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:23.916 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:24.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:24.049 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:24.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:24.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:24.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:24.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:24.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:50:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:50:24] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T14:50:24.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:24.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:24.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:24.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:50:24.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:24.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:24.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:24.898 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:25.000 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.031 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:25.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:25.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:25.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:25.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:25.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:25.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:50:25.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:25.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:25.549 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:25 smithi003 ceph-mon[104007]: pgmap v870: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 10340/170634 objects degraded (6.060%); 2050/170634 objects misplaced (1.201%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:50:25.549 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:25 smithi003 ceph-mon[107278]: pgmap v870: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 10340/170634 objects degraded (6.060%); 2050/170634 objects misplaced (1.201%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45869.0:120796 9.12 9:4fa44c23:::benchmark_data_smithi003_156985_object120795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:43:18.870541+0000", 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 426.67556070299997, 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4219450079999998, 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.550 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45869", 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 120796 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.870541+0000", 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.870541+0000", 2024-09-13T14:50:25.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.870543+0000", 2024-09-13T14:50:25.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.04e-06 2024-09-13T14:50:25.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.870565+0000", 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1973000000000001e-05 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.870577+0000", 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1719000000000001e-05 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.870584+0000", 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2320000000000004e-06 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.871181+0000", 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00059714999999999996 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:18.871190+0000", 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0820000000000005e-06 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.854930+0000", 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98373961099999996 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:19.854938+0000", 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2910000000000007e-06 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:26.288937+0000", 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4339986409999996 2024-09-13T14:50:25.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:43:26.292486+0000", 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035492689999999999 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:240147 9.8 e216/179 9:15bf2b63:::benchmark_data_smithi003_156985_object119231:head v 216'9057, mlcod=177'5253)", 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:35.179698+0000", 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 230.366403453, 2024-09-13T14:50:25.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21772921100000001, 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.179698+0000", 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.179698+0000", 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.179699+0000", 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8999999999999998e-07 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.179703+0000", 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4700000000000004e-06 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.179709+0000", 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.071e-06 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.179713+0000", 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0210000000000001e-06 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.184779+0000", 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050656709999999999 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.184805+0000", 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6279e-05 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.397409+0000", 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21260379700000001 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.397428+0000", 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8612000000000001e-05 2024-09-13T14:50:25.560 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:240163 9.8 e216/179 9:117c80e2:::benchmark_data_smithi003_156985_object119247:head v 216'9058, mlcod=177'5253)", 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:35.181334+0000", 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 230.36476785299999, 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21610876100000001, 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.181334+0000", 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.181334+0000", 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.181335+0000", 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6700000000000002e-07 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.181341+0000", 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.455e-06 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.181353+0000", 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1228e-05 2024-09-13T14:50:25.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.181359+0000", 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.601e-06 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.388444+0000", 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20708484499999999 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.388498+0000", 2024-09-13T14:50:25.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3810000000000001e-05 2024-09-13T14:50:25.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.397430+0000", 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089320979999999994 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.397443+0000", 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2857000000000001e-05 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45869.0:240195 9.f e216/152 9:f301f0c3:::benchmark_data_smithi003_156985_object119279:head v 216'14679, mlcod=216'14677)", 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:46:35.194073+0000", 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 230.352028819, 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.206836145, 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.194073+0000", 2024-09-13T14:50:25.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.194073+0000", 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.194074+0000", 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.001e-06 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.194081+0000", 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1679999999999998e-06 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.386436+0000", 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19235459199999999 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.386446+0000", 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0579e-05 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.386561+0000", 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114992 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.386582+0000", 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0469999999999999e-05 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.400823+0000", 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014241512 2024-09-13T14:50:25.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:46:35.400909+0000", 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5830999999999999e-05 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:34969 10.b e238/229 10:d27681eb:::benchmark_data_smithi003_197468_object34968:head v 238'2158, mlcod=238'2156)", 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:51.767910+0000", 2024-09-13T14:50:25.571 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 153.77819207300001, 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.097294981000000003, 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.767910+0000", 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.767910+0000", 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.767911+0000", 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1100000000000004e-07 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.767985+0000", 2024-09-13T14:50:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4707000000000001e-05 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.768000+0000", 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4780000000000001e-05 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.768009+0000", 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6989999999999992e-06 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.768071+0000", 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1976000000000004e-05 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.768092+0000", 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1182000000000001e-05 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.865161+0000", 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.097069039999999995 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.865205+0000", 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3686000000000002e-05 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:34970 10.b e238/229 10:d1615903:::benchmark_data_smithi003_197468_object34969:head v 238'2159, mlcod=238'2156)", 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:51.768913+0000", 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 153.77718881999999, 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.096329866, 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.576 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.577 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.768913+0000", 2024-09-13T14:50:25.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.768913+0000", 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.768914+0000", 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1800000000000005e-07 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.768984+0000", 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9558e-05 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.768995+0000", 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1847e-05 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.769006+0000", 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0951000000000001e-05 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.769383+0000", 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037641900000000002 2024-09-13T14:50:25.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.769402+0000", 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9258000000000001e-05 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.865210+0000", 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.095807798999999999 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.865243+0000", 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3216000000000002e-05 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:34976 10.2 e238/229 10:4dbe6cac:::benchmark_data_smithi003_197468_object34975:head v 238'2204, mlcod=238'2202)", 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:51.771621+0000", 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 153.77448140499999, 2024-09-13T14:50:25.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.094189755, 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.771621+0000", 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.771621+0000", 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.771622+0000", 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1880000000000001e-06 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.771651+0000", 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9042000000000001e-05 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.862624+0000", 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.090973706000000001 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.862638+0000", 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3413e-05 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.862705+0000", 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6921999999999994e-05 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.862729+0000", 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3730000000000001e-05 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.865763+0000", 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030346240000000001 2024-09-13T14:50:25.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:51.865810+0000", 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7129999999999998e-05 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:76314 10.2 e243/229 10:4841c7b7:::benchmark_data_smithi003_197468_object76313:head v 243'4828, mlcod=243'4824)", 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:48:21.123513+0000", 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 124.42258928699999, 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027632879999999999, 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.586 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.123513+0000", 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.123513+0000", 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.123513+0000", 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9599999999999999e-07 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.123557+0000", 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3677999999999998e-05 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.123570+0000", 2024-09-13T14:50:25.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2741e-05 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.123576+0000", 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0419999999999999e-06 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.126691+0000", 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031155750000000002 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.126746+0000", 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4330999999999999e-05 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.150966+0000", 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024220512 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.151146+0000", 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017930499999999999 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.590 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:76572 10.0 e243/229 10:091b4821:::benchmark_data_smithi003_197468_object76571:head v 243'4733, mlcod=243'4731)", 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:48:21.364899+0000", 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 124.18120249, 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024871126, 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.364899+0000", 2024-09-13T14:50:25.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.364899+0000", 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.364900+0000", 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0100000000000001e-06 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.364942+0000", 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1863999999999998e-05 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.364961+0000", 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9116000000000001e-05 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.364970+0000", 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.473e-06 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.365163+0000", 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019344499999999999 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.365246+0000", 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2789999999999998e-05 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.389721+0000", 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024474523000000001 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:21.389771+0000", 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9904999999999997e-05 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:101645 10.2 e250/229 10:4d8d6088:::benchmark_data_smithi003_197468_object101644:head v 250'6399, mlcod=250'6395)", 2024-09-13T14:50:25.595 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:48:50.047538+0000", 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 95.49856355, 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025083013000000001, 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.047538+0000", 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.047538+0000", 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.047539+0000", 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3399999999999999e-07 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.047566+0000", 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6474000000000001e-05 2024-09-13T14:50:25.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.047575+0000", 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2900000000000008e-06 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.047581+0000", 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7440000000000004e-06 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.056202+0000", 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0086219499999999998 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.056271+0000", 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8532e-05 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.072365+0000", 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016093580999999999 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:50.072621+0000", 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025680800000000003 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:103098 10.2 e250/229 10:421da409:::benchmark_data_smithi003_197468_object103097:head v 250'6503, mlcod=250'6501)", 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:48:51.790239+0000", 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 93.755862811, 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040377009999999998, 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.790239+0000", 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.790239+0000", 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.790240+0000", 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.238e-06 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.790283+0000", 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2704000000000003e-05 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.790303+0000", 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0369000000000001e-05 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.790312+0000", 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7309999999999999e-06 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.790494+0000", 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018157899999999999 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.790545+0000", 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1004000000000002e-05 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.830444+0000", 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039899724999999997 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.830616+0000", 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017165999999999999 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:103102 10.2 e250/229 10:47d6f782:::benchmark_data_smithi003_197468_object103101:head v 250'6504, mlcod=250'6501)", 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:48:51.793798+0000", 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 93.752303577000006, 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.037012978000000002, 2024-09-13T14:50:25.605 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.793798+0000", 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.793798+0000", 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.793800+0000", 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.294e-06 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.793860+0000", 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0214000000000003e-05 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.793875+0000", 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4687e-05 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.793882+0000", 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9580000000000001e-06 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.809914+0000", 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016032115999999999 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.809962+0000", 2024-09-13T14:50:25.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8032999999999999e-05 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.830621+0000", 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020659819999999999 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.830811+0000", 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018985599999999999 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:103113 10.2 e250/229 10:4245687b:::benchmark_data_smithi003_197468_object103112:head v 250'6508, mlcod=250'6501)", 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:48:51.819040+0000", 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 93.727062121000003, 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026381913, 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.819040+0000", 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.819040+0000", 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.819041+0000", 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2330000000000001e-06 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.819110+0000", 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9282999999999994e-05 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.819133+0000", 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2316000000000001e-05 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.819144+0000", 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1129e-05 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.838552+0000", 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019408491 2024-09-13T14:50:25.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.838573+0000", 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1066000000000001e-05 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.845197+0000", 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066239580000000001 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:48:51.845422+0000", 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022443699999999999 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:122398 10.b e253/229 10:d5c3e660:::benchmark_data_smithi003_197468_object5669:head v 253'7688, mlcod=253'7685)", 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:16.230664+0000", 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 69.315438274000002, 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025275181000000001, 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.616 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.230664+0000", 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.230664+0000", 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.230665+0000", 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.114e-06 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.230676+0000", 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0757000000000001e-05 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.232695+0000", 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020196379999999998 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.232709+0000", 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3478e-05 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.233374+0000", 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00066490600000000005 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.233477+0000", 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103378 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.255664+0000", 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022187235999999999 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:16.255939+0000", 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027467399999999998 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.620 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:175786 10.0 e256/229 10:0d6990e2:::benchmark_data_smithi003_197468_object59057:head v 256'10850, mlcod=256'10848)", 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:50:18.342361+0000", 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.2037413130000001, 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028437985999999998, 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.342361+0000", 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.342361+0000", 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.342362+0000", 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.339e-06 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.342374+0000", 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1678999999999999e-05 2024-09-13T14:50:25.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.342395+0000", 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1098e-05 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.342404+0000", 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2199999999999998e-06 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.342481+0000", 2024-09-13T14:50:25.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6662999999999994e-05 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.342501+0000", 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0086e-05 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.370755+0000", 2024-09-13T14:50:25.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028254540000000002 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:18.370799+0000", 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3361e-05 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:178081 10.b e256/229 10:dc20a7be:::benchmark_data_smithi003_197468_object61352:head v 256'11174, mlcod=256'11171)", 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:50:21.000467+0000", 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.5456353739999997, 2024-09-13T14:50:25.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027851417999999999, 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.000467+0000", 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.000467+0000", 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.000468+0000", 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1599999999999999e-06 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.000477+0000", 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1390000000000001e-06 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.000494+0000", 2024-09-13T14:50:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6657999999999999e-05 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.000501+0000", 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.289e-06 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.003036+0000", 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025354520000000001 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.003114+0000", 2024-09-13T14:50:25.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8127000000000006e-05 2024-09-13T14:50:25.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.028270+0000", 2024-09-13T14:50:25.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025155734999999999 2024-09-13T14:50:25.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.028318+0000", 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7858000000000002e-05 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:178083 10.b e256/229 10:d9aa9373:::benchmark_data_smithi003_197468_object61354:head v 256'11176, mlcod=256'11171)", 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:50:21.007107+0000", 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.538995152, 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025071078, 2024-09-13T14:50:25.632 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.007107+0000", 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.007107+0000", 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.007107+0000", 2024-09-13T14:50:25.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5099999999999999e-07 2024-09-13T14:50:25.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.007115+0000", 2024-09-13T14:50:25.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1879999999999996e-06 2024-09-13T14:50:25.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.007127+0000", 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.259e-05 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.007133+0000", 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.823e-06 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.014068+0000", 2024-09-13T14:50:25.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069353059999999996 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.014089+0000", 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1044999999999999e-05 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.032127+0000", 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018037926999999999 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:21.032178+0000", 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0547999999999998e-05 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:180334 10.0 e257/229 10:05fdc067:::benchmark_data_smithi003_197468_object63605:head v 257'11134, mlcod=257'11132)", 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:50:23.436301+0000", 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.109800973, 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025457565000000001, 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.637 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.436301+0000", 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.436301+0000", 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.436302+0000", 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2500000000000005e-07 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.436307+0000", 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3600000000000004e-06 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.436318+0000", 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1156e-05 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.436324+0000", 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8590000000000001e-06 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.439928+0000", 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003603443 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.439988+0000", 2024-09-13T14:50:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0059999999999998e-05 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.461710+0000", 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021722439999999999 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:23.461759+0000", 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8522000000000001e-05 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:181957 10.b e257/229 10:d050b1a4:::benchmark_data_smithi003_197468_object65228:head v 257'11436, mlcod=257'11433)", 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:50:25.253843+0000", 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.29225873499999999, 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.031197144999999999, 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.253843+0000", 2024-09-13T14:50:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.253843+0000", 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.253843+0000", 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2399999999999999e-07 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.253856+0000", 2024-09-13T14:50:25.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2105e-05 2024-09-13T14:50:25.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.253862+0000", 2024-09-13T14:50:25.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2829999999999998e-06 2024-09-13T14:50:25.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.253866+0000", 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0380000000000001e-06 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.262867+0000", 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0090015489999999993 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.262920+0000", 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3072999999999998e-05 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.284861+0000", 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021940103999999998 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.285040+0000", 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017976900000000001 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:181963 10.b e257/229 10:dca39fbe:::benchmark_data_smithi003_197468_object65234:head v 257'11437, mlcod=257'11433)", 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:50:25.257700+0000", 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.28840243799999998, 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027549725000000001, 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.257700+0000", 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.257700+0000", 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.257701+0000", 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1030000000000001e-06 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.257710+0000", 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4690000000000003e-06 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.257738+0000", 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7906999999999999e-05 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.257747+0000", 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.772e-06 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.269087+0000", 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011339831 2024-09-13T14:50:25.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.269139+0000", 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2389000000000003e-05 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.285044+0000", 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015904667000000001 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:25.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:50:25.285249+0000", 2024-09-13T14:50:25.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000205587 2024-09-13T14:50:25.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.651 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:25.651 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:25.651 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:50:25.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:25 smithi019 ceph-mon[91296]: pgmap v870: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 10340/170634 objects degraded (6.060%); 2050/170634 objects misplaced (1.201%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:50:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:25.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:25.791 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:25.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:25.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:26.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:26.125 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:26.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:26.273 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:26.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:26.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:26.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:26.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:26.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:26.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10340/170634 objects degraded (6.060%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:26.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10340/170634 objects degraded (6.060%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:26.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10340/170634 objects degraded (6.060%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:26.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:26.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:26.814 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:26.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:26.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:50:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:26.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:27.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:27.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:27.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:27.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:27.377 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:27.417 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:27.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:27.520 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:27 smithi003 ceph-mon[107278]: pgmap v871: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 867 op/s; 10497/157221 objects degraded (6.677%); 1773/157221 objects misplaced (1.128%); 1.6 MiB/s, 24 objects/s recovering 2024-09-13T14:50:27.521 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:27 smithi003 ceph-mon[104007]: pgmap v871: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 867 op/s; 10497/157221 objects degraded (6.677%); 1773/157221 objects misplaced (1.128%); 1.6 MiB/s, 24 objects/s recovering 2024-09-13T14:50:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:27.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:27.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:27.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:27 smithi019 ceph-mon[91296]: pgmap v871: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 867 op/s; 10497/157221 objects degraded (6.677%); 1773/157221 objects misplaced (1.128%); 1.6 MiB/s, 24 objects/s recovering 2024-09-13T14:50:27.708 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:50:27.709 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:27.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:27.752 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:27.752 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:27.752 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:50:27.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:27.754 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:27.799 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:50:27.799 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:27.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:27.835 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:50:27.835 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:27.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:27.869 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:50:27.869 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:27.897 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:50:27.898 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:27.932 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:50:27.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:27.964 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:50:27.964 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:27.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:28.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:28.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:28.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:28.274 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:28.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:28.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:28.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/584124687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:28.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:28.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:28.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/584124687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:28.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:28.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:28.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:28.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:28.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:28.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/584124687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:28.802 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:28.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:28.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:28.973 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:29.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:29.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:29.211 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:29.248 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:29.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:29.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:29.483 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:29.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:29.541 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:29 smithi003 ceph-mon[107278]: pgmap v872: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 10497/157221 objects degraded (6.677%); 1773/157221 objects misplaced (1.128%); 1.6 MiB/s, 24 objects/s recovering 2024-09-13T14:50:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:29 smithi003 ceph-mon[104007]: pgmap v872: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 10497/157221 objects degraded (6.677%); 1773/157221 objects misplaced (1.128%); 1.6 MiB/s, 24 objects/s recovering 2024-09-13T14:50:29.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.670 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:29.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:29.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:29 smithi019 ceph-mon[91296]: pgmap v872: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 10497/157221 objects degraded (6.677%); 1773/157221 objects misplaced (1.128%); 1.6 MiB/s, 24 objects/s recovering 2024-09-13T14:50:29.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:29.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:29.912 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:29.950 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:29.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:30.052 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.091 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:30.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:30.192 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:30.236 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:30.338 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.379 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:30.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:30.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:30.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:30.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:30.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:50:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:50:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:30.783 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:30.812 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:30.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:50:30.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:30.938 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:30.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:31.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.073 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:31.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:31.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:31.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:31.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:31.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:31.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:31.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[107278]: pgmap v873: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 10617/146082 objects degraded (7.268%); 1518/146082 objects misplaced (1.039%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:50:31.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:31.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:31.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10617/146082 objects degraded (7.268%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:31.431 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:50:31.431 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:50:31.431 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:31.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[104007]: pgmap v873: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 10617/146082 objects degraded (7.268%); 1518/146082 objects misplaced (1.039%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:50:31.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:31.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:31.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10617/146082 objects degraded (7.268%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:31.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:50:31.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:50:31.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:31.476 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:31.507 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:31.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:31.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:31.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:31 smithi019 ceph-mon[91296]: pgmap v873: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 10617/146082 objects degraded (7.268%); 1518/146082 objects misplaced (1.039%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:50:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10617/146082 objects degraded (7.268%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:50:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:50:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:50:31.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:31.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:31.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:31.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:32.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:32.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:32.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:32.201 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:32.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:32.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:32.439 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:32.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:32.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:32.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:32.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:32.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:32.749 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:32.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:32.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:32.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:32.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.042 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:33.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:33.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:33.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:33.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.315 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:33.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:33.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:33.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:33.570 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:33 smithi003 ceph-mon[107278]: pgmap v874: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 10617/144447 objects degraded (7.350%); 1518/144447 objects misplaced (1.051%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:50:33.570 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:33 smithi003 ceph-mon[104007]: pgmap v874: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 10617/144447 objects degraded (7.350%); 1518/144447 objects misplaced (1.051%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:50:33.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:33.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:33.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:33 smithi019 ceph-mon[91296]: pgmap v874: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 10617/144447 objects degraded (7.350%); 1518/144447 objects misplaced (1.051%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:50:33.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:33.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:33.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:33.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:50:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:50:34.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:34.036 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:34.138 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:34.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:34.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:50:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:50:34] "GET /metrics HTTP/1.1" 200 36555 "" "Prometheus/2.51.0" 2024-09-13T14:50:34.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:34.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:34.422 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:34.464 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:34.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:34.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:34.713 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:34.747 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:34.760 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":875,"stamp":"2024-09-13T14:50:33.803723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3154346168,"num_objects":48149,"num_object_clones":0,"num_object_copies":144447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10617,"num_objects_misplaced":1518,"num_objects_unfound":0,"num_objects_dirty":48149,"num_whiteouts":0,"num_read":15,"num_read_kb":35,"num_write":185402,"num_write_kb":7470571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3569,"num_bytes_recovered":233598513,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114976,"ondisk_log_size":114976,"up":327,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":333,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16138448,"kb_used_data":10027600,"kb_used_omap":29,"kb_used_meta":6110754,"kb_avail":733822768,"statfs":{"total":767960285184,"available":751434514432,"internally_reserved":0,"allocated":10268262400,"data_stored":10257493080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6257412622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,2,6,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-572063744,"num_objects":-8729,"num_object_clones":0,"num_object_copies":-26187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":277,"num_objects_misplaced":-532,"num_objects_unfound":0,"num_objects_dirty":-8729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8729,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":16711680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008910"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":393,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.231404+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:50:21.231404+0000","last_peered":"2024-09-13T14:50:21.231404+0000","last_clean":"2024-09-13T14:50:21.231404+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:50:21.231404+0000","last_undegraded":"2024-09-13T14:50:21.231404+0000","last_fullsized":"2024-09-13T14:50:21.231404+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":229,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.154541+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:50:05.154541+0000","last_peered":"2024-09-13T14:50:05.154541+0000","last_clean":"2024-09-13T14:50:05.154541+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:50:05.154541+0000","last_undegraded":"2024-09-13T14:50:05.154541+0000","last_fullsized":"2024-09-13T14:50:05.154541+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":20,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163113+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:50:05.163113+0000","last_peered":"2024-09-13T14:50:05.163113+0000","last_clean":"2024-09-13T14:50:05.163113+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:50:05.163113+0000","last_undegraded":"2024-09-13T14:50:05.163113+0000","last_fullsized":"2024-09-13T14:50:05.163113+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":351,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.232306+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:50:21.232306+0000","last_peered":"2024-09-13T14:50:21.232306+0000","last_clean":"2024-09-13T14:50:21.232306+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:50:21.232306+0000","last_undegraded":"2024-09-13T14:50:21.232306+0000","last_fullsized":"2024-09-13T14:50:21.232306+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":203,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163797+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:50:05.163797+0000","last_peered":"2024-09-13T14:50:05.163797+0000","last_clean":"2024-09-13T14:50:05.163797+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:50:05.163797+0000","last_undegraded":"2024-09-13T14:50:05.163797+0000","last_fullsized":"2024-09-13T14:50:05.163797+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":236,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.159754+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:50:05.159754+0000","last_peered":"2024-09-13T14:50:05.159754+0000","last_clean":"2024-09-13T14:50:05.159754+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:50:05.159754+0000","last_undegraded":"2024-09-13T14:50:05.159754+0000","last_fullsized":"2024-09-13T14:50:05.159754+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":220,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163531+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:50:05.163531+0000","last_peered":"2024-09-13T14:50:05.163531+0000","last_clean":"2024-09-13T14:50:05.163531+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:50:05.163531+0000","last_undegraded":"2024-09-13T14:50:05.163531+0000","last_fullsized":"2024-09-13T14:50:05.163531+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":266,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.155315+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:50:05.155315+0000","last_peered":"2024-09-13T14:50:05.155315+0000","last_clean":"2024-09-13T14:50:05.155315+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:50:05.155315+0000","last_undegraded":"2024-09-13T14:50:05.155315+0000","last_fullsized":"2024-09-13T14:50:05.155315+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":380,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.234224+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:50:21.234224+0000","last_peered":"2024-09-13T14:50:21.234224+0000","last_clean":"2024-09-13T14:50:21.234224+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:50:21.234224+0000","last_undegraded":"2024-09-13T14:50:21.234224+0000","last_fullsized":"2024-09-13T14:50:21.234224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":376,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.227830+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:50:21.227830+0000","last_peered":"2024-09-13T14:50:21.227830+0000","last_clean":"2024-09-13T14:50:21.227830+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:50:21.227830+0000","last_undegraded":"2024-09-13T14:50:21.227830+0000","last_fullsized":"2024-09-13T14:50:21.227830+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":238,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.159881+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:50:05.159881+0000","last_peered":"2024-09-13T14:50:05.159881+0000","last_clean":"2024-09-13T14:50:05.159881+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:50:05.159881+0000","last_undegraded":"2024-09-13T14:50:05.159881+0000","last_fullsized":"2024-09-13T14:50:05.159881+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":318,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.233260+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:50:21.233260+0000","last_peered":"2024-09-13T14:50:21.233260+0000","last_clean":"2024-09-13T14:50:21.233260+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:50:21.233260+0000","last_undegraded":"2024-09-13T14:50:21.233260+0000","last_fullsized":"2024-09-13T14:50:21.233260+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":32,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.238120+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:50:21.238120+0000","last_peered":"2024-09-13T14:50:21.238120+0000","last_clean":"2024-09-13T14:50:21.238120+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:50:21.238120+0000","last_undegraded":"2024-09-13T14:50:21.238120+0000","last_fullsized":"2024-09-13T14:50:21.238120+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":411,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.237503+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:50:21.237503+0000","last_peered":"2024-09-13T14:50:21.237503+0000","last_clean":"2024-09-13T14:50:21.237503+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:50:21.237503+0000","last_undegraded":"2024-09-13T14:50:21.237503+0000","last_fullsized":"2024-09-13T14:50:21.237503+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":350,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.231602+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:50:21.231602+0000","last_peered":"2024-09-13T14:50:21.231602+0000","last_clean":"2024-09-13T14:50:21.231602+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:50:21.231602+0000","last_undegraded":"2024-09-13T14:50:21.231602+0000","last_fullsized":"2024-09-13T14:50:21.231602+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":219,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.156063+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:50:05.156063+0000","last_peered":"2024-09-13T14:50:05.156063+0000","last_clean":"2024-09-13T14:50:05.156063+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:50:05.156063+0000","last_undegraded":"2024-09-13T14:50:05.156063+0000","last_fullsized":"2024-09-13T14:50:05.156063+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":318,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.231436+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:50:21.231436+0000","last_peered":"2024-09-13T14:50:21.231436+0000","last_clean":"2024-09-13T14:50:21.231436+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:50:21.231436+0000","last_undegraded":"2024-09-13T14:50:21.231436+0000","last_fullsized":"2024-09-13T14:50:21.231436+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":394,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.231714+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:50:21.231714+0000","last_peered":"2024-09-13T14:50:21.231714+0000","last_clean":"2024-09-13T14:50:21.231714+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:50:21.231714+0000","last_undegraded":"2024-09-13T14:50:21.231714+0000","last_fullsized":"2024-09-13T14:50:21.231714+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":323,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.231979+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:50:21.231979+0000","last_peered":"2024-09-13T14:50:21.231979+0000","last_clean":"2024-09-13T14:50:21.231979+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:50:21.231979+0000","last_undegraded":"2024-09-13T14:50:21.231979+0000","last_fullsized":"2024-09-13T14:50:21.231979+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163005+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:50:05.163005+0000","last_peered":"2024-09-13T14:50:05.163005+0000","last_clean":"2024-09-13T14:50:05.163005+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:50:05.163005+0000","last_undegraded":"2024-09-13T14:50:05.163005+0000","last_fullsized":"2024-09-13T14:50:05.163005+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":73,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.227606+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:50:21.227606+0000","last_peered":"2024-09-13T14:50:21.227606+0000","last_clean":"2024-09-13T14:50:21.227606+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:50:21.227606+0000","last_undegraded":"2024-09-13T14:50:21.227606+0000","last_fullsized":"2024-09-13T14:50:21.227606+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":20,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.160747+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:50:05.160747+0000","last_peered":"2024-09-13T14:50:05.160747+0000","last_clean":"2024-09-13T14:50:05.160747+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:50:05.160747+0000","last_undegraded":"2024-09-13T14:50:05.160747+0000","last_fullsized":"2024-09-13T14:50:05.160747+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":158,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.150143+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:50:05.150143+0000","last_peered":"2024-09-13T14:50:05.150143+0000","last_clean":"2024-09-13T14:50:05.150143+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:50:05.150143+0000","last_undegraded":"2024-09-13T14:50:05.150143+0000","last_fullsized":"2024-09-13T14:50:05.150143+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":226,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.164538+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:50:05.164538+0000","last_peered":"2024-09-13T14:50:05.164538+0000","last_clean":"2024-09-13T14:50:05.164538+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:50:05.164538+0000","last_undegraded":"2024-09-13T14:50:05.164538+0000","last_fullsized":"2024-09-13T14:50:05.164538+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":219,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.160785+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:50:05.160785+0000","last_peered":"2024-09-13T14:50:05.160785+0000","last_clean":"2024-09-13T14:50:05.160785+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:50:05.160785+0000","last_undegraded":"2024-09-13T14:50:05.160785+0000","last_fullsized":"2024-09-13T14:50:05.160785+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"257'11431","reported_seq":11467,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:28.901254+0000","last_change":"2024-09-13T14:47:20.107455+0000","last_active":"2024-09-13T14:50:28.901254+0000","last_peered":"2024-09-13T14:50:28.901254+0000","last_clean":"2024-09-13T14:50:28.901254+0000","last_became_active":"2024-09-13T14:47:20.107142+0000","last_became_peered":"2024-09-13T14:47:20.107142+0000","last_unstale":"2024-09-13T14:50:28.901254+0000","last_undegraded":"2024-09-13T14:50:28.901254+0000","last_fullsized":"2024-09-13T14:50:28.901254+0000","mapping_epoch":229,"log_start":"243'4700","ondisk_log_start":"243'4700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6731,"log_dups_size":0,"ondisk_log_size":6731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:19.189519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199032832,"num_objects":3037,"num_object_clones":0,"num_object_copies":9111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":462976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":202,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163358+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:50:05.163358+0000","last_peered":"2024-09-13T14:50:05.163358+0000","last_clean":"2024-09-13T14:50:05.163358+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:50:05.163358+0000","last_undegraded":"2024-09-13T14:50:05.163358+0000","last_fullsized":"2024-09-13T14:50:05.163358+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":252,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.156199+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:50:05.156199+0000","last_peered":"2024-09-13T14:50:05.156199+0000","last_clean":"2024-09-13T14:50:05.156199+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:50:05.156199+0000","last_undegraded":"2024-09-13T14:50:05.156199+0000","last_fullsized":"2024-09-13T14:50:05.156199+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":392,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.237649+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:50:21.237649+0000","last_peered":"2024-09-13T14:50:21.237649+0000","last_clean":"2024-09-13T14:50:21.237649+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:50:21.237649+0000","last_undegraded":"2024-09-13T14:50:21.237649+0000","last_fullsized":"2024-09-13T14:50:21.237649+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":80,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.232960+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:50:21.232960+0000","last_peered":"2024-09-13T14:50:21.232960+0000","last_clean":"2024-09-13T14:50:21.232960+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:50:21.232960+0000","last_undegraded":"2024-09-13T14:50:21.232960+0000","last_fullsized":"2024-09-13T14:50:21.232960+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"257'11512","reported_seq":11548,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:29.123505+0000","last_change":"2024-09-13T14:47:20.054114+0000","last_active":"2024-09-13T14:50:29.123505+0000","last_peered":"2024-09-13T14:50:29.123505+0000","last_clean":"2024-09-13T14:50:29.123505+0000","last_became_active":"2024-09-13T14:47:20.053767+0000","last_became_peered":"2024-09-13T14:47:20.053767+0000","last_unstale":"2024-09-13T14:50:29.123505+0000","last_undegraded":"2024-09-13T14:50:29.123505+0000","last_fullsized":"2024-09-13T14:50:29.123505+0000","mapping_epoch":229,"log_start":"245'5200","ondisk_log_start":"245'5200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:48.035191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202506240,"num_objects":3090,"num_object_clones":0,"num_object_copies":9270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11512,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":361,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.231524+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:50:21.231524+0000","last_peered":"2024-09-13T14:50:21.231524+0000","last_clean":"2024-09-13T14:50:21.231524+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:50:21.231524+0000","last_undegraded":"2024-09-13T14:50:21.231524+0000","last_fullsized":"2024-09-13T14:50:21.231524+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":93,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.150980+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:50:05.150980+0000","last_peered":"2024-09-13T14:50:05.150980+0000","last_clean":"2024-09-13T14:50:05.150980+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:50:05.150980+0000","last_undegraded":"2024-09-13T14:50:05.150980+0000","last_fullsized":"2024-09-13T14:50:05.150980+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":350,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.235932+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:50:21.235932+0000","last_peered":"2024-09-13T14:50:21.235932+0000","last_clean":"2024-09-13T14:50:21.235932+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:50:21.235932+0000","last_undegraded":"2024-09-13T14:50:21.235932+0000","last_fullsized":"2024-09-13T14:50:21.235932+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":40,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.161583+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:50:05.161583+0000","last_peered":"2024-09-13T14:50:05.161583+0000","last_clean":"2024-09-13T14:50:05.161583+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:50:05.161583+0000","last_undegraded":"2024-09-13T14:50:05.161583+0000","last_fullsized":"2024-09-13T14:50:05.161583+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"257'11647","reported_seq":11684,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:28.864378+0000","last_change":"2024-09-13T14:47:20.052876+0000","last_active":"2024-09-13T14:50:28.864378+0000","last_peered":"2024-09-13T14:50:28.864378+0000","last_clean":"2024-09-13T14:50:28.864378+0000","last_became_active":"2024-09-13T14:47:20.052566+0000","last_became_peered":"2024-09-13T14:47:20.052566+0000","last_unstale":"2024-09-13T14:50:28.864378+0000","last_undegraded":"2024-09-13T14:50:28.864378+0000","last_fullsized":"2024-09-13T14:50:28.864378+0000","mapping_epoch":229,"log_start":"244'4900","ondisk_log_start":"244'4900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6747,"log_dups_size":0,"ondisk_log_size":6747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:56.317980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195297304,"num_objects":2981,"num_object_clones":0,"num_object_copies":8943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11647,"num_write_kb":468033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":379,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.234136+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:50:21.234136+0000","last_peered":"2024-09-13T14:50:21.234136+0000","last_clean":"2024-09-13T14:50:21.234136+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:50:21.234136+0000","last_undegraded":"2024-09-13T14:50:21.234136+0000","last_fullsized":"2024-09-13T14:50:21.234136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":232,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.161061+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:50:05.161061+0000","last_peered":"2024-09-13T14:50:05.161061+0000","last_clean":"2024-09-13T14:50:05.161061+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:50:05.161061+0000","last_undegraded":"2024-09-13T14:50:05.161061+0000","last_fullsized":"2024-09-13T14:50:05.161061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":334,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.227708+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:50:21.227708+0000","last_peered":"2024-09-13T14:50:21.227708+0000","last_clean":"2024-09-13T14:50:21.227708+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:50:21.227708+0000","last_undegraded":"2024-09-13T14:50:21.227708+0000","last_fullsized":"2024-09-13T14:50:21.227708+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":193,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.150730+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:50:05.150730+0000","last_peered":"2024-09-13T14:50:05.150730+0000","last_clean":"2024-09-13T14:50:05.150730+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:50:05.150730+0000","last_undegraded":"2024-09-13T14:50:05.150730+0000","last_fullsized":"2024-09-13T14:50:05.150730+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"257'11671","reported_seq":11707,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:29.042715+0000","last_change":"2024-09-13T14:47:20.052419+0000","last_active":"2024-09-13T14:50:29.042715+0000","last_peered":"2024-09-13T14:50:29.042715+0000","last_clean":"2024-09-13T14:50:29.042715+0000","last_became_active":"2024-09-13T14:47:20.051845+0000","last_became_peered":"2024-09-13T14:47:20.051845+0000","last_unstale":"2024-09-13T14:50:29.042715+0000","last_undegraded":"2024-09-13T14:50:29.042715+0000","last_fullsized":"2024-09-13T14:50:29.042715+0000","mapping_epoch":229,"log_start":"243'4814","ondisk_log_start":"243'4814","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.420332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198508544,"num_objects":3029,"num_object_clones":0,"num_object_copies":9087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11671,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163054+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:50:05.163054+0000","last_peered":"2024-09-13T14:50:05.163054+0000","last_clean":"2024-09-13T14:50:05.163054+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:50:05.163054+0000","last_undegraded":"2024-09-13T14:50:05.163054+0000","last_fullsized":"2024-09-13T14:50:05.163054+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":192,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.234027+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:50:21.234027+0000","last_peered":"2024-09-13T14:50:21.234027+0000","last_clean":"2024-09-13T14:50:21.234027+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:50:21.234027+0000","last_undegraded":"2024-09-13T14:50:21.234027+0000","last_fullsized":"2024-09-13T14:50:21.234027+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":235,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163861+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:50:05.163861+0000","last_peered":"2024-09-13T14:50:05.163861+0000","last_clean":"2024-09-13T14:50:05.163861+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:50:05.163861+0000","last_undegraded":"2024-09-13T14:50:05.163861+0000","last_fullsized":"2024-09-13T14:50:05.163861+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":213,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.155926+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:50:05.155926+0000","last_peered":"2024-09-13T14:50:05.155926+0000","last_clean":"2024-09-13T14:50:05.155926+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:50:05.155926+0000","last_undegraded":"2024-09-13T14:50:05.155926+0000","last_fullsized":"2024-09-13T14:50:05.155926+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"257'11419","reported_seq":11455,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:28.901256+0000","last_change":"2024-09-13T14:47:20.051919+0000","last_active":"2024-09-13T14:50:28.901256+0000","last_peered":"2024-09-13T14:50:28.901256+0000","last_clean":"2024-09-13T14:50:28.901256+0000","last_became_active":"2024-09-13T14:47:20.051626+0000","last_became_peered":"2024-09-13T14:47:20.051626+0000","last_unstale":"2024-09-13T14:50:28.901256+0000","last_undegraded":"2024-09-13T14:50:28.901256+0000","last_fullsized":"2024-09-13T14:50:28.901256+0000","mapping_epoch":229,"log_start":"243'4700","ondisk_log_start":"243'4700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:12.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196935680,"num_objects":3005,"num_object_clones":0,"num_object_copies":9015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11419,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":246,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.227822+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:50:21.227822+0000","last_peered":"2024-09-13T14:50:21.227822+0000","last_clean":"2024-09-13T14:50:21.227822+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:50:21.227822+0000","last_undegraded":"2024-09-13T14:50:21.227822+0000","last_fullsized":"2024-09-13T14:50:21.227822+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":220,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.160853+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:50:05.160853+0000","last_peered":"2024-09-13T14:50:05.160853+0000","last_clean":"2024-09-13T14:50:05.160853+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:50:05.160853+0000","last_undegraded":"2024-09-13T14:50:05.160853+0000","last_fullsized":"2024-09-13T14:50:05.160853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":231,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163347+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:50:05.163347+0000","last_peered":"2024-09-13T14:50:05.163347+0000","last_clean":"2024-09-13T14:50:05.163347+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:50:05.163347+0000","last_undegraded":"2024-09-13T14:50:05.163347+0000","last_fullsized":"2024-09-13T14:50:05.163347+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":40,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.171702+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:50:05.171702+0000","last_peered":"2024-09-13T14:50:05.171702+0000","last_clean":"2024-09-13T14:50:05.171702+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:50:05.171702+0000","last_undegraded":"2024-09-13T14:50:05.171702+0000","last_fullsized":"2024-09-13T14:50:05.171702+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"257'11533","reported_seq":15199,"reported_epoch":257,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:29.216042+0000","last_change":"2024-09-13T14:49:05.872778+0000","last_active":"2024-09-13T14:50:29.216042+0000","last_peered":"2024-09-13T14:50:29.216042+0000","last_clean":"2024-09-13T14:48:37.826901+0000","last_became_active":"2024-09-13T14:48:40.041761+0000","last_became_peered":"2024-09-13T14:48:40.041761+0000","last_unstale":"2024-09-13T14:50:29.216042+0000","last_undegraded":"2024-09-13T14:48:39.850843+0000","last_fullsized":"2024-09-13T14:48:39.825201+0000","mapping_epoch":249,"log_start":"239'3400","ondisk_log_start":"239'3400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8133,"log_dups_size":0,"ondisk_log_size":8133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200212480,"num_objects":3055,"num_object_clones":0,"num_object_copies":9165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2441,"num_objects_misplaced":1518,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11533,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":117964800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":1518},{"shards":"2,7","objects":-904}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":151,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.173847+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:50:05.173847+0000","last_peered":"2024-09-13T14:50:05.173847+0000","last_clean":"2024-09-13T14:50:05.173847+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:50:05.173847+0000","last_undegraded":"2024-09-13T14:50:05.173847+0000","last_fullsized":"2024-09-13T14:50:05.173847+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":186,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.226468+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:50:21.226468+0000","last_peered":"2024-09-13T14:50:21.226468+0000","last_clean":"2024-09-13T14:50:21.226468+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:50:21.226468+0000","last_undegraded":"2024-09-13T14:50:21.226468+0000","last_fullsized":"2024-09-13T14:50:21.226468+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":377,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.247848+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:50:21.247848+0000","last_peered":"2024-09-13T14:50:21.247848+0000","last_clean":"2024-09-13T14:50:21.247848+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:50:21.247848+0000","last_undegraded":"2024-09-13T14:50:21.247848+0000","last_fullsized":"2024-09-13T14:50:21.247848+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.160175+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:50:05.160175+0000","last_peered":"2024-09-13T14:50:05.160175+0000","last_clean":"2024-09-13T14:50:05.160175+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:50:05.160175+0000","last_undegraded":"2024-09-13T14:50:05.160175+0000","last_fullsized":"2024-09-13T14:50:05.160175+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"257'11797","reported_seq":11864,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:30.357510+0000","last_change":"2024-09-13T14:47:20.049747+0000","last_active":"2024-09-13T14:50:30.357510+0000","last_peered":"2024-09-13T14:50:30.357510+0000","last_clean":"2024-09-13T14:50:30.357510+0000","last_became_active":"2024-09-13T14:47:20.049448+0000","last_became_peered":"2024-09-13T14:47:20.049448+0000","last_unstale":"2024-09-13T14:50:30.357510+0000","last_undegraded":"2024-09-13T14:50:30.357510+0000","last_fullsized":"2024-09-13T14:50:30.357510+0000","mapping_epoch":229,"log_start":"243'4800","ondisk_log_start":"243'4800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:11.585800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191823872,"num_objects":2927,"num_object_clones":0,"num_object_copies":8781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11797,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":400,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.237469+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:50:21.237469+0000","last_peered":"2024-09-13T14:50:21.237469+0000","last_clean":"2024-09-13T14:50:21.237469+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:50:21.237469+0000","last_undegraded":"2024-09-13T14:50:21.237469+0000","last_fullsized":"2024-09-13T14:50:21.237469+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":410,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.236821+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:50:21.236821+0000","last_peered":"2024-09-13T14:50:21.236821+0000","last_clean":"2024-09-13T14:50:21.236821+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:50:21.236821+0000","last_undegraded":"2024-09-13T14:50:21.236821+0000","last_fullsized":"2024-09-13T14:50:21.236821+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":269,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.160354+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:50:05.160354+0000","last_peered":"2024-09-13T14:50:05.160354+0000","last_clean":"2024-09-13T14:50:05.160354+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:50:05.160354+0000","last_undegraded":"2024-09-13T14:50:05.160354+0000","last_fullsized":"2024-09-13T14:50:05.160354+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":193,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.150749+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:50:05.150749+0000","last_peered":"2024-09-13T14:50:05.150749+0000","last_clean":"2024-09-13T14:50:05.150749+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:50:05.150749+0000","last_undegraded":"2024-09-13T14:50:05.150749+0000","last_fullsized":"2024-09-13T14:50:05.150749+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"257'11622","reported_seq":11658,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:28.917120+0000","last_change":"2024-09-13T14:47:20.106371+0000","last_active":"2024-09-13T14:50:28.917120+0000","last_peered":"2024-09-13T14:50:28.917120+0000","last_clean":"2024-09-13T14:50:28.917120+0000","last_became_active":"2024-09-13T14:47:20.105934+0000","last_became_peered":"2024-09-13T14:47:20.105934+0000","last_unstale":"2024-09-13T14:50:28.917120+0000","last_undegraded":"2024-09-13T14:50:28.917120+0000","last_fullsized":"2024-09-13T14:50:28.917120+0000","mapping_epoch":229,"log_start":"244'4900","ondisk_log_start":"244'4900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6722,"log_dups_size":0,"ondisk_log_size":6722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:54.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198180864,"num_objects":3024,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11622,"num_write_kb":468672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":202,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.150638+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:50:05.150638+0000","last_peered":"2024-09-13T14:50:05.150638+0000","last_clean":"2024-09-13T14:50:05.150638+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:50:05.150638+0000","last_undegraded":"2024-09-13T14:50:05.150638+0000","last_fullsized":"2024-09-13T14:50:05.150638+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":309,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.231344+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:50:21.231344+0000","last_peered":"2024-09-13T14:50:21.231344+0000","last_clean":"2024-09-13T14:50:21.231344+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:50:21.231344+0000","last_undegraded":"2024-09-13T14:50:21.231344+0000","last_fullsized":"2024-09-13T14:50:21.231344+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":230,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.165045+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:50:05.165045+0000","last_peered":"2024-09-13T14:50:05.165045+0000","last_clean":"2024-09-13T14:50:05.165045+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:50:05.165045+0000","last_undegraded":"2024-09-13T14:50:05.165045+0000","last_fullsized":"2024-09-13T14:50:05.165045+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.159562+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:50:05.159562+0000","last_peered":"2024-09-13T14:50:05.159562+0000","last_clean":"2024-09-13T14:50:05.159562+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:50:05.159562+0000","last_undegraded":"2024-09-13T14:50:05.159562+0000","last_fullsized":"2024-09-13T14:50:05.159562+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"257'11479","reported_seq":11515,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:29.134079+0000","last_change":"2024-09-13T14:47:20.050773+0000","last_active":"2024-09-13T14:50:29.134079+0000","last_peered":"2024-09-13T14:50:29.134079+0000","last_clean":"2024-09-13T14:50:29.134079+0000","last_became_active":"2024-09-13T14:47:20.050243+0000","last_became_peered":"2024-09-13T14:47:20.050243+0000","last_unstale":"2024-09-13T14:50:29.134079+0000","last_undegraded":"2024-09-13T14:50:29.134079+0000","last_fullsized":"2024-09-13T14:50:29.134079+0000","mapping_epoch":229,"log_start":"246'5200","ondisk_log_start":"246'5200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:42.724078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194183168,"num_objects":2963,"num_object_clones":0,"num_object_copies":8889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.234604+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:50:21.234604+0000","last_peered":"2024-09-13T14:50:21.234604+0000","last_clean":"2024-09-13T14:50:21.234604+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:50:21.234604+0000","last_undegraded":"2024-09-13T14:50:21.234604+0000","last_fullsized":"2024-09-13T14:50:21.234604+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":320,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.231695+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:50:21.231695+0000","last_peered":"2024-09-13T14:50:21.231695+0000","last_clean":"2024-09-13T14:50:21.231695+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:50:21.231695+0000","last_undegraded":"2024-09-13T14:50:21.231695+0000","last_fullsized":"2024-09-13T14:50:21.231695+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":40,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.162987+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:50:05.162987+0000","last_peered":"2024-09-13T14:50:05.162987+0000","last_clean":"2024-09-13T14:50:05.162987+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:50:05.162987+0000","last_undegraded":"2024-09-13T14:50:05.162987+0000","last_fullsized":"2024-09-13T14:50:05.162987+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"257'11430","reported_seq":11465,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:28.931155+0000","last_change":"2024-09-13T14:47:20.106351+0000","last_active":"2024-09-13T14:50:28.931155+0000","last_peered":"2024-09-13T14:50:28.931155+0000","last_clean":"2024-09-13T14:50:28.931155+0000","last_became_active":"2024-09-13T14:47:20.105946+0000","last_became_peered":"2024-09-13T14:47:20.105946+0000","last_unstale":"2024-09-13T14:50:28.931155+0000","last_undegraded":"2024-09-13T14:50:28.931155+0000","last_fullsized":"2024-09-13T14:50:28.931155+0000","mapping_epoch":229,"log_start":"243'4700","ondisk_log_start":"243'4700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6730,"log_dups_size":0,"ondisk_log_size":6730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:07.223637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197656576,"num_objects":3016,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11430,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":239,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163798+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:50:05.163798+0000","last_peered":"2024-09-13T14:50:05.163798+0000","last_clean":"2024-09-13T14:50:05.163798+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:50:05.163798+0000","last_undegraded":"2024-09-13T14:50:05.163798+0000","last_fullsized":"2024-09-13T14:50:05.163798+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":220,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163678+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:50:05.163678+0000","last_peered":"2024-09-13T14:50:05.163678+0000","last_clean":"2024-09-13T14:50:05.163678+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:50:05.163678+0000","last_undegraded":"2024-09-13T14:50:05.163678+0000","last_fullsized":"2024-09-13T14:50:05.163678+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":370,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.233939+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:50:21.233939+0000","last_peered":"2024-09-13T14:50:21.233939+0000","last_clean":"2024-09-13T14:50:21.233939+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:50:21.233939+0000","last_undegraded":"2024-09-13T14:50:21.233939+0000","last_fullsized":"2024-09-13T14:50:21.233939+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"257'11690","reported_seq":15265,"reported_epoch":257,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:29.240889+0000","last_change":"2024-09-13T14:49:05.871518+0000","last_active":"2024-09-13T14:50:29.240889+0000","last_peered":"2024-09-13T14:50:29.240889+0000","last_clean":"2024-09-13T14:48:09.848772+0000","last_became_active":"2024-09-13T14:48:12.232150+0000","last_became_peered":"2024-09-13T14:48:12.232150+0000","last_unstale":"2024-09-13T14:50:29.240889+0000","last_undegraded":"2024-09-13T14:49:05.858877+0000","last_fullsized":"2024-09-13T14:48:11.643512+0000","mapping_epoch":242,"log_start":"239'3500","ondisk_log_start":"239'3500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8190,"log_dups_size":0,"ondisk_log_size":8190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191758336,"num_objects":2926,"num_object_clones":0,"num_object_copies":8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11690,"num_write_kb":467712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":114753536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,5","objects":499}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":236,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.161774+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:50:05.161774+0000","last_peered":"2024-09-13T14:50:05.161774+0000","last_clean":"2024-09-13T14:50:05.161774+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:50:05.161774+0000","last_undegraded":"2024-09-13T14:50:05.161774+0000","last_fullsized":"2024-09-13T14:50:05.161774+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":240,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.150208+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:50:05.150208+0000","last_peered":"2024-09-13T14:50:05.150208+0000","last_clean":"2024-09-13T14:50:05.150208+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:50:05.150208+0000","last_undegraded":"2024-09-13T14:50:05.150208+0000","last_fullsized":"2024-09-13T14:50:05.150208+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":22,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.162707+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:50:05.162707+0000","last_peered":"2024-09-13T14:50:05.162707+0000","last_clean":"2024-09-13T14:50:05.162707+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:50:05.162707+0000","last_undegraded":"2024-09-13T14:50:05.162707+0000","last_fullsized":"2024-09-13T14:50:05.162707+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"257'11705","reported_seq":11741,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:29.113691+0000","last_change":"2024-09-13T14:47:20.106142+0000","last_active":"2024-09-13T14:50:29.113691+0000","last_peered":"2024-09-13T14:50:29.113691+0000","last_clean":"2024-09-13T14:50:29.113691+0000","last_became_active":"2024-09-13T14:47:20.105758+0000","last_became_peered":"2024-09-13T14:47:20.105758+0000","last_unstale":"2024-09-13T14:50:29.113691+0000","last_undegraded":"2024-09-13T14:50:29.113691+0000","last_fullsized":"2024-09-13T14:50:29.113691+0000","mapping_epoch":229,"log_start":"247'5400","ondisk_log_start":"247'5400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6305,"log_dups_size":0,"ondisk_log_size":6305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:51.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199950336,"num_objects":3051,"num_object_clones":0,"num_object_copies":9153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11705,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":219,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163799+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:50:05.163799+0000","last_peered":"2024-09-13T14:50:05.163799+0000","last_clean":"2024-09-13T14:50:05.163799+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:50:05.163799+0000","last_undegraded":"2024-09-13T14:50:05.163799+0000","last_fullsized":"2024-09-13T14:50:05.163799+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":349,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.238804+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:50:21.238804+0000","last_peered":"2024-09-13T14:50:21.238804+0000","last_clean":"2024-09-13T14:50:21.238804+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:50:21.238804+0000","last_undegraded":"2024-09-13T14:50:21.238804+0000","last_fullsized":"2024-09-13T14:50:21.238804+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":40,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163219+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:50:05.163219+0000","last_peered":"2024-09-13T14:50:05.163219+0000","last_clean":"2024-09-13T14:50:05.163219+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:50:05.163219+0000","last_undegraded":"2024-09-13T14:50:05.163219+0000","last_fullsized":"2024-09-13T14:50:05.163219+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"257'11591","reported_seq":11659,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:29.204396+0000","last_change":"2024-09-13T14:47:20.107374+0000","last_active":"2024-09-13T14:50:29.204396+0000","last_peered":"2024-09-13T14:50:29.204396+0000","last_clean":"2024-09-13T14:50:29.204396+0000","last_became_active":"2024-09-13T14:47:20.107057+0000","last_became_peered":"2024-09-13T14:47:20.107057+0000","last_unstale":"2024-09-13T14:50:29.204396+0000","last_undegraded":"2024-09-13T14:50:29.204396+0000","last_fullsized":"2024-09-13T14:50:29.204396+0000","mapping_epoch":229,"log_start":"239'3400","ondisk_log_start":"239'3400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8191,"log_dups_size":0,"ondisk_log_size":8191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:05.814456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199032832,"num_objects":3037,"num_object_clones":0,"num_object_copies":9111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11591,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":389,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.234039+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:50:21.234039+0000","last_peered":"2024-09-13T14:50:21.234039+0000","last_clean":"2024-09-13T14:50:21.234039+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:50:21.234039+0000","last_undegraded":"2024-09-13T14:50:21.234039+0000","last_fullsized":"2024-09-13T14:50:21.234039+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":218,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163596+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:50:05.163596+0000","last_peered":"2024-09-13T14:50:05.163596+0000","last_clean":"2024-09-13T14:50:05.163596+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:50:05.163596+0000","last_undegraded":"2024-09-13T14:50:05.163596+0000","last_fullsized":"2024-09-13T14:50:05.163596+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":40,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.162873+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:50:05.162873+0000","last_peered":"2024-09-13T14:50:05.162873+0000","last_clean":"2024-09-13T14:50:05.162873+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:50:05.162873+0000","last_undegraded":"2024-09-13T14:50:05.162873+0000","last_fullsized":"2024-09-13T14:50:05.162873+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"257'11589","reported_seq":11657,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:30.402059+0000","last_change":"2024-09-13T14:47:20.108173+0000","last_active":"2024-09-13T14:50:30.402059+0000","last_peered":"2024-09-13T14:50:30.402059+0000","last_clean":"2024-09-13T14:50:30.402059+0000","last_became_active":"2024-09-13T14:47:20.107879+0000","last_became_peered":"2024-09-13T14:47:20.107879+0000","last_unstale":"2024-09-13T14:50:30.402059+0000","last_undegraded":"2024-09-13T14:50:30.402059+0000","last_fullsized":"2024-09-13T14:50:30.402059+0000","mapping_epoch":229,"log_start":"243'4600","ondisk_log_start":"243'4600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6989,"log_dups_size":0,"ondisk_log_size":6989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:50.152706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190906368,"num_objects":2913,"num_object_clones":0,"num_object_copies":8739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11589,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":232,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.150115+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:50:05.150115+0000","last_peered":"2024-09-13T14:50:05.150115+0000","last_clean":"2024-09-13T14:50:05.150115+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:50:05.150115+0000","last_undegraded":"2024-09-13T14:50:05.150115+0000","last_fullsized":"2024-09-13T14:50:05.150115+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":233,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.160990+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:50:05.160990+0000","last_peered":"2024-09-13T14:50:05.160990+0000","last_clean":"2024-09-13T14:50:05.160990+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:50:05.160990+0000","last_undegraded":"2024-09-13T14:50:05.160990+0000","last_fullsized":"2024-09-13T14:50:05.160990+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":193,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.155998+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:50:05.155998+0000","last_peered":"2024-09-13T14:50:05.155998+0000","last_clean":"2024-09-13T14:50:05.155998+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:50:05.155998+0000","last_undegraded":"2024-09-13T14:50:05.155998+0000","last_fullsized":"2024-09-13T14:50:05.155998+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"257'11423","reported_seq":11462,"reported_epoch":257,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:28.925608+0000","last_change":"2024-09-13T14:48:40.078958+0000","last_active":"2024-09-13T14:50:28.925608+0000","last_peered":"2024-09-13T14:50:28.925608+0000","last_clean":"2024-09-13T14:48:37.811111+0000","last_became_active":"2024-09-13T14:48:40.052181+0000","last_became_peered":"2024-09-13T14:48:40.052181+0000","last_unstale":"2024-09-13T14:50:28.925608+0000","last_undegraded":"2024-09-13T14:48:39.849823+0000","last_fullsized":"2024-09-13T14:48:39.826167+0000","mapping_epoch":249,"log_start":"243'4700","ondisk_log_start":"243'4700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6723,"log_dups_size":0,"ondisk_log_size":6723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:49.521146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200998912,"num_objects":3067,"num_object_clones":0,"num_object_copies":9201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11423,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":1571},{"shards":"3,6","objects":-2682}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":246,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.164209+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:50:05.164209+0000","last_peered":"2024-09-13T14:50:05.164209+0000","last_clean":"2024-09-13T14:50:05.164209+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:50:05.164209+0000","last_undegraded":"2024-09-13T14:50:05.164209+0000","last_fullsized":"2024-09-13T14:50:05.164209+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":223,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.161337+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:50:05.161337+0000","last_peered":"2024-09-13T14:50:05.161337+0000","last_clean":"2024-09-13T14:50:05.161337+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:50:05.161337+0000","last_undegraded":"2024-09-13T14:50:05.161337+0000","last_fullsized":"2024-09-13T14:50:05.161337+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":77,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.233050+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:50:21.233050+0000","last_peered":"2024-09-13T14:50:21.233050+0000","last_clean":"2024-09-13T14:50:21.233050+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:50:21.233050+0000","last_undegraded":"2024-09-13T14:50:21.233050+0000","last_fullsized":"2024-09-13T14:50:21.233050+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"257'11792","reported_seq":11827,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:29.719121+0000","last_change":"2024-09-13T14:47:20.051824+0000","last_active":"2024-09-13T14:50:29.719121+0000","last_peered":"2024-09-13T14:50:29.719121+0000","last_clean":"2024-09-13T14:50:29.719121+0000","last_became_active":"2024-09-13T14:47:20.051487+0000","last_became_peered":"2024-09-13T14:47:20.051487+0000","last_unstale":"2024-09-13T14:50:29.719121+0000","last_undegraded":"2024-09-13T14:50:29.719121+0000","last_fullsized":"2024-09-13T14:50:29.719121+0000","mapping_epoch":229,"log_start":"239'3600","ondisk_log_start":"239'3600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8192,"log_dups_size":0,"ondisk_log_size":8192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:11.533543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196739072,"num_objects":3002,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11792,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163956+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:50:05.163956+0000","last_peered":"2024-09-13T14:50:05.163956+0000","last_clean":"2024-09-13T14:50:05.163956+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:50:05.163956+0000","last_undegraded":"2024-09-13T14:50:05.163956+0000","last_fullsized":"2024-09-13T14:50:05.163956+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.161586+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:50:05.161586+0000","last_peered":"2024-09-13T14:50:05.161586+0000","last_clean":"2024-09-13T14:50:05.161586+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:50:05.161586+0000","last_undegraded":"2024-09-13T14:50:05.161586+0000","last_fullsized":"2024-09-13T14:50:05.161586+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.160820+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:50:05.160820+0000","last_peered":"2024-09-13T14:50:05.160820+0000","last_clean":"2024-09-13T14:50:05.160820+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:50:05.160820+0000","last_undegraded":"2024-09-13T14:50:05.160820+0000","last_fullsized":"2024-09-13T14:50:05.160820+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":268,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.149797+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:50:05.149797+0000","last_peered":"2024-09-13T14:50:05.149797+0000","last_clean":"2024-09-13T14:50:05.149797+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:50:05.149797+0000","last_undegraded":"2024-09-13T14:50:05.149797+0000","last_fullsized":"2024-09-13T14:50:05.149797+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":358,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.235511+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:50:21.235511+0000","last_peered":"2024-09-13T14:50:21.235511+0000","last_clean":"2024-09-13T14:50:21.235511+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:50:21.235511+0000","last_undegraded":"2024-09-13T14:50:21.235511+0000","last_fullsized":"2024-09-13T14:50:21.235511+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":50,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163126+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:50:05.163126+0000","last_peered":"2024-09-13T14:50:05.163126+0000","last_clean":"2024-09-13T14:50:05.163126+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:50:05.163126+0000","last_undegraded":"2024-09-13T14:50:05.163126+0000","last_fullsized":"2024-09-13T14:50:05.163126+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":214,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163307+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:50:05.163307+0000","last_peered":"2024-09-13T14:50:05.163307+0000","last_clean":"2024-09-13T14:50:05.163307+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:50:05.163307+0000","last_undegraded":"2024-09-13T14:50:05.163307+0000","last_fullsized":"2024-09-13T14:50:05.163307+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":402,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.237523+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:50:21.237523+0000","last_peered":"2024-09-13T14:50:21.237523+0000","last_clean":"2024-09-13T14:50:21.237523+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:50:21.237523+0000","last_undegraded":"2024-09-13T14:50:21.237523+0000","last_fullsized":"2024-09-13T14:50:21.237523+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":73,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.238625+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:50:21.238625+0000","last_peered":"2024-09-13T14:50:21.238625+0000","last_clean":"2024-09-13T14:50:21.238625+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:50:21.238625+0000","last_undegraded":"2024-09-13T14:50:21.238625+0000","last_fullsized":"2024-09-13T14:50:21.238625+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":409,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.231842+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:50:21.231842+0000","last_peered":"2024-09-13T14:50:21.231842+0000","last_clean":"2024-09-13T14:50:21.231842+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:50:21.231842+0000","last_undegraded":"2024-09-13T14:50:21.231842+0000","last_fullsized":"2024-09-13T14:50:21.231842+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.163729+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:50:05.163729+0000","last_peered":"2024-09-13T14:50:05.163729+0000","last_clean":"2024-09-13T14:50:05.163729+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:50:05.163729+0000","last_undegraded":"2024-09-13T14:50:05.163729+0000","last_fullsized":"2024-09-13T14:50:05.163729+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":40,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.164027+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:50:05.164027+0000","last_peered":"2024-09-13T14:50:05.164027+0000","last_clean":"2024-09-13T14:50:05.164027+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:50:05.164027+0000","last_undegraded":"2024-09-13T14:50:05.164027+0000","last_fullsized":"2024-09-13T14:50:05.164027+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":239,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:50:05.160453+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:50:05.160453+0000","last_peered":"2024-09-13T14:50:05.160453+0000","last_clean":"2024-09-13T14:50:05.160453+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:50:05.160453+0000","last_undegraded":"2024-09-13T14:50:05.160453+0000","last_fullsized":"2024-09-13T14:50:05.160453+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":68,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.233636+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:50:21.233636+0000","last_peered":"2024-09-13T14:50:21.233636+0000","last_clean":"2024-09-13T14:50:21.233636+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:50:21.233636+0000","last_undegraded":"2024-09-13T14:50:21.233636+0000","last_fullsized":"2024-09-13T14:50:21.233636+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":366,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.236944+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:50:21.236944+0000","last_peered":"2024-09-13T14:50:21.236944+0000","last_clean":"2024-09-13T14:50:21.236944+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:50:21.236944+0000","last_undegraded":"2024-09-13T14:50:21.236944+0000","last_fullsized":"2024-09-13T14:50:21.236944+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3153723416,"num_objects":48123,"num_object_clones":0,"num_object_copies":144369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10617,"num_objects_misplaced":1518,"num_objects_unfound":0,"num_objects_dirty":48123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185331,"num_write_kb":7470465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3551,"num_bytes_recovered":232718336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10157961216,"data_stored":10157949000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112817,"ondisk_log_size":112817,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365169,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1691076,"kb_used_data":781692,"kb_used_omap":1,"kb_used_meta":909374,"kb_avail":92054076,"statfs":{"total":95995035648,"available":94263373824,"internally_reserved":0,"allocated":800452608,"data_stored":799098894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931199320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561399,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2277440,"kb_used_data":1383928,"kb_used_omap":1,"kb_used_meta":893502,"kb_avail":91467712,"statfs":{"total":95995035648,"available":93662937088,"internally_reserved":0,"allocated":1417142272,"data_stored":1415792501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914946392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790331,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680308,"kb_used_data":775084,"kb_used_omap":1,"kb_used_meta":905214,"kb_avail":92064844,"statfs":{"total":95995035648,"available":94274400256,"internally_reserved":0,"allocated":793686016,"data_stored":792344627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":926939480},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986559,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3099396,"kb_used_data":2204988,"kb_used_omap":1,"kb_used_meta":894398,"kb_avail":90645756,"statfs":{"total":95995035648,"available":92821254144,"internally_reserved":0,"allocated":2257907712,"data_stored":2256554183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915863896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519715,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1437108,"kb_used_data":902828,"kb_used_omap":4,"kb_used_meta":534267,"kb_avail":92308044,"statfs":{"total":95995035648,"available":94523437056,"internally_reserved":0,"allocated":924495872,"data_stored":923154493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":547089856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304630,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2019320,"kb_used_data":1532656,"kb_used_omap":6,"kb_used_meta":486649,"kb_avail":91725832,"statfs":{"total":95995035648,"available":93927251968,"internally_reserved":0,"allocated":1569439744,"data_stored":1568094783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":498329031},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827934,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346164,"kb_used_data":552108,"kb_used_omap":5,"kb_used_meta":794042,"kb_avail":92398988,"statfs":{"total":95995035648,"available":94616563712,"internally_reserved":0,"allocated":565358592,"data_stored":564016640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":813099244},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285798,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2587636,"kb_used_data":1894316,"kb_used_omap":5,"kb_used_meta":693306,"kb_avail":91157516,"statfs":{"total":95995035648,"available":93345296384,"internally_reserved":0,"allocated":1939779584,"data_stored":1938436959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":709945403},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1921454080,"data_stored":1921450008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":558432256,"data_stored":558432256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1565065216,"data_stored":1565065216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":918159360,"data_stored":918159360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2231963648,"data_stored":2231959576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":778960896,"data_stored":778960896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1391595520,"data_stored":1391591448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":792330240,"data_stored":792330240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:50:34.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:50:34.849 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.880 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:34.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:34.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:35.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:35.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:35.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:35.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:35.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:35 smithi003 ceph-mon[107278]: pgmap v875: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 10617/144447 objects degraded (7.350%); 1518/144447 objects misplaced (1.051%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:35.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:35 smithi003 ceph-mon[107278]: from='client.46358 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:35.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:35 smithi003 ceph-mon[104007]: pgmap v875: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 10617/144447 objects degraded (7.350%); 1518/144447 objects misplaced (1.051%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:35.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:35 smithi003 ceph-mon[104007]: from='client.46358 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:35.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:35.324 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:35.324 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:35.324 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:35.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:35.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:35.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:35.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:35 smithi019 ceph-mon[91296]: pgmap v875: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 10617/144447 objects degraded (7.350%); 1518/144447 objects misplaced (1.051%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:50:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:35 smithi019 ceph-mon[91296]: from='client.46358 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:35.713 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:35.747 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:35.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:35.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:35.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:35.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:36.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:36.126 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:36.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:36.265 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:50:36.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:36.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:36.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:50:36.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:36.440 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:36.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:36.547 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:36.577 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:36.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:36.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:36 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10617/144447 objects degraded (7.350%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:36 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/865996472' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:36 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:36.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:36 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10617/144447 objects degraded (7.350%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:36.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:36 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/865996472' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:36 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:36 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10617/144447 objects degraded (7.350%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:36 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/865996472' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:36 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:36.717 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:36.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:36.861 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:36.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:37.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:37.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:37.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:37.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.277 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:37.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:37.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:37 smithi003 ceph-mon[104007]: pgmap v876: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 10739/130977 objects degraded (8.199%); 1263/130977 objects misplaced (0.964%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T14:50:37.290 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:37 smithi003 ceph-mon[107278]: pgmap v876: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 10739/130977 objects degraded (8.199%); 1263/130977 objects misplaced (0.964%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T14:50:37.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:37.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:37.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:37.551 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:37.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:37 smithi019 ceph-mon[91296]: pgmap v876: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 10739/130977 objects degraded (8.199%); 1263/130977 objects misplaced (0.964%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T14:50:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:37 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:37 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:37 smithi019 ceph-mon[91296]: osdmap e258: 8 total, 8 up, 8 in 2024-09-13T14:50:37.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:37 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:37 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:37 smithi003 ceph-mon[107278]: osdmap e258: 8 total, 8 up, 8 in 2024-09-13T14:50:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:37 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:37 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:37 smithi003 ceph-mon[104007]: osdmap e258: 8 total, 8 up, 8 in 2024-09-13T14:50:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:37.789 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:37.826 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:37.928 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:37.966 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:38.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:38.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:38.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.249 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:38.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:38.350 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:38.386 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:38.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:38.487 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:38.524 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:38.626 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.657 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:38.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:38.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:38.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:38.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:38.927 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:39.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:39.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:39.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:39.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:39.307 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:39.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:39.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:39.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:39.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:39.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:39.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:39.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:39.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:39 smithi019 ceph-mon[91296]: pgmap v878: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 10739/130977 objects degraded (8.199%); 1263/130977 objects misplaced (0.964%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:39.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:39 smithi003 ceph-mon[104007]: pgmap v878: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 10739/130977 objects degraded (8.199%); 1263/130977 objects misplaced (0.964%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:39.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:39 smithi003 ceph-mon[107278]: pgmap v878: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 10739/130977 objects degraded (8.199%); 1263/130977 objects misplaced (0.964%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:39.743 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:39.782 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:39.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:39.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:40.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:40.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:40.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:40.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.192 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:40.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:40.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:40.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:40.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:40.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:40.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:40.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:50:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:50:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:40.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:40.720 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:40.745 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:40.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:40.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:40.883 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:40.985 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:41.011 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:41.113 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:41.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:41.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:41.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:41.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:41.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:41.424 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:41 smithi003 ceph-mon[107278]: pgmap v879: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10892/119859 objects degraded (9.087%); 976/119859 objects misplaced (0.814%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10892/119859 objects degraded (9.087%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:41.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:41 smithi003 ceph-mon[104007]: pgmap v879: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10892/119859 objects degraded (9.087%); 976/119859 objects misplaced (0.814%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:41.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10892/119859 objects degraded (9.087%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:41.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:41.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:41.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:41 smithi019 ceph-mon[91296]: pgmap v879: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10892/119859 objects degraded (9.087%); 976/119859 objects misplaced (0.814%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10892/119859 objects degraded (9.087%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:41.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:41.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:41.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:41.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:41.979 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:42.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:42.117 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:42.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:42.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:42.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:42.362 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:42.402 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:42.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:42.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:42.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:42.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:42.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:42.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:42.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:42.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:43.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:43.096 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:43.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:50:43.164 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:43.198 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:43.228 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:43.330 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:43.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:43 smithi003 ceph-mon[107278]: pgmap v880: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10892/118245 objects degraded (9.211%); 976/118245 objects misplaced (0.825%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:43.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:43.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:43 smithi003 ceph-mon[104007]: pgmap v880: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10892/118245 objects degraded (9.211%); 976/118245 objects misplaced (0.825%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:43.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:43.363 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:43.465 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:43.497 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:43.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:43.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:43.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:43.641 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:43 smithi019 ceph-mon[91296]: pgmap v880: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10892/118245 objects degraded (9.211%); 976/118245 objects misplaced (0.825%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:43.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:43.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:43.877 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.913 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:43.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:44.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:44.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:44.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:44.160 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:44.199 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:44.300 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:44.327 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:50:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:50:44] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.51.0" 2024-09-13T14:50:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.333 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:44.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:44.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:44.473 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:44.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:44.579 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:44 smithi003 ceph-mon[107278]: pgmap v881: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10892/118245 objects degraded (9.211%); 976/118245 objects misplaced (0.825%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:44.579 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:44 smithi003 ceph-mon[104007]: pgmap v881: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10892/118245 objects degraded (9.211%); 976/118245 objects misplaced (0.825%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:44.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:44.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:44 smithi019 ceph-mon[91296]: pgmap v881: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10892/118245 objects degraded (9.211%); 976/118245 objects misplaced (0.825%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:50:44.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:44.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:44.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:44.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:44.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:45.027 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:45.129 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:45.168 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:45.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:45.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:45.424 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.462 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:45.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:45.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:45.600 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:45.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:45.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:45.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.875 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:45.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:45.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10892/118245 objects degraded (9.211%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:45.928 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10892/118245 objects degraded (9.211%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:45.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:46.020 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:46.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:46.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10892/118245 objects degraded (9.211%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:46.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:46.296 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:46.397 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:46.437 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:46.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:46.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:46.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:46.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:46.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:46.873 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:46.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:46 smithi003 ceph-mon[104007]: pgmap v882: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 11014/104541 objects degraded (10.536%); 714/104541 objects misplaced (0.683%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:50:46.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:46 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3444985517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:46.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:46 smithi003 ceph-mon[107278]: pgmap v882: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 11014/104541 objects degraded (10.536%); 714/104541 objects misplaced (0.683%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:50:46.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:46 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3444985517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:46.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:47.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:47.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:47.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:46 smithi019 ceph-mon[91296]: pgmap v882: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 11014/104541 objects degraded (10.536%); 714/104541 objects misplaced (0.683%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:50:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:46 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3444985517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:47.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:47.259 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:50:47.259 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:50:47.259 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:50:47.259 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:50:47.259 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:50:47.259 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:50:47.259 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:50:47.259 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:50:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:47.294 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:47.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.436 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:47.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:47.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:47.577 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:47.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.718 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:47.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:47.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:47.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:47.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:48.002 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:48.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:48.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:48.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:48.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:48.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:48.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:48.409 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:48.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:48.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:48.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:48.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:48.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:48.686 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:48.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:48.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:48.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:48 smithi003 ceph-mon[104007]: pgmap v883: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 11014/104541 objects degraded (10.536%); 714/104541 objects misplaced (0.683%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:50:48.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:48 smithi003 ceph-mon[107278]: pgmap v883: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 11014/104541 objects degraded (10.536%); 714/104541 objects misplaced (0.683%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:50:48.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:48.951 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:50:48.951 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11014/104541 objects degraded (10.536%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1275,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":258,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238227,"num_in_osds":8,"osd_in_since":1726237182,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":106},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":109,"num_pools":5,"num_objects":34847,"data_bytes":2282586296,"bytes_used":14165426176,"bytes_avail":753794859008,"bytes_total":767960285184,"degraded_objects":11014,"degraded_total":104541,"degraded_ratio":0.10535579342076314,"misplaced_objects":714,"misplaced_total":104541,"misplaced_ratio":0.0068298562286566991,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":1554694,"recovering_keys_per_sec":0,"num_objects_recovered":274,"num_bytes_recovered":17956864,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":762},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.zmhiel","status":"up:active","gid":14646}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-13T14:43:29.603041+0000","services":{}},"progress_events":{"70b0e26c-ee48-4914-b6cc-9ce75e26a89c":{"message":"Global Recovery Event (2m)\n [===========================.] (remaining: 3s)","progress":0.97247707843780518,"add_to_ceph_s":true},"b67cb405-2ef0-4d69-99ef-7eee6a6810fd":{"message":"Upgrade to 19.3.0-4697-gd58f7306 (9m)\n [============................] (remaining: 11m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-13T14:50:48.967 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:49.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:49.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:48 smithi019 ceph-mon[91296]: pgmap v883: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 11014/104541 objects degraded (10.536%); 714/104541 objects misplaced (0.683%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:50:49.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.247 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:49.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:49.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:49.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:49.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:49.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:49.666 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:49.671 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:50:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:50:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:50:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:50:49.671 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:50:49.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:49.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:49.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:49.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:50.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:50.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:50.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:50:50.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/701609703' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:50:50.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/701609703' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:50:50.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/701609703' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:50:50.196 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.235 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:50.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:50.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:50.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:50.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:50:50.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:50.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:50.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.681 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:50:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:50:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:50.713 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:50.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:50.815 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.853 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:50.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:50.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:51.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:51.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:50 smithi019 ceph-mon[91296]: pgmap v884: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 11161/92655 objects degraded (12.046%); 422/92655 objects misplaced (0.455%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:50:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11161/92655 objects degraded (12.046%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:51.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:50 smithi003 ceph-mon[104007]: pgmap v884: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 11161/92655 objects degraded (12.046%); 422/92655 objects misplaced (0.455%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:50:51.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11161/92655 objects degraded (12.046%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:51.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:50 smithi003 ceph-mon[107278]: pgmap v884: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 11161/92655 objects degraded (12.046%); 422/92655 objects misplaced (0.455%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:50:51.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11161/92655 objects degraded (12.046%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:51.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:51.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:51.375 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.412 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:51.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:51.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.556 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:51.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:51.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:51.700 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:51.797 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:50:51.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:50:51.802 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:51.826 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:51.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:51.858 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:51.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:51.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:52.086 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:52.115 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:52.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:52.253 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:52.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:52.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:52.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:52.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:52.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:52.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:52.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:52.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:52.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.994 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:52.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:53.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:52 smithi003 ceph-mon[107278]: pgmap v885: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 11161/90957 objects degraded (12.271%); 422/90957 objects misplaced (0.464%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:50:53.013 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:52 smithi003 ceph-mon[104007]: pgmap v885: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 11161/90957 objects degraded (12.271%); 422/90957 objects misplaced (0.464%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:50:53.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:53.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:53.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:52 smithi019 ceph-mon[91296]: pgmap v885: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 11161/90957 objects degraded (12.271%); 422/90957 objects misplaced (0.464%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:50:53.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:53.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:53.375 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:53.412 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:53.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:53.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:53.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:53.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:53.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:53.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.797 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:53.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:53.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:53.912 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:50:53.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:53.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:54.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:54.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/79929373' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:53 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:50:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:50:54] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.51.0" 2024-09-13T14:50:54.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/79929373' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:53 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/79929373' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:53 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:54.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:54.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:54.363 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:54.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:54.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:54.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:54.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:54.741 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:54.780 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:54.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.921 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:54.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:55.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.059 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:55.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:55.161 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:54 smithi019 ceph-mon[91296]: pgmap v886: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 11161/90957 objects degraded (12.271%); 422/90957 objects misplaced (0.464%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:50:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:54 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:54 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:54 smithi019 ceph-mon[91296]: osdmap e259: 8 total, 8 up, 8 in 2024-09-13T14:50:55.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:54 smithi003 ceph-mon[107278]: pgmap v886: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 11161/90957 objects degraded (12.271%); 422/90957 objects misplaced (0.464%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:50:55.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:54 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:55.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:54 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:55.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:54 smithi003 ceph-mon[107278]: osdmap e259: 8 total, 8 up, 8 in 2024-09-13T14:50:55.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:54 smithi003 ceph-mon[104007]: pgmap v886: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 11161/90957 objects degraded (12.271%); 422/90957 objects misplaced (0.464%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:50:55.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:54 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:55.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:54 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:55.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:54 smithi003 ceph-mon[104007]: osdmap e259: 8 total, 8 up, 8 in 2024-09-13T14:50:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:55.306 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:55.341 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:55.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:55.476 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:55.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:55.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:55.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:55.721 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:55.766 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:55.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:55.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:56.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:56.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:56.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11161/90957 objects degraded (12.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:56.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11161/90957 objects degraded (12.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:56.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11161/90957 objects degraded (12.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:56.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:56.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:56.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:56.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:56.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:56.562 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:56.602 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:56.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:56.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:56.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:56.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:56.981 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:57.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:56 smithi003 ceph-mon[107278]: pgmap v888: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 11295/78168 objects degraded (14.450%); 143/78168 objects misplaced (0.183%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:50:57.013 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:56 smithi003 ceph-mon[104007]: pgmap v888: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 11295/78168 objects degraded (14.450%); 143/78168 objects misplaced (0.183%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:57.123 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:57.166 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:56 smithi019 ceph-mon[91296]: pgmap v888: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 11295/78168 objects degraded (14.450%); 143/78168 objects misplaced (0.183%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:50:57.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:57.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:57.341 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:50:57.341 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:27877 10.3 10:cfe350c3:::benchmark_data_smithi003_197468_object27876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:48.004684+0000", 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 189.33316128800001, 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28990059800000001, 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.342 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 27877 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004684+0000", 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004684+0000", 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004685+0000", 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2270000000000001e-06 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004706+0000", 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0746000000000001e-05 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004717+0000", 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1207e-05 2024-09-13T14:50:57.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004724+0000", 2024-09-13T14:50:57.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4259999999999997e-06 2024-09-13T14:50:57.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004791+0000", 2024-09-13T14:50:57.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6811999999999997e-05 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.004983+0000", 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000192098 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005103+0000", 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011979 2024-09-13T14:50:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.007476+0000", 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002372847 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.008772+0000", 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012964540000000001 2024-09-13T14:50:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294492+0000", 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28571956199999998 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294540+0000", 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8170999999999997e-05 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294584+0000", 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4258000000000002e-05 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:27878 10.3 10:c94d1851:::benchmark_data_smithi003_197468_object27877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:48.005297+0000", 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 189.33254818, 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.289465476, 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 27878 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005297+0000", 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005297+0000", 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005298+0000", 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.111e-06 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005319+0000", 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1474000000000001e-05 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005330+0000", 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0824e-05 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005337+0000", 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2989999999999999e-06 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.006220+0000", 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00088279299999999999 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.006329+0000", 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108799 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.006453+0000", 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012355600000000001 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.008786+0000", 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023338579999999999 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.010360+0000", 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015731569999999999 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294646+0000", 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.284286078 2024-09-13T14:50:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294711+0000", 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5222000000000007e-05 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294762+0000", 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1304999999999998e-05 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:27879 10.3 10:ca6535d7:::benchmark_data_smithi003_197468_object27878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:47:48.005359+0000", 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 189.33248607300001, 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28957026200000002, 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 27879 2024-09-13T14:50:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005359+0000", 2024-09-13T14:50:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005359+0000", 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005359+0000", 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7899999999999998e-07 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005387+0000", 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7073999999999999e-05 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005395+0000", 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4200000000000007e-06 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.005401+0000", 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9649999999999996e-06 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.007509+0000", 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021081400000000001 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.007692+0000", 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000183315 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.007820+0000", 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012717199999999999 2024-09-13T14:50:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.008930+0000", 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011109340000000001 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.010476+0000", 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001545645 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294816+0000", 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28434024699999999 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294888+0000", 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1768999999999997e-05 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:47:48.294929+0000", 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1001999999999998e-05 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:117289 10.3 10:cb5621ab:::benchmark_data_smithi003_197468_object560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:09.758640+0000", 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 107.579205058, 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.29505568399999998, 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 117289 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.758640+0000", 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.758640+0000", 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.758641+0000", 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2389999999999999e-06 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.758647+0000", 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8610000000000003e-06 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.758661+0000", 2024-09-13T14:50:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3638999999999999e-05 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.758668+0000", 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0210000000000002e-06 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.758731+0000", 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2887999999999998e-05 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.758907+0000", 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017676500000000001 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.759065+0000", 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015796899999999999 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.765400+0000", 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063350469999999999 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.767656+0000", 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022559949999999998 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.051563+0000", 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28390654100000001 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.051621+0000", 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7825999999999998e-05 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.053696+0000", 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002074893 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:117290 10.3 10:c834c0a4:::benchmark_data_smithi003_197468_object561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:09.760012+0000", 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 107.577832505, 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.29514077300000002, 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 117290 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.760012+0000", 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.760012+0000", 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.760013+0000", 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9899999999999999e-07 2024-09-13T14:50:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.760017+0000", 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0990000000000001e-06 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.760030+0000", 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2805e-05 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.760035+0000", 2024-09-13T14:50:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3460000000000001e-06 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.762107+0000", 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002071916 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.762239+0000", 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131402 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.762345+0000", 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010581699999999999 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.765895+0000", 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035500750000000002 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.768093+0000", 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021987090000000001 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.053754+0000", 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28566013299999998 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.053809+0000", 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5037000000000002e-05 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.055153+0000", 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013446350000000001 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:117296 10.5 10:a30b8338:::benchmark_data_smithi003_197468_object567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:09.769023+0000", 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 107.568822289, 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28267649700000003, 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 117296 2024-09-13T14:50:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.769023+0000", 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.769023+0000", 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.769024+0000", 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.212e-06 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.769028+0000", 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5449999999999997e-06 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.769041+0000", 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2306999999999999e-05 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.769047+0000", 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7399999999999998e-06 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.769116+0000", 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8201999999999995e-05 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.769263+0000", 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014724900000000001 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.769387+0000", 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012437200000000001 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.773250+0000", 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003862482 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.773351+0000", 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101403 2024-09-13T14:50:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.051607+0000", 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27825550599999999 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.051653+0000", 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6703000000000003e-05 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.051699+0000", 2024-09-13T14:50:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5775999999999997e-05 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:117299 10.5 10:aeeb0ff6:::benchmark_data_smithi003_197468_object570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:09.773921+0000", 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 107.563923843, 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27792557000000001, 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 117299 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.773921+0000", 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.773921+0000", 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.773922+0000", 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.356e-06 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.773927+0000", 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9409999999999998e-06 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.773938+0000", 2024-09-13T14:50:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.101e-05 2024-09-13T14:50:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.773944+0000", 2024-09-13T14:50:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4620000000000004e-06 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.773998+0000", 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3909999999999997e-05 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.774111+0000", 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011343 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.774185+0000", 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3645000000000004e-05 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.777335+0000", 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003150482 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.777664+0000", 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032883000000000001 2024-09-13T14:50:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.051761+0000", 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27409721399999998 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.051810+0000", 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8900999999999998e-05 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.051847+0000", 2024-09-13T14:50:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6389e-05 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:117311 10.3 10:cefb7836:::benchmark_data_smithi003_197468_object582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:09.804064+0000", 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 107.53378069999999, 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26134798100000001, 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 117311 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.804064+0000", 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.804064+0000", 2024-09-13T14:50:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.804066+0000", 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.432e-06 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.804071+0000", 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4500000000000003e-06 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.804084+0000", 2024-09-13T14:50:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3288000000000001e-05 2024-09-13T14:50:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.804091+0000", 2024-09-13T14:50:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3210000000000001e-06 2024-09-13T14:50:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.804148+0000", 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7417000000000003e-05 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.804262+0000", 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113405 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.804359+0000", 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7244999999999995e-05 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.807481+0000", 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031217340000000001 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.807821+0000", 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000340501 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.063427+0000", 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.25560553000000003 2024-09-13T14:50:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.063493+0000", 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6283000000000002e-05 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.065412+0000", 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019193750000000001 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:123357 10.5 10:a6ed905f:::benchmark_data_smithi003_197468_object6628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.382026+0000", 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 99.955819168999994, 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26545722700000002, 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123357 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382026+0000", 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382026+0000", 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382027+0000", 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.178e-06 2024-09-13T14:50:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382032+0000", 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2619999999999998e-06 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382046+0000", 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3526999999999999e-05 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382051+0000", 2024-09-13T14:50:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3720000000000004e-06 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382102+0000", 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0949999999999998e-05 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382212+0000", 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109595 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:50:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382298+0000", 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6633999999999998e-05 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.386415+0000", 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041170679999999998 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.388064+0000", 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00164824 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647385+0000", 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.25932095999999999 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647441+0000", 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6444999999999999e-05 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647483+0000", 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1996000000000002e-05 2024-09-13T14:50:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:123364 10.3 10:c110b7c7:::benchmark_data_smithi003_197468_object6635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.388847+0000", 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 99.948998093, 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.282522891, 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123364 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.388847+0000", 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.388847+0000", 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.388848+0000", 2024-09-13T14:50:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3829999999999999e-06 2024-09-13T14:50:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.388859+0000", 2024-09-13T14:50:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0550000000000001e-05 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.388872+0000", 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3542e-05 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.388879+0000", 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.092e-06 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.388938+0000", 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8532000000000001e-05 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389040+0000", 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101575 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389127+0000", 2024-09-13T14:50:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7892999999999993e-05 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.392109+0000", 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002981788 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.396307+0000", 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041974559999999996 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.669340+0000", 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27303324299999998 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.669393+0000", 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3414000000000001e-05 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.671370+0000", 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019764230000000002 2024-09-13T14:50:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:123369 10.3 10:c5745efe:::benchmark_data_smithi003_197468_object6640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.397017+0000", 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 99.940827815000006, 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28821777199999998, 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123369 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.397017+0000", 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.397017+0000", 2024-09-13T14:50:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.397019+0000", 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5090000000000001e-06 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.397023+0000", 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8149999999999996e-06 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.397035+0000", 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1748e-05 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.397042+0000", 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2830000000000004e-06 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.397093+0000", 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0871999999999997e-05 2024-09-13T14:50:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.397199+0000", 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010566200000000001 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.397272+0000", 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2548999999999999e-05 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.400474+0000", 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003202281 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.402668+0000", 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021943259999999999 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.683257+0000", 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28058921100000001 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.683316+0000", 2024-09-13T14:50:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8363999999999999e-05 2024-09-13T14:50:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.685235+0000", 2024-09-13T14:50:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019191519999999999 2024-09-13T14:50:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:123383 10.3 10:c6ef3114:::benchmark_data_smithi003_197468_object6654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.414141+0000", 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 99.923704338999997, 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27489696299999999, 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123383 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.414141+0000", 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.414141+0000", 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.414142+0000", 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0419999999999999e-06 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.414145+0000", 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8330000000000004e-06 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.414155+0000", 2024-09-13T14:50:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0769999999999997e-06 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.414160+0000", 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7440000000000004e-06 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.414211+0000", 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0414999999999998e-05 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.414301+0000", 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9983999999999998e-05 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.414377+0000", 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6750999999999998e-05 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.417529+0000", 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00315169 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.423217+0000", 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056882369999999996 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.685286+0000", 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26206882300000001 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.685340+0000", 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4174e-05 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.689038+0000", 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036971930000000001 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:123386 10.3 10:c4cdea6b:::benchmark_data_smithi003_197468_object6657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.418118+0000", 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 99.919726487999995, 2024-09-13T14:50:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27298570999999999, 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123386 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.418118+0000", 2024-09-13T14:50:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.418118+0000", 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.418120+0000", 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1999999999999999e-06 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.418125+0000", 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0549999999999999e-06 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.418137+0000", 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1918e-05 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.418143+0000", 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5400000000000001e-06 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.418205+0000", 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1600000000000007e-05 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.418369+0000", 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000164034 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.418523+0000", 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000154332 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.421687+0000", 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003163375 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.423283+0000", 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015964219999999999 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.689160+0000", 2024-09-13T14:50:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26587681400000002 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.689281+0000", 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012143500000000001 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.691104+0000", 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018229850000000001 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149380 10.3 10:c6822046:::benchmark_data_smithi003_197468_object32651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.296643+0000", 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 70.041201882999999, 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34575353399999997, 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149380 2024-09-13T14:50:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296643+0000", 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296643+0000", 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296645+0000", 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.761e-06 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296651+0000", 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5180000000000001e-06 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296665+0000", 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4066000000000001e-05 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296674+0000", 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6840000000000002e-06 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296741+0000", 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7305999999999996e-05 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296904+0000", 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016293199999999999 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297060+0000", 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015613700000000001 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.310299+0000", 2024-09-13T14:50:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013238409 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.313242+0000", 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029432899999999999 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.640267+0000", 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32702447200000001 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.640356+0000", 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9588999999999999e-05 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642397+0000", 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020403700000000001 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149384 10.3 10:c87a8b94:::benchmark_data_smithi003_197468_object32655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:50:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.299373+0000", 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 70.038471963999996, 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34479961399999998, 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149384 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299373+0000", 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299373+0000", 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299374+0000", 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3060000000000001e-06 2024-09-13T14:50:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299379+0000", 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0159999999999999e-06 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299420+0000", 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0426999999999998e-05 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299427+0000", 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0639999999999996e-06 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.304870+0000", 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054428130000000003 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305026+0000", 2024-09-13T14:50:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156138 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305177+0000", 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015097900000000001 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.310738+0000", 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055609120000000003 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.313250+0000", 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025120889999999999 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642486+0000", 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.329236468 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642558+0000", 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1736000000000006e-05 2024-09-13T14:50:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644173+0000", 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016146660000000001 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149387 10.5 10:a0bfa116:::benchmark_data_smithi003_197468_object32658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.303456+0000", 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 70.034389039999994, 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34075042799999999, 2024-09-13T14:50:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149387 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303456+0000", 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303456+0000", 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303457+0000", 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.443e-06 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303463+0000", 2024-09-13T14:50:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6219999999999997e-06 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303508+0000", 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4771999999999998e-05 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303540+0000", 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2500999999999999e-05 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.311300+0000", 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077601040000000003 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.311420+0000", 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119769 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.311533+0000", 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112631 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316488+0000", 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049549090000000004 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.321528+0000", 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050398550000000002 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644049+0000", 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32252128600000002 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644163+0000", 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114039 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644206+0000", 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3497e-05 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149402 10.5 10:a23e644e:::benchmark_data_smithi003_197468_object32673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.316465+0000", 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 70.021380386000004, 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32805168899999998, 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149402 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316465+0000", 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316465+0000", 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316466+0000", 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1689999999999999e-06 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316470+0000", 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9720000000000003e-06 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316479+0000", 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0100000000000001e-06 2024-09-13T14:50:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316484+0000", 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1970000000000004e-06 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316606+0000", 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012250100000000001 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316764+0000", 2024-09-13T14:50:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015716499999999999 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316876+0000", 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112608 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.321155+0000", 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042787850000000002 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.321559+0000", 2024-09-13T14:50:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000404254 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644242+0000", 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.322682523 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644300+0000", 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8136e-05 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644516+0000", 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000216369 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149405 10.3 10:cbffac49:::benchmark_data_smithi003_197468_object32676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.322647+0000", 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 70.015197619000006, 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32315936899999997, 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149405 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322647+0000", 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322647+0000", 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322648+0000", 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1319999999999999e-06 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322653+0000", 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6190000000000002e-06 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322664+0000", 2024-09-13T14:50:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1286e-05 2024-09-13T14:50:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322670+0000", 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2830000000000001e-06 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322718+0000", 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7972000000000001e-05 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.322896+0000", 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017793799999999999 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.323005+0000", 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108967 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.326477+0000", 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034722799999999999 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.328146+0000", 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016687990000000001 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644235+0000", 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.31608894599999998 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644314+0000", 2024-09-13T14:50:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9530000000000006e-05 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.645807+0000", 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014926169999999999 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149411 10.5 10:a7018860:::benchmark_data_smithi003_197468_object32682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.333505+0000", 2024-09-13T14:50:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 70.004339522999999, 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.31276074799999998, 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149411 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.333505+0000", 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.333505+0000", 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.333507+0000", 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.457e-06 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.333513+0000", 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6760000000000004e-06 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.333525+0000", 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2593e-05 2024-09-13T14:50:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.333532+0000", 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3709999999999996e-06 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.333586+0000", 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3999999999999998e-05 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.333674+0000", 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8633000000000003e-05 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.333745+0000", 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1174000000000007e-05 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.338101+0000", 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043555920000000001 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.338407+0000", 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000306385 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644695+0000", 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.30628771399999999 2024-09-13T14:50:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646229+0000", 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015339539999999999 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646266+0000", 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7199000000000001e-05 2024-09-13T14:50:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149424 10.3 10:c6e36560:::benchmark_data_smithi003_197468_object32695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.367504+0000", 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 69.970341164999994, 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.29034110899999999, 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:50:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:50:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:50:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149424 2024-09-13T14:50:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:50:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:50:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.367504+0000", 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.367504+0000", 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:50:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.367505+0000", 2024-09-13T14:50:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.28e-06 2024-09-13T14:50:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:50:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.367510+0000", 2024-09-13T14:50:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2480000000000004e-06 2024-09-13T14:50:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.367522+0000", 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.135e-05 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.367526+0000", 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.746e-06 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.367578+0000", 2024-09-13T14:50:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2006000000000003e-05 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.367695+0000", 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011691 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.367799+0000", 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010356700000000001 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.371032+0000", 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032326159999999998 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.371217+0000", 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185701 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.655899+0000", 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28468187700000003 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.655958+0000", 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8984999999999999e-05 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.657845+0000", 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018868229999999999 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:50:57.480 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:50:57.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:57.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:57.634 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:57.736 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:57.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.779 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:57.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:57.847 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:57.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:57.915 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.017 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:58.050 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:58.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:58.191 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:58.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:58.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:58.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:58.433 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:58.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:58.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:58.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:50:58.575 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:58.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.717 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:58.746 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:58.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:50:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:58.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:58.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:58.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:58.993 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:59.028 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:59.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:59.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:50:58 smithi003 ceph-mon[107278]: pgmap v889: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 11295/78168 objects degraded (14.450%); 143/78168 objects misplaced (0.183%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:50:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:50:58 smithi003 ceph-mon[104007]: pgmap v889: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 11295/78168 objects degraded (14.450%); 143/78168 objects misplaced (0.183%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:50:59.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:59.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:59.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:59.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:50:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:59.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:50:58 smithi019 ceph-mon[91296]: pgmap v889: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 11295/78168 objects degraded (14.450%); 143/78168 objects misplaced (0.183%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:50:59.518 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:59.541 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:59.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:59.683 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:59.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:59.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:59.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:59.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:59.946 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:50:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:00.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:00.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:00.181 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:00.221 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:00.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:00.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:00.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:00.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:00.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.621 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:00.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:51:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:51:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:00.722 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:00.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:00.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:00.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:01.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:01.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:01.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:01.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:01.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.309 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:01.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:01.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:01.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:00 smithi019 ceph-mon[91296]: pgmap v890: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 11275/66738 objects degraded (16.894%); 1.3 MiB/s, 30 objects/s recovering 2024-09-13T14:51:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11275/66738 objects degraded (16.894%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:01.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:00 smithi003 ceph-mon[104007]: pgmap v890: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 11275/66738 objects degraded (16.894%); 1.3 MiB/s, 30 objects/s recovering 2024-09-13T14:51:01.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11275/66738 objects degraded (16.894%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:01.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:00 smithi003 ceph-mon[107278]: pgmap v890: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 11275/66738 objects degraded (16.894%); 1.3 MiB/s, 30 objects/s recovering 2024-09-13T14:51:01.436 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11275/66738 objects degraded (16.894%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:01.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:01.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:01.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:01.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:01.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:01.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.882 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:01.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:01.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:02.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:02.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:02.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:02.260 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:02.287 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:02.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:02.428 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:02.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:02.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:02.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:02.663 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:02.699 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:02.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:02.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:02.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.978 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:02.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:03.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:03.115 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:03.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:02 smithi003 ceph-mon[107278]: pgmap v891: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11275/65217 objects degraded (17.288%); 1.3 MiB/s, 30 objects/s recovering 2024-09-13T14:51:03.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:02 smithi003 ceph-mon[104007]: pgmap v891: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11275/65217 objects degraded (17.288%); 1.3 MiB/s, 30 objects/s recovering 2024-09-13T14:51:03.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:03.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:03.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:03.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:03.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:02 smithi019 ceph-mon[91296]: pgmap v891: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11275/65217 objects degraded (17.288%); 1.3 MiB/s, 30 objects/s recovering 2024-09-13T14:51:03.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:03.557 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:03.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:03.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:03.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:03.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:03.848 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:03.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.974 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:03.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:04.076 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:04.113 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:04.215 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:04.252 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:04.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:04.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:04.397 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:04.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:04.400 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:51:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:51:04] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-09-13T14:51:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:04.502 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:04.540 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:04.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:04.685 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:04.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:04.808 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:04.910 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:04.943 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:05.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:05.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:05.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:05.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:04 smithi003 ceph-mon[107278]: pgmap v892: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11275/65217 objects degraded (17.288%); 1.3 MiB/s, 30 objects/s recovering 2024-09-13T14:51:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:04 smithi003 ceph-mon[104007]: pgmap v892: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11275/65217 objects degraded (17.288%); 1.3 MiB/s, 30 objects/s recovering 2024-09-13T14:51:05.190 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:05.232 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:05.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:05.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:04 smithi019 ceph-mon[91296]: pgmap v892: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11275/65217 objects degraded (17.288%); 1.3 MiB/s, 30 objects/s recovering 2024-09-13T14:51:05.476 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:05.514 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:05.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:05.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:05.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:05.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:05.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:05.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:06.049 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:06.087 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:06.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 11275/65217 objects degraded (17.288%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:06.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 11275/65217 objects degraded (17.288%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:06.189 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:06.232 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:06.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:06.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 11275/65217 objects degraded (17.288%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:06.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:06.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:06.602 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:51:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:06.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:06.634 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:51:06.634 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:06.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:06.674 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:51:06.674 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:06.717 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:51:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:06.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:06.764 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:51:06.764 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:06.789 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:06.813 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:51:06.813 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:06.847 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:51:06.847 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:06.885 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:51:06.886 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:06.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:06.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:07.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:07.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:51:07.169 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:06 smithi003 ceph-mon[107278]: pgmap v893: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 11125/52224 objects degraded (21.302%); 1.1 MiB/s, 38 objects/s recovering 2024-09-13T14:51:07.169 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:06 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2466148660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:07.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:06 smithi003 ceph-mon[104007]: pgmap v893: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 11125/52224 objects degraded (21.302%); 1.1 MiB/s, 38 objects/s recovering 2024-09-13T14:51:07.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:06 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2466148660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:07.175 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:51:07.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:07.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:07.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:07.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:07.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:06 smithi019 ceph-mon[91296]: pgmap v893: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 11125/52224 objects degraded (21.302%); 1.1 MiB/s, 38 objects/s recovering 2024-09-13T14:51:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:06 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2466148660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:07.462 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:07.497 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:07.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:07.633 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:07.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:07.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:07.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:07.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:08.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:08.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:08.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:08.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:07 smithi003 ceph-mon[107278]: from='client.36357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:08.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:07 smithi003 ceph-mon[104007]: from='client.36357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:08.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:08.305 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:08.335 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:08.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:07 smithi019 ceph-mon[91296]: from='client.36357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:08.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:08.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:08.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:08.720 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:08.751 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:08.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.888 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:08.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:08.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:09.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:09.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:08 smithi003 ceph-mon[107278]: pgmap v894: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 11125/52224 objects degraded (21.302%); 368 KiB/s, 26 objects/s recovering 2024-09-13T14:51:09.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:08 smithi003 ceph-mon[104007]: pgmap v894: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 11125/52224 objects degraded (21.302%); 368 KiB/s, 26 objects/s recovering 2024-09-13T14:51:09.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:09.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:09.278 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:09.315 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:09.417 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:08 smithi019 ceph-mon[91296]: pgmap v894: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 11125/52224 objects degraded (21.302%); 368 KiB/s, 26 objects/s recovering 2024-09-13T14:51:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.456 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:09.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:09.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:09.589 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:09.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:09.723 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:09.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:09.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:09.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:10.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:10.009 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:51:10.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:10.141 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:10.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:10.333 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1026330461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:10.333 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:09 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:10.333 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:51:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:51:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:10.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:10.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:10.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:10.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1026330461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:10 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:10.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1026330461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:10.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:09 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:10.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:10.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:10.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:10.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:10.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:10.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:10.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:51:10.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:11.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:11.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:11.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[104007]: pgmap v895: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 858 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 10918/41247 objects degraded (26.470%); 368 KiB/s, 43 objects/s recovering 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[104007]: osdmap e260: 8 total, 8 up, 8 in 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10918/41247 objects degraded (26.470%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[107278]: pgmap v895: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 858 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 10918/41247 objects degraded (26.470%); 368 KiB/s, 43 objects/s recovering 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[107278]: osdmap e260: 8 total, 8 up, 8 in 2024-09-13T14:51:11.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10918/41247 objects degraded (26.470%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:11.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:11.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:11.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:11.400 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:10 smithi019 ceph-mon[91296]: pgmap v895: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 858 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 10918/41247 objects degraded (26.470%); 368 KiB/s, 43 objects/s recovering 2024-09-13T14:51:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:11 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:11 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:11 smithi019 ceph-mon[91296]: osdmap e260: 8 total, 8 up, 8 in 2024-09-13T14:51:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10918/41247 objects degraded (26.470%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:11.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.443 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:11.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:11.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:11.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:11.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:11.683 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:11.720 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:11.819 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":893,"stamp":"2024-09-13T14:51:05.822173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1139703992,"num_objects":17408,"num_object_clones":0,"num_object_copies":52224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17408,"num_whiteouts":0,"num_read":15,"num_read_kb":35,"num_write":216143,"num_write_kb":7470571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4579,"num_bytes_recovered":283799089,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115017,"ondisk_log_size":115017,"up":327,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":333,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10592528,"kb_used_data":4448208,"kb_used_omap":29,"kb_used_meta":6144098,"kb_avail":739368688,"statfs":{"total":767960285184,"available":757113536512,"internally_reserved":0,"allocated":4554964992,"data_stored":4544071528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6291556878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,10,6,13,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-846135296,"num_objects":-12911,"num_object_clones":0,"num_object_copies":-38733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-36,"num_objects_misplaced":-422,"num_objects_unfound":0,"num_objects_dirty":-12911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12911,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":14024704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.913335"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":397,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921331+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:50:53.921331+0000","last_peered":"2024-09-13T14:50:53.921331+0000","last_clean":"2024-09-13T14:50:53.921331+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:50:53.921331+0000","last_undegraded":"2024-09-13T14:50:53.921331+0000","last_fullsized":"2024-09-13T14:50:53.921331+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":231,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.269198+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:50:36.269198+0000","last_peered":"2024-09-13T14:50:36.269198+0000","last_clean":"2024-09-13T14:50:36.269198+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:50:36.269198+0000","last_undegraded":"2024-09-13T14:50:36.269198+0000","last_fullsized":"2024-09-13T14:50:36.269198+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":22,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.277255+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:50:36.277255+0000","last_peered":"2024-09-13T14:50:36.277255+0000","last_clean":"2024-09-13T14:50:36.277255+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:50:36.277255+0000","last_undegraded":"2024-09-13T14:50:36.277255+0000","last_fullsized":"2024-09-13T14:50:36.277255+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":355,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925585+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:50:53.925585+0000","last_peered":"2024-09-13T14:50:53.925585+0000","last_clean":"2024-09-13T14:50:53.925585+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:50:53.925585+0000","last_undegraded":"2024-09-13T14:50:53.925585+0000","last_fullsized":"2024-09-13T14:50:53.925585+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":205,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.277349+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:50:36.277349+0000","last_peered":"2024-09-13T14:50:36.277349+0000","last_clean":"2024-09-13T14:50:36.277349+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:50:36.277349+0000","last_undegraded":"2024-09-13T14:50:36.277349+0000","last_fullsized":"2024-09-13T14:50:36.277349+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":238,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.269776+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:50:36.269776+0000","last_peered":"2024-09-13T14:50:36.269776+0000","last_clean":"2024-09-13T14:50:36.269776+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:50:36.269776+0000","last_undegraded":"2024-09-13T14:50:36.269776+0000","last_fullsized":"2024-09-13T14:50:36.269776+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":222,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.277492+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:50:36.277492+0000","last_peered":"2024-09-13T14:50:36.277492+0000","last_clean":"2024-09-13T14:50:36.277492+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:50:36.277492+0000","last_undegraded":"2024-09-13T14:50:36.277492+0000","last_fullsized":"2024-09-13T14:50:36.277492+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":268,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.269251+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:50:36.269251+0000","last_peered":"2024-09-13T14:50:36.269251+0000","last_clean":"2024-09-13T14:50:36.269251+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:50:36.269251+0000","last_undegraded":"2024-09-13T14:50:36.269251+0000","last_fullsized":"2024-09-13T14:50:36.269251+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":384,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.922403+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:50:53.922403+0000","last_peered":"2024-09-13T14:50:53.922403+0000","last_clean":"2024-09-13T14:50:53.922403+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:50:53.922403+0000","last_undegraded":"2024-09-13T14:50:53.922403+0000","last_fullsized":"2024-09-13T14:50:53.922403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":380,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.910150+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:50:53.910150+0000","last_peered":"2024-09-13T14:50:53.910150+0000","last_clean":"2024-09-13T14:50:53.910150+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:50:53.910150+0000","last_undegraded":"2024-09-13T14:50:53.910150+0000","last_fullsized":"2024-09-13T14:50:53.910150+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":240,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.269899+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:50:36.269899+0000","last_peered":"2024-09-13T14:50:36.269899+0000","last_clean":"2024-09-13T14:50:36.269899+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:50:36.269899+0000","last_undegraded":"2024-09-13T14:50:36.269899+0000","last_fullsized":"2024-09-13T14:50:36.269899+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":322,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925576+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:50:53.925576+0000","last_peered":"2024-09-13T14:50:53.925576+0000","last_clean":"2024-09-13T14:50:53.925576+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:50:53.925576+0000","last_undegraded":"2024-09-13T14:50:53.925576+0000","last_fullsized":"2024-09-13T14:50:53.925576+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":37,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.912090+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:50:53.912090+0000","last_peered":"2024-09-13T14:50:53.912090+0000","last_clean":"2024-09-13T14:50:53.912090+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:50:53.912090+0000","last_undegraded":"2024-09-13T14:50:53.912090+0000","last_fullsized":"2024-09-13T14:50:53.912090+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":415,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921064+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:50:53.921064+0000","last_peered":"2024-09-13T14:50:53.921064+0000","last_clean":"2024-09-13T14:50:53.921064+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:50:53.921064+0000","last_undegraded":"2024-09-13T14:50:53.921064+0000","last_fullsized":"2024-09-13T14:50:53.921064+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":354,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925329+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:50:53.925329+0000","last_peered":"2024-09-13T14:50:53.925329+0000","last_clean":"2024-09-13T14:50:53.925329+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:50:53.925329+0000","last_undegraded":"2024-09-13T14:50:53.925329+0000","last_fullsized":"2024-09-13T14:50:53.925329+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":221,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.275676+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:50:36.275676+0000","last_peered":"2024-09-13T14:50:36.275676+0000","last_clean":"2024-09-13T14:50:36.275676+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:50:36.275676+0000","last_undegraded":"2024-09-13T14:50:36.275676+0000","last_fullsized":"2024-09-13T14:50:36.275676+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":322,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925150+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:50:53.925150+0000","last_peered":"2024-09-13T14:50:53.925150+0000","last_clean":"2024-09-13T14:50:53.925150+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:50:53.925150+0000","last_undegraded":"2024-09-13T14:50:53.925150+0000","last_fullsized":"2024-09-13T14:50:53.925150+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":399,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921577+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:50:53.921577+0000","last_peered":"2024-09-13T14:50:53.921577+0000","last_clean":"2024-09-13T14:50:53.921577+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:50:53.921577+0000","last_undegraded":"2024-09-13T14:50:53.921577+0000","last_fullsized":"2024-09-13T14:50:53.921577+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":327,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925225+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:50:53.925225+0000","last_peered":"2024-09-13T14:50:53.925225+0000","last_clean":"2024-09-13T14:50:53.925225+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:50:53.925225+0000","last_undegraded":"2024-09-13T14:50:53.925225+0000","last_fullsized":"2024-09-13T14:50:53.925225+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.277233+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:50:36.277233+0000","last_peered":"2024-09-13T14:50:36.277233+0000","last_clean":"2024-09-13T14:50:36.277233+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:50:36.277233+0000","last_undegraded":"2024-09-13T14:50:36.277233+0000","last_fullsized":"2024-09-13T14:50:36.277233+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":77,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.909951+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:50:53.909951+0000","last_peered":"2024-09-13T14:50:53.909951+0000","last_clean":"2024-09-13T14:50:53.909951+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:50:53.909951+0000","last_undegraded":"2024-09-13T14:50:53.909951+0000","last_fullsized":"2024-09-13T14:50:53.909951+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":22,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.284882+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:50:36.284882+0000","last_peered":"2024-09-13T14:50:36.284882+0000","last_clean":"2024-09-13T14:50:36.284882+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:50:36.284882+0000","last_undegraded":"2024-09-13T14:50:36.284882+0000","last_fullsized":"2024-09-13T14:50:36.284882+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":160,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.274698+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:50:36.274698+0000","last_peered":"2024-09-13T14:50:36.274698+0000","last_clean":"2024-09-13T14:50:36.274698+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:50:36.274698+0000","last_undegraded":"2024-09-13T14:50:36.274698+0000","last_fullsized":"2024-09-13T14:50:36.274698+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":228,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.270844+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:50:36.270844+0000","last_peered":"2024-09-13T14:50:36.270844+0000","last_clean":"2024-09-13T14:50:36.270844+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:50:36.270844+0000","last_undegraded":"2024-09-13T14:50:36.270844+0000","last_fullsized":"2024-09-13T14:50:36.270844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":221,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.269318+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:50:36.269318+0000","last_peered":"2024-09-13T14:50:36.269318+0000","last_clean":"2024-09-13T14:50:36.269318+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:50:36.269318+0000","last_undegraded":"2024-09-13T14:50:36.269318+0000","last_fullsized":"2024-09-13T14:50:36.269318+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"259'13411","reported_seq":13449,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:03.998100+0000","last_change":"2024-09-13T14:47:20.107455+0000","last_active":"2024-09-13T14:51:03.998100+0000","last_peered":"2024-09-13T14:51:03.998100+0000","last_clean":"2024-09-13T14:51:03.998100+0000","last_became_active":"2024-09-13T14:47:20.107142+0000","last_became_peered":"2024-09-13T14:47:20.107142+0000","last_unstale":"2024-09-13T14:51:03.998100+0000","last_undegraded":"2024-09-13T14:51:03.998100+0000","last_fullsized":"2024-09-13T14:51:03.998100+0000","mapping_epoch":229,"log_start":"251'6700","ondisk_log_start":"251'6700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:19.189519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69271552,"num_objects":1057,"num_object_clones":0,"num_object_copies":3171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13411,"num_write_kb":462976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":204,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.268847+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:50:36.268847+0000","last_peered":"2024-09-13T14:50:36.268847+0000","last_clean":"2024-09-13T14:50:36.268847+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:50:36.268847+0000","last_undegraded":"2024-09-13T14:50:36.268847+0000","last_fullsized":"2024-09-13T14:50:36.268847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":254,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.275803+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:50:36.275803+0000","last_peered":"2024-09-13T14:50:36.275803+0000","last_clean":"2024-09-13T14:50:36.275803+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:50:36.275803+0000","last_undegraded":"2024-09-13T14:50:36.275803+0000","last_fullsized":"2024-09-13T14:50:36.275803+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":397,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921271+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:50:53.921271+0000","last_peered":"2024-09-13T14:50:53.921271+0000","last_clean":"2024-09-13T14:50:53.921271+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:50:53.921271+0000","last_undegraded":"2024-09-13T14:50:53.921271+0000","last_fullsized":"2024-09-13T14:50:53.921271+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":84,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925807+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:50:53.925807+0000","last_peered":"2024-09-13T14:50:53.925807+0000","last_clean":"2024-09-13T14:50:53.925807+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:50:53.925807+0000","last_undegraded":"2024-09-13T14:50:53.925807+0000","last_fullsized":"2024-09-13T14:50:53.925807+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"259'13518","reported_seq":13556,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:04.167502+0000","last_change":"2024-09-13T14:47:20.054114+0000","last_active":"2024-09-13T14:51:04.167502+0000","last_peered":"2024-09-13T14:51:04.167502+0000","last_clean":"2024-09-13T14:51:04.167502+0000","last_became_active":"2024-09-13T14:47:20.053767+0000","last_became_peered":"2024-09-13T14:47:20.053767+0000","last_unstale":"2024-09-13T14:51:04.167502+0000","last_undegraded":"2024-09-13T14:51:04.167502+0000","last_fullsized":"2024-09-13T14:51:04.167502+0000","mapping_epoch":229,"log_start":"253'7200","ondisk_log_start":"253'7200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:48.035191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13518,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":365,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925242+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:50:53.925242+0000","last_peered":"2024-09-13T14:50:53.925242+0000","last_clean":"2024-09-13T14:50:53.925242+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:50:53.925242+0000","last_undegraded":"2024-09-13T14:50:53.925242+0000","last_fullsized":"2024-09-13T14:50:53.925242+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.275363+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:50:36.275363+0000","last_peered":"2024-09-13T14:50:36.275363+0000","last_clean":"2024-09-13T14:50:36.275363+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:50:36.275363+0000","last_undegraded":"2024-09-13T14:50:36.275363+0000","last_fullsized":"2024-09-13T14:50:36.275363+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":354,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.928615+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:50:53.928615+0000","last_peered":"2024-09-13T14:50:53.928615+0000","last_clean":"2024-09-13T14:50:53.928615+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:50:53.928615+0000","last_undegraded":"2024-09-13T14:50:53.928615+0000","last_fullsized":"2024-09-13T14:50:53.928615+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":42,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.269110+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:50:36.269110+0000","last_peered":"2024-09-13T14:50:36.269110+0000","last_clean":"2024-09-13T14:50:36.269110+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:50:36.269110+0000","last_undegraded":"2024-09-13T14:50:36.269110+0000","last_fullsized":"2024-09-13T14:50:36.269110+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"259'13534","reported_seq":13573,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:03.964023+0000","last_change":"2024-09-13T14:47:20.052876+0000","last_active":"2024-09-13T14:51:03.964023+0000","last_peered":"2024-09-13T14:51:03.964023+0000","last_clean":"2024-09-13T14:51:03.964023+0000","last_became_active":"2024-09-13T14:47:20.052566+0000","last_became_peered":"2024-09-13T14:47:20.052566+0000","last_unstale":"2024-09-13T14:51:03.964023+0000","last_undegraded":"2024-09-13T14:51:03.964023+0000","last_fullsized":"2024-09-13T14:51:03.964023+0000","mapping_epoch":229,"log_start":"251'6800","ondisk_log_start":"251'6800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6734,"log_dups_size":0,"ondisk_log_size":6734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:56.317980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71630872,"num_objects":1094,"num_object_clones":0,"num_object_copies":3282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13534,"num_write_kb":468033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":383,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.922319+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:50:53.922319+0000","last_peered":"2024-09-13T14:50:53.922319+0000","last_clean":"2024-09-13T14:50:53.922319+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:50:53.922319+0000","last_undegraded":"2024-09-13T14:50:53.922319+0000","last_fullsized":"2024-09-13T14:50:53.922319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":234,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.285776+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:50:36.285776+0000","last_peered":"2024-09-13T14:50:36.285776+0000","last_clean":"2024-09-13T14:50:36.285776+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:50:36.285776+0000","last_undegraded":"2024-09-13T14:50:36.285776+0000","last_fullsized":"2024-09-13T14:50:36.285776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":338,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.910058+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:50:53.910058+0000","last_peered":"2024-09-13T14:50:53.910058+0000","last_clean":"2024-09-13T14:50:53.910058+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:50:53.910058+0000","last_undegraded":"2024-09-13T14:50:53.910058+0000","last_fullsized":"2024-09-13T14:50:53.910058+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":195,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.275033+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:50:36.275033+0000","last_peered":"2024-09-13T14:50:36.275033+0000","last_clean":"2024-09-13T14:50:36.275033+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:50:36.275033+0000","last_undegraded":"2024-09-13T14:50:36.275033+0000","last_fullsized":"2024-09-13T14:50:36.275033+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"259'13592","reported_seq":13630,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:04.021623+0000","last_change":"2024-09-13T14:47:20.052419+0000","last_active":"2024-09-13T14:51:04.021623+0000","last_peered":"2024-09-13T14:51:04.021623+0000","last_clean":"2024-09-13T14:51:04.021623+0000","last_became_active":"2024-09-13T14:47:20.051845+0000","last_became_peered":"2024-09-13T14:47:20.051845+0000","last_unstale":"2024-09-13T14:51:04.021623+0000","last_undegraded":"2024-09-13T14:51:04.021623+0000","last_fullsized":"2024-09-13T14:51:04.021623+0000","mapping_epoch":229,"log_start":"251'6714","ondisk_log_start":"251'6714","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.420332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72613888,"num_objects":1108,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13592,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":20,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.276735+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:50:36.276735+0000","last_peered":"2024-09-13T14:50:36.276735+0000","last_clean":"2024-09-13T14:50:36.276735+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:50:36.276735+0000","last_undegraded":"2024-09-13T14:50:36.276735+0000","last_fullsized":"2024-09-13T14:50:36.276735+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":196,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.929459+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:50:53.929459+0000","last_peered":"2024-09-13T14:50:53.929459+0000","last_clean":"2024-09-13T14:50:53.929459+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:50:53.929459+0000","last_undegraded":"2024-09-13T14:50:53.929459+0000","last_fullsized":"2024-09-13T14:50:53.929459+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":237,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.277413+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:50:36.277413+0000","last_peered":"2024-09-13T14:50:36.277413+0000","last_clean":"2024-09-13T14:50:36.277413+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:50:36.277413+0000","last_undegraded":"2024-09-13T14:50:36.277413+0000","last_fullsized":"2024-09-13T14:50:36.277413+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":215,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.275522+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:50:36.275522+0000","last_peered":"2024-09-13T14:50:36.275522+0000","last_clean":"2024-09-13T14:50:36.275522+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:50:36.275522+0000","last_undegraded":"2024-09-13T14:50:36.275522+0000","last_fullsized":"2024-09-13T14:50:36.275522+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"259'13323","reported_seq":13361,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:03.965311+0000","last_change":"2024-09-13T14:47:20.051919+0000","last_active":"2024-09-13T14:51:03.965311+0000","last_peered":"2024-09-13T14:51:03.965311+0000","last_clean":"2024-09-13T14:51:03.965311+0000","last_became_active":"2024-09-13T14:47:20.051626+0000","last_became_peered":"2024-09-13T14:47:20.051626+0000","last_unstale":"2024-09-13T14:51:03.965311+0000","last_undegraded":"2024-09-13T14:51:03.965311+0000","last_fullsized":"2024-09-13T14:51:03.965311+0000","mapping_epoch":229,"log_start":"251'6600","ondisk_log_start":"251'6600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6723,"log_dups_size":0,"ondisk_log_size":6723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:12.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72155136,"num_objects":1101,"num_object_clones":0,"num_object_copies":3303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13323,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":250,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.910235+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:50:53.910235+0000","last_peered":"2024-09-13T14:50:53.910235+0000","last_clean":"2024-09-13T14:50:53.910235+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:50:53.910235+0000","last_undegraded":"2024-09-13T14:50:53.910235+0000","last_fullsized":"2024-09-13T14:50:53.910235+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":222,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.269393+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:50:36.269393+0000","last_peered":"2024-09-13T14:50:36.269393+0000","last_clean":"2024-09-13T14:50:36.269393+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:50:36.269393+0000","last_undegraded":"2024-09-13T14:50:36.269393+0000","last_fullsized":"2024-09-13T14:50:36.269393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":233,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.277115+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:50:36.277115+0000","last_peered":"2024-09-13T14:50:36.277115+0000","last_clean":"2024-09-13T14:50:36.277115+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:50:36.277115+0000","last_undegraded":"2024-09-13T14:50:36.277115+0000","last_fullsized":"2024-09-13T14:50:36.277115+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":42,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.276677+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:50:36.276677+0000","last_peered":"2024-09-13T14:50:36.276677+0000","last_clean":"2024-09-13T14:50:36.276677+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:50:36.276677+0000","last_undegraded":"2024-09-13T14:50:36.276677+0000","last_fullsized":"2024-09-13T14:50:36.276677+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"259'13484","reported_seq":19176,"reported_epoch":259,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:04.385088+0000","last_change":"2024-09-13T14:49:05.872778+0000","last_active":"2024-09-13T14:51:04.385088+0000","last_peered":"2024-09-13T14:51:04.385088+0000","last_clean":"2024-09-13T14:48:37.826901+0000","last_became_active":"2024-09-13T14:48:40.041761+0000","last_became_peered":"2024-09-13T14:48:40.041761+0000","last_unstale":"2024-09-13T14:51:04.385088+0000","last_undegraded":"2024-09-13T14:48:39.850843+0000","last_fullsized":"2024-09-13T14:48:39.825201+0000","mapping_epoch":249,"log_start":"247'5300","ondisk_log_start":"247'5300","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8184,"log_dups_size":0,"ondisk_log_size":8184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72351744,"num_objects":1104,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13484,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":168165376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,7","objects":-1845}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":153,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.280571+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:50:36.280571+0000","last_peered":"2024-09-13T14:50:36.280571+0000","last_clean":"2024-09-13T14:50:36.280571+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:50:36.280571+0000","last_undegraded":"2024-09-13T14:50:36.280571+0000","last_fullsized":"2024-09-13T14:50:36.280571+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":190,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.910447+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:50:53.910447+0000","last_peered":"2024-09-13T14:50:53.910447+0000","last_clean":"2024-09-13T14:50:53.910447+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:50:53.910447+0000","last_undegraded":"2024-09-13T14:50:53.910447+0000","last_fullsized":"2024-09-13T14:50:53.910447+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":381,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921934+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:50:53.921934+0000","last_peered":"2024-09-13T14:50:53.921934+0000","last_clean":"2024-09-13T14:50:53.921934+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:50:53.921934+0000","last_undegraded":"2024-09-13T14:50:53.921934+0000","last_fullsized":"2024-09-13T14:50:53.921934+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":16,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.284443+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:50:36.284443+0000","last_peered":"2024-09-13T14:50:36.284443+0000","last_clean":"2024-09-13T14:50:36.284443+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:50:36.284443+0000","last_undegraded":"2024-09-13T14:50:36.284443+0000","last_fullsized":"2024-09-13T14:50:36.284443+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"259'13678","reported_seq":13750,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:05.509354+0000","last_change":"2024-09-13T14:47:20.049747+0000","last_active":"2024-09-13T14:51:05.509354+0000","last_peered":"2024-09-13T14:51:05.509354+0000","last_clean":"2024-09-13T14:51:05.509354+0000","last_became_active":"2024-09-13T14:47:20.049448+0000","last_became_peered":"2024-09-13T14:47:20.049448+0000","last_unstale":"2024-09-13T14:51:05.509354+0000","last_undegraded":"2024-09-13T14:51:05.509354+0000","last_fullsized":"2024-09-13T14:51:05.509354+0000","mapping_epoch":229,"log_start":"251'6700","ondisk_log_start":"251'6700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:11.585800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68550656,"num_objects":1046,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13678,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":405,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921021+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:50:53.921021+0000","last_peered":"2024-09-13T14:50:53.921021+0000","last_clean":"2024-09-13T14:50:53.921021+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:50:53.921021+0000","last_undegraded":"2024-09-13T14:50:53.921021+0000","last_fullsized":"2024-09-13T14:50:53.921021+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":414,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921890+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:50:53.921890+0000","last_peered":"2024-09-13T14:50:53.921890+0000","last_clean":"2024-09-13T14:50:53.921890+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:50:53.921890+0000","last_undegraded":"2024-09-13T14:50:53.921890+0000","last_fullsized":"2024-09-13T14:50:53.921890+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":271,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.284860+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:50:36.284860+0000","last_peered":"2024-09-13T14:50:36.284860+0000","last_clean":"2024-09-13T14:50:36.284860+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:50:36.284860+0000","last_undegraded":"2024-09-13T14:50:36.284860+0000","last_fullsized":"2024-09-13T14:50:36.284860+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":195,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.275101+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:50:36.275101+0000","last_peered":"2024-09-13T14:50:36.275101+0000","last_clean":"2024-09-13T14:50:36.275101+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:50:36.275101+0000","last_undegraded":"2024-09-13T14:50:36.275101+0000","last_fullsized":"2024-09-13T14:50:36.275101+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"259'13551","reported_seq":13589,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:03.984261+0000","last_change":"2024-09-13T14:47:20.106371+0000","last_active":"2024-09-13T14:51:03.984261+0000","last_peered":"2024-09-13T14:51:03.984261+0000","last_clean":"2024-09-13T14:51:03.984261+0000","last_became_active":"2024-09-13T14:47:20.105934+0000","last_became_peered":"2024-09-13T14:47:20.105934+0000","last_unstale":"2024-09-13T14:51:03.984261+0000","last_undegraded":"2024-09-13T14:51:03.984261+0000","last_fullsized":"2024-09-13T14:51:03.984261+0000","mapping_epoch":229,"log_start":"251'6800","ondisk_log_start":"251'6800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6751,"log_dups_size":0,"ondisk_log_size":6751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:54.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":3285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13551,"num_write_kb":468672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":204,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.274703+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:50:36.274703+0000","last_peered":"2024-09-13T14:50:36.274703+0000","last_clean":"2024-09-13T14:50:36.274703+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:50:36.274703+0000","last_undegraded":"2024-09-13T14:50:36.274703+0000","last_fullsized":"2024-09-13T14:50:36.274703+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":313,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925050+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:50:53.925050+0000","last_peered":"2024-09-13T14:50:53.925050+0000","last_clean":"2024-09-13T14:50:53.925050+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:50:53.925050+0000","last_undegraded":"2024-09-13T14:50:53.925050+0000","last_fullsized":"2024-09-13T14:50:53.925050+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":232,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.278383+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:50:36.278383+0000","last_peered":"2024-09-13T14:50:36.278383+0000","last_clean":"2024-09-13T14:50:36.278383+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:50:36.278383+0000","last_undegraded":"2024-09-13T14:50:36.278383+0000","last_fullsized":"2024-09-13T14:50:36.278383+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":16,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.284489+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:50:36.284489+0000","last_peered":"2024-09-13T14:50:36.284489+0000","last_clean":"2024-09-13T14:50:36.284489+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:50:36.284489+0000","last_undegraded":"2024-09-13T14:50:36.284489+0000","last_fullsized":"2024-09-13T14:50:36.284489+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"259'13346","reported_seq":13384,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:04.088098+0000","last_change":"2024-09-13T14:47:20.050773+0000","last_active":"2024-09-13T14:51:04.088098+0000","last_peered":"2024-09-13T14:51:04.088098+0000","last_clean":"2024-09-13T14:51:04.088098+0000","last_became_active":"2024-09-13T14:47:20.050243+0000","last_became_peered":"2024-09-13T14:47:20.050243+0000","last_unstale":"2024-09-13T14:51:04.088098+0000","last_undegraded":"2024-09-13T14:51:04.088098+0000","last_fullsized":"2024-09-13T14:51:04.088098+0000","mapping_epoch":229,"log_start":"251'7000","ondisk_log_start":"251'7000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:42.724078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71827456,"num_objects":1096,"num_object_clones":0,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13346,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":37,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.911147+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:50:53.911147+0000","last_peered":"2024-09-13T14:50:53.911147+0000","last_clean":"2024-09-13T14:50:53.911147+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:50:53.911147+0000","last_undegraded":"2024-09-13T14:50:53.911147+0000","last_fullsized":"2024-09-13T14:50:53.911147+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":324,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925420+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:50:53.925420+0000","last_peered":"2024-09-13T14:50:53.925420+0000","last_clean":"2024-09-13T14:50:53.925420+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:50:53.925420+0000","last_undegraded":"2024-09-13T14:50:53.925420+0000","last_fullsized":"2024-09-13T14:50:53.925420+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":42,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.276928+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:50:36.276928+0000","last_peered":"2024-09-13T14:50:36.276928+0000","last_clean":"2024-09-13T14:50:36.276928+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:50:36.276928+0000","last_undegraded":"2024-09-13T14:50:36.276928+0000","last_fullsized":"2024-09-13T14:50:36.276928+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"259'13369","reported_seq":13407,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:04.015691+0000","last_change":"2024-09-13T14:47:20.106351+0000","last_active":"2024-09-13T14:51:04.015691+0000","last_peered":"2024-09-13T14:51:04.015691+0000","last_clean":"2024-09-13T14:51:04.015691+0000","last_became_active":"2024-09-13T14:47:20.105946+0000","last_became_peered":"2024-09-13T14:47:20.105946+0000","last_unstale":"2024-09-13T14:51:04.015691+0000","last_undegraded":"2024-09-13T14:51:04.015691+0000","last_fullsized":"2024-09-13T14:51:04.015691+0000","mapping_epoch":229,"log_start":"251'6700","ondisk_log_start":"251'6700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6669,"log_dups_size":0,"ondisk_log_size":6669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:07.223637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70582272,"num_objects":1077,"num_object_clones":0,"num_object_copies":3231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13369,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":241,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.272661+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:50:36.272661+0000","last_peered":"2024-09-13T14:50:36.272661+0000","last_clean":"2024-09-13T14:50:36.272661+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:50:36.272661+0000","last_undegraded":"2024-09-13T14:50:36.272661+0000","last_fullsized":"2024-09-13T14:50:36.272661+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":222,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.269065+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:50:36.269065+0000","last_peered":"2024-09-13T14:50:36.269065+0000","last_clean":"2024-09-13T14:50:36.269065+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:50:36.269065+0000","last_undegraded":"2024-09-13T14:50:36.269065+0000","last_fullsized":"2024-09-13T14:50:36.269065+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":375,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.922200+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:50:53.922200+0000","last_peered":"2024-09-13T14:50:53.922200+0000","last_clean":"2024-09-13T14:50:53.922200+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:50:53.922200+0000","last_undegraded":"2024-09-13T14:50:53.922200+0000","last_fullsized":"2024-09-13T14:50:53.922200+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"259'13567","reported_seq":17148,"reported_epoch":259,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:04.310197+0000","last_change":"2024-09-13T14:49:05.871518+0000","last_active":"2024-09-13T14:51:04.310197+0000","last_peered":"2024-09-13T14:51:04.310197+0000","last_clean":"2024-09-13T14:48:09.848772+0000","last_became_active":"2024-09-13T14:48:12.232150+0000","last_became_peered":"2024-09-13T14:48:12.232150+0000","last_unstale":"2024-09-13T14:51:04.310197+0000","last_undegraded":"2024-09-13T14:49:05.858877+0000","last_fullsized":"2024-09-13T14:48:11.643512+0000","mapping_epoch":242,"log_start":"247'5400","ondisk_log_start":"247'5400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8167,"log_dups_size":0,"ondisk_log_size":8167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68747264,"num_objects":1049,"num_object_clones":0,"num_object_copies":3147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13567,"num_write_kb":467712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":114753536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,5","objects":-1378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":238,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.269306+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:50:36.269306+0000","last_peered":"2024-09-13T14:50:36.269306+0000","last_clean":"2024-09-13T14:50:36.269306+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:50:36.269306+0000","last_undegraded":"2024-09-13T14:50:36.269306+0000","last_fullsized":"2024-09-13T14:50:36.269306+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":242,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.274766+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:50:36.274766+0000","last_peered":"2024-09-13T14:50:36.274766+0000","last_clean":"2024-09-13T14:50:36.274766+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:50:36.274766+0000","last_undegraded":"2024-09-13T14:50:36.274766+0000","last_fullsized":"2024-09-13T14:50:36.274766+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":24,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.276613+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:50:36.276613+0000","last_peered":"2024-09-13T14:50:36.276613+0000","last_clean":"2024-09-13T14:50:36.276613+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:50:36.276613+0000","last_undegraded":"2024-09-13T14:50:36.276613+0000","last_fullsized":"2024-09-13T14:50:36.276613+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"259'13629","reported_seq":13665,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:04.154359+0000","last_change":"2024-09-13T14:47:20.106142+0000","last_active":"2024-09-13T14:51:04.154359+0000","last_peered":"2024-09-13T14:51:04.154359+0000","last_clean":"2024-09-13T14:51:04.154359+0000","last_became_active":"2024-09-13T14:47:20.105758+0000","last_became_peered":"2024-09-13T14:47:20.105758+0000","last_unstale":"2024-09-13T14:51:04.154359+0000","last_undegraded":"2024-09-13T14:51:04.154359+0000","last_fullsized":"2024-09-13T14:51:04.154359+0000","mapping_epoch":229,"log_start":"253'7300","ondisk_log_start":"253'7300","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:51.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73859072,"num_objects":1127,"num_object_clones":0,"num_object_copies":3381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13629,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":221,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.270715+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:50:36.270715+0000","last_peered":"2024-09-13T14:50:36.270715+0000","last_clean":"2024-09-13T14:50:36.270715+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:50:36.270715+0000","last_undegraded":"2024-09-13T14:50:36.270715+0000","last_fullsized":"2024-09-13T14:50:36.270715+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":354,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925628+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:50:53.925628+0000","last_peered":"2024-09-13T14:50:53.925628+0000","last_clean":"2024-09-13T14:50:53.925628+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:50:53.925628+0000","last_undegraded":"2024-09-13T14:50:53.925628+0000","last_fullsized":"2024-09-13T14:50:53.925628+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":42,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.287290+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:50:36.287290+0000","last_peered":"2024-09-13T14:50:36.287290+0000","last_clean":"2024-09-13T14:50:36.287290+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:50:36.287290+0000","last_undegraded":"2024-09-13T14:50:36.287290+0000","last_fullsized":"2024-09-13T14:50:36.287290+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"259'13521","reported_seq":13593,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:04.358873+0000","last_change":"2024-09-13T14:47:20.107374+0000","last_active":"2024-09-13T14:51:04.358873+0000","last_peered":"2024-09-13T14:51:04.358873+0000","last_clean":"2024-09-13T14:51:04.358873+0000","last_became_active":"2024-09-13T14:47:20.107057+0000","last_became_peered":"2024-09-13T14:47:20.107057+0000","last_unstale":"2024-09-13T14:51:04.358873+0000","last_undegraded":"2024-09-13T14:51:04.358873+0000","last_fullsized":"2024-09-13T14:51:04.358873+0000","mapping_epoch":229,"log_start":"247'5400","ondisk_log_start":"247'5400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8121,"log_dups_size":0,"ondisk_log_size":8121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:05.814456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72548352,"num_objects":1107,"num_object_clones":0,"num_object_copies":3321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13521,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":393,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.922245+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:50:53.922245+0000","last_peered":"2024-09-13T14:50:53.922245+0000","last_clean":"2024-09-13T14:50:53.922245+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:50:53.922245+0000","last_undegraded":"2024-09-13T14:50:53.922245+0000","last_fullsized":"2024-09-13T14:50:53.922245+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":220,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.277554+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:50:36.277554+0000","last_peered":"2024-09-13T14:50:36.277554+0000","last_clean":"2024-09-13T14:50:36.277554+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:50:36.277554+0000","last_undegraded":"2024-09-13T14:50:36.277554+0000","last_fullsized":"2024-09-13T14:50:36.277554+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":42,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.272492+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:50:36.272492+0000","last_peered":"2024-09-13T14:50:36.272492+0000","last_clean":"2024-09-13T14:50:36.272492+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:50:36.272492+0000","last_undegraded":"2024-09-13T14:50:36.272492+0000","last_fullsized":"2024-09-13T14:50:36.272492+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"259'13468","reported_seq":13540,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:05.473670+0000","last_change":"2024-09-13T14:47:20.108173+0000","last_active":"2024-09-13T14:51:05.473670+0000","last_peered":"2024-09-13T14:51:05.473670+0000","last_clean":"2024-09-13T14:51:05.473670+0000","last_became_active":"2024-09-13T14:47:20.107879+0000","last_became_peered":"2024-09-13T14:47:20.107879+0000","last_unstale":"2024-09-13T14:51:05.473670+0000","last_undegraded":"2024-09-13T14:51:05.473670+0000","last_fullsized":"2024-09-13T14:51:05.473670+0000","mapping_epoch":229,"log_start":"250'6400","ondisk_log_start":"250'6400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7068,"log_dups_size":0,"ondisk_log_size":7068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:50.152706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67764224,"num_objects":1034,"num_object_clones":0,"num_object_copies":3102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13468,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":234,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.274631+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:50:36.274631+0000","last_peered":"2024-09-13T14:50:36.274631+0000","last_clean":"2024-09-13T14:50:36.274631+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:50:36.274631+0000","last_undegraded":"2024-09-13T14:50:36.274631+0000","last_fullsized":"2024-09-13T14:50:36.274631+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":235,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.285712+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:50:36.285712+0000","last_peered":"2024-09-13T14:50:36.285712+0000","last_clean":"2024-09-13T14:50:36.285712+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:50:36.285712+0000","last_undegraded":"2024-09-13T14:50:36.285712+0000","last_fullsized":"2024-09-13T14:50:36.285712+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":195,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.275615+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:50:36.275615+0000","last_peered":"2024-09-13T14:50:36.275615+0000","last_clean":"2024-09-13T14:50:36.275615+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:50:36.275615+0000","last_undegraded":"2024-09-13T14:50:36.275615+0000","last_fullsized":"2024-09-13T14:50:36.275615+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"259'13373","reported_seq":13413,"reported_epoch":259,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:04.000670+0000","last_change":"2024-09-13T14:48:40.078958+0000","last_active":"2024-09-13T14:51:04.000670+0000","last_peered":"2024-09-13T14:51:04.000670+0000","last_clean":"2024-09-13T14:48:37.811111+0000","last_became_active":"2024-09-13T14:48:40.052181+0000","last_became_peered":"2024-09-13T14:48:40.052181+0000","last_unstale":"2024-09-13T14:51:04.000670+0000","last_undegraded":"2024-09-13T14:48:39.849823+0000","last_fullsized":"2024-09-13T14:48:39.826167+0000","mapping_epoch":249,"log_start":"251'6700","ondisk_log_start":"251'6700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:49.521146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73203712,"num_objects":1117,"num_object_clones":0,"num_object_copies":3351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13373,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"3,6","objects":-4632}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":248,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.268925+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:50:36.268925+0000","last_peered":"2024-09-13T14:50:36.268925+0000","last_clean":"2024-09-13T14:50:36.268925+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:50:36.268925+0000","last_undegraded":"2024-09-13T14:50:36.268925+0000","last_fullsized":"2024-09-13T14:50:36.268925+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":225,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.285176+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:50:36.285176+0000","last_peered":"2024-09-13T14:50:36.285176+0000","last_clean":"2024-09-13T14:50:36.285176+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:50:36.285176+0000","last_undegraded":"2024-09-13T14:50:36.285176+0000","last_fullsized":"2024-09-13T14:50:36.285176+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":81,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925915+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:50:53.925915+0000","last_peered":"2024-09-13T14:50:53.925915+0000","last_clean":"2024-09-13T14:50:53.925915+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:50:53.925915+0000","last_undegraded":"2024-09-13T14:50:53.925915+0000","last_fullsized":"2024-09-13T14:50:53.925915+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"259'13708","reported_seq":13746,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:51:04.716938+0000","last_change":"2024-09-13T14:47:20.051824+0000","last_active":"2024-09-13T14:51:04.716938+0000","last_peered":"2024-09-13T14:51:04.716938+0000","last_clean":"2024-09-13T14:51:04.716938+0000","last_became_active":"2024-09-13T14:47:20.051487+0000","last_became_peered":"2024-09-13T14:47:20.051487+0000","last_unstale":"2024-09-13T14:51:04.716938+0000","last_undegraded":"2024-09-13T14:51:04.716938+0000","last_fullsized":"2024-09-13T14:51:04.716938+0000","mapping_epoch":229,"log_start":"247'5500","ondisk_log_start":"247'5500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8208,"log_dups_size":0,"ondisk_log_size":8208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:11.533543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71172096,"num_objects":1086,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13708,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.276714+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:50:36.276714+0000","last_peered":"2024-09-13T14:50:36.276714+0000","last_clean":"2024-09-13T14:50:36.276714+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:50:36.276714+0000","last_undegraded":"2024-09-13T14:50:36.276714+0000","last_fullsized":"2024-09-13T14:50:36.276714+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":20,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.285402+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:50:36.285402+0000","last_peered":"2024-09-13T14:50:36.285402+0000","last_clean":"2024-09-13T14:50:36.285402+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:50:36.285402+0000","last_undegraded":"2024-09-13T14:50:36.285402+0000","last_fullsized":"2024-09-13T14:50:36.285402+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":16,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.285551+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:50:36.285551+0000","last_peered":"2024-09-13T14:50:36.285551+0000","last_clean":"2024-09-13T14:50:36.285551+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:50:36.285551+0000","last_undegraded":"2024-09-13T14:50:36.285551+0000","last_fullsized":"2024-09-13T14:50:36.285551+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":270,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.275340+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:50:36.275340+0000","last_peered":"2024-09-13T14:50:36.275340+0000","last_clean":"2024-09-13T14:50:36.275340+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:50:36.275340+0000","last_undegraded":"2024-09-13T14:50:36.275340+0000","last_fullsized":"2024-09-13T14:50:36.275340+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":362,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.928480+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:50:53.928480+0000","last_peered":"2024-09-13T14:50:53.928480+0000","last_clean":"2024-09-13T14:50:53.928480+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:50:53.928480+0000","last_undegraded":"2024-09-13T14:50:53.928480+0000","last_fullsized":"2024-09-13T14:50:53.928480+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":52,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.276815+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:50:36.276815+0000","last_peered":"2024-09-13T14:50:36.276815+0000","last_clean":"2024-09-13T14:50:36.276815+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:50:36.276815+0000","last_undegraded":"2024-09-13T14:50:36.276815+0000","last_fullsized":"2024-09-13T14:50:36.276815+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":216,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.287437+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:50:36.287437+0000","last_peered":"2024-09-13T14:50:36.287437+0000","last_clean":"2024-09-13T14:50:36.287437+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:50:36.287437+0000","last_undegraded":"2024-09-13T14:50:36.287437+0000","last_fullsized":"2024-09-13T14:50:36.287437+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":407,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921140+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:50:53.921140+0000","last_peered":"2024-09-13T14:50:53.921140+0000","last_clean":"2024-09-13T14:50:53.921140+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:50:53.921140+0000","last_undegraded":"2024-09-13T14:50:53.921140+0000","last_fullsized":"2024-09-13T14:50:53.921140+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":77,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925465+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:50:53.925465+0000","last_peered":"2024-09-13T14:50:53.925465+0000","last_clean":"2024-09-13T14:50:53.925465+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:50:53.925465+0000","last_undegraded":"2024-09-13T14:50:53.925465+0000","last_fullsized":"2024-09-13T14:50:53.925465+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":413,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.922764+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:50:53.922764+0000","last_peered":"2024-09-13T14:50:53.922764+0000","last_clean":"2024-09-13T14:50:53.922764+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:50:53.922764+0000","last_undegraded":"2024-09-13T14:50:53.922764+0000","last_fullsized":"2024-09-13T14:50:53.922764+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":20,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.277278+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:50:36.277278+0000","last_peered":"2024-09-13T14:50:36.277278+0000","last_clean":"2024-09-13T14:50:36.277278+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:50:36.277278+0000","last_undegraded":"2024-09-13T14:50:36.277278+0000","last_fullsized":"2024-09-13T14:50:36.277278+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":42,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.276800+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:50:36.276800+0000","last_peered":"2024-09-13T14:50:36.276800+0000","last_clean":"2024-09-13T14:50:36.276800+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:50:36.276800+0000","last_undegraded":"2024-09-13T14:50:36.276800+0000","last_fullsized":"2024-09-13T14:50:36.276800+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":241,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.284620+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:50:36.284620+0000","last_peered":"2024-09-13T14:50:36.284620+0000","last_clean":"2024-09-13T14:50:36.284620+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:50:36.284620+0000","last_undegraded":"2024-09-13T14:50:36.284620+0000","last_fullsized":"2024-09-13T14:50:36.284620+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":72,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921378+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:50:53.921378+0000","last_peered":"2024-09-13T14:50:53.921378+0000","last_clean":"2024-09-13T14:50:53.921378+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:50:53.921378+0000","last_undegraded":"2024-09-13T14:50:53.921378+0000","last_fullsized":"2024-09-13T14:50:53.921378+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":371,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921744+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:50:53.921744+0000","last_peered":"2024-09-13T14:50:53.921744+0000","last_clean":"2024-09-13T14:50:53.921744+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:50:53.921744+0000","last_undegraded":"2024-09-13T14:50:53.921744+0000","last_fullsized":"2024-09-13T14:50:53.921744+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1139081240,"num_objects":17382,"num_object_clones":0,"num_object_copies":52146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":216072,"num_write_kb":7470465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4561,"num_bytes_recovered":282918912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4461965312,"data_stored":4461953096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112858,"ondisk_log_size":112858,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365176,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1197924,"kb_used_data":292700,"kb_used_omap":1,"kb_used_meta":905214,"kb_avail":92547228,"statfs":{"total":95995035648,"available":94768361472,"internally_reserved":0,"allocated":299724800,"data_stored":298355568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":926939480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561406,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1404896,"kb_used_data":511320,"kb_used_omap":1,"kb_used_meta":893502,"kb_avail":92340256,"statfs":{"total":95995035648,"available":94556422144,"internally_reserved":0,"allocated":523591680,"data_stored":522226391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914946392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,9,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790338,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1247188,"kb_used_data":346124,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":92497964,"statfs":{"total":95995035648,"available":94717915136,"internally_reserved":0,"allocated":354430976,"data_stored":353074069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986566,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2109668,"kb_used_data":1206876,"kb_used_omap":1,"kb_used_meta":902718,"kb_avail":91635484,"statfs":{"total":95995035648,"available":93834735616,"internally_reserved":0,"allocated":1235841024,"data_stored":1234471977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924383576},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519721,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":909396,"kb_used_data":375116,"kb_used_omap":4,"kb_used_meta":534267,"kb_avail":92835756,"statfs":{"total":95995035648,"available":95063814144,"internally_reserved":0,"allocated":384118784,"data_stored":382761887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":547089856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304637,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1301336,"kb_used_data":804944,"kb_used_omap":6,"kb_used_meta":496377,"kb_avail":92443816,"statfs":{"total":95995035648,"available":94662467584,"internally_reserved":0,"allocated":824262656,"data_stored":822902177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":508290503},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827940,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1031188,"kb_used_data":231564,"kb_used_omap":5,"kb_used_meta":799610,"kb_avail":92713964,"statfs":{"total":95995035648,"available":94939099136,"internally_reserved":0,"allocated":237121536,"data_stored":235764066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":818800876},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285805,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1390932,"kb_used_data":679564,"kb_used_omap":5,"kb_used_meta":711354,"kb_avail":92354220,"statfs":{"total":95995035648,"available":94570721280,"internally_reserved":0,"allocated":695873536,"data_stored":694515393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":728426555},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":679022592,"data_stored":679018520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233832448,"data_stored":233832448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":800522240,"data_stored":800522240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":380043264,"data_stored":380043264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1225527296,"data_stored":1225523224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":350683136,"data_stored":350683136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":500240384,"data_stored":500236312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":292093952,"data_stored":292093952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:11.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:11.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:11.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:11.962 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:11.999 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:12.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:12.140 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:12.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:12.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:12.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:12.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:12.547 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:12.579 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:12.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:12.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:12.818 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:12.850 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:12.926 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:12.953 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:12.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:13.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:13.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:13 smithi003 ceph-mon[107278]: pgmap v897: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 825 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 10918/39672 objects degraded (27.521%); 0 B/s, 35 objects/s recovering 2024-09-13T14:51:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:13.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:13.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:13 smithi003 ceph-mon[104007]: pgmap v897: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 825 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 10918/39672 objects degraded (27.521%); 0 B/s, 35 objects/s recovering 2024-09-13T14:51:13.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:13.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:13.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:13.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:13.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:13 smithi019 ceph-mon[91296]: pgmap v897: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 825 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 10918/39672 objects degraded (27.521%); 0 B/s, 35 objects/s recovering 2024-09-13T14:51:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:13.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:13.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:13.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:13.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:13.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:13.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:13.975 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:14.073 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:14.073 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:14.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:14.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:14.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:14.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:14.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:14.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:14.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:14.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:14.413 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:51:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:51:14] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:51:14.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:14.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:14.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:14.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:14.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:14.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:14.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:14.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:14.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:51:14.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:51:14.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:15.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:15.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:15.132 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:15.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:15 smithi003 ceph-mon[107278]: pgmap v898: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 825 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 10918/39672 objects degraded (27.521%); 0 B/s, 35 objects/s recovering 2024-09-13T14:51:15.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:15 smithi003 ceph-mon[104007]: pgmap v898: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 825 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 10918/39672 objects degraded (27.521%); 0 B/s, 35 objects/s recovering 2024-09-13T14:51:15.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:15.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:15.392 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:15.425 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:15.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:15 smithi019 ceph-mon[91296]: pgmap v898: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 825 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 10918/39672 objects degraded (27.521%); 0 B/s, 35 objects/s recovering 2024-09-13T14:51:15.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:15.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:15.586 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:15.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:15.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:15.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:15.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:15.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.020 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:16.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:16.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:16.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:16.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:16.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10918/39672 objects degraded (27.521%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:16.260 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10918/39672 objects degraded (27.521%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:16.282 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:16.317 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:16.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:16.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10918/39672 objects degraded (27.521%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:16.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:16.558 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:16.600 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:16.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:16.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:16.796 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:51:16.796 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:51:16.796 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:51:16.796 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:51:16.796 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:51:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:16.797 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:16.797 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:16.797 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:16.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:51:16.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:51:16.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:51:16.797 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:16.797 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:16.797 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:16.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:16.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:16.977 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:17.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:17.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:17.016 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:17.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:17.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:17 smithi003 ceph-mon[107278]: pgmap v899: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 539 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 10739/25926 objects degraded (41.422%); 0 B/s, 38 objects/s recovering 2024-09-13T14:51:17.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:17 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3708501818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:17.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:17 smithi003 ceph-mon[104007]: pgmap v899: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 539 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 10739/25926 objects degraded (41.422%); 0 B/s, 38 objects/s recovering 2024-09-13T14:51:17.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:17 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3708501818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:17.155 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:17.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:17.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.291 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:17.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:17.392 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:17.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:17 smithi019 ceph-mon[91296]: pgmap v899: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 539 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 10739/25926 objects degraded (41.422%); 0 B/s, 38 objects/s recovering 2024-09-13T14:51:17.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:17 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3708501818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:17.442 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:17.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:17.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:17.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:17.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:17.837 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.873 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:17.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:17.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:18.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:18.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:18.154 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:18.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:18.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:18.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:18.402 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.446 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:18.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:18.547 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:18.584 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:18.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:18.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:18.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:18.862 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:18.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:19.103 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.149 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:19.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:19.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:19.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:19.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:19 smithi003 ceph-mon[107278]: pgmap v900: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 539 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 10739/25926 objects degraded (41.422%); 0 B/s, 38 objects/s recovering 2024-09-13T14:51:19.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:19 smithi003 ceph-mon[104007]: pgmap v900: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 539 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 10739/25926 objects degraded (41.422%); 0 B/s, 38 objects/s recovering 2024-09-13T14:51:19.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:19 smithi019 ceph-mon[91296]: pgmap v900: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 539 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 10739/25926 objects degraded (41.422%); 0 B/s, 38 objects/s recovering 2024-09-13T14:51:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:19.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:19.536 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:19.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:51:19.581 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:19.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:19.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:19.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:19.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:19.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:19.970 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:20.005 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:20.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:20.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:20.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:20.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:20.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:20.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:20.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:20.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:20.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:51:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:51:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:20.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:20.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:20.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:21.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:21.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:21.237 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:21.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:21.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:21 smithi003 ceph-mon[104007]: pgmap v901: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 295 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 901 op/s; 10548/14202 objects degraded (74.271%); 0 B/s, 36 objects/s recovering 2024-09-13T14:51:21.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10548/14202 objects degraded (74.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:21.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:21 smithi003 ceph-mon[107278]: pgmap v901: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 295 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 901 op/s; 10548/14202 objects degraded (74.271%); 0 B/s, 36 objects/s recovering 2024-09-13T14:51:21.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10548/14202 objects degraded (74.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:21.338 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:21.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.410 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:21.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:21 smithi019 ceph-mon[91296]: pgmap v901: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 295 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 901 op/s; 10548/14202 objects degraded (74.271%); 0 B/s, 36 objects/s recovering 2024-09-13T14:51:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10548/14202 objects degraded (74.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:21.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:21.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:21.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:21.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:21.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:22.050 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:51:22.052 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:51:22.065 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:51:22.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:22.109 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:22.211 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:22.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.252 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:22.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:22.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:22.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2584892423' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:22 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:22.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2584892423' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:22.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:22 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:22.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2584892423' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:22.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:22 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:22.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:22.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:22.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:22.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:22.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:22.817 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:22.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:22.886 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:51:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:51:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:51:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:51:22.887 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:51:22.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:22.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:23.084 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:23.120 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[107278]: pgmap v902: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 259 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 10548/12477 objects degraded (84.540%); 0 B/s, 31 objects/s recovering 2024-09-13T14:51:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[107278]: from='client.36369 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[107278]: osdmap e261: 8 total, 8 up, 8 in 2024-09-13T14:51:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[104007]: pgmap v902: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 259 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 10548/12477 objects degraded (84.540%); 0 B/s, 31 objects/s recovering 2024-09-13T14:51:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[104007]: from='client.36369 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:23.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:23.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:23.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:23 smithi003 ceph-mon[104007]: osdmap e261: 8 total, 8 up, 8 in 2024-09-13T14:51:23.221 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:23.261 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:23.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:23.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.399 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:23.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:23 smithi019 ceph-mon[91296]: pgmap v902: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 259 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 10548/12477 objects degraded (84.540%); 0 B/s, 31 objects/s recovering 2024-09-13T14:51:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:23 smithi019 ceph-mon[91296]: from='client.36369 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:23 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:23 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:23 smithi019 ceph-mon[91296]: osdmap e261: 8 total, 8 up, 8 in 2024-09-13T14:51:23.501 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.536 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:23.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:23.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:23.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:23.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:23.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:51:23.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:23.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:24.049 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 116726 objects 2024-09-13T14:51:24.049 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :134.947 2024-09-13T14:51:24.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.086 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:24.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:24.187 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:24.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:24.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:24.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:24.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:51:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:51:24] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:51:24.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:24.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:24.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:24.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:24.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:24.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:24.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:24.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:24.952 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":902,"stamp":"2024-09-13T14:51:21.831501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":271417528,"num_objects":4159,"num_object_clones":0,"num_object_copies":12477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":15,"num_read_kb":35,"num_write":229392,"num_write_kb":7470571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5156,"num_bytes_recovered":283799089,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114766,"ondisk_log_size":114766,"up":327,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":333,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8100880,"kb_used_data":1925840,"kb_used_omap":29,"kb_used_meta":6174626,"kb_avail":741860336,"statfs":{"total":767960285184,"available":759664984064,"internally_reserved":0,"allocated":1972060160,"data_stored":1961104624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6322817550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,3,7,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-628490240,"num_objects":-9590,"num_object_clones":0,"num_object_copies":-28770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9590,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":370,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.834206"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":399,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.004160+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:51:10.004160+0000","last_peered":"2024-09-13T14:51:10.004160+0000","last_clean":"2024-09-13T14:51:10.004160+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:51:10.004160+0000","last_undegraded":"2024-09-13T14:51:10.004160+0000","last_fullsized":"2024-09-13T14:51:10.004160+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":232,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.923935+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:50:53.923935+0000","last_peered":"2024-09-13T14:50:53.923935+0000","last_clean":"2024-09-13T14:50:53.923935+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:50:53.923935+0000","last_undegraded":"2024-09-13T14:50:53.923935+0000","last_fullsized":"2024-09-13T14:50:53.923935+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":23,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919426+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:50:53.919426+0000","last_peered":"2024-09-13T14:50:53.919426+0000","last_clean":"2024-09-13T14:50:53.919426+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:50:53.919426+0000","last_undegraded":"2024-09-13T14:50:53.919426+0000","last_fullsized":"2024-09-13T14:50:53.919426+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":357,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.014693+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:51:10.014693+0000","last_peered":"2024-09-13T14:51:10.014693+0000","last_clean":"2024-09-13T14:51:10.014693+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:51:10.014693+0000","last_undegraded":"2024-09-13T14:51:10.014693+0000","last_fullsized":"2024-09-13T14:51:10.014693+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":206,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918565+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:50:53.918565+0000","last_peered":"2024-09-13T14:50:53.918565+0000","last_clean":"2024-09-13T14:50:53.918565+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:50:53.918565+0000","last_undegraded":"2024-09-13T14:50:53.918565+0000","last_fullsized":"2024-09-13T14:50:53.918565+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":239,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.924147+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:50:53.924147+0000","last_peered":"2024-09-13T14:50:53.924147+0000","last_clean":"2024-09-13T14:50:53.924147+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:50:53.924147+0000","last_undegraded":"2024-09-13T14:50:53.924147+0000","last_fullsized":"2024-09-13T14:50:53.924147+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":223,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918836+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:50:53.918836+0000","last_peered":"2024-09-13T14:50:53.918836+0000","last_clean":"2024-09-13T14:50:53.918836+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:50:53.918836+0000","last_undegraded":"2024-09-13T14:50:53.918836+0000","last_fullsized":"2024-09-13T14:50:53.918836+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":269,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.924380+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:50:53.924380+0000","last_peered":"2024-09-13T14:50:53.924380+0000","last_clean":"2024-09-13T14:50:53.924380+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:50:53.924380+0000","last_undegraded":"2024-09-13T14:50:53.924380+0000","last_fullsized":"2024-09-13T14:50:53.924380+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":386,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.005696+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:51:10.005696+0000","last_peered":"2024-09-13T14:51:10.005696+0000","last_clean":"2024-09-13T14:51:10.005696+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:51:10.005696+0000","last_undegraded":"2024-09-13T14:51:10.005696+0000","last_fullsized":"2024-09-13T14:51:10.005696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":382,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.998958+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:51:09.998958+0000","last_peered":"2024-09-13T14:51:09.998958+0000","last_clean":"2024-09-13T14:51:09.998958+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:51:09.998958+0000","last_undegraded":"2024-09-13T14:51:09.998958+0000","last_fullsized":"2024-09-13T14:51:09.998958+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":241,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.924286+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:50:53.924286+0000","last_peered":"2024-09-13T14:50:53.924286+0000","last_clean":"2024-09-13T14:50:53.924286+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:50:53.924286+0000","last_undegraded":"2024-09-13T14:50:53.924286+0000","last_fullsized":"2024-09-13T14:50:53.924286+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":324,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.018087+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:51:10.018087+0000","last_peered":"2024-09-13T14:51:10.018087+0000","last_clean":"2024-09-13T14:51:10.018087+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:51:10.018087+0000","last_undegraded":"2024-09-13T14:51:10.018087+0000","last_fullsized":"2024-09-13T14:51:10.018087+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.999932+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:51:09.999932+0000","last_peered":"2024-09-13T14:51:09.999932+0000","last_clean":"2024-09-13T14:51:09.999932+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:51:09.999932+0000","last_undegraded":"2024-09-13T14:51:09.999932+0000","last_fullsized":"2024-09-13T14:51:09.999932+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":417,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.004647+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:51:10.004647+0000","last_peered":"2024-09-13T14:51:10.004647+0000","last_clean":"2024-09-13T14:51:10.004647+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:51:10.004647+0000","last_undegraded":"2024-09-13T14:51:10.004647+0000","last_fullsized":"2024-09-13T14:51:10.004647+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":356,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.014446+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:51:10.014446+0000","last_peered":"2024-09-13T14:51:10.014446+0000","last_clean":"2024-09-13T14:51:10.014446+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:51:10.014446+0000","last_undegraded":"2024-09-13T14:51:10.014446+0000","last_fullsized":"2024-09-13T14:51:10.014446+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":222,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.922681+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:50:53.922681+0000","last_peered":"2024-09-13T14:50:53.922681+0000","last_clean":"2024-09-13T14:50:53.922681+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:50:53.922681+0000","last_undegraded":"2024-09-13T14:50:53.922681+0000","last_fullsized":"2024-09-13T14:50:53.922681+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":325,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.014281+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:51:10.014281+0000","last_peered":"2024-09-13T14:51:10.014281+0000","last_clean":"2024-09-13T14:51:10.014281+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:51:10.014281+0000","last_undegraded":"2024-09-13T14:51:10.014281+0000","last_fullsized":"2024-09-13T14:51:10.014281+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":401,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.004528+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:51:10.004528+0000","last_peered":"2024-09-13T14:51:10.004528+0000","last_clean":"2024-09-13T14:51:10.004528+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:51:10.004528+0000","last_undegraded":"2024-09-13T14:51:10.004528+0000","last_fullsized":"2024-09-13T14:51:10.004528+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":330,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.014468+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:51:10.014468+0000","last_peered":"2024-09-13T14:51:10.014468+0000","last_clean":"2024-09-13T14:51:10.014468+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:51:10.014468+0000","last_undegraded":"2024-09-13T14:51:10.014468+0000","last_fullsized":"2024-09-13T14:51:10.014468+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919344+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:50:53.919344+0000","last_peered":"2024-09-13T14:50:53.919344+0000","last_clean":"2024-09-13T14:50:53.919344+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:50:53.919344+0000","last_undegraded":"2024-09-13T14:50:53.919344+0000","last_fullsized":"2024-09-13T14:50:53.919344+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":79,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.998798+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:51:09.998798+0000","last_peered":"2024-09-13T14:51:09.998798+0000","last_clean":"2024-09-13T14:51:09.998798+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:51:09.998798+0000","last_undegraded":"2024-09-13T14:51:09.998798+0000","last_fullsized":"2024-09-13T14:51:09.998798+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":23,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.920975+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:50:53.920975+0000","last_peered":"2024-09-13T14:50:53.920975+0000","last_clean":"2024-09-13T14:50:53.920975+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:50:53.920975+0000","last_undegraded":"2024-09-13T14:50:53.920975+0000","last_fullsized":"2024-09-13T14:50:53.920975+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":161,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.920090+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:50:53.920090+0000","last_peered":"2024-09-13T14:50:53.920090+0000","last_clean":"2024-09-13T14:50:53.920090+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:50:53.920090+0000","last_undegraded":"2024-09-13T14:50:53.920090+0000","last_fullsized":"2024-09-13T14:50:53.920090+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":229,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.924352+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:50:53.924352+0000","last_peered":"2024-09-13T14:50:53.924352+0000","last_clean":"2024-09-13T14:50:53.924352+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:50:53.924352+0000","last_undegraded":"2024-09-13T14:50:53.924352+0000","last_fullsized":"2024-09-13T14:50:53.924352+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":222,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.924445+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:50:53.924445+0000","last_peered":"2024-09-13T14:50:53.924445+0000","last_clean":"2024-09-13T14:50:53.924445+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:50:53.924445+0000","last_undegraded":"2024-09-13T14:50:53.924445+0000","last_fullsized":"2024-09-13T14:50:53.924445+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"260'14198","reported_seq":14236,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:19.021781+0000","last_change":"2024-09-13T14:47:20.107455+0000","last_active":"2024-09-13T14:51:19.021781+0000","last_peered":"2024-09-13T14:51:19.021781+0000","last_clean":"2024-09-13T14:51:19.021781+0000","last_became_active":"2024-09-13T14:47:20.107142+0000","last_became_peered":"2024-09-13T14:47:20.107142+0000","last_unstale":"2024-09-13T14:51:19.021781+0000","last_undegraded":"2024-09-13T14:51:19.021781+0000","last_fullsized":"2024-09-13T14:51:19.021781+0000","mapping_epoch":229,"log_start":"253'7500","ondisk_log_start":"253'7500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6698,"log_dups_size":0,"ondisk_log_size":6698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:19.189519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17694720,"num_objects":270,"num_object_clones":0,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14198,"num_write_kb":462976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":205,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.926788+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:50:53.926788+0000","last_peered":"2024-09-13T14:50:53.926788+0000","last_clean":"2024-09-13T14:50:53.926788+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:50:53.926788+0000","last_undegraded":"2024-09-13T14:50:53.926788+0000","last_fullsized":"2024-09-13T14:50:53.926788+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":255,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.922812+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:50:53.922812+0000","last_peered":"2024-09-13T14:50:53.922812+0000","last_clean":"2024-09-13T14:50:53.922812+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:50:53.922812+0000","last_undegraded":"2024-09-13T14:50:53.922812+0000","last_fullsized":"2024-09-13T14:50:53.922812+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":399,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.004555+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:51:10.004555+0000","last_peered":"2024-09-13T14:51:10.004555+0000","last_clean":"2024-09-13T14:51:10.004555+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:51:10.004555+0000","last_undegraded":"2024-09-13T14:51:10.004555+0000","last_fullsized":"2024-09-13T14:51:10.004555+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":87,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.017592+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:51:10.017592+0000","last_peered":"2024-09-13T14:51:10.017592+0000","last_clean":"2024-09-13T14:51:10.017592+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:51:10.017592+0000","last_undegraded":"2024-09-13T14:51:10.017592+0000","last_fullsized":"2024-09-13T14:51:10.017592+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"260'14339","reported_seq":14377,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:19.205412+0000","last_change":"2024-09-13T14:47:20.054114+0000","last_active":"2024-09-13T14:51:19.205412+0000","last_peered":"2024-09-13T14:51:19.205412+0000","last_clean":"2024-09-13T14:51:19.205412+0000","last_became_active":"2024-09-13T14:47:20.053767+0000","last_became_peered":"2024-09-13T14:47:20.053767+0000","last_unstale":"2024-09-13T14:51:19.205412+0000","last_undegraded":"2024-09-13T14:51:19.205412+0000","last_fullsized":"2024-09-13T14:51:19.205412+0000","mapping_epoch":229,"log_start":"254'8000","ondisk_log_start":"254'8000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6339,"log_dups_size":0,"ondisk_log_size":6339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:48.035191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":263,"num_object_clones":0,"num_object_copies":789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14339,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":368,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.014371+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:51:10.014371+0000","last_peered":"2024-09-13T14:51:10.014371+0000","last_clean":"2024-09-13T14:51:10.014371+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:51:10.014371+0000","last_undegraded":"2024-09-13T14:51:10.014371+0000","last_fullsized":"2024-09-13T14:51:10.014371+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":96,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.920567+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:50:53.920567+0000","last_peered":"2024-09-13T14:50:53.920567+0000","last_clean":"2024-09-13T14:50:53.920567+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:50:53.920567+0000","last_undegraded":"2024-09-13T14:50:53.920567+0000","last_fullsized":"2024-09-13T14:50:53.920567+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":356,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.019022+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:51:10.019022+0000","last_peered":"2024-09-13T14:51:10.019022+0000","last_clean":"2024-09-13T14:51:10.019022+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:51:10.019022+0000","last_undegraded":"2024-09-13T14:51:10.019022+0000","last_fullsized":"2024-09-13T14:51:10.019022+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":43,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.924955+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:50:53.924955+0000","last_peered":"2024-09-13T14:50:53.924955+0000","last_clean":"2024-09-13T14:50:53.924955+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:50:53.924955+0000","last_undegraded":"2024-09-13T14:50:53.924955+0000","last_fullsized":"2024-09-13T14:50:53.924955+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"260'14343","reported_seq":14383,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:18.999999+0000","last_change":"2024-09-13T14:47:20.052876+0000","last_active":"2024-09-13T14:51:18.999999+0000","last_peered":"2024-09-13T14:51:18.999999+0000","last_clean":"2024-09-13T14:51:18.999999+0000","last_became_active":"2024-09-13T14:47:20.052566+0000","last_became_peered":"2024-09-13T14:47:20.052566+0000","last_unstale":"2024-09-13T14:51:18.999999+0000","last_undegraded":"2024-09-13T14:51:18.999999+0000","last_fullsized":"2024-09-13T14:51:18.999999+0000","mapping_epoch":229,"log_start":"253'7600","ondisk_log_start":"253'7600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6743,"log_dups_size":0,"ondisk_log_size":6743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:56.317980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18612248,"num_objects":285,"num_object_clones":0,"num_object_copies":855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14343,"num_write_kb":468033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":385,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.005561+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:51:10.005561+0000","last_peered":"2024-09-13T14:51:10.005561+0000","last_clean":"2024-09-13T14:51:10.005561+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:51:10.005561+0000","last_undegraded":"2024-09-13T14:51:10.005561+0000","last_fullsized":"2024-09-13T14:51:10.005561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":235,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921304+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:50:53.921304+0000","last_peered":"2024-09-13T14:50:53.921304+0000","last_clean":"2024-09-13T14:50:53.921304+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:50:53.921304+0000","last_undegraded":"2024-09-13T14:50:53.921304+0000","last_fullsized":"2024-09-13T14:50:53.921304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":340,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.998843+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:51:09.998843+0000","last_peered":"2024-09-13T14:51:09.998843+0000","last_clean":"2024-09-13T14:51:09.998843+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:51:09.998843+0000","last_undegraded":"2024-09-13T14:51:09.998843+0000","last_fullsized":"2024-09-13T14:51:09.998843+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":196,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.920331+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:50:53.920331+0000","last_peered":"2024-09-13T14:50:53.920331+0000","last_clean":"2024-09-13T14:50:53.920331+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:50:53.920331+0000","last_undegraded":"2024-09-13T14:50:53.920331+0000","last_fullsized":"2024-09-13T14:50:53.920331+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"260'14408","reported_seq":14447,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:19.051839+0000","last_change":"2024-09-13T14:47:20.052419+0000","last_active":"2024-09-13T14:51:19.051839+0000","last_peered":"2024-09-13T14:51:19.051839+0000","last_clean":"2024-09-13T14:51:19.051839+0000","last_became_active":"2024-09-13T14:47:20.051845+0000","last_became_peered":"2024-09-13T14:47:20.051845+0000","last_unstale":"2024-09-13T14:51:19.051839+0000","last_undegraded":"2024-09-13T14:51:19.051839+0000","last_fullsized":"2024-09-13T14:51:19.051839+0000","mapping_epoch":229,"log_start":"253'7514","ondisk_log_start":"253'7514","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6894,"log_dups_size":0,"ondisk_log_size":6894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.420332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19136512,"num_objects":292,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14408,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919167+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:50:53.919167+0000","last_peered":"2024-09-13T14:50:53.919167+0000","last_clean":"2024-09-13T14:50:53.919167+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:50:53.919167+0000","last_undegraded":"2024-09-13T14:50:53.919167+0000","last_fullsized":"2024-09-13T14:50:53.919167+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":198,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.010086+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:51:10.010086+0000","last_peered":"2024-09-13T14:51:10.010086+0000","last_clean":"2024-09-13T14:51:10.010086+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:51:10.010086+0000","last_undegraded":"2024-09-13T14:51:10.010086+0000","last_fullsized":"2024-09-13T14:51:10.010086+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":238,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918561+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:50:53.918561+0000","last_peered":"2024-09-13T14:50:53.918561+0000","last_clean":"2024-09-13T14:50:53.918561+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:50:53.918561+0000","last_undegraded":"2024-09-13T14:50:53.918561+0000","last_fullsized":"2024-09-13T14:50:53.918561+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":216,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.920720+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:50:53.920720+0000","last_peered":"2024-09-13T14:50:53.920720+0000","last_clean":"2024-09-13T14:50:53.920720+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:50:53.920720+0000","last_undegraded":"2024-09-13T14:50:53.920720+0000","last_fullsized":"2024-09-13T14:50:53.920720+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"260'14183","reported_seq":14220,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:19.006234+0000","last_change":"2024-09-13T14:47:20.051919+0000","last_active":"2024-09-13T14:51:19.006234+0000","last_peered":"2024-09-13T14:51:19.006234+0000","last_clean":"2024-09-13T14:51:19.006234+0000","last_became_active":"2024-09-13T14:47:20.051626+0000","last_became_peered":"2024-09-13T14:47:20.051626+0000","last_unstale":"2024-09-13T14:51:19.006234+0000","last_undegraded":"2024-09-13T14:51:19.006234+0000","last_fullsized":"2024-09-13T14:51:19.006234+0000","mapping_epoch":229,"log_start":"253'7500","ondisk_log_start":"253'7500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6683,"log_dups_size":0,"ondisk_log_size":6683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:12.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15794176,"num_objects":241,"num_object_clones":0,"num_object_copies":723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14183,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":252,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.999021+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:51:09.999021+0000","last_peered":"2024-09-13T14:51:09.999021+0000","last_clean":"2024-09-13T14:51:09.999021+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:51:09.999021+0000","last_undegraded":"2024-09-13T14:51:09.999021+0000","last_fullsized":"2024-09-13T14:51:09.999021+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":223,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.924512+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:50:53.924512+0000","last_peered":"2024-09-13T14:50:53.924512+0000","last_clean":"2024-09-13T14:50:53.924512+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:50:53.924512+0000","last_undegraded":"2024-09-13T14:50:53.924512+0000","last_fullsized":"2024-09-13T14:50:53.924512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":234,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919482+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:50:53.919482+0000","last_peered":"2024-09-13T14:50:53.919482+0000","last_clean":"2024-09-13T14:50:53.919482+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:50:53.919482+0000","last_undegraded":"2024-09-13T14:50:53.919482+0000","last_fullsized":"2024-09-13T14:50:53.919482+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":43,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.926441+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:50:53.926441+0000","last_peered":"2024-09-13T14:50:53.926441+0000","last_clean":"2024-09-13T14:50:53.926441+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:50:53.926441+0000","last_undegraded":"2024-09-13T14:50:53.926441+0000","last_fullsized":"2024-09-13T14:50:53.926441+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"260'14328","reported_seq":21175,"reported_epoch":260,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:19.459835+0000","last_change":"2024-09-13T14:49:05.872778+0000","last_active":"2024-09-13T14:51:19.459835+0000","last_peered":"2024-09-13T14:51:19.459835+0000","last_clean":"2024-09-13T14:48:37.826901+0000","last_became_active":"2024-09-13T14:48:40.041761+0000","last_became_peered":"2024-09-13T14:48:40.041761+0000","last_unstale":"2024-09-13T14:51:19.459835+0000","last_undegraded":"2024-09-13T14:48:39.850843+0000","last_fullsized":"2024-09-13T14:48:39.825201+0000","mapping_epoch":249,"log_start":"250'6200","ondisk_log_start":"250'6200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8128,"log_dups_size":0,"ondisk_log_size":8128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17039360,"num_objects":260,"num_object_clones":0,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14328,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3387,"num_bytes_recovered":168165376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,7","objects":-2112}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":154,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925180+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:50:53.925180+0000","last_peered":"2024-09-13T14:50:53.925180+0000","last_clean":"2024-09-13T14:50:53.925180+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:50:53.925180+0000","last_undegraded":"2024-09-13T14:50:53.925180+0000","last_fullsized":"2024-09-13T14:50:53.925180+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":192,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.999442+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:51:09.999442+0000","last_peered":"2024-09-13T14:51:09.999442+0000","last_clean":"2024-09-13T14:51:09.999442+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:51:09.999442+0000","last_undegraded":"2024-09-13T14:51:09.999442+0000","last_fullsized":"2024-09-13T14:51:09.999442+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":383,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.001317+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:51:10.001317+0000","last_peered":"2024-09-13T14:51:10.001317+0000","last_clean":"2024-09-13T14:51:10.001317+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:51:10.001317+0000","last_undegraded":"2024-09-13T14:51:10.001317+0000","last_fullsized":"2024-09-13T14:51:10.001317+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":17,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919026+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:50:53.919026+0000","last_peered":"2024-09-13T14:50:53.919026+0000","last_clean":"2024-09-13T14:50:53.919026+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:50:53.919026+0000","last_undegraded":"2024-09-13T14:50:53.919026+0000","last_fullsized":"2024-09-13T14:50:53.919026+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"260'14522","reported_seq":14595,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.561477+0000","last_change":"2024-09-13T14:47:20.049747+0000","last_active":"2024-09-13T14:51:20.561477+0000","last_peered":"2024-09-13T14:51:20.561477+0000","last_clean":"2024-09-13T14:51:20.561477+0000","last_became_active":"2024-09-13T14:47:20.049448+0000","last_became_peered":"2024-09-13T14:47:20.049448+0000","last_unstale":"2024-09-13T14:51:20.561477+0000","last_undegraded":"2024-09-13T14:51:20.561477+0000","last_fullsized":"2024-09-13T14:51:20.561477+0000","mapping_epoch":229,"log_start":"253'7500","ondisk_log_start":"253'7500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:11.585800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14522,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":407,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.004356+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:51:10.004356+0000","last_peered":"2024-09-13T14:51:10.004356+0000","last_clean":"2024-09-13T14:51:10.004356+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:51:10.004356+0000","last_undegraded":"2024-09-13T14:51:10.004356+0000","last_fullsized":"2024-09-13T14:51:10.004356+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":416,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.005389+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:51:10.005389+0000","last_peered":"2024-09-13T14:51:10.005389+0000","last_clean":"2024-09-13T14:51:10.005389+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:51:10.005389+0000","last_undegraded":"2024-09-13T14:51:10.005389+0000","last_fullsized":"2024-09-13T14:51:10.005389+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":272,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918991+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:50:53.918991+0000","last_peered":"2024-09-13T14:50:53.918991+0000","last_clean":"2024-09-13T14:50:53.918991+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:50:53.918991+0000","last_undegraded":"2024-09-13T14:50:53.918991+0000","last_fullsized":"2024-09-13T14:50:53.918991+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":196,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.920450+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:50:53.920450+0000","last_peered":"2024-09-13T14:50:53.920450+0000","last_clean":"2024-09-13T14:50:53.920450+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:50:53.920450+0000","last_undegraded":"2024-09-13T14:50:53.920450+0000","last_fullsized":"2024-09-13T14:50:53.920450+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"260'14399","reported_seq":14438,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:18.921000+0000","last_change":"2024-09-13T14:47:20.106371+0000","last_active":"2024-09-13T14:51:18.921000+0000","last_peered":"2024-09-13T14:51:18.921000+0000","last_clean":"2024-09-13T14:51:18.921000+0000","last_became_active":"2024-09-13T14:47:20.105934+0000","last_became_peered":"2024-09-13T14:47:20.105934+0000","last_unstale":"2024-09-13T14:51:18.921000+0000","last_undegraded":"2024-09-13T14:51:18.921000+0000","last_fullsized":"2024-09-13T14:51:18.921000+0000","mapping_epoch":229,"log_start":"253'7700","ondisk_log_start":"253'7700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6699,"log_dups_size":0,"ondisk_log_size":6699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:54.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":247,"num_object_clones":0,"num_object_copies":741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14399,"num_write_kb":468672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":205,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921610+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:50:53.921610+0000","last_peered":"2024-09-13T14:50:53.921610+0000","last_clean":"2024-09-13T14:50:53.921610+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:50:53.921610+0000","last_undegraded":"2024-09-13T14:50:53.921610+0000","last_fullsized":"2024-09-13T14:50:53.921610+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":315,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.014783+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:51:10.014783+0000","last_peered":"2024-09-13T14:51:10.014783+0000","last_clean":"2024-09-13T14:51:10.014783+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:51:10.014783+0000","last_undegraded":"2024-09-13T14:51:10.014783+0000","last_fullsized":"2024-09-13T14:51:10.014783+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":233,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919184+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:50:53.919184+0000","last_peered":"2024-09-13T14:50:53.919184+0000","last_clean":"2024-09-13T14:50:53.919184+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:50:53.919184+0000","last_undegraded":"2024-09-13T14:50:53.919184+0000","last_fullsized":"2024-09-13T14:50:53.919184+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":17,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918351+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:50:53.918351+0000","last_peered":"2024-09-13T14:50:53.918351+0000","last_clean":"2024-09-13T14:50:53.918351+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:50:53.918351+0000","last_undegraded":"2024-09-13T14:50:53.918351+0000","last_fullsized":"2024-09-13T14:50:53.918351+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"260'14177","reported_seq":14216,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:19.183442+0000","last_change":"2024-09-13T14:47:20.050773+0000","last_active":"2024-09-13T14:51:19.183442+0000","last_peered":"2024-09-13T14:51:19.183442+0000","last_clean":"2024-09-13T14:51:19.183442+0000","last_became_active":"2024-09-13T14:47:20.050243+0000","last_became_peered":"2024-09-13T14:47:20.050243+0000","last_unstale":"2024-09-13T14:51:19.183442+0000","last_undegraded":"2024-09-13T14:51:19.183442+0000","last_fullsized":"2024-09-13T14:51:19.183442+0000","mapping_epoch":229,"log_start":"254'7900","ondisk_log_start":"254'7900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6277,"log_dups_size":0,"ondisk_log_size":6277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:42.724078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":265,"num_object_clones":0,"num_object_copies":795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14177,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":39,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.999250+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:51:09.999250+0000","last_peered":"2024-09-13T14:51:09.999250+0000","last_clean":"2024-09-13T14:51:09.999250+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:51:09.999250+0000","last_undegraded":"2024-09-13T14:51:09.999250+0000","last_fullsized":"2024-09-13T14:51:09.999250+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":326,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.014532+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:51:10.014532+0000","last_peered":"2024-09-13T14:51:10.014532+0000","last_clean":"2024-09-13T14:51:10.014532+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:51:10.014532+0000","last_undegraded":"2024-09-13T14:51:10.014532+0000","last_fullsized":"2024-09-13T14:51:10.014532+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":43,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919093+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:50:53.919093+0000","last_peered":"2024-09-13T14:50:53.919093+0000","last_clean":"2024-09-13T14:50:53.919093+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:50:53.919093+0000","last_undegraded":"2024-09-13T14:50:53.919093+0000","last_fullsized":"2024-09-13T14:50:53.919093+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"260'14178","reported_seq":14217,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:18.995983+0000","last_change":"2024-09-13T14:47:20.106351+0000","last_active":"2024-09-13T14:51:18.995983+0000","last_peered":"2024-09-13T14:51:18.995983+0000","last_clean":"2024-09-13T14:51:18.995983+0000","last_became_active":"2024-09-13T14:47:20.105946+0000","last_became_peered":"2024-09-13T14:47:20.105946+0000","last_unstale":"2024-09-13T14:51:18.995983+0000","last_undegraded":"2024-09-13T14:51:18.995983+0000","last_fullsized":"2024-09-13T14:51:18.995983+0000","mapping_epoch":229,"log_start":"253'7500","ondisk_log_start":"253'7500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6678,"log_dups_size":0,"ondisk_log_size":6678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:07.223637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":268,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14178,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":242,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925108+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:50:53.925108+0000","last_peered":"2024-09-13T14:50:53.925108+0000","last_clean":"2024-09-13T14:50:53.925108+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:50:53.925108+0000","last_undegraded":"2024-09-13T14:50:53.925108+0000","last_fullsized":"2024-09-13T14:50:53.925108+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":223,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.930214+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:50:53.930214+0000","last_peered":"2024-09-13T14:50:53.930214+0000","last_clean":"2024-09-13T14:50:53.930214+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:50:53.930214+0000","last_undegraded":"2024-09-13T14:50:53.930214+0000","last_fullsized":"2024-09-13T14:50:53.930214+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":377,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.004997+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:51:10.004997+0000","last_peered":"2024-09-13T14:51:10.004997+0000","last_clean":"2024-09-13T14:51:10.004997+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:51:10.004997+0000","last_undegraded":"2024-09-13T14:51:10.004997+0000","last_fullsized":"2024-09-13T14:51:10.004997+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"260'14379","reported_seq":17962,"reported_epoch":260,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:19.430600+0000","last_change":"2024-09-13T14:49:05.871518+0000","last_active":"2024-09-13T14:51:19.430600+0000","last_peered":"2024-09-13T14:51:19.430600+0000","last_clean":"2024-09-13T14:48:09.848772+0000","last_became_active":"2024-09-13T14:48:12.232150+0000","last_became_peered":"2024-09-13T14:48:12.232150+0000","last_unstale":"2024-09-13T14:51:19.430600+0000","last_undegraded":"2024-09-13T14:49:05.858877+0000","last_fullsized":"2024-09-13T14:48:11.643512+0000","mapping_epoch":242,"log_start":"250'6200","ondisk_log_start":"250'6200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8179,"log_dups_size":0,"ondisk_log_size":8179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15532032,"num_objects":237,"num_object_clones":0,"num_object_copies":711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14379,"num_write_kb":467712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1751,"num_bytes_recovered":114753536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,5","objects":-2190}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":239,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925164+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:50:53.925164+0000","last_peered":"2024-09-13T14:50:53.925164+0000","last_clean":"2024-09-13T14:50:53.925164+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:50:53.925164+0000","last_undegraded":"2024-09-13T14:50:53.925164+0000","last_fullsized":"2024-09-13T14:50:53.925164+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":243,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.920162+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:50:53.920162+0000","last_peered":"2024-09-13T14:50:53.920162+0000","last_clean":"2024-09-13T14:50:53.920162+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:50:53.920162+0000","last_undegraded":"2024-09-13T14:50:53.920162+0000","last_fullsized":"2024-09-13T14:50:53.920162+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":25,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918478+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:50:53.918478+0000","last_peered":"2024-09-13T14:50:53.918478+0000","last_clean":"2024-09-13T14:50:53.918478+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:50:53.918478+0000","last_undegraded":"2024-09-13T14:50:53.918478+0000","last_fullsized":"2024-09-13T14:50:53.918478+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"260'14486","reported_seq":14524,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:19.178904+0000","last_change":"2024-09-13T14:47:20.106142+0000","last_active":"2024-09-13T14:51:19.178904+0000","last_peered":"2024-09-13T14:51:19.178904+0000","last_clean":"2024-09-13T14:51:19.178904+0000","last_became_active":"2024-09-13T14:47:20.105758+0000","last_became_peered":"2024-09-13T14:47:20.105758+0000","last_unstale":"2024-09-13T14:51:19.178904+0000","last_undegraded":"2024-09-13T14:51:19.178904+0000","last_fullsized":"2024-09-13T14:51:19.178904+0000","mapping_epoch":229,"log_start":"254'8200","ondisk_log_start":"254'8200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:51.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17694720,"num_objects":270,"num_object_clones":0,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":222,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.924198+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:50:53.924198+0000","last_peered":"2024-09-13T14:50:53.924198+0000","last_clean":"2024-09-13T14:50:53.924198+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:50:53.924198+0000","last_undegraded":"2024-09-13T14:50:53.924198+0000","last_fullsized":"2024-09-13T14:50:53.924198+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":356,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.014999+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:51:10.014999+0000","last_peered":"2024-09-13T14:51:10.014999+0000","last_clean":"2024-09-13T14:51:10.014999+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:51:10.014999+0000","last_undegraded":"2024-09-13T14:51:10.014999+0000","last_fullsized":"2024-09-13T14:51:10.014999+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":43,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925508+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:50:53.925508+0000","last_peered":"2024-09-13T14:50:53.925508+0000","last_clean":"2024-09-13T14:50:53.925508+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:50:53.925508+0000","last_undegraded":"2024-09-13T14:50:53.925508+0000","last_fullsized":"2024-09-13T14:50:53.925508+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"260'14366","reported_seq":14440,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:19.402327+0000","last_change":"2024-09-13T14:47:20.107374+0000","last_active":"2024-09-13T14:51:19.402327+0000","last_peered":"2024-09-13T14:51:19.402327+0000","last_clean":"2024-09-13T14:51:19.402327+0000","last_became_active":"2024-09-13T14:47:20.107057+0000","last_became_peered":"2024-09-13T14:47:20.107057+0000","last_unstale":"2024-09-13T14:51:19.402327+0000","last_undegraded":"2024-09-13T14:51:19.402327+0000","last_fullsized":"2024-09-13T14:51:19.402327+0000","mapping_epoch":229,"log_start":"250'6200","ondisk_log_start":"250'6200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8166,"log_dups_size":0,"ondisk_log_size":8166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:05.814456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17170432,"num_objects":262,"num_object_clones":0,"num_object_copies":786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14366,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":395,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.005471+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:51:10.005471+0000","last_peered":"2024-09-13T14:51:10.005471+0000","last_clean":"2024-09-13T14:51:10.005471+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:51:10.005471+0000","last_undegraded":"2024-09-13T14:51:10.005471+0000","last_fullsized":"2024-09-13T14:51:10.005471+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":221,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918933+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:50:53.918933+0000","last_peered":"2024-09-13T14:50:53.918933+0000","last_clean":"2024-09-13T14:50:53.918933+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:50:53.918933+0000","last_undegraded":"2024-09-13T14:50:53.918933+0000","last_fullsized":"2024-09-13T14:50:53.918933+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":43,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.924876+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:50:53.924876+0000","last_peered":"2024-09-13T14:50:53.924876+0000","last_clean":"2024-09-13T14:50:53.924876+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:50:53.924876+0000","last_undegraded":"2024-09-13T14:50:53.924876+0000","last_fullsized":"2024-09-13T14:50:53.924876+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"260'14296","reported_seq":14370,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.583745+0000","last_change":"2024-09-13T14:47:20.108173+0000","last_active":"2024-09-13T14:51:20.583745+0000","last_peered":"2024-09-13T14:51:20.583745+0000","last_clean":"2024-09-13T14:51:20.583745+0000","last_became_active":"2024-09-13T14:47:20.107879+0000","last_became_peered":"2024-09-13T14:47:20.107879+0000","last_unstale":"2024-09-13T14:51:20.583745+0000","last_undegraded":"2024-09-13T14:51:20.583745+0000","last_fullsized":"2024-09-13T14:51:20.583745+0000","mapping_epoch":229,"log_start":"253'7300","ondisk_log_start":"253'7300","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6996,"log_dups_size":0,"ondisk_log_size":6996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:50.152706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13500416,"num_objects":206,"num_object_clones":0,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14296,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":235,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.920017+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:50:53.920017+0000","last_peered":"2024-09-13T14:50:53.920017+0000","last_clean":"2024-09-13T14:50:53.920017+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:50:53.920017+0000","last_undegraded":"2024-09-13T14:50:53.920017+0000","last_fullsized":"2024-09-13T14:50:53.920017+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":236,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921223+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:50:53.921223+0000","last_peered":"2024-09-13T14:50:53.921223+0000","last_clean":"2024-09-13T14:50:53.921223+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:50:53.921223+0000","last_undegraded":"2024-09-13T14:50:53.921223+0000","last_fullsized":"2024-09-13T14:50:53.921223+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":196,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.922614+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:50:53.922614+0000","last_peered":"2024-09-13T14:50:53.922614+0000","last_clean":"2024-09-13T14:50:53.922614+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:50:53.922614+0000","last_undegraded":"2024-09-13T14:50:53.922614+0000","last_fullsized":"2024-09-13T14:50:53.922614+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"260'14185","reported_seq":14227,"reported_epoch":260,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:19.001064+0000","last_change":"2024-09-13T14:48:40.078958+0000","last_active":"2024-09-13T14:51:19.001064+0000","last_peered":"2024-09-13T14:51:19.001064+0000","last_clean":"2024-09-13T14:48:37.811111+0000","last_became_active":"2024-09-13T14:48:40.052181+0000","last_became_peered":"2024-09-13T14:48:40.052181+0000","last_unstale":"2024-09-13T14:51:19.001064+0000","last_undegraded":"2024-09-13T14:48:39.849823+0000","last_fullsized":"2024-09-13T14:48:39.826167+0000","mapping_epoch":249,"log_start":"253'7500","ondisk_log_start":"253'7500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":6685,"log_dups_size":0,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:49.521146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19988480,"num_objects":305,"num_object_clones":0,"num_object_copies":915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14185,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"3,6","objects":-5444}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":249,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.930182+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:50:53.930182+0000","last_peered":"2024-09-13T14:50:53.930182+0000","last_clean":"2024-09-13T14:50:53.930182+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:50:53.930182+0000","last_undegraded":"2024-09-13T14:50:53.930182+0000","last_fullsized":"2024-09-13T14:50:53.930182+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":226,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919516+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:50:53.919516+0000","last_peered":"2024-09-13T14:50:53.919516+0000","last_clean":"2024-09-13T14:50:53.919516+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:50:53.919516+0000","last_undegraded":"2024-09-13T14:50:53.919516+0000","last_fullsized":"2024-09-13T14:50:53.919516+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":83,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.017661+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:51:10.017661+0000","last_peered":"2024-09-13T14:51:10.017661+0000","last_clean":"2024-09-13T14:51:10.017661+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:51:10.017661+0000","last_undegraded":"2024-09-13T14:51:10.017661+0000","last_fullsized":"2024-09-13T14:51:10.017661+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"260'14534","reported_seq":14573,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:19.784755+0000","last_change":"2024-09-13T14:47:20.051824+0000","last_active":"2024-09-13T14:51:19.784755+0000","last_peered":"2024-09-13T14:51:19.784755+0000","last_clean":"2024-09-13T14:51:19.784755+0000","last_became_active":"2024-09-13T14:47:20.051487+0000","last_became_peered":"2024-09-13T14:47:20.051487+0000","last_unstale":"2024-09-13T14:51:19.784755+0000","last_undegraded":"2024-09-13T14:51:19.784755+0000","last_fullsized":"2024-09-13T14:51:19.784755+0000","mapping_epoch":229,"log_start":"250'6400","ondisk_log_start":"250'6400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:19.021989+0000","last_clean_scrub_stamp":"2024-09-13T14:47:19.021989+0000","objects_scrubbed":0,"log_size":8134,"log_dups_size":0,"ondisk_log_size":8134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:11.533543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17039360,"num_objects":260,"num_object_clones":0,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14534,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918562+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:50:53.918562+0000","last_peered":"2024-09-13T14:50:53.918562+0000","last_clean":"2024-09-13T14:50:53.918562+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:50:53.918562+0000","last_undegraded":"2024-09-13T14:50:53.918562+0000","last_fullsized":"2024-09-13T14:50:53.918562+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":21,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919762+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:50:53.919762+0000","last_peered":"2024-09-13T14:50:53.919762+0000","last_clean":"2024-09-13T14:50:53.919762+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:50:53.919762+0000","last_undegraded":"2024-09-13T14:50:53.919762+0000","last_fullsized":"2024-09-13T14:50:53.919762+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":17,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.921051+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:50:53.921051+0000","last_peered":"2024-09-13T14:50:53.921051+0000","last_clean":"2024-09-13T14:50:53.921051+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:50:53.921051+0000","last_undegraded":"2024-09-13T14:50:53.921051+0000","last_fullsized":"2024-09-13T14:50:53.921051+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":271,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.920633+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:50:53.920633+0000","last_peered":"2024-09-13T14:50:53.920633+0000","last_clean":"2024-09-13T14:50:53.920633+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:50:53.920633+0000","last_undegraded":"2024-09-13T14:50:53.920633+0000","last_fullsized":"2024-09-13T14:50:53.920633+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":364,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.018884+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:51:10.018884+0000","last_peered":"2024-09-13T14:51:10.018884+0000","last_clean":"2024-09-13T14:51:10.018884+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:51:10.018884+0000","last_undegraded":"2024-09-13T14:51:10.018884+0000","last_fullsized":"2024-09-13T14:51:10.018884+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":53,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919241+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:50:53.919241+0000","last_peered":"2024-09-13T14:50:53.919241+0000","last_clean":"2024-09-13T14:50:53.919241+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:50:53.919241+0000","last_undegraded":"2024-09-13T14:50:53.919241+0000","last_fullsized":"2024-09-13T14:50:53.919241+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":217,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.925963+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:50:53.925963+0000","last_peered":"2024-09-13T14:50:53.925963+0000","last_clean":"2024-09-13T14:50:53.925963+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:50:53.925963+0000","last_undegraded":"2024-09-13T14:50:53.925963+0000","last_fullsized":"2024-09-13T14:50:53.925963+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":409,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.004460+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:51:10.004460+0000","last_peered":"2024-09-13T14:51:10.004460+0000","last_clean":"2024-09-13T14:51:10.004460+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:51:10.004460+0000","last_undegraded":"2024-09-13T14:51:10.004460+0000","last_fullsized":"2024-09-13T14:51:10.004460+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":79,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.014814+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:51:10.014814+0000","last_peered":"2024-09-13T14:51:10.014814+0000","last_clean":"2024-09-13T14:51:10.014814+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:51:10.014814+0000","last_undegraded":"2024-09-13T14:51:10.014814+0000","last_fullsized":"2024-09-13T14:51:10.014814+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":415,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.009986+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:51:10.009986+0000","last_peered":"2024-09-13T14:51:10.009986+0000","last_clean":"2024-09-13T14:51:10.009986+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:51:10.009986+0000","last_undegraded":"2024-09-13T14:51:10.009986+0000","last_fullsized":"2024-09-13T14:51:10.009986+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918458+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:50:53.918458+0000","last_peered":"2024-09-13T14:50:53.918458+0000","last_clean":"2024-09-13T14:50:53.918458+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:50:53.918458+0000","last_undegraded":"2024-09-13T14:50:53.918458+0000","last_fullsized":"2024-09-13T14:50:53.918458+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":43,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.918638+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:50:53.918638+0000","last_peered":"2024-09-13T14:50:53.918638+0000","last_clean":"2024-09-13T14:50:53.918638+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:50:53.918638+0000","last_undegraded":"2024-09-13T14:50:53.918638+0000","last_fullsized":"2024-09-13T14:50:53.918638+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":242,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:50:53.919207+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:50:53.919207+0000","last_peered":"2024-09-13T14:50:53.919207+0000","last_clean":"2024-09-13T14:50:53.919207+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:50:53.919207+0000","last_undegraded":"2024-09-13T14:50:53.919207+0000","last_fullsized":"2024-09-13T14:50:53.919207+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":74,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.008999+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:51:10.008999+0000","last_peered":"2024-09-13T14:51:10.008999+0000","last_clean":"2024-09-13T14:51:10.008999+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:51:10.008999+0000","last_undegraded":"2024-09-13T14:51:10.008999+0000","last_fullsized":"2024-09-13T14:51:10.008999+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":373,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.005481+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:51:10.005481+0000","last_peered":"2024-09-13T14:51:10.005481+0000","last_clean":"2024-09-13T14:51:10.005481+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:51:10.005481+0000","last_undegraded":"2024-09-13T14:51:10.005481+0000","last_fullsized":"2024-09-13T14:51:10.005481+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":270794776,"num_objects":4133,"num_object_clones":0,"num_object_copies":12399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229321,"num_write_kb":7470465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5138,"num_bytes_recovered":282918912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1854287872,"data_stored":1854275656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112607,"ondisk_log_size":112607,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365179,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":980148,"kb_used_data":79084,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":92765004,"statfs":{"total":95995035648,"available":94991364096,"internally_reserved":0,"allocated":80982016,"data_stored":79605025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561409,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1042736,"kb_used_data":144936,"kb_used_omap":1,"kb_used_meta":897662,"kb_avail":92702416,"statfs":{"total":95995035648,"available":94927273984,"internally_reserved":0,"allocated":148414464,"data_stored":147041416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919206232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790341,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":998244,"kb_used_data":96988,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":92746908,"statfs":{"total":95995035648,"available":94972833792,"internally_reserved":0,"allocated":99315712,"data_stored":97951046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986569,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1676660,"kb_used_data":782252,"kb_used_omap":1,"kb_used_meta":894398,"kb_avail":92068492,"statfs":{"total":95995035648,"available":94278135808,"internally_reserved":0,"allocated":801026048,"data_stored":799649242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915863896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519724,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":651876,"kb_used_data":117596,"kb_used_omap":4,"kb_used_meta":534267,"kb_avail":93093276,"statfs":{"total":95995035648,"available":95327514624,"internally_reserved":0,"allocated":120418304,"data_stored":119053648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":547089856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304640,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":986280,"kb_used_data":489888,"kb_used_omap":6,"kb_used_meta":496377,"kb_avail":92758872,"statfs":{"total":95995035648,"available":94985084928,"internally_reserved":0,"allocated":501645312,"data_stored":500277074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":508290503},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827943,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907620,"kb_used_data":81628,"kb_used_omap":5,"kb_used_meta":825978,"kb_avail":92837532,"statfs":{"total":95995035648,"available":95065632768,"internally_reserved":0,"allocated":83587072,"data_stored":82221843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":845801708},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285808,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":857316,"kb_used_data":133468,"kb_used_omap":5,"kb_used_meta":723834,"kb_avail":92887836,"statfs":{"total":95995035648,"available":95117144064,"internally_reserved":0,"allocated":136671232,"data_stored":135305330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":741206075},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125505536,"data_stored":125501464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70254592,"data_stored":70254592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":476250112,"data_stored":476250112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112984064,"data_stored":112984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":793579520,"data_stored":793575448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":77070336,"data_stored":77070336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":123670528,"data_stored":123666456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":74973184,"data_stored":74973184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:24.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:51:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.961 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:24.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:25.083 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:25.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:25 smithi003 ceph-mon[104007]: pgmap v904: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 259 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 905 op/s; 10548/12477 objects degraded (84.540%); 0 B/s, 36 objects/s recovering 2024-09-13T14:51:25.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:25 smithi003 ceph-mon[107278]: pgmap v904: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 259 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 905 op/s; 10548/12477 objects degraded (84.540%); 0 B/s, 36 objects/s recovering 2024-09-13T14:51:25.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:25.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:25.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:25.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:25 smithi019 ceph-mon[91296]: pgmap v904: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 259 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 905 op/s; 10548/12477 objects degraded (84.540%); 0 B/s, 36 objects/s recovering 2024-09-13T14:51:25.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:25.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.584 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:25.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:25.651 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:25.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:25.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:25.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:25.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:25.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:25.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:26.058 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:26.081 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:26.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:26.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:26.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:26.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:26.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 10548/12477 objects degraded (84.540%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:26.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 10548/12477 objects degraded (84.540%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 10548/12477 objects degraded (84.540%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:26.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:26.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:26.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:26.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:51:26.558 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:26.580 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:26.682 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:26.705 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:26.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:26.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:26.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:26.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:27.060 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:27.083 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:27.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:27 smithi003 ceph-mon[107278]: pgmap v905: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 672 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 10289/84 objects degraded (12248.810%); 0 B/s, 44 objects/s recovering 2024-09-13T14:51:27.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:27 smithi003 ceph-mon[104007]: pgmap v905: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 672 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 10289/84 objects degraded (12248.810%); 0 B/s, 44 objects/s recovering 2024-09-13T14:51:27.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:27.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:27.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:27.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:27.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:27 smithi019 ceph-mon[91296]: pgmap v905: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 672 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 10289/84 objects degraded (12248.810%); 0 B/s, 44 objects/s recovering 2024-09-13T14:51:27.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:27.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:27.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:27.580 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:27.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:27.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:27.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:27.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:27.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:27.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:28.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:28.081 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:28.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:28.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:28.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:28.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:28.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:28.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:28.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:28.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:28.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:28.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:28.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:28.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:28.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:28.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.955 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:28.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:29.056 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:29.078 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:29.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:29.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:28 smithi019 ceph-mon[91296]: pgmap v906: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 672 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 10289/84 objects degraded (12248.810%); 0 B/s, 44 objects/s recovering 2024-09-13T14:51:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:28 smithi003 ceph-mon[104007]: pgmap v906: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 672 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 10289/84 objects degraded (12248.810%); 0 B/s, 44 objects/s recovering 2024-09-13T14:51:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:29.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:28 smithi003 ceph-mon[107278]: pgmap v906: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 672 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 10289/84 objects degraded (12248.810%); 0 B/s, 44 objects/s recovering 2024-09-13T14:51:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:29.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:29.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:29.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:29.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.456 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:29.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:29.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:29.996 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-13T14:51:29.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-13T14:51:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:30.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:30.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:30.144 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:30.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:30.245 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:30.268 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:30.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:30.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:30.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:30.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:30.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.646 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:30.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:51:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:51:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:30.747 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.771 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:30.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:30.872 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.899 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:30.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:31.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:31.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:31.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:31.126 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:31.149 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:30 smithi019 ceph-mon[91296]: pgmap v907: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 470 op/s; 9143/78 objects degraded (11721.795%); 0 B/s, 140 objects/s recovering 2024-09-13T14:51:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 9143/78 objects degraded (11721.795%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:31.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:30 smithi003 ceph-mon[107278]: pgmap v907: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 470 op/s; 9143/78 objects degraded (11721.795%); 0 B/s, 140 objects/s recovering 2024-09-13T14:51:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 9143/78 objects degraded (11721.795%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:30 smithi003 ceph-mon[104007]: pgmap v907: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 470 op/s; 9143/78 objects degraded (11721.795%); 0 B/s, 140 objects/s recovering 2024-09-13T14:51:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 9143/78 objects degraded (11721.795%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:31.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:31.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:31.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:31.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:31.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:31.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:31.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:31.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:31.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:31.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:31.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:31.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:31.902 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:32.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:32.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:32.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:32.139 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:51:32.139 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:51:32.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:51:32.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:32.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:32.376 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:32.399 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:32.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:32.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:32.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:32.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:32.773 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:32.875 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.905 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:32.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:33.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:33.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:33.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:33.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:33.159 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:33.161 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:32 smithi019 ceph-mon[91296]: pgmap v908: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 413 op/s; 9143/78 objects degraded (11721.795%); 0 B/s, 140 objects/s recovering 2024-09-13T14:51:33.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:32 smithi003 ceph-mon[107278]: pgmap v908: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 413 op/s; 9143/78 objects degraded (11721.795%); 0 B/s, 140 objects/s recovering 2024-09-13T14:51:33.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:32 smithi003 ceph-mon[104007]: pgmap v908: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 413 op/s; 9143/78 objects degraded (11721.795%); 0 B/s, 140 objects/s recovering 2024-09-13T14:51:33.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:33.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:33.410 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:33.512 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:33.534 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:33.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:33.761 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:33.785 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:33.887 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:33.914 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:34.016 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:34.038 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:34.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:33 smithi003 ceph-mon[107278]: osdmap e262: 8 total, 8 up, 8 in 2024-09-13T14:51:34.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:33 smithi003 ceph-mon[104007]: osdmap e262: 8 total, 8 up, 8 in 2024-09-13T14:51:34.140 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:34.164 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:34.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:33 smithi019 ceph-mon[91296]: osdmap e262: 8 total, 8 up, 8 in 2024-09-13T14:51:34.266 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:34.291 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:34.347 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:51:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:51:34] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-09-13T14:51:34.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:34.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:34.519 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:34.541 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:34.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:34.644 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:34.667 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:34.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:34.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:34.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:34.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:34 smithi003 ceph-mon[104007]: pgmap v910: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 413 op/s; 9143/78 objects degraded (11721.795%); 0 B/s, 140 objects/s recovering 2024-09-13T14:51:34.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:34 smithi003 ceph-mon[104007]: osdmap e263: 8 total, 8 up, 8 in 2024-09-13T14:51:34.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:35.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:35.044 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:35.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:35.156 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:35.157 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:35.178 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:51:35.179 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:35.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:34 smithi003 ceph-mon[107278]: pgmap v910: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 413 op/s; 9143/78 objects degraded (11721.795%); 0 B/s, 140 objects/s recovering 2024-09-13T14:51:35.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:34 smithi003 ceph-mon[107278]: osdmap e263: 8 total, 8 up, 8 in 2024-09-13T14:51:35.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:35.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:35.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:34 smithi019 ceph-mon[91296]: pgmap v910: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 413 op/s; 9143/78 objects degraded (11721.795%); 0 B/s, 140 objects/s recovering 2024-09-13T14:51:35.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:34 smithi019 ceph-mon[91296]: osdmap e263: 8 total, 8 up, 8 in 2024-09-13T14:51:35.200 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:51:35.200 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:35.282 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:35.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.431 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:35.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:35.532 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:35.559 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:35.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:35.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:35.787 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.810 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:35.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:35.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:35.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:35 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2897346260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:35.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:35 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2897346260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:36.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:36.058 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:36.160 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:36.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:35 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2897346260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:36.182 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:36.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.308 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:36.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:36.409 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:36.433 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:36.480 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:51:36.480 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8060/78 objects degraded (10333.333%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1322,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":263,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238227,"num_in_osds":8,"osd_in_since":1726237182,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":107},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":109,"num_pools":5,"num_objects":26,"data_bytes":622752,"bytes_used":7407341568,"bytes_avail":760552943616,"bytes_total":767960285184,"degraded_objects":8060,"degraded_total":78,"degraded_ratio":103.33333333333331,"recovering_objects_per_sec":278,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.zmhiel","status":"up:active","gid":14646}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-13T14:43:29.603041+0000","services":{}},"progress_events":{"70b0e26c-ee48-4914-b6cc-9ce75e26a89c":{"message":"Global Recovery Event (3m)\n [===========================.] (remaining: 5s)","progress":0.97247707843780518,"add_to_ceph_s":true},"b67cb405-2ef0-4d69-99ef-7eee6a6810fd":{"message":"Upgrade to 19.3.0-4697-gd58f7306 (9m)\n [============................] (remaining: 11m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-13T14:51:36.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:36.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:36.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:36.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:36.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:36.811 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:36.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:36.913 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:36.935 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:37.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:37.060 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:37.162 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:36 smithi019 ceph-mon[91296]: pgmap v912: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 0 op/s; 8060/78 objects degraded (10333.333%); 0 B/s, 278 objects/s recovering 2024-09-13T14:51:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:36 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 8060/78 objects degraded (10333.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:36 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:36 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:36 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1235520129' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:51:37.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[104007]: pgmap v912: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 0 op/s; 8060/78 objects degraded (10333.333%); 0 B/s, 278 objects/s recovering 2024-09-13T14:51:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 8060/78 objects degraded (10333.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1235520129' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:51:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[107278]: pgmap v912: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 0 op/s; 8060/78 objects degraded (10333.333%); 0 B/s, 278 objects/s recovering 2024-09-13T14:51:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 8060/78 objects degraded (10333.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:36 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1235520129' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:51:37.184 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:37.286 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:37.309 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:37.411 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:37.433 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:37.515 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:37.515 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:37.535 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:37.558 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:37.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:37.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:37.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:37.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:37.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:37.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:51:38.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:38.061 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:38.163 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:38.186 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:38.274 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:51:38.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:38.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:38.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:38.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:38.436 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:38.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:38.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:38.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:38.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:38.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:38.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:38.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:38.899 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:51:38.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:38.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:38.931 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:38 smithi019 ceph-mon[91296]: pgmap v913: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 8060/78 objects degraded (10333.333%); 0 B/s, 135 objects/s recovering 2024-09-13T14:51:38.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:39.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:39.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:38 smithi003 ceph-mon[107278]: pgmap v913: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 8060/78 objects degraded (10333.333%); 0 B/s, 135 objects/s recovering 2024-09-13T14:51:39.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:38 smithi003 ceph-mon[104007]: pgmap v913: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 8060/78 objects degraded (10333.333%); 0 B/s, 135 objects/s recovering 2024-09-13T14:51:39.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:39.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:39.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.343 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:39.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:39.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:39.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:39.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:39.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:39.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.845 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:39.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:40.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:40.095 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:40.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:40.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:40.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:40.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:40.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:40.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:40.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:40.528 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:51:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:51:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:40.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:40.598 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:40.694 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_5' removed 2024-09-13T14:51:40.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:40.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:40.849 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:40.872 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:40.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:40.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:41.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:41.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:40 smithi019 ceph-mon[91296]: pgmap v914: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 7628/78 objects degraded (9779.487%); 0 B/s, 189 objects/s recovering 2024-09-13T14:51:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:40 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2907753290' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:51:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:40 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:51:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:40 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:51:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:40 smithi019 ceph-mon[91296]: osdmap e264: 8 total, 8 up, 8 in 2024-09-13T14:51:41.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[107278]: pgmap v914: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 7628/78 objects degraded (9779.487%); 0 B/s, 189 objects/s recovering 2024-09-13T14:51:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2907753290' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:51:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:51:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:51:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[107278]: osdmap e264: 8 total, 8 up, 8 in 2024-09-13T14:51:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[104007]: pgmap v914: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 7628/78 objects degraded (9779.487%); 0 B/s, 189 objects/s recovering 2024-09-13T14:51:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2907753290' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:51:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:51:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:51:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:40 smithi003 ceph-mon[104007]: osdmap e264: 8 total, 8 up, 8 in 2024-09-13T14:51:41.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:41.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:41.348 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:41.373 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:41.475 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:41.497 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:41.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:41.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:41.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.747 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:41.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:41.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:41.858 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:41.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:41.883 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:51:41.884 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:51:41.884 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-09-13T14:51:41.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-09-13T14:51:41.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:42.098 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:42.119 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:41 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7628/78 objects degraded (9779.487%), 2 pgs degraded, 2 pgs undersized) 2024-09-13T14:51:42.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:41 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7628/78 objects degraded (9779.487%), 2 pgs degraded, 2 pgs undersized) 2024-09-13T14:51:42.120 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:42.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:42.245 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:42.347 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:42.352 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:41 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7628/78 objects degraded (9779.487%), 2 pgs degraded, 2 pgs undersized) 2024-09-13T14:51:42.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:42.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:42.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:42.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:42.620 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:42.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.741 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:42.741 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:42.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:42.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:42.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:42.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:42.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:43.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:43.099 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:43.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:43.145 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:43.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:42 smithi003 ceph-mon[107278]: pgmap v916: 93 pgs: 93 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:51:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:42 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2076629988' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:43.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:42 smithi003 ceph-mon[104007]: pgmap v916: 93 pgs: 93 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:51:43.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:43.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:43.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:42 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2076629988' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:43.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:43.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:43.336 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:42 smithi019 ceph-mon[91296]: pgmap v916: 93 pgs: 93 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:51:43.336 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:43.337 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:43.337 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:42 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2076629988' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:43.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:43.361 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:51:43.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:43.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:43.499 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:43.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:43.620 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:43.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:43.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:43.747 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:43.793 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:51:43.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:43.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:43.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:43.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:44.073 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:43 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:44.073 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2076629988' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:44.073 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:43 smithi019 ceph-mon[91296]: osdmap e265: 8 total, 8 up, 8 in 2024-09-13T14:51:44.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:43 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:44.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2076629988' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:44.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:43 smithi003 ceph-mon[107278]: osdmap e265: 8 total, 8 up, 8 in 2024-09-13T14:51:44.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:43 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:44.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2076629988' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:44.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:43 smithi003 ceph-mon[104007]: osdmap e265: 8 total, 8 up, 8 in 2024-09-13T14:51:44.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:44.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:44.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:44.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:44.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:44.368 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:44.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:51:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:51:44] "GET /metrics HTTP/1.1" 200 34112 "" "Prometheus/2.51.0" 2024-09-13T14:51:44.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.493 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:44.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:44.594 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:44.616 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:44.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:44.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:44.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:44.870 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:44.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:44.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:45.098 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:45.121 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:44 smithi019 ceph-mon[91296]: pgmap v918: 93 pgs: 93 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:51:45.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:45.246 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:45.323 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:44 smithi003 ceph-mon[107278]: pgmap v918: 93 pgs: 93 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:51:45.324 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:44 smithi003 ceph-mon[104007]: pgmap v918: 93 pgs: 93 active+clean; 608 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:51:45.348 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:45.372 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:45.438 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:45.439 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:45.474 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:45.497 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:45.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:45.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:45.723 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.747 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:45.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:45.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:45.871 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:45.878 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:51:45.878 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:51:45.878 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:51:45.878 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:51:45.879 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:51:45.879 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:51:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:45.880 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:51:45.880 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:51:45.880 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:45.880 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:45.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:45.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:46.076 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:45 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1167606324' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:46.097 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:46.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:51:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.121 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:46.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:46.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:46.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:46.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:46.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:46.375 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:46.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:45 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1167606324' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:46.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:45 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1167606324' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:46.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:46.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:46.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:46.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:46.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:46.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:46.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:46.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:46.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:46.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:46.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:47.097 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:47.119 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:47.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:46 smithi019 ceph-mon[91296]: pgmap v919: 93 pgs: 93 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:47.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:46 smithi003 ceph-mon[107278]: pgmap v919: 93 pgs: 93 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:47.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:46 smithi003 ceph-mon[104007]: pgmap v919: 93 pgs: 93 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:47.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:47.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:47.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:47.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:47.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:47.498 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:47.599 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:47.623 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:47.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:47.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:47.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:47.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:47.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:47.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:47.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:48.038 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:51:48.038 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T14:51:48.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117660 9.10 9:0b6a8aa7:::benchmark_data_smithi003_156985_object117659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.630465+0000", 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.40491942200003, 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96999332500000002, 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.039 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117660 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.630465+0000", 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.630465+0000", 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.630462+0000", 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.630568+0000", 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106061 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.630583+0000", 2024-09-13T14:51:48.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5041999999999999e-05 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.630590+0000", 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5089999999999999e-06 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.630869+0000", 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00027941899999999999 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.630906+0000", 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7064000000000003e-05 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.581074+0000", 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95016762300000002 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.581210+0000", 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013632 2024-09-13T14:51:48.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.581384+0000", 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000173878 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.586991+0000", 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0056063229999999999 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600349+0000", 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013358139 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600382+0000", 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3606000000000003e-05 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600415+0000", 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2351999999999998e-05 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600458+0000", 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3176999999999997e-05 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117700 9.10 9:0a5018e5:::benchmark_data_smithi003_156985_object117699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.663221+0000", 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.37216283400005, 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93738011700000001, 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.046 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117700 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663221+0000", 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663221+0000", 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663219+0000", 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663285+0000", 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5883999999999995e-05 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663303+0000", 2024-09-13T14:51:48.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7929999999999999e-05 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663310+0000", 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5579999999999997e-06 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663974+0000", 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00066456100000000004 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663984+0000", 2024-09-13T14:51:48.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0051000000000001e-05 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.582890+0000", 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.918905478 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.583023+0000", 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013343 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.583173+0000", 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015030499999999999 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.589615+0000", 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0064419330000000004 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600501+0000", 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010885959000000001 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.051 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600539+0000", 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7494999999999999e-05 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600566+0000", 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7367000000000001e-05 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600601+0000", 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5114000000000003e-05 2024-09-13T14:51:48.052 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117701 9.10 9:080708b2:::benchmark_data_smithi003_156985_object117700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.663327+0000", 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.37205686499999, 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.937442475, 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.053 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117701 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663327+0000", 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663327+0000", 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663327+0000", 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663392+0000", 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5494999999999995e-05 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663404+0000", 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2275e-05 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.663409+0000", 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6569999999999996e-06 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.665627+0000", 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022178480000000001 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.665645+0000", 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8142e-05 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.584764+0000", 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.91911868799999996 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.584884+0000", 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012027399999999999 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.585041+0000", 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015730200000000001 2024-09-13T14:51:48.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.058 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.589623+0000", 2024-09-13T14:51:48.058 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0045816590000000001 2024-09-13T14:51:48.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.060 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600677+0000", 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011053746999999999 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600707+0000", 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0830999999999997e-05 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600734+0000", 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6857999999999999e-05 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600770+0000", 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5311999999999997e-05 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117713 9.10 9:09f06dc8:::benchmark_data_smithi003_156985_object117712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.672406+0000", 2024-09-13T14:51:48.062 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.36297783999999, 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92867329499999995, 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117713 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.672406+0000", 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.672406+0000", 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.672404+0000", 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.672433+0000", 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9040999999999999e-05 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.672452+0000", 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8873999999999999e-05 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.672458+0000", 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3160000000000002e-06 2024-09-13T14:51:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.672640+0000", 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00018115699999999999 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.672679+0000", 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9768999999999998e-05 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.585888+0000", 2024-09-13T14:51:48.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.91320911599999999 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.586012+0000", 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000123891 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.586173+0000", 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016028 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.067 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.591536+0000", 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0053636439999999999 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.600970+0000", 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0094335349999999998 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601018+0000", 2024-09-13T14:51:48.068 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7814e-05 2024-09-13T14:51:48.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.070 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.070 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.070 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601060+0000", 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2423999999999999e-05 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601079+0000", 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9378999999999999e-05 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117725 9.10 9:0a83be25:::benchmark_data_smithi003_156985_object117724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.071 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.682193+0000", 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.35319096199999, 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.918997061, 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117725 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.682193+0000", 2024-09-13T14:51:48.072 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.682193+0000", 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.682191+0000", 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.073 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.682223+0000", 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1429e-05 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.682240+0000", 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6952e-05 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.682245+0000", 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9259999999999997e-06 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.074 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.682315+0000", 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9560999999999993e-05 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.682379+0000", 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4117999999999998e-05 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.587039+0000", 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.90466020800000002 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.587158+0000", 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118303 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.587295+0000", 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000137405 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.591543+0000", 2024-09-13T14:51:48.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0042479939999999997 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601117+0000", 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0095735549999999992 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601146+0000", 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9740999999999999e-05 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.077 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601171+0000", 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5018999999999999e-05 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601190+0000", 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8728000000000001e-05 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117752 9.10 9:08223de5:::benchmark_data_smithi003_156985_object117751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.708022+0000", 2024-09-13T14:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.32736185500005, 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.89326906299999997, 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117752 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.708022+0000", 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.708022+0000", 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.708020+0000", 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.080 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.708053+0000", 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3268000000000001e-05 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.708076+0000", 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2963999999999999e-05 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.708084+0000", 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7200000000000006e-06 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.708230+0000", 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014582399999999999 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.708240+0000", 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0521e-05 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.588484+0000", 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.88024334599999998 2024-09-13T14:51:48.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.588608+0000", 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000124358 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.588760+0000", 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000152196 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.593685+0000", 2024-09-13T14:51:48.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0049249580000000001 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601224+0000", 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0075385699999999996 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601250+0000", 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6071000000000002e-05 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.084 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601273+0000", 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3728999999999999e-05 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601291+0000", 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7730000000000001e-05 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117810 9.10 9:0b82ab88:::benchmark_data_smithi003_156985_object117809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.085 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.757446+0000", 2024-09-13T14:51:48.086 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.27793766000002, 2024-09-13T14:51:48.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.84395679800000001, 2024-09-13T14:51:48.086 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.086 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.086 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.086 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.086 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.086 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117810 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.757446+0000", 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.757446+0000", 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.757444+0000", 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.757482+0000", 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8087999999999999e-05 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.757504+0000", 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1061999999999999e-05 2024-09-13T14:51:48.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.757511+0000", 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0010000000000004e-06 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.757656+0000", 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014510900000000001 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.089 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.757680+0000", 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4749e-05 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.589659+0000", 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.83197884200000005 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.589741+0000", 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2280000000000005e-05 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.090 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.589834+0000", 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2726000000000006e-05 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.593694+0000", 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0038593080000000001 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601319+0000", 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0076253340000000001 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601346+0000", 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7520000000000001e-05 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601370+0000", 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3402000000000001e-05 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601403+0000", 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3348e-05 2024-09-13T14:51:48.092 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117827 9.10 9:0d5d0b41:::benchmark_data_smithi003_156985_object117826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.774040+0000", 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.26134398299996, 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.82752425900000004, 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.093 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117827 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.774040+0000", 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.774040+0000", 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.774038+0000", 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.774096+0000", 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8452999999999998e-05 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.774115+0000", 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8898000000000001e-05 2024-09-13T14:51:48.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.774121+0000", 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1380000000000004e-06 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.774258+0000", 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013681899999999999 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.774267+0000", 2024-09-13T14:51:48.096 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3219999999999999e-06 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.590615+0000", 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.81634721499999996 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.590685+0000", 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0094999999999999e-05 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.097 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.590768+0000", 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3607000000000001e-05 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.595772+0000", 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0050033100000000004 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601458+0000", 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0056863870000000002 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601501+0000", 2024-09-13T14:51:48.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2715e-05 2024-09-13T14:51:48.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.099 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601530+0000", 2024-09-13T14:51:48.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8989999999999999e-05 2024-09-13T14:51:48.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.100 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601564+0000", 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4622000000000002e-05 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117835 9.10 9:0eb495a4:::benchmark_data_smithi003_156985_object117834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.784668+0000", 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.25071620599999, 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.81737546000000005, 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.101 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.102 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.102 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.102 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117835 2024-09-13T14:51:48.102 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.102 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.102 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.102 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.102 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.784668+0000", 2024-09-13T14:51:48.102 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.784668+0000", 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.784666+0000", 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.784699+0000", 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3652999999999998e-05 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.784720+0000", 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0530000000000002e-05 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.784728+0000", 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0150000000000003e-06 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.104 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.784810+0000", 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1997999999999997e-05 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.784821+0000", 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1233000000000001e-05 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.591585+0000", 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.80676363799999995 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.105 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.591702+0000", 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011694200000000001 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.591851+0000", 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000148868 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.595778+0000", 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039276190000000002 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.106 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601919+0000", 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.006140805 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.601958+0000", 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8936999999999997e-05 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602007+0000", 2024-09-13T14:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8900000000000003e-05 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602043+0000", 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6371999999999997e-05 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117838 9.10 9:0c7b26aa:::benchmark_data_smithi003_156985_object117837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.785374+0000", 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.25000999500003, 2024-09-13T14:51:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.816894444, 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117838 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.785374+0000", 2024-09-13T14:51:48.109 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.785374+0000", 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.785373+0000", 2024-09-13T14:51:48.110 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:51:48.111 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.111 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.111 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.111 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.785435+0000", 2024-09-13T14:51:48.111 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1309999999999994e-05 2024-09-13T14:51:48.111 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.111 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.111 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.112 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.785446+0000", 2024-09-13T14:51:48.112 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1005999999999999e-05 2024-09-13T14:51:48.112 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.112 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.112 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.112 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.785450+0000", 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4229999999999998e-06 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.785496+0000", 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6144999999999999e-05 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.785506+0000", 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0254999999999999e-05 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.113 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.592632+0000", 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.80712519000000005 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.592715+0000", 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3257999999999995e-05 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.592796+0000", 2024-09-13T14:51:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1508000000000006e-05 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.597772+0000", 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0049757880000000001 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602104+0000", 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0043322810000000003 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.115 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602143+0000", 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8886e-05 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602227+0000", 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3882999999999996e-05 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602268+0000", 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1186000000000002e-05 2024-09-13T14:51:48.116 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117921 9.10 9:0f4bf890:::benchmark_data_smithi003_156985_object117920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.875676+0000", 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.15970831300001, 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.726772258, 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.117 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117921 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.875676+0000", 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.875676+0000", 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.875674+0000", 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.875736+0000", 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2210000000000002e-05 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.119 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.875751+0000", 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4547e-05 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.875757+0000", 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8150000000000002e-06 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.875826+0000", 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9089000000000002e-05 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.875835+0000", 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.0410000000000003e-06 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.593741+0000", 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.71790683799999999 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.121 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.593812+0000", 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0316999999999998e-05 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.593912+0000", 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.9746000000000003e-05 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.597779+0000", 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0038671700000000001 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.122 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602336+0000", 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0045573280000000002 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602369+0000", 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3408000000000003e-05 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602411+0000", 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1433000000000001e-05 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602448+0000", 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7063000000000001e-05 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:117955 9.10 9:0a267b93:::benchmark_data_smithi003_156985_object117954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:15.914475+0000", 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.12090868799999, 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.68818549500000004, 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.124 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117955 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.914475+0000", 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.125 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.126 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.126 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.914475+0000", 2024-09-13T14:51:48.126 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.126 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.126 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.126 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.126 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.914474+0000", 2024-09-13T14:51:48.126 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.914537+0000", 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3484000000000004e-05 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.914554+0000", 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6739e-05 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.127 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.914562+0000", 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2539999999999992e-06 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.914628+0000", 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5843e-05 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:15.914639+0000", 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0761000000000001e-05 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.594699+0000", 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.68006074800000005 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.594822+0000", 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012261600000000001 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.129 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.594982+0000", 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015994300000000001 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.599987+0000", 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0050050169999999996 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602515+0000", 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0025276230000000001 2024-09-13T14:51:48.130 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602573+0000", 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8780999999999998e-05 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602628+0000", 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4676000000000003e-05 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602661+0000", 2024-09-13T14:51:48.131 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2651999999999998e-05 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:118044 9.10 9:099297fa:::benchmark_data_smithi003_156985_object118043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:43:16.022660+0000", 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 512.012723553, 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.58009502899999998, 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:51:48.132 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:51:48.133 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 118044 2024-09-13T14:51:48.133 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.133 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.133 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.133 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.134 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.022660+0000", 2024-09-13T14:51:48.134 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.134 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.134 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.134 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.134 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.022660+0000", 2024-09-13T14:51:48.134 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.134 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.134 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.022659+0000", 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.022704+0000", 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5643999999999999e-05 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.022726+0000", 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1050999999999999e-05 2024-09-13T14:51:48.135 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.022733+0000", 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6979999999999998e-06 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.022820+0000", 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6611000000000001e-05 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:51:48.136 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.022831+0000", 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1544e-05 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.595815+0000", 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.57298342099999999 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.595897+0000", 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2280999999999994e-05 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.137 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.595985+0000", 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7502000000000003e-05 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.599995+0000", 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0040103259999999998 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602696+0000", 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027015590000000001 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.138 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602712+0000", 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5733999999999999e-05 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602739+0000", 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6548999999999999e-05 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:43:16.602755+0000", 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6708e-05 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.139 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79800 10.1 10:8fcf9e35:::benchmark_data_smithi003_197468_object79799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.374328+0000", 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 203.66105630999999, 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30145008699999998, 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:51:48.140 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79800 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.374328+0000", 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.374328+0000", 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.141 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.374326+0000", 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.374359+0000", 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2302999999999999e-05 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.374374+0000", 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4999e-05 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.142 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.374379+0000", 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7710000000000003e-06 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.374448+0000", 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8238999999999995e-05 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.374680+0000", 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00023207100000000001 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.143 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.144 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:51:48.144 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:48.145 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.374908+0000", 2024-09-13T14:51:48.145 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00022835299999999999 2024-09-13T14:51:48.145 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.145 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.145 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.145 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.381816+0000", 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0069075969999999997 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.384767+0000", 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029512190000000002 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.675690+0000", 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29092350700000003 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.146 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.675741+0000", 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0565000000000001e-05 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.675778+0000", 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6931999999999998e-05 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.147 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79802 10.1 10:8531ce40:::benchmark_data_smithi003_197468_object79801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.376134+0000", 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 203.65925032300001, 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29977895599999999, 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79802 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.148 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.376134+0000", 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.376134+0000", 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.376132+0000", 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.149 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.378313+0000", 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0021813000000000002 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.378327+0000", 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3361e-05 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.378333+0000", 2024-09-13T14:51:48.150 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1179999999999997e-06 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.378415+0000", 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1786000000000006e-05 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.378541+0000", 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012600999999999999 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:51:48.151 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.378674+0000", 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013392100000000001 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.381842+0000", 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0031671619999999998 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.384807+0000", 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029654479999999999 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.675818+0000", 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29101100699999999 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.675865+0000", 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6788e-05 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.675913+0000", 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7759000000000001e-05 2024-09-13T14:51:48.153 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79812 10.2 10:4873bc01:::benchmark_data_smithi003_197468_object79811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.385193+0000", 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 203.650191256, 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29071687200000002, 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:51:48.154 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79812 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385193+0000", 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385193+0000", 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.155 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385192+0000", 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385273+0000", 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1082999999999996e-05 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385284+0000", 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1460999999999999e-05 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385290+0000", 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7899999999999996e-06 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385356+0000", 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5955000000000001e-05 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385478+0000", 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012235300000000001 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.157 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385608+0000", 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012998999999999999 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.388359+0000", 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027507460000000001 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.389432+0000", 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001072649 2024-09-13T14:51:48.158 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.675828+0000", 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28639640599999999 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.675868+0000", 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9938000000000003e-05 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.675910+0000", 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1381999999999998e-05 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79815 10.1 10:8d925557:::benchmark_data_smithi003_197468_object79814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.387835+0000", 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 203.647549156, 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29446334000000002, 2024-09-13T14:51:48.160 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.161 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.161 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.161 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:51:48.161 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:51:48.161 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79815 2024-09-13T14:51:48.161 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.161 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.161 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.387835+0000", 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.387835+0000", 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.387833+0000", 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.162 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.387861+0000", 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8000999999999998e-05 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.387879+0000", 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7881e-05 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.387893+0000", 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.464e-05 2024-09-13T14:51:48.163 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.387941+0000", 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7508e-05 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.388056+0000", 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000114838 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:51:48.164 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.388180+0000", 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000124289 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.394046+0000", 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0058660659999999996 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.394358+0000", 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00031189500000000002 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.165 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.682210+0000", 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28785206400000002 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.682262+0000", 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1476e-05 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.682298+0000", 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6560000000000002e-05 2024-09-13T14:51:48.166 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79817 10.1 10:8739a6d0:::benchmark_data_smithi003_197468_object79816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.390265+0000", 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 203.64511914299999, 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29214368699999999, 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:51:48.167 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79817 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390265+0000", 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390265+0000", 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.168 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390264+0000", 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390313+0000", 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9236000000000001e-05 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390331+0000", 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8031000000000001e-05 2024-09-13T14:51:48.169 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390337+0000", 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.84e-06 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390907+0000", 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00057078200000000004 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.170 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391101+0000", 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000193353 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391245+0000", 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000144649 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.394243+0000", 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029977010000000002 2024-09-13T14:51:48.171 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.394432+0000", 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000188961 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.682331+0000", 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28789855399999997 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.172 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.682377+0000", 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6366000000000003e-05 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.682409+0000", 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1551999999999999e-05 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:211567 10.2 10:4f67d55c:::benchmark_data_smithi003_197468_object94838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:50:59.069429+0000", 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 48.965954922999998, 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.208739855, 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211567 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069429+0000", 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.174 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069429+0000", 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069429+0000", 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069431+0000", 2024-09-13T14:51:48.175 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.599e-06 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069439+0000", 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5079999999999999e-06 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069443+0000", 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8600000000000003e-06 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.176 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069490+0000", 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7778999999999998e-05 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069592+0000", 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101412 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069710+0000", 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118328 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.177 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073241+0000", 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0035303719999999999 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.077160+0000", 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039191169999999997 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.278063+0000", 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.20090324400000001 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.178 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.278122+0000", 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9210999999999999e-05 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.278169+0000", 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6844000000000001e-05 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.179 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:211573 10.2 10:49dbb05f:::benchmark_data_smithi003_197468_object94844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:50:59.073455+0000", 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 48.961929288999997, 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21269655600000001, 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211573 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:51:48.180 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073455+0000", 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073455+0000", 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073453+0000", 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.181 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073459+0000", 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8039999999999998e-06 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073469+0000", 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8859999999999999e-06 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073474+0000", 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1320000000000002e-06 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.182 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073519+0000", 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4366000000000002e-05 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073624+0000", 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000105553 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073732+0000", 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000107787 2024-09-13T14:51:48.183 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.077346+0000", 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036141430000000002 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.079713+0000", 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0023672340000000002 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.286072+0000", 2024-09-13T14:51:48.184 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.206358184 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.286116+0000", 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4048000000000002e-05 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.286151+0000", 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5629000000000001e-05 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.185 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:51:48.186 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:51:48.186 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:51:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.209 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:48.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:48.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:48.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:48.458 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:48.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:48.582 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:48.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:48.706 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:48.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:48.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:48.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:51:48.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:48.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:49.029 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:51:49.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:49.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:49.143 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_6' created 2024-09-13T14:51:49.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:49.209 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:49.291 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:49.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:49.315 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:51:49.315 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:49 smithi003 ceph-mon[107278]: pgmap v920: 93 pgs: 93 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:49.315 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:49 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:49.315 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/423133258' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-13T14:51:49.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:49 smithi003 ceph-mon[104007]: pgmap v920: 93 pgs: 93 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:49.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:49 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:49.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/423133258' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-13T14:51:49.319 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:51:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.335 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:49.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:49.413 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:49 smithi019 ceph-mon[91296]: pgmap v920: 93 pgs: 93 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:49.413 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:49 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:49.413 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/423133258' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-13T14:51:49.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:49.458 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:49.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:49.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:49.582 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:49.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:49.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:49.708 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:49.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:49.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:49.935 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.959 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:49.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:50.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:50.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:50.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:51:50.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:50.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:50 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/423133258' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-13T14:51:50.213 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:50 smithi019 ceph-mon[91296]: osdmap e266: 8 total, 8 up, 8 in 2024-09-13T14:51:50.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-09-13T14:51:50.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:50.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:50.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:50 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/423133258' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-13T14:51:50.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:50 smithi003 ceph-mon[107278]: osdmap e266: 8 total, 8 up, 8 in 2024-09-13T14:51:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:50 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/423133258' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-13T14:51:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:50 smithi003 ceph-mon[104007]: osdmap e266: 8 total, 8 up, 8 in 2024-09-13T14:51:50.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:50.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:50.566 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:51:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:51:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:50.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:50.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:50.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:50.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:50.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:50.852 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:50.931 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":921,"stamp":"2024-09-13T14:51:49.142646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622752,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":71,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":880177,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2159,"ondisk_log_size":2159,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6862956,"kb_used_data":635468,"kb_used_omap":29,"kb_used_meta":6227362,"kb_avail":743098260,"statfs":{"total":767960285184,"available":760932618240,"internally_reserved":0,"allocated":650719232,"data_stored":639502470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6376819214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.453822"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":409,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265336+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:51:44.265336+0000","last_peered":"2024-09-13T14:51:44.265336+0000","last_clean":"2024-09-13T14:51:44.265336+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:51:44.265336+0000","last_undegraded":"2024-09-13T14:51:44.265336+0000","last_fullsized":"2024-09-13T14:51:44.265336+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":236,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.905258+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:51:33.905258+0000","last_peered":"2024-09-13T14:51:33.905258+0000","last_clean":"2024-09-13T14:51:33.905258+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:51:33.905258+0000","last_undegraded":"2024-09-13T14:51:33.905258+0000","last_fullsized":"2024-09-13T14:51:33.905258+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":28,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.927497+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:51:40.927497+0000","last_peered":"2024-09-13T14:51:40.927497+0000","last_clean":"2024-09-13T14:51:40.927497+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:51:40.927497+0000","last_undegraded":"2024-09-13T14:51:40.927497+0000","last_fullsized":"2024-09-13T14:51:40.927497+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":368,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839676+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:51:43.839676+0000","last_peered":"2024-09-13T14:51:43.839676+0000","last_clean":"2024-09-13T14:51:43.839676+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:51:43.839676+0000","last_undegraded":"2024-09-13T14:51:43.839676+0000","last_fullsized":"2024-09-13T14:51:43.839676+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":211,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.927679+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:51:40.927679+0000","last_peered":"2024-09-13T14:51:40.927679+0000","last_clean":"2024-09-13T14:51:40.927679+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:51:40.927679+0000","last_undegraded":"2024-09-13T14:51:40.927679+0000","last_fullsized":"2024-09-13T14:51:40.927679+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":243,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.905768+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:51:33.905768+0000","last_peered":"2024-09-13T14:51:33.905768+0000","last_clean":"2024-09-13T14:51:33.905768+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:51:33.905768+0000","last_undegraded":"2024-09-13T14:51:33.905768+0000","last_fullsized":"2024-09-13T14:51:33.905768+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":228,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.860415+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:51:40.860415+0000","last_peered":"2024-09-13T14:51:40.860415+0000","last_clean":"2024-09-13T14:51:40.860415+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:51:40.860415+0000","last_undegraded":"2024-09-13T14:51:40.860415+0000","last_fullsized":"2024-09-13T14:51:40.860415+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":273,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.905933+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:51:33.905933+0000","last_peered":"2024-09-13T14:51:33.905933+0000","last_clean":"2024-09-13T14:51:33.905933+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:51:33.905933+0000","last_undegraded":"2024-09-13T14:51:33.905933+0000","last_fullsized":"2024-09-13T14:51:33.905933+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":397,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265741+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:51:44.265741+0000","last_peered":"2024-09-13T14:51:44.265741+0000","last_clean":"2024-09-13T14:51:44.265741+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:51:44.265741+0000","last_undegraded":"2024-09-13T14:51:44.265741+0000","last_fullsized":"2024-09-13T14:51:44.265741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":392,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265174+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:51:44.265174+0000","last_peered":"2024-09-13T14:51:44.265174+0000","last_clean":"2024-09-13T14:51:44.265174+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:51:44.265174+0000","last_undegraded":"2024-09-13T14:51:44.265174+0000","last_fullsized":"2024-09-13T14:51:44.265174+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":245,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.905928+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:51:33.905928+0000","last_peered":"2024-09-13T14:51:33.905928+0000","last_clean":"2024-09-13T14:51:33.905928+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:51:33.905928+0000","last_undegraded":"2024-09-13T14:51:33.905928+0000","last_fullsized":"2024-09-13T14:51:33.905928+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":334,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839795+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:51:43.839795+0000","last_peered":"2024-09-13T14:51:43.839795+0000","last_clean":"2024-09-13T14:51:43.839795+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:51:43.839795+0000","last_undegraded":"2024-09-13T14:51:43.839795+0000","last_fullsized":"2024-09-13T14:51:43.839795+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":49,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265032+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:51:44.265032+0000","last_peered":"2024-09-13T14:51:44.265032+0000","last_clean":"2024-09-13T14:51:44.265032+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:51:44.265032+0000","last_undegraded":"2024-09-13T14:51:44.265032+0000","last_fullsized":"2024-09-13T14:51:44.265032+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":428,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265644+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:51:44.265644+0000","last_peered":"2024-09-13T14:51:44.265644+0000","last_clean":"2024-09-13T14:51:44.265644+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:51:44.265644+0000","last_undegraded":"2024-09-13T14:51:44.265644+0000","last_fullsized":"2024-09-13T14:51:44.265644+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":367,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839493+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:51:43.839493+0000","last_peered":"2024-09-13T14:51:43.839493+0000","last_clean":"2024-09-13T14:51:43.839493+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:51:43.839493+0000","last_undegraded":"2024-09-13T14:51:43.839493+0000","last_fullsized":"2024-09-13T14:51:43.839493+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":227,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.329745+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:51:41.329745+0000","last_peered":"2024-09-13T14:51:41.329745+0000","last_clean":"2024-09-13T14:51:41.329745+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:51:41.329745+0000","last_undegraded":"2024-09-13T14:51:41.329745+0000","last_fullsized":"2024-09-13T14:51:41.329745+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":335,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839373+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:51:43.839373+0000","last_peered":"2024-09-13T14:51:43.839373+0000","last_clean":"2024-09-13T14:51:43.839373+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:51:43.839373+0000","last_undegraded":"2024-09-13T14:51:43.839373+0000","last_fullsized":"2024-09-13T14:51:43.839373+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":411,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265267+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:51:44.265267+0000","last_peered":"2024-09-13T14:51:44.265267+0000","last_clean":"2024-09-13T14:51:44.265267+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:51:44.265267+0000","last_undegraded":"2024-09-13T14:51:44.265267+0000","last_fullsized":"2024-09-13T14:51:44.265267+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":340,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839485+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:51:43.839485+0000","last_peered":"2024-09-13T14:51:43.839485+0000","last_clean":"2024-09-13T14:51:43.839485+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:51:43.839485+0000","last_undegraded":"2024-09-13T14:51:43.839485+0000","last_fullsized":"2024-09-13T14:51:43.839485+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.927488+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:51:40.927488+0000","last_peered":"2024-09-13T14:51:40.927488+0000","last_clean":"2024-09-13T14:51:40.927488+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:51:40.927488+0000","last_undegraded":"2024-09-13T14:51:40.927488+0000","last_fullsized":"2024-09-13T14:51:40.927488+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":89,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.264980+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:51:44.264980+0000","last_peered":"2024-09-13T14:51:44.264980+0000","last_clean":"2024-09-13T14:51:44.264980+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:51:44.264980+0000","last_undegraded":"2024-09-13T14:51:44.264980+0000","last_fullsized":"2024-09-13T14:51:44.264980+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":28,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.147603+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:51:41.147603+0000","last_peered":"2024-09-13T14:51:41.147603+0000","last_clean":"2024-09-13T14:51:41.147603+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:51:41.147603+0000","last_undegraded":"2024-09-13T14:51:41.147603+0000","last_fullsized":"2024-09-13T14:51:41.147603+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":166,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.328743+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:51:41.328743+0000","last_peered":"2024-09-13T14:51:41.328743+0000","last_clean":"2024-09-13T14:51:41.328743+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:51:41.328743+0000","last_undegraded":"2024-09-13T14:51:41.328743+0000","last_fullsized":"2024-09-13T14:51:41.328743+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":233,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.906343+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:51:33.906343+0000","last_peered":"2024-09-13T14:51:33.906343+0000","last_clean":"2024-09-13T14:51:33.906343+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:51:33.906343+0000","last_undegraded":"2024-09-13T14:51:33.906343+0000","last_fullsized":"2024-09-13T14:51:33.906343+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":226,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.905967+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:51:33.905967+0000","last_peered":"2024-09-13T14:51:33.905967+0000","last_clean":"2024-09-13T14:51:33.905967+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:51:33.905967+0000","last_undegraded":"2024-09-13T14:51:33.905967+0000","last_fullsized":"2024-09-13T14:51:33.905967+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":209,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.905366+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:51:33.905366+0000","last_peered":"2024-09-13T14:51:33.905366+0000","last_clean":"2024-09-13T14:51:33.905366+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:51:33.905366+0000","last_undegraded":"2024-09-13T14:51:33.905366+0000","last_fullsized":"2024-09-13T14:51:33.905366+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":260,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.335236+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:51:41.335236+0000","last_peered":"2024-09-13T14:51:41.335236+0000","last_clean":"2024-09-13T14:51:41.335236+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:51:41.335236+0000","last_undegraded":"2024-09-13T14:51:41.335236+0000","last_fullsized":"2024-09-13T14:51:41.335236+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":409,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265803+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:51:44.265803+0000","last_peered":"2024-09-13T14:51:44.265803+0000","last_clean":"2024-09-13T14:51:44.265803+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:51:44.265803+0000","last_undegraded":"2024-09-13T14:51:44.265803+0000","last_fullsized":"2024-09-13T14:51:44.265803+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":97,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839705+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:51:43.839705+0000","last_peered":"2024-09-13T14:51:43.839705+0000","last_clean":"2024-09-13T14:51:43.839705+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:51:43.839705+0000","last_undegraded":"2024-09-13T14:51:43.839705+0000","last_fullsized":"2024-09-13T14:51:43.839705+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":378,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839408+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:51:43.839408+0000","last_peered":"2024-09-13T14:51:43.839408+0000","last_clean":"2024-09-13T14:51:43.839408+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:51:43.839408+0000","last_undegraded":"2024-09-13T14:51:43.839408+0000","last_fullsized":"2024-09-13T14:51:43.839408+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":101,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.329500+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:51:41.329500+0000","last_peered":"2024-09-13T14:51:41.329500+0000","last_clean":"2024-09-13T14:51:41.329500+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:51:41.329500+0000","last_undegraded":"2024-09-13T14:51:41.329500+0000","last_fullsized":"2024-09-13T14:51:41.329500+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":366,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.840104+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:51:43.840104+0000","last_peered":"2024-09-13T14:51:43.840104+0000","last_clean":"2024-09-13T14:51:43.840104+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:51:43.840104+0000","last_undegraded":"2024-09-13T14:51:43.840104+0000","last_fullsized":"2024-09-13T14:51:43.840104+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":47,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.906261+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:51:33.906261+0000","last_peered":"2024-09-13T14:51:33.906261+0000","last_clean":"2024-09-13T14:51:33.906261+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:51:33.906261+0000","last_undegraded":"2024-09-13T14:51:33.906261+0000","last_fullsized":"2024-09-13T14:51:33.906261+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":395,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265652+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:51:44.265652+0000","last_peered":"2024-09-13T14:51:44.265652+0000","last_clean":"2024-09-13T14:51:44.265652+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:51:44.265652+0000","last_undegraded":"2024-09-13T14:51:44.265652+0000","last_fullsized":"2024-09-13T14:51:44.265652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":240,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.148358+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:51:41.148358+0000","last_peered":"2024-09-13T14:51:41.148358+0000","last_clean":"2024-09-13T14:51:41.148358+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:51:41.148358+0000","last_undegraded":"2024-09-13T14:51:41.148358+0000","last_fullsized":"2024-09-13T14:51:41.148358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":350,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265082+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:51:44.265082+0000","last_peered":"2024-09-13T14:51:44.265082+0000","last_clean":"2024-09-13T14:51:44.265082+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:51:44.265082+0000","last_undegraded":"2024-09-13T14:51:44.265082+0000","last_fullsized":"2024-09-13T14:51:44.265082+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":201,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.329401+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:51:41.329401+0000","last_peered":"2024-09-13T14:51:41.329401+0000","last_clean":"2024-09-13T14:51:41.329401+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:51:41.329401+0000","last_undegraded":"2024-09-13T14:51:41.329401+0000","last_fullsized":"2024-09-13T14:51:41.329401+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.860103+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:51:40.860103+0000","last_peered":"2024-09-13T14:51:40.860103+0000","last_clean":"2024-09-13T14:51:40.860103+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:51:40.860103+0000","last_undegraded":"2024-09-13T14:51:40.860103+0000","last_fullsized":"2024-09-13T14:51:40.860103+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":208,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265250+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:51:44.265250+0000","last_peered":"2024-09-13T14:51:44.265250+0000","last_clean":"2024-09-13T14:51:44.265250+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:51:44.265250+0000","last_undegraded":"2024-09-13T14:51:44.265250+0000","last_fullsized":"2024-09-13T14:51:44.265250+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":243,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.859399+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:51:40.859399+0000","last_peered":"2024-09-13T14:51:40.859399+0000","last_clean":"2024-09-13T14:51:40.859399+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:51:40.859399+0000","last_undegraded":"2024-09-13T14:51:40.859399+0000","last_fullsized":"2024-09-13T14:51:40.859399+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":221,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.329575+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:51:41.329575+0000","last_peered":"2024-09-13T14:51:41.329575+0000","last_clean":"2024-09-13T14:51:41.329575+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:51:41.329575+0000","last_undegraded":"2024-09-13T14:51:41.329575+0000","last_fullsized":"2024-09-13T14:51:41.329575+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":262,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265261+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:51:44.265261+0000","last_peered":"2024-09-13T14:51:44.265261+0000","last_clean":"2024-09-13T14:51:44.265261+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:51:44.265261+0000","last_undegraded":"2024-09-13T14:51:44.265261+0000","last_fullsized":"2024-09-13T14:51:44.265261+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":227,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.906010+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:51:33.906010+0000","last_peered":"2024-09-13T14:51:33.906010+0000","last_clean":"2024-09-13T14:51:33.906010+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:51:33.906010+0000","last_undegraded":"2024-09-13T14:51:33.906010+0000","last_fullsized":"2024-09-13T14:51:33.906010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":239,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.860354+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:51:40.860354+0000","last_peered":"2024-09-13T14:51:40.860354+0000","last_clean":"2024-09-13T14:51:40.860354+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:51:40.860354+0000","last_undegraded":"2024-09-13T14:51:40.860354+0000","last_fullsized":"2024-09-13T14:51:40.860354+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":48,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.859539+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:51:40.859539+0000","last_peered":"2024-09-13T14:51:40.859539+0000","last_clean":"2024-09-13T14:51:40.859539+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:51:40.859539+0000","last_undegraded":"2024-09-13T14:51:40.859539+0000","last_fullsized":"2024-09-13T14:51:40.859539+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":159,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.928356+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:51:40.928356+0000","last_peered":"2024-09-13T14:51:40.928356+0000","last_clean":"2024-09-13T14:51:40.928356+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:51:40.928356+0000","last_undegraded":"2024-09-13T14:51:40.928356+0000","last_fullsized":"2024-09-13T14:51:40.928356+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":203,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.264911+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:51:44.264911+0000","last_peered":"2024-09-13T14:51:44.264911+0000","last_clean":"2024-09-13T14:51:44.264911+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:51:44.264911+0000","last_undegraded":"2024-09-13T14:51:44.264911+0000","last_fullsized":"2024-09-13T14:51:44.264911+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":393,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265365+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:51:44.265365+0000","last_peered":"2024-09-13T14:51:44.265365+0000","last_clean":"2024-09-13T14:51:44.265365+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:51:44.265365+0000","last_undegraded":"2024-09-13T14:51:44.265365+0000","last_fullsized":"2024-09-13T14:51:44.265365+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":22,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.147364+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:51:41.147364+0000","last_peered":"2024-09-13T14:51:41.147364+0000","last_clean":"2024-09-13T14:51:41.147364+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:51:41.147364+0000","last_undegraded":"2024-09-13T14:51:41.147364+0000","last_fullsized":"2024-09-13T14:51:41.147364+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":417,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265563+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:51:44.265563+0000","last_peered":"2024-09-13T14:51:44.265563+0000","last_clean":"2024-09-13T14:51:44.265563+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:51:44.265563+0000","last_undegraded":"2024-09-13T14:51:44.265563+0000","last_fullsized":"2024-09-13T14:51:44.265563+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":426,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265147+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:51:44.265147+0000","last_peered":"2024-09-13T14:51:44.265147+0000","last_clean":"2024-09-13T14:51:44.265147+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:51:44.265147+0000","last_undegraded":"2024-09-13T14:51:44.265147+0000","last_fullsized":"2024-09-13T14:51:44.265147+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":277,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.148384+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:51:41.148384+0000","last_peered":"2024-09-13T14:51:41.148384+0000","last_clean":"2024-09-13T14:51:41.148384+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:51:41.148384+0000","last_undegraded":"2024-09-13T14:51:41.148384+0000","last_fullsized":"2024-09-13T14:51:41.148384+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":201,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.328816+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:51:41.328816+0000","last_peered":"2024-09-13T14:51:41.328816+0000","last_clean":"2024-09-13T14:51:41.328816+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:51:41.328816+0000","last_undegraded":"2024-09-13T14:51:41.328816+0000","last_fullsized":"2024-09-13T14:51:41.328816+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":210,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.329322+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:51:41.329322+0000","last_peered":"2024-09-13T14:51:41.329322+0000","last_clean":"2024-09-13T14:51:41.329322+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:51:41.329322+0000","last_undegraded":"2024-09-13T14:51:41.329322+0000","last_fullsized":"2024-09-13T14:51:41.329322+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":326,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839521+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:51:43.839521+0000","last_peered":"2024-09-13T14:51:43.839521+0000","last_clean":"2024-09-13T14:51:43.839521+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:51:43.839521+0000","last_undegraded":"2024-09-13T14:51:43.839521+0000","last_fullsized":"2024-09-13T14:51:43.839521+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":238,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.860345+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:51:40.860345+0000","last_peered":"2024-09-13T14:51:40.860345+0000","last_clean":"2024-09-13T14:51:40.860345+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:51:40.860345+0000","last_undegraded":"2024-09-13T14:51:40.860345+0000","last_fullsized":"2024-09-13T14:51:40.860345+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":22,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.148170+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:51:41.148170+0000","last_peered":"2024-09-13T14:51:41.148170+0000","last_clean":"2024-09-13T14:51:41.148170+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:51:41.148170+0000","last_undegraded":"2024-09-13T14:51:41.148170+0000","last_fullsized":"2024-09-13T14:51:41.148170+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":49,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265769+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:51:44.265769+0000","last_peered":"2024-09-13T14:51:44.265769+0000","last_clean":"2024-09-13T14:51:44.265769+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:51:44.265769+0000","last_undegraded":"2024-09-13T14:51:44.265769+0000","last_fullsized":"2024-09-13T14:51:44.265769+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":336,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839501+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:51:43.839501+0000","last_peered":"2024-09-13T14:51:43.839501+0000","last_clean":"2024-09-13T14:51:43.839501+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:51:43.839501+0000","last_undegraded":"2024-09-13T14:51:43.839501+0000","last_fullsized":"2024-09-13T14:51:43.839501+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":48,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.860103+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:51:40.860103+0000","last_peered":"2024-09-13T14:51:40.860103+0000","last_clean":"2024-09-13T14:51:40.860103+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:51:40.860103+0000","last_undegraded":"2024-09-13T14:51:40.860103+0000","last_fullsized":"2024-09-13T14:51:40.860103+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":246,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.906595+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:51:33.906595+0000","last_peered":"2024-09-13T14:51:33.906595+0000","last_clean":"2024-09-13T14:51:33.906595+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:51:33.906595+0000","last_undegraded":"2024-09-13T14:51:33.906595+0000","last_fullsized":"2024-09-13T14:51:33.906595+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":227,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.905596+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:51:33.905596+0000","last_peered":"2024-09-13T14:51:33.905596+0000","last_clean":"2024-09-13T14:51:33.905596+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:51:33.905596+0000","last_undegraded":"2024-09-13T14:51:33.905596+0000","last_fullsized":"2024-09-13T14:51:33.905596+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":387,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265465+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:51:44.265465+0000","last_peered":"2024-09-13T14:51:44.265465+0000","last_clean":"2024-09-13T14:51:44.265465+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:51:44.265465+0000","last_undegraded":"2024-09-13T14:51:44.265465+0000","last_fullsized":"2024-09-13T14:51:44.265465+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":243,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.906337+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:51:33.906337+0000","last_peered":"2024-09-13T14:51:33.906337+0000","last_clean":"2024-09-13T14:51:33.906337+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:51:33.906337+0000","last_undegraded":"2024-09-13T14:51:33.906337+0000","last_fullsized":"2024-09-13T14:51:33.906337+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":248,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.328805+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:51:41.328805+0000","last_peered":"2024-09-13T14:51:41.328805+0000","last_clean":"2024-09-13T14:51:41.328805+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:51:41.328805+0000","last_undegraded":"2024-09-13T14:51:41.328805+0000","last_fullsized":"2024-09-13T14:51:41.328805+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":30,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.859444+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:51:40.859444+0000","last_peered":"2024-09-13T14:51:40.859444+0000","last_clean":"2024-09-13T14:51:40.859444+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:51:40.859444+0000","last_undegraded":"2024-09-13T14:51:40.859444+0000","last_fullsized":"2024-09-13T14:51:40.859444+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":226,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.906160+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:51:33.906160+0000","last_peered":"2024-09-13T14:51:33.906160+0000","last_clean":"2024-09-13T14:51:33.906160+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:51:33.906160+0000","last_undegraded":"2024-09-13T14:51:33.906160+0000","last_fullsized":"2024-09-13T14:51:33.906160+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":366,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839721+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:51:43.839721+0000","last_peered":"2024-09-13T14:51:43.839721+0000","last_clean":"2024-09-13T14:51:43.839721+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:51:43.839721+0000","last_undegraded":"2024-09-13T14:51:43.839721+0000","last_fullsized":"2024-09-13T14:51:43.839721+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":48,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.147998+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:51:41.147998+0000","last_peered":"2024-09-13T14:51:41.147998+0000","last_clean":"2024-09-13T14:51:41.147998+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:51:41.147998+0000","last_undegraded":"2024-09-13T14:51:41.147998+0000","last_fullsized":"2024-09-13T14:51:41.147998+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":406,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265605+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:51:44.265605+0000","last_peered":"2024-09-13T14:51:44.265605+0000","last_clean":"2024-09-13T14:51:44.265605+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:51:44.265605+0000","last_undegraded":"2024-09-13T14:51:44.265605+0000","last_fullsized":"2024-09-13T14:51:44.265605+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":226,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.860508+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:51:40.860508+0000","last_peered":"2024-09-13T14:51:40.860508+0000","last_clean":"2024-09-13T14:51:40.860508+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:51:40.860508+0000","last_undegraded":"2024-09-13T14:51:40.860508+0000","last_fullsized":"2024-09-13T14:51:40.860508+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":47,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.906508+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:51:33.906508+0000","last_peered":"2024-09-13T14:51:33.906508+0000","last_clean":"2024-09-13T14:51:33.906508+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:51:33.906508+0000","last_undegraded":"2024-09-13T14:51:33.906508+0000","last_fullsized":"2024-09-13T14:51:33.906508+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":240,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.328679+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:51:41.328679+0000","last_peered":"2024-09-13T14:51:41.328679+0000","last_clean":"2024-09-13T14:51:41.328679+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:51:41.328679+0000","last_undegraded":"2024-09-13T14:51:41.328679+0000","last_fullsized":"2024-09-13T14:51:41.328679+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":241,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.148282+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:51:41.148282+0000","last_peered":"2024-09-13T14:51:41.148282+0000","last_clean":"2024-09-13T14:51:41.148282+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:51:41.148282+0000","last_undegraded":"2024-09-13T14:51:41.148282+0000","last_fullsized":"2024-09-13T14:51:41.148282+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":201,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.329664+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:51:41.329664+0000","last_peered":"2024-09-13T14:51:41.329664+0000","last_clean":"2024-09-13T14:51:41.329664+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:51:41.329664+0000","last_undegraded":"2024-09-13T14:51:41.329664+0000","last_fullsized":"2024-09-13T14:51:41.329664+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":253,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:51:33.905445+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:51:33.905445+0000","last_peered":"2024-09-13T14:51:33.905445+0000","last_clean":"2024-09-13T14:51:33.905445+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:51:33.905445+0000","last_undegraded":"2024-09-13T14:51:33.905445+0000","last_fullsized":"2024-09-13T14:51:33.905445+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":231,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.148081+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:51:41.148081+0000","last_peered":"2024-09-13T14:51:41.148081+0000","last_clean":"2024-09-13T14:51:41.148081+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:51:41.148081+0000","last_undegraded":"2024-09-13T14:51:41.148081+0000","last_fullsized":"2024-09-13T14:51:41.148081+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":94,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839678+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:51:43.839678+0000","last_peered":"2024-09-13T14:51:43.839678+0000","last_clean":"2024-09-13T14:51:43.839678+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:51:43.839678+0000","last_undegraded":"2024-09-13T14:51:43.839678+0000","last_fullsized":"2024-09-13T14:51:43.839678+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:51:49.136139+0000","last_change":"2024-09-13T14:51:49.136139+0000","last_active":"2024-09-13T14:51:49.136139+0000","last_peered":"2024-09-13T14:51:49.136139+0000","last_clean":"2024-09-13T14:51:49.136139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:51:49.136139+0000","last_undegraded":"2024-09-13T14:51:49.136139+0000","last_fullsized":"2024-09-13T14:51:49.136139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":36,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.859488+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:51:40.859488+0000","last_peered":"2024-09-13T14:51:40.859488+0000","last_clean":"2024-09-13T14:51:40.859488+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:51:40.859488+0000","last_undegraded":"2024-09-13T14:51:40.859488+0000","last_fullsized":"2024-09-13T14:51:40.859488+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":26,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.148647+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:51:41.148647+0000","last_peered":"2024-09-13T14:51:41.148647+0000","last_clean":"2024-09-13T14:51:41.148647+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:51:41.148647+0000","last_undegraded":"2024-09-13T14:51:41.148647+0000","last_fullsized":"2024-09-13T14:51:41.148647+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":22,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.147691+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:51:41.147691+0000","last_peered":"2024-09-13T14:51:41.147691+0000","last_clean":"2024-09-13T14:51:41.147691+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:51:41.147691+0000","last_undegraded":"2024-09-13T14:51:41.147691+0000","last_fullsized":"2024-09-13T14:51:41.147691+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":276,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.328725+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:51:41.328725+0000","last_peered":"2024-09-13T14:51:41.328725+0000","last_clean":"2024-09-13T14:51:41.328725+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:51:41.328725+0000","last_undegraded":"2024-09-13T14:51:41.328725+0000","last_fullsized":"2024-09-13T14:51:41.328725+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":375,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.840010+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:51:43.840010+0000","last_peered":"2024-09-13T14:51:43.840010+0000","last_clean":"2024-09-13T14:51:43.840010+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:51:43.840010+0000","last_undegraded":"2024-09-13T14:51:43.840010+0000","last_fullsized":"2024-09-13T14:51:43.840010+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":58,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.860227+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:51:40.860227+0000","last_peered":"2024-09-13T14:51:40.860227+0000","last_clean":"2024-09-13T14:51:40.860227+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:51:40.860227+0000","last_undegraded":"2024-09-13T14:51:40.860227+0000","last_fullsized":"2024-09-13T14:51:40.860227+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":222,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.147113+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:51:41.147113+0000","last_peered":"2024-09-13T14:51:41.147113+0000","last_clean":"2024-09-13T14:51:41.147113+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:51:41.147113+0000","last_undegraded":"2024-09-13T14:51:41.147113+0000","last_fullsized":"2024-09-13T14:51:41.147113+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":419,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265718+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:51:44.265718+0000","last_peered":"2024-09-13T14:51:44.265718+0000","last_clean":"2024-09-13T14:51:44.265718+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:51:44.265718+0000","last_undegraded":"2024-09-13T14:51:44.265718+0000","last_fullsized":"2024-09-13T14:51:44.265718+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":90,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:43.839591+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:51:43.839591+0000","last_peered":"2024-09-13T14:51:43.839591+0000","last_clean":"2024-09-13T14:51:43.839591+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:51:43.839591+0000","last_undegraded":"2024-09-13T14:51:43.839591+0000","last_fullsized":"2024-09-13T14:51:43.839591+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":425,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265876+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:51:44.265876+0000","last_peered":"2024-09-13T14:51:44.265876+0000","last_clean":"2024-09-13T14:51:44.265876+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:51:44.265876+0000","last_undegraded":"2024-09-13T14:51:44.265876+0000","last_fullsized":"2024-09-13T14:51:44.265876+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.927421+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:51:40.927421+0000","last_peered":"2024-09-13T14:51:40.927421+0000","last_clean":"2024-09-13T14:51:40.927421+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:51:40.927421+0000","last_undegraded":"2024-09-13T14:51:40.927421+0000","last_fullsized":"2024-09-13T14:51:40.927421+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":48,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:40.859545+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:51:40.859545+0000","last_peered":"2024-09-13T14:51:40.859545+0000","last_clean":"2024-09-13T14:51:40.859545+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:51:40.859545+0000","last_undegraded":"2024-09-13T14:51:40.859545+0000","last_fullsized":"2024-09-13T14:51:40.859545+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":247,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:51:41.147118+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:51:41.147118+0000","last_peered":"2024-09-13T14:51:41.147118+0000","last_clean":"2024-09-13T14:51:41.147118+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:51:41.147118+0000","last_undegraded":"2024-09-13T14:51:41.147118+0000","last_fullsized":"2024-09-13T14:51:41.147118+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265088+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:51:44.265088+0000","last_peered":"2024-09-13T14:51:44.265088+0000","last_clean":"2024-09-13T14:51:44.265088+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:51:44.265088+0000","last_undegraded":"2024-09-13T14:51:44.265088+0000","last_fullsized":"2024-09-13T14:51:44.265088+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":383,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-13T14:51:44.265256+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:51:44.265256+0000","last_peered":"2024-09-13T14:51:44.265256+0000","last_clean":"2024-09-13T14:51:44.265256+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:51:44.265256+0000","last_undegraded":"2024-09-13T14:51:44.265256+0000","last_fullsized":"2024-09-13T14:51:44.265256+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365185,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908548,"kb_used_data":3320,"kb_used_omap":1,"kb_used_meta":905214,"kb_avail":92836604,"statfs":{"total":95995035648,"available":95064682496,"internally_reserved":0,"allocated":3399680,"data_stored":1987480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":926939480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561415,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":892988,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":889342,"kb_avail":92852164,"statfs":{"total":95995035648,"available":95080615936,"internally_reserved":0,"allocated":3719168,"data_stored":2314983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910686552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790346,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":904616,"kb_used_data":3548,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":92840536,"statfs":{"total":95995035648,"available":95068708864,"internally_reserved":0,"allocated":3633152,"data_stored":2237392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986574,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1513328,"kb_used_data":610600,"kb_used_omap":1,"kb_used_meta":902718,"kb_avail":92231824,"statfs":{"total":95995035648,"available":94445387776,"internally_reserved":0,"allocated":625254400,"data_stored":623850572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924383576},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519730,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542324,"kb_used_data":3880,"kb_used_omap":4,"kb_used_meta":538427,"kb_avail":93202828,"statfs":{"total":95995035648,"available":95439695872,"internally_reserved":0,"allocated":3973120,"data_stored":2573255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":551349696},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304645,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":526328,"kb_used_data":3564,"kb_used_omap":6,"kb_used_meta":522745,"kb_avail":93218824,"statfs":{"total":95995035648,"available":95456075776,"internally_reserved":0,"allocated":3649536,"data_stored":2246089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":535291335},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827949,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837620,"kb_used_data":3304,"kb_used_omap":5,"kb_used_meta":834298,"kb_avail":92907532,"statfs":{"total":95995035648,"available":95137312768,"internally_reserved":0,"allocated":3383296,"data_stored":1982858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":854321388},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285813,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":737204,"kb_used_data":3620,"kb_used_omap":5,"kb_used_meta":733562,"kb_avail":93007948,"statfs":{"total":95995035648,"available":95240138752,"internally_reserved":0,"allocated":3706880,"data_stored":2309841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":751167547},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:50.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:50.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:51.078 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:51:51.079 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:51:51.079 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T14:51:51.079 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:51:51.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.104 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:51.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:51.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:51 smithi003 ceph-mon[107278]: from='client.46439 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:51.166 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:51 smithi003 ceph-mon[107278]: pgmap v922: 109 pgs: 5 creating+peering, 11 unknown, 93 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:51.166 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:51 smithi003 ceph-mon[107278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:51:51.166 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:51 smithi003 ceph-mon[107278]: osdmap e267: 8 total, 8 up, 8 in 2024-09-13T14:51:51.166 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:51 smithi003 ceph-mon[104007]: from='client.46439 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:51.166 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:51 smithi003 ceph-mon[104007]: pgmap v922: 109 pgs: 5 creating+peering, 11 unknown, 93 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:51.166 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:51 smithi003 ceph-mon[104007]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:51:51.166 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:51 smithi003 ceph-mon[104007]: osdmap e267: 8 total, 8 up, 8 in 2024-09-13T14:51:51.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:51.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:51.281 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:51.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.353 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:51.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:51.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:51 smithi019 ceph-mon[91296]: from='client.46439 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:51 smithi019 ceph-mon[91296]: pgmap v922: 109 pgs: 5 creating+peering, 11 unknown, 93 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:51 smithi019 ceph-mon[91296]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:51:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:51 smithi019 ceph-mon[91296]: osdmap e267: 8 total, 8 up, 8 in 2024-09-13T14:51:51.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:51.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:51.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:51.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:51.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:51.682 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:51:51.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:51.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:51.795 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:51.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:51.937 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:51.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:51.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:51.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:52.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:52.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:52.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:52.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:52.309 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:52 smithi019 ceph-mon[91296]: osdmap e268: 8 total, 8 up, 8 in 2024-09-13T14:51:52.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:52.348 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:52.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:52 smithi003 ceph-mon[107278]: osdmap e268: 8 total, 8 up, 8 in 2024-09-13T14:51:52.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:52 smithi003 ceph-mon[104007]: osdmap e268: 8 total, 8 up, 8 in 2024-09-13T14:51:52.450 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.473 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:52.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:52.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:52.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:52.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:52.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:52.824 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:52.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.848 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:52.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:52.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:52.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:53.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:53.096 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:53.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:53.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:53.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:53.357 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:53 smithi003 ceph-mon[107278]: pgmap v925: 109 pgs: 5 creating+peering, 10 unknown, 94 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:53.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:53 smithi003 ceph-mon[104007]: pgmap v925: 109 pgs: 5 creating+peering, 10 unknown, 94 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:53.414 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:53 smithi019 ceph-mon[91296]: pgmap v925: 109 pgs: 5 creating+peering, 10 unknown, 94 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:53.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.473 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:53.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:53.525 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:51:53.525 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:51:53.525 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:51:53.525 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:51:53.525 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:51:53.575 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.603 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:53.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:53.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:53.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:53.828 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:53.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:53.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:53.975 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:54.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.100 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:54.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:54.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:54.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:51:54.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:54.327 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:54.328 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:51:54.328 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:51:54.328 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:51:54.328 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:51:54.328 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:51:54.329 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:51:54.329 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:51:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:54.337 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:51:54.337 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:54.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:54.429 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:51:54] "GET /metrics HTTP/1.1" 200 34112 "" "Prometheus/2.51.0" 2024-09-13T14:51:54.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:54.461 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:54.475 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:54.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:54.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[104007]: pgmap v926: 109 pgs: 5 creating+peering, 5 unknown, 99 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2625676105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:54.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:54.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:54.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[107278]: pgmap v926: 109 pgs: 5 creating+peering, 5 unknown, 99 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:54.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:54.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:54.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:54.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:54 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2625676105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:54.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:54.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.726 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:54.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:54.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.850 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:54.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:54.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:54.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:54.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:54 smithi019 ceph-mon[91296]: pgmap v926: 109 pgs: 5 creating+peering, 5 unknown, 99 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:54.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:54.866 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:54.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:54.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:54 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2625676105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:54.893 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:54.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:54.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:55.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:55.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:55.200 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.225 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:55.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:55.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:55.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:55.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:55.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:55.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:55.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:55.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:55.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:55.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.849 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:55.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:55.950 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:55.973 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:56.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.098 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:56.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:56.172 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:56.172 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:56.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:56.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:56.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.349 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:56.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:56.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:56.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:56.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:56.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:56.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:56.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:56.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:51:56.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:56.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:56.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:56.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:56.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:57.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.099 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:57.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:57.129 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:56 smithi019 ceph-mon[91296]: pgmap v927: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:57.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:56 smithi003 ceph-mon[104007]: pgmap v927: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:57.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:56 smithi003 ceph-mon[107278]: pgmap v927: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:57.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:57.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:57.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:57.346 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:57.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:57.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:51:57.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:57.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:57.475 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:57.574 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.598 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:57.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:57.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:57.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:57.824 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.847 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:57.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:57.903 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-09-13T14:51:57.948 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:57.970 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:58.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:58.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:58.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:58.133 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:57 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3460002288' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:58.133 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:57 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:58.133 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:58.133 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:57 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3460002288' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:57 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:57 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3460002288' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:57 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:51:58.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:58.197 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.221 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:58.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:58.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:58.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:58.410 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:58.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.471 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:58.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:58.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:58.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:58.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:58.659 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:58.659 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:58.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:58.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:58.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:58.842 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:58.908 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:51:58.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:58.967 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:59.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:59.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:58 smithi019 ceph-mon[91296]: pgmap v928: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:58 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:51:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:58 smithi019 ceph-mon[91296]: osdmap e269: 8 total, 8 up, 8 in 2024-09-13T14:51:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:58 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2820847578' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:58 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[107278]: pgmap v928: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:51:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[107278]: osdmap e269: 8 total, 8 up, 8 in 2024-09-13T14:51:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2820847578' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[104007]: pgmap v928: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:51:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:51:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[104007]: osdmap e269: 8 total, 8 up, 8 in 2024-09-13T14:51:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2820847578' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:59.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:58 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:59.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:59.217 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:59.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:51:59.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:59.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:59.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:59.470 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:51:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:59.566 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:59.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:59.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:59.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:59.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:59.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:59.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:59.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:59.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:51:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:59.987 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:59 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:59.987 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:59 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:59.988 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:51:59 smithi019 ceph-mon[91296]: osdmap e270: 8 total, 8 up, 8 in 2024-09-13T14:52:00.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:00.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:00.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:59 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:59 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:52:00.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:51:59 smithi003 ceph-mon[107278]: osdmap e270: 8 total, 8 up, 8 in 2024-09-13T14:52:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:59 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:59 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:52:00.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:51:59 smithi003 ceph-mon[104007]: osdmap e270: 8 total, 8 up, 8 in 2024-09-13T14:52:00.200 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:00.223 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:00.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:00.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:00.407 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:52:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:52:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:00.449 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-09-13T14:52:00.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:00.746 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:52:00.746 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:52:00.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:00.832 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T14:52:00.832 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:52:00.833 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_229782 2024-09-13T14:52:00.833 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:52:00.833 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:52:00.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:00.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:00 smithi003 ceph-mon[104007]: pgmap v931: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:52:00.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:00 smithi003 ceph-mon[104007]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:52:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:00.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:01.007 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:00 smithi019 ceph-mon[91296]: pgmap v931: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:52:01.007 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:00 smithi019 ceph-mon[91296]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:52:01.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:01.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:01.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:00 smithi003 ceph-mon[107278]: pgmap v931: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:52:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:00 smithi003 ceph-mon[107278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:52:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:01.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:01.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:01.321 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:01.321 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:01.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:01.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:01.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:01.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:01.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:01.739 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:01.775 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:01.833 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3663 3647 227.936 227.938 0.00438704 0.00437259 2024-09-13T14:52:01.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:01.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:02.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:52:02.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:02.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:02.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:02.205 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:02.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:52:02.309 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:02.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:52:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:52:02.440 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:52:02.448 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.487 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:02.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:02.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:02.621 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:02.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:02.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:02.836 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 6756 6740 210.337 193.312 0.00442899 0.00474259 2024-09-13T14:52:02.883 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:02.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:03.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:03.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:03.090 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:02 smithi003 ceph-mon[104007]: pgmap v932: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:52:03.090 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:02 smithi003 ceph-mon[107278]: pgmap v932: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:52:03.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:03.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:02 smithi019 ceph-mon[91296]: pgmap v932: 109 pgs: 109 active+clean; 608 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:52:03.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:03.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:03.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:03.430 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.468 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:03.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:03.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:03.585 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:03.591 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":932,"stamp":"2024-09-13T14:52:01.842089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622752,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":71,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":880177,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2159,"ondisk_log_size":2159,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6828096,"kb_used_data":578388,"kb_used_omap":29,"kb_used_meta":6249570,"kb_avail":743133120,"statfs":{"total":767960285184,"available":760968314880,"internally_reserved":0,"allocated":592269312,"data_stored":580808182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6399560206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001993"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":419,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.967246+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:51:59.967246+0000","last_peered":"2024-09-13T14:51:59.967246+0000","last_clean":"2024-09-13T14:51:59.967246+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:51:59.967246+0000","last_undegraded":"2024-09-13T14:51:59.967246+0000","last_fullsized":"2024-09-13T14:51:59.967246+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":241,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.046604+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:51:53.046604+0000","last_peered":"2024-09-13T14:51:53.046604+0000","last_clean":"2024-09-13T14:51:53.046604+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:51:53.046604+0000","last_undegraded":"2024-09-13T14:51:53.046604+0000","last_fullsized":"2024-09-13T14:51:53.046604+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":33,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544506+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:51:58.544506+0000","last_peered":"2024-09-13T14:51:58.544506+0000","last_clean":"2024-09-13T14:51:58.544506+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:51:58.544506+0000","last_undegraded":"2024-09-13T14:51:58.544506+0000","last_fullsized":"2024-09-13T14:51:58.544506+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":374,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440825+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:51:51.440825+0000","last_peered":"2024-09-13T14:51:51.440825+0000","last_clean":"2024-09-13T14:51:51.440825+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:51:51.440825+0000","last_undegraded":"2024-09-13T14:51:51.440825+0000","last_fullsized":"2024-09-13T14:51:51.440825+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":216,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544485+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:51:58.544485+0000","last_peered":"2024-09-13T14:51:58.544485+0000","last_clean":"2024-09-13T14:51:58.544485+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:51:58.544485+0000","last_undegraded":"2024-09-13T14:51:58.544485+0000","last_fullsized":"2024-09-13T14:51:58.544485+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":248,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.046757+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:51:53.046757+0000","last_peered":"2024-09-13T14:51:53.046757+0000","last_clean":"2024-09-13T14:51:53.046757+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:51:53.046757+0000","last_undegraded":"2024-09-13T14:51:53.046757+0000","last_fullsized":"2024-09-13T14:51:53.046757+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":233,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544297+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:51:58.544297+0000","last_peered":"2024-09-13T14:51:58.544297+0000","last_clean":"2024-09-13T14:51:58.544297+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:51:58.544297+0000","last_undegraded":"2024-09-13T14:51:58.544297+0000","last_fullsized":"2024-09-13T14:51:58.544297+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":278,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.047035+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:51:53.047035+0000","last_peered":"2024-09-13T14:51:53.047035+0000","last_clean":"2024-09-13T14:51:53.047035+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:51:53.047035+0000","last_undegraded":"2024-09-13T14:51:53.047035+0000","last_fullsized":"2024-09-13T14:51:53.047035+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":407,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.966914+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:51:59.966914+0000","last_peered":"2024-09-13T14:51:59.966914+0000","last_clean":"2024-09-13T14:51:59.966914+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:51:59.966914+0000","last_undegraded":"2024-09-13T14:51:59.966914+0000","last_fullsized":"2024-09-13T14:51:59.966914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":402,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.289239+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:51:59.289239+0000","last_peered":"2024-09-13T14:51:59.289239+0000","last_clean":"2024-09-13T14:51:59.289239+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:51:59.289239+0000","last_undegraded":"2024-09-13T14:51:59.289239+0000","last_fullsized":"2024-09-13T14:51:59.289239+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":250,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.046855+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:51:53.046855+0000","last_peered":"2024-09-13T14:51:53.046855+0000","last_clean":"2024-09-13T14:51:53.046855+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:51:53.046855+0000","last_undegraded":"2024-09-13T14:51:53.046855+0000","last_fullsized":"2024-09-13T14:51:53.046855+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":340,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440616+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:51:51.440616+0000","last_peered":"2024-09-13T14:51:51.440616+0000","last_clean":"2024-09-13T14:51:51.440616+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:51:51.440616+0000","last_undegraded":"2024-09-13T14:51:51.440616+0000","last_fullsized":"2024-09-13T14:51:51.440616+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":59,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.288905+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:51:59.288905+0000","last_peered":"2024-09-13T14:51:59.288905+0000","last_clean":"2024-09-13T14:51:59.288905+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:51:59.288905+0000","last_undegraded":"2024-09-13T14:51:59.288905+0000","last_fullsized":"2024-09-13T14:51:59.288905+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":438,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.967273+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:51:59.967273+0000","last_peered":"2024-09-13T14:51:59.967273+0000","last_clean":"2024-09-13T14:51:59.967273+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:51:59.967273+0000","last_undegraded":"2024-09-13T14:51:59.967273+0000","last_fullsized":"2024-09-13T14:51:59.967273+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":373,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440507+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:51:51.440507+0000","last_peered":"2024-09-13T14:51:51.440507+0000","last_clean":"2024-09-13T14:51:51.440507+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:51:51.440507+0000","last_undegraded":"2024-09-13T14:51:51.440507+0000","last_fullsized":"2024-09-13T14:51:51.440507+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":231,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230660+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:51:51.230660+0000","last_peered":"2024-09-13T14:51:51.230660+0000","last_clean":"2024-09-13T14:51:51.230660+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:51:51.230660+0000","last_undegraded":"2024-09-13T14:51:51.230660+0000","last_fullsized":"2024-09-13T14:51:51.230660+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":341,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440262+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:51:51.440262+0000","last_peered":"2024-09-13T14:51:51.440262+0000","last_clean":"2024-09-13T14:51:51.440262+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:51:51.440262+0000","last_undegraded":"2024-09-13T14:51:51.440262+0000","last_fullsized":"2024-09-13T14:51:51.440262+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":421,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.967454+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:51:59.967454+0000","last_peered":"2024-09-13T14:51:59.967454+0000","last_clean":"2024-09-13T14:51:59.967454+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:51:59.967454+0000","last_undegraded":"2024-09-13T14:51:59.967454+0000","last_fullsized":"2024-09-13T14:51:59.967454+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":346,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440469+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:51:51.440469+0000","last_peered":"2024-09-13T14:51:51.440469+0000","last_clean":"2024-09-13T14:51:51.440469+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:51:51.440469+0000","last_undegraded":"2024-09-13T14:51:51.440469+0000","last_fullsized":"2024-09-13T14:51:51.440469+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":31,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544490+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:51:58.544490+0000","last_peered":"2024-09-13T14:51:58.544490+0000","last_clean":"2024-09-13T14:51:58.544490+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:51:58.544490+0000","last_undegraded":"2024-09-13T14:51:58.544490+0000","last_fullsized":"2024-09-13T14:51:58.544490+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":100,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.289106+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:51:59.289106+0000","last_peered":"2024-09-13T14:51:59.289106+0000","last_clean":"2024-09-13T14:51:59.289106+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:51:59.289106+0000","last_undegraded":"2024-09-13T14:51:59.289106+0000","last_fullsized":"2024-09-13T14:51:59.289106+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":32,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315581+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:51:53.315581+0000","last_peered":"2024-09-13T14:51:53.315581+0000","last_clean":"2024-09-13T14:51:53.315581+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:51:53.315581+0000","last_undegraded":"2024-09-13T14:51:53.315581+0000","last_fullsized":"2024-09-13T14:51:53.315581+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":170,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230447+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:51:51.230447+0000","last_peered":"2024-09-13T14:51:51.230447+0000","last_clean":"2024-09-13T14:51:51.230447+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:51:51.230447+0000","last_undegraded":"2024-09-13T14:51:51.230447+0000","last_fullsized":"2024-09-13T14:51:51.230447+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":238,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.046540+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:51:53.046540+0000","last_peered":"2024-09-13T14:51:53.046540+0000","last_clean":"2024-09-13T14:51:53.046540+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:51:53.046540+0000","last_undegraded":"2024-09-13T14:51:53.046540+0000","last_fullsized":"2024-09-13T14:51:53.046540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":231,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.047069+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:51:53.047069+0000","last_peered":"2024-09-13T14:51:53.047069+0000","last_clean":"2024-09-13T14:51:53.047069+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:51:53.047069+0000","last_undegraded":"2024-09-13T14:51:53.047069+0000","last_fullsized":"2024-09-13T14:51:53.047069+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.047416+0000","last_change":"2024-09-13T14:51:50.160931+0000","last_active":"2024-09-13T14:51:53.047416+0000","last_peered":"2024-09-13T14:51:53.047416+0000","last_clean":"2024-09-13T14:51:53.047416+0000","last_became_active":"2024-09-13T14:51:50.160636+0000","last_became_peered":"2024-09-13T14:51:50.160636+0000","last_unstale":"2024-09-13T14:51:53.047416+0000","last_undegraded":"2024-09-13T14:51:53.047416+0000","last_fullsized":"2024-09-13T14:51:53.047416+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:05.011657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":214,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.046637+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:51:53.046637+0000","last_peered":"2024-09-13T14:51:53.046637+0000","last_clean":"2024-09-13T14:51:53.046637+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:51:53.046637+0000","last_undegraded":"2024-09-13T14:51:53.046637+0000","last_fullsized":"2024-09-13T14:51:53.046637+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":264,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230686+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:51:51.230686+0000","last_peered":"2024-09-13T14:51:51.230686+0000","last_clean":"2024-09-13T14:51:51.230686+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:51:51.230686+0000","last_undegraded":"2024-09-13T14:51:51.230686+0000","last_fullsized":"2024-09-13T14:51:51.230686+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":419,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.967349+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:51:59.967349+0000","last_peered":"2024-09-13T14:51:59.967349+0000","last_clean":"2024-09-13T14:51:59.967349+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:51:59.967349+0000","last_undegraded":"2024-09-13T14:51:59.967349+0000","last_fullsized":"2024-09-13T14:51:59.967349+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":103,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440492+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:51:51.440492+0000","last_peered":"2024-09-13T14:51:51.440492+0000","last_clean":"2024-09-13T14:51:51.440492+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:51:51.440492+0000","last_undegraded":"2024-09-13T14:51:51.440492+0000","last_fullsized":"2024-09-13T14:51:51.440492+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.316055+0000","last_change":"2024-09-13T14:51:50.161207+0000","last_active":"2024-09-13T14:51:53.316055+0000","last_peered":"2024-09-13T14:51:53.316055+0000","last_clean":"2024-09-13T14:51:53.316055+0000","last_became_active":"2024-09-13T14:51:50.160931+0000","last_became_peered":"2024-09-13T14:51:50.160931+0000","last_unstale":"2024-09-13T14:51:53.316055+0000","last_undegraded":"2024-09-13T14:51:53.316055+0000","last_fullsized":"2024-09-13T14:51:53.316055+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:28.030946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":384,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440361+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:51:51.440361+0000","last_peered":"2024-09-13T14:51:51.440361+0000","last_clean":"2024-09-13T14:51:51.440361+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:51:51.440361+0000","last_undegraded":"2024-09-13T14:51:51.440361+0000","last_fullsized":"2024-09-13T14:51:51.440361+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":105,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230592+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:51:51.230592+0000","last_peered":"2024-09-13T14:51:51.230592+0000","last_clean":"2024-09-13T14:51:51.230592+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:51:51.230592+0000","last_undegraded":"2024-09-13T14:51:51.230592+0000","last_fullsized":"2024-09-13T14:51:51.230592+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":372,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.441129+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:51:51.441129+0000","last_peered":"2024-09-13T14:51:51.441129+0000","last_clean":"2024-09-13T14:51:51.441129+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:51:51.441129+0000","last_undegraded":"2024-09-13T14:51:51.441129+0000","last_fullsized":"2024-09-13T14:51:51.441129+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":52,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.047323+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:51:53.047323+0000","last_peered":"2024-09-13T14:51:53.047323+0000","last_clean":"2024-09-13T14:51:53.047323+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:51:53.047323+0000","last_undegraded":"2024-09-13T14:51:53.047323+0000","last_fullsized":"2024-09-13T14:51:53.047323+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":19,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.289138+0000","last_change":"2024-09-13T14:51:50.159795+0000","last_active":"2024-09-13T14:51:59.289138+0000","last_peered":"2024-09-13T14:51:59.289138+0000","last_clean":"2024-09-13T14:51:59.289138+0000","last_became_active":"2024-09-13T14:51:50.159494+0000","last_became_peered":"2024-09-13T14:51:50.159494+0000","last_unstale":"2024-09-13T14:51:59.289138+0000","last_undegraded":"2024-09-13T14:51:59.289138+0000","last_fullsized":"2024-09-13T14:51:59.289138+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:22.123702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":405,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.966906+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:51:59.966906+0000","last_peered":"2024-09-13T14:51:59.966906+0000","last_clean":"2024-09-13T14:51:59.966906+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:51:59.966906+0000","last_undegraded":"2024-09-13T14:51:59.966906+0000","last_fullsized":"2024-09-13T14:51:59.966906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":244,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315825+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:51:53.315825+0000","last_peered":"2024-09-13T14:51:53.315825+0000","last_clean":"2024-09-13T14:51:53.315825+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:51:53.315825+0000","last_undegraded":"2024-09-13T14:51:53.315825+0000","last_fullsized":"2024-09-13T14:51:53.315825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":361,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.289270+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:51:59.289270+0000","last_peered":"2024-09-13T14:51:59.289270+0000","last_clean":"2024-09-13T14:51:59.289270+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:51:59.289270+0000","last_undegraded":"2024-09-13T14:51:59.289270+0000","last_fullsized":"2024-09-13T14:51:59.289270+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":205,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230453+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:51:51.230453+0000","last_peered":"2024-09-13T14:51:51.230453+0000","last_clean":"2024-09-13T14:51:51.230453+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:51:51.230453+0000","last_undegraded":"2024-09-13T14:51:51.230453+0000","last_fullsized":"2024-09-13T14:51:51.230453+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":19,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.289299+0000","last_change":"2024-09-13T14:51:50.162139+0000","last_active":"2024-09-13T14:51:59.289299+0000","last_peered":"2024-09-13T14:51:59.289299+0000","last_clean":"2024-09-13T14:51:59.289299+0000","last_became_active":"2024-09-13T14:51:50.161828+0000","last_became_peered":"2024-09-13T14:51:50.161828+0000","last_unstale":"2024-09-13T14:51:59.289299+0000","last_undegraded":"2024-09-13T14:51:59.289299+0000","last_fullsized":"2024-09-13T14:51:59.289299+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:18.448351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544263+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:51:58.544263+0000","last_peered":"2024-09-13T14:51:58.544263+0000","last_clean":"2024-09-13T14:51:58.544263+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:51:58.544263+0000","last_undegraded":"2024-09-13T14:51:58.544263+0000","last_fullsized":"2024-09-13T14:51:58.544263+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":219,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.967166+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:51:59.967166+0000","last_peered":"2024-09-13T14:51:59.967166+0000","last_clean":"2024-09-13T14:51:59.967166+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:51:59.967166+0000","last_undegraded":"2024-09-13T14:51:59.967166+0000","last_fullsized":"2024-09-13T14:51:59.967166+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":248,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544546+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:51:58.544546+0000","last_peered":"2024-09-13T14:51:58.544546+0000","last_clean":"2024-09-13T14:51:58.544546+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:51:58.544546+0000","last_undegraded":"2024-09-13T14:51:58.544546+0000","last_fullsized":"2024-09-13T14:51:58.544546+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":225,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230562+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:51:51.230562+0000","last_peered":"2024-09-13T14:51:51.230562+0000","last_clean":"2024-09-13T14:51:51.230562+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:51:51.230562+0000","last_undegraded":"2024-09-13T14:51:51.230562+0000","last_fullsized":"2024-09-13T14:51:51.230562+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":14,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440563+0000","last_change":"2024-09-13T14:51:50.161719+0000","last_active":"2024-09-13T14:51:51.440563+0000","last_peered":"2024-09-13T14:51:51.440563+0000","last_clean":"2024-09-13T14:51:51.440563+0000","last_became_active":"2024-09-13T14:51:50.161426+0000","last_became_peered":"2024-09-13T14:51:50.161426+0000","last_unstale":"2024-09-13T14:51:51.440563+0000","last_undegraded":"2024-09-13T14:51:51.440563+0000","last_fullsized":"2024-09-13T14:51:51.440563+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.259535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":273,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.289375+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:51:59.289375+0000","last_peered":"2024-09-13T14:51:59.289375+0000","last_clean":"2024-09-13T14:51:59.289375+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:51:59.289375+0000","last_undegraded":"2024-09-13T14:51:59.289375+0000","last_fullsized":"2024-09-13T14:51:59.289375+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":232,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.047118+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:51:53.047118+0000","last_peered":"2024-09-13T14:51:53.047118+0000","last_clean":"2024-09-13T14:51:53.047118+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:51:53.047118+0000","last_undegraded":"2024-09-13T14:51:53.047118+0000","last_fullsized":"2024-09-13T14:51:53.047118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":244,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544428+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:51:58.544428+0000","last_peered":"2024-09-13T14:51:58.544428+0000","last_clean":"2024-09-13T14:51:58.544428+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:51:58.544428+0000","last_undegraded":"2024-09-13T14:51:58.544428+0000","last_fullsized":"2024-09-13T14:51:58.544428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":53,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544111+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:51:58.544111+0000","last_peered":"2024-09-13T14:51:58.544111+0000","last_clean":"2024-09-13T14:51:58.544111+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:51:58.544111+0000","last_undegraded":"2024-09-13T14:51:58.544111+0000","last_fullsized":"2024-09-13T14:51:58.544111+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440400+0000","last_change":"2024-09-13T14:51:50.163583+0000","last_active":"2024-09-13T14:51:51.440400+0000","last_peered":"2024-09-13T14:51:51.440400+0000","last_clean":"2024-09-13T14:51:51.440400+0000","last_became_active":"2024-09-13T14:51:50.163295+0000","last_became_peered":"2024-09-13T14:51:50.163295+0000","last_unstale":"2024-09-13T14:51:51.440400+0000","last_undegraded":"2024-09-13T14:51:51.440400+0000","last_fullsized":"2024-09-13T14:51:51.440400+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.168486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":164,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544588+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:51:58.544588+0000","last_peered":"2024-09-13T14:51:58.544588+0000","last_clean":"2024-09-13T14:51:58.544588+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:51:58.544588+0000","last_undegraded":"2024-09-13T14:51:58.544588+0000","last_fullsized":"2024-09-13T14:51:58.544588+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":213,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.288906+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:51:59.288906+0000","last_peered":"2024-09-13T14:51:59.288906+0000","last_clean":"2024-09-13T14:51:59.288906+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:51:59.288906+0000","last_undegraded":"2024-09-13T14:51:59.288906+0000","last_fullsized":"2024-09-13T14:51:59.288906+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":403,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.289104+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:51:59.289104+0000","last_peered":"2024-09-13T14:51:59.289104+0000","last_clean":"2024-09-13T14:51:59.289104+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:51:59.289104+0000","last_undegraded":"2024-09-13T14:51:59.289104+0000","last_fullsized":"2024-09-13T14:51:59.289104+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":26,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315301+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:51:53.315301+0000","last_peered":"2024-09-13T14:51:53.315301+0000","last_clean":"2024-09-13T14:51:53.315301+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:51:53.315301+0000","last_undegraded":"2024-09-13T14:51:53.315301+0000","last_fullsized":"2024-09-13T14:51:53.315301+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440407+0000","last_change":"2024-09-13T14:51:50.162553+0000","last_active":"2024-09-13T14:51:51.440407+0000","last_peered":"2024-09-13T14:51:51.440407+0000","last_clean":"2024-09-13T14:51:51.440407+0000","last_became_active":"2024-09-13T14:51:50.162256+0000","last_became_peered":"2024-09-13T14:51:50.162256+0000","last_unstale":"2024-09-13T14:51:51.440407+0000","last_undegraded":"2024-09-13T14:51:51.440407+0000","last_fullsized":"2024-09-13T14:51:51.440407+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:20.561325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":427,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.967178+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:51:59.967178+0000","last_peered":"2024-09-13T14:51:59.967178+0000","last_clean":"2024-09-13T14:51:59.967178+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:51:59.967178+0000","last_undegraded":"2024-09-13T14:51:59.967178+0000","last_fullsized":"2024-09-13T14:51:59.967178+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":436,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.966643+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:51:59.966643+0000","last_peered":"2024-09-13T14:51:59.966643+0000","last_clean":"2024-09-13T14:51:59.966643+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:51:59.966643+0000","last_undegraded":"2024-09-13T14:51:59.966643+0000","last_fullsized":"2024-09-13T14:51:59.966643+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":281,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315559+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:51:53.315559+0000","last_peered":"2024-09-13T14:51:53.315559+0000","last_clean":"2024-09-13T14:51:53.315559+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:51:53.315559+0000","last_undegraded":"2024-09-13T14:51:53.315559+0000","last_fullsized":"2024-09-13T14:51:53.315559+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":205,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230282+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:51:51.230282+0000","last_peered":"2024-09-13T14:51:51.230282+0000","last_clean":"2024-09-13T14:51:51.230282+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:51:51.230282+0000","last_undegraded":"2024-09-13T14:51:51.230282+0000","last_fullsized":"2024-09-13T14:51:51.230282+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440501+0000","last_change":"2024-09-13T14:51:50.164744+0000","last_active":"2024-09-13T14:51:51.440501+0000","last_peered":"2024-09-13T14:51:51.440501+0000","last_clean":"2024-09-13T14:51:51.440501+0000","last_became_active":"2024-09-13T14:51:50.164428+0000","last_became_peered":"2024-09-13T14:51:50.164428+0000","last_unstale":"2024-09-13T14:51:51.440501+0000","last_undegraded":"2024-09-13T14:51:51.440501+0000","last_fullsized":"2024-09-13T14:51:51.440501+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:42.472246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":214,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230605+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:51:51.230605+0000","last_peered":"2024-09-13T14:51:51.230605+0000","last_clean":"2024-09-13T14:51:51.230605+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:51:51.230605+0000","last_undegraded":"2024-09-13T14:51:51.230605+0000","last_fullsized":"2024-09-13T14:51:51.230605+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":332,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440417+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:51:51.440417+0000","last_peered":"2024-09-13T14:51:51.440417+0000","last_clean":"2024-09-13T14:51:51.440417+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:51:51.440417+0000","last_undegraded":"2024-09-13T14:51:51.440417+0000","last_fullsized":"2024-09-13T14:51:51.440417+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":243,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544217+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:51:58.544217+0000","last_peered":"2024-09-13T14:51:58.544217+0000","last_clean":"2024-09-13T14:51:58.544217+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:51:58.544217+0000","last_undegraded":"2024-09-13T14:51:58.544217+0000","last_fullsized":"2024-09-13T14:51:58.544217+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":26,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315128+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:51:53.315128+0000","last_peered":"2024-09-13T14:51:53.315128+0000","last_clean":"2024-09-13T14:51:53.315128+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:51:53.315128+0000","last_undegraded":"2024-09-13T14:51:53.315128+0000","last_fullsized":"2024-09-13T14:51:53.315128+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.441059+0000","last_change":"2024-09-13T14:51:50.159479+0000","last_active":"2024-09-13T14:51:51.441059+0000","last_peered":"2024-09-13T14:51:51.441059+0000","last_clean":"2024-09-13T14:51:51.441059+0000","last_became_active":"2024-09-13T14:51:50.159183+0000","last_became_peered":"2024-09-13T14:51:50.159183+0000","last_unstale":"2024-09-13T14:51:51.441059+0000","last_undegraded":"2024-09-13T14:51:51.441059+0000","last_fullsized":"2024-09-13T14:51:51.441059+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:31.284043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":59,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.289460+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:51:59.289460+0000","last_peered":"2024-09-13T14:51:59.289460+0000","last_clean":"2024-09-13T14:51:59.289460+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:51:59.289460+0000","last_undegraded":"2024-09-13T14:51:59.289460+0000","last_fullsized":"2024-09-13T14:51:59.289460+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":342,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440699+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:51:51.440699+0000","last_peered":"2024-09-13T14:51:51.440699+0000","last_clean":"2024-09-13T14:51:51.440699+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:51:51.440699+0000","last_undegraded":"2024-09-13T14:51:51.440699+0000","last_fullsized":"2024-09-13T14:51:51.440699+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":53,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544150+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:51:58.544150+0000","last_peered":"2024-09-13T14:51:58.544150+0000","last_clean":"2024-09-13T14:51:58.544150+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:51:58.544150+0000","last_undegraded":"2024-09-13T14:51:58.544150+0000","last_fullsized":"2024-09-13T14:51:58.544150+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.229989+0000","last_change":"2024-09-13T14:51:50.162786+0000","last_active":"2024-09-13T14:51:51.229989+0000","last_peered":"2024-09-13T14:51:51.229989+0000","last_clean":"2024-09-13T14:51:51.229989+0000","last_became_active":"2024-09-13T14:51:50.162300+0000","last_became_peered":"2024-09-13T14:51:50.162300+0000","last_unstale":"2024-09-13T14:51:51.229989+0000","last_undegraded":"2024-09-13T14:51:51.229989+0000","last_fullsized":"2024-09-13T14:51:51.229989+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:16:10.631643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":251,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.047490+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:51:53.047490+0000","last_peered":"2024-09-13T14:51:53.047490+0000","last_clean":"2024-09-13T14:51:53.047490+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:51:53.047490+0000","last_undegraded":"2024-09-13T14:51:53.047490+0000","last_fullsized":"2024-09-13T14:51:53.047490+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":232,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.046789+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:51:53.046789+0000","last_peered":"2024-09-13T14:51:53.046789+0000","last_clean":"2024-09-13T14:51:53.046789+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:51:53.046789+0000","last_undegraded":"2024-09-13T14:51:53.046789+0000","last_fullsized":"2024-09-13T14:51:53.046789+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":397,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.966721+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:51:59.966721+0000","last_peered":"2024-09-13T14:51:59.966721+0000","last_clean":"2024-09-13T14:51:59.966721+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:51:59.966721+0000","last_undegraded":"2024-09-13T14:51:59.966721+0000","last_fullsized":"2024-09-13T14:51:59.966721+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":19,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.288981+0000","last_change":"2024-09-13T14:51:50.163483+0000","last_active":"2024-09-13T14:51:59.288981+0000","last_peered":"2024-09-13T14:51:59.288981+0000","last_clean":"2024-09-13T14:51:59.288981+0000","last_became_active":"2024-09-13T14:51:50.163168+0000","last_became_peered":"2024-09-13T14:51:50.163168+0000","last_unstale":"2024-09-13T14:51:59.288981+0000","last_undegraded":"2024-09-13T14:51:59.288981+0000","last_fullsized":"2024-09-13T14:51:59.288981+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:29.480939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":248,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.047480+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:51:53.047480+0000","last_peered":"2024-09-13T14:51:53.047480+0000","last_clean":"2024-09-13T14:51:53.047480+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:51:53.047480+0000","last_undegraded":"2024-09-13T14:51:53.047480+0000","last_fullsized":"2024-09-13T14:51:53.047480+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":252,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230608+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:51:51.230608+0000","last_peered":"2024-09-13T14:51:51.230608+0000","last_clean":"2024-09-13T14:51:51.230608+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:51:51.230608+0000","last_undegraded":"2024-09-13T14:51:51.230608+0000","last_fullsized":"2024-09-13T14:51:51.230608+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":35,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544141+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:51:58.544141+0000","last_peered":"2024-09-13T14:51:58.544141+0000","last_clean":"2024-09-13T14:51:58.544141+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:51:58.544141+0000","last_undegraded":"2024-09-13T14:51:58.544141+0000","last_fullsized":"2024-09-13T14:51:58.544141+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230536+0000","last_change":"2024-09-13T14:51:50.162628+0000","last_active":"2024-09-13T14:51:51.230536+0000","last_peered":"2024-09-13T14:51:51.230536+0000","last_clean":"2024-09-13T14:51:51.230536+0000","last_became_active":"2024-09-13T14:51:50.162356+0000","last_became_peered":"2024-09-13T14:51:50.162356+0000","last_unstale":"2024-09-13T14:51:51.230536+0000","last_undegraded":"2024-09-13T14:51:51.230536+0000","last_fullsized":"2024-09-13T14:51:51.230536+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:43.465982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":231,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.047111+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:51:53.047111+0000","last_peered":"2024-09-13T14:51:53.047111+0000","last_clean":"2024-09-13T14:51:53.047111+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:51:53.047111+0000","last_undegraded":"2024-09-13T14:51:53.047111+0000","last_fullsized":"2024-09-13T14:51:53.047111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":372,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440913+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:51:51.440913+0000","last_peered":"2024-09-13T14:51:51.440913+0000","last_clean":"2024-09-13T14:51:51.440913+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:51:51.440913+0000","last_undegraded":"2024-09-13T14:51:51.440913+0000","last_fullsized":"2024-09-13T14:51:51.440913+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":52,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315328+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:51:53.315328+0000","last_peered":"2024-09-13T14:51:53.315328+0000","last_clean":"2024-09-13T14:51:53.315328+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:51:53.315328+0000","last_undegraded":"2024-09-13T14:51:53.315328+0000","last_fullsized":"2024-09-13T14:51:53.315328+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315534+0000","last_change":"2024-09-13T14:51:50.161603+0000","last_active":"2024-09-13T14:51:53.315534+0000","last_peered":"2024-09-13T14:51:53.315534+0000","last_clean":"2024-09-13T14:51:53.315534+0000","last_became_active":"2024-09-13T14:51:50.161319+0000","last_became_peered":"2024-09-13T14:51:50.161319+0000","last_unstale":"2024-09-13T14:51:53.315534+0000","last_undegraded":"2024-09-13T14:51:53.315534+0000","last_fullsized":"2024-09-13T14:51:53.315534+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:55.947148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":416,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.966757+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:51:59.966757+0000","last_peered":"2024-09-13T14:51:59.966757+0000","last_clean":"2024-09-13T14:51:59.966757+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:51:59.966757+0000","last_undegraded":"2024-09-13T14:51:59.966757+0000","last_fullsized":"2024-09-13T14:51:59.966757+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":231,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544388+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:51:58.544388+0000","last_peered":"2024-09-13T14:51:58.544388+0000","last_clean":"2024-09-13T14:51:58.544388+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:51:58.544388+0000","last_undegraded":"2024-09-13T14:51:58.544388+0000","last_fullsized":"2024-09-13T14:51:58.544388+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":52,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.047252+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:51:53.047252+0000","last_peered":"2024-09-13T14:51:53.047252+0000","last_clean":"2024-09-13T14:51:53.047252+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:51:53.047252+0000","last_undegraded":"2024-09-13T14:51:53.047252+0000","last_fullsized":"2024-09-13T14:51:53.047252+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315704+0000","last_change":"2024-09-13T14:51:50.161149+0000","last_active":"2024-09-13T14:51:53.315704+0000","last_peered":"2024-09-13T14:51:53.315704+0000","last_clean":"2024-09-13T14:51:53.315704+0000","last_became_active":"2024-09-13T14:51:50.160859+0000","last_became_peered":"2024-09-13T14:51:50.160859+0000","last_unstale":"2024-09-13T14:51:53.315704+0000","last_undegraded":"2024-09-13T14:51:53.315704+0000","last_fullsized":"2024-09-13T14:51:53.315704+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:40.847290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":244,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230342+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:51:51.230342+0000","last_peered":"2024-09-13T14:51:51.230342+0000","last_clean":"2024-09-13T14:51:51.230342+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:51:51.230342+0000","last_undegraded":"2024-09-13T14:51:51.230342+0000","last_fullsized":"2024-09-13T14:51:51.230342+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":245,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315824+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:51:53.315824+0000","last_peered":"2024-09-13T14:51:53.315824+0000","last_clean":"2024-09-13T14:51:53.315824+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:51:53.315824+0000","last_undegraded":"2024-09-13T14:51:53.315824+0000","last_fullsized":"2024-09-13T14:51:53.315824+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":205,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230639+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:51:51.230639+0000","last_peered":"2024-09-13T14:51:51.230639+0000","last_clean":"2024-09-13T14:51:51.230639+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:51:51.230639+0000","last_undegraded":"2024-09-13T14:51:51.230639+0000","last_fullsized":"2024-09-13T14:51:51.230639+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":19,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.967237+0000","last_change":"2024-09-13T14:51:50.162266+0000","last_active":"2024-09-13T14:51:59.967237+0000","last_peered":"2024-09-13T14:51:59.967237+0000","last_clean":"2024-09-13T14:51:59.967237+0000","last_became_active":"2024-09-13T14:51:50.161674+0000","last_became_peered":"2024-09-13T14:51:50.161674+0000","last_unstale":"2024-09-13T14:51:59.967237+0000","last_undegraded":"2024-09-13T14:51:59.967237+0000","last_fullsized":"2024-09-13T14:51:59.967237+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:59.738238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":258,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.046683+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:51:53.046683+0000","last_peered":"2024-09-13T14:51:53.046683+0000","last_clean":"2024-09-13T14:51:53.046683+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:51:53.046683+0000","last_undegraded":"2024-09-13T14:51:53.046683+0000","last_fullsized":"2024-09-13T14:51:53.046683+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":235,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315939+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:51:53.315939+0000","last_peered":"2024-09-13T14:51:53.315939+0000","last_clean":"2024-09-13T14:51:53.315939+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:51:53.315939+0000","last_undegraded":"2024-09-13T14:51:53.315939+0000","last_fullsized":"2024-09-13T14:51:53.315939+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":100,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440566+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:51:51.440566+0000","last_peered":"2024-09-13T14:51:51.440566+0000","last_clean":"2024-09-13T14:51:51.440566+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:51:51.440566+0000","last_undegraded":"2024-09-13T14:51:51.440566+0000","last_fullsized":"2024-09-13T14:51:51.440566+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544018+0000","last_change":"2024-09-13T14:51:50.159778+0000","last_active":"2024-09-13T14:51:58.544018+0000","last_peered":"2024-09-13T14:51:58.544018+0000","last_clean":"2024-09-13T14:51:58.544018+0000","last_became_active":"2024-09-13T14:51:50.159462+0000","last_became_peered":"2024-09-13T14:51:50.159462+0000","last_unstale":"2024-09-13T14:51:58.544018+0000","last_undegraded":"2024-09-13T14:51:58.544018+0000","last_fullsized":"2024-09-13T14:51:58.544018+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:20.012384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":41,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544263+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:51:58.544263+0000","last_peered":"2024-09-13T14:51:58.544263+0000","last_clean":"2024-09-13T14:51:58.544263+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:51:58.544263+0000","last_undegraded":"2024-09-13T14:51:58.544263+0000","last_fullsized":"2024-09-13T14:51:58.544263+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":30,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.316032+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:51:53.316032+0000","last_peered":"2024-09-13T14:51:53.316032+0000","last_clean":"2024-09-13T14:51:53.316032+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:51:53.316032+0000","last_undegraded":"2024-09-13T14:51:53.316032+0000","last_fullsized":"2024-09-13T14:51:53.316032+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":26,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315582+0000","last_change":"2024-09-13T14:49:05.826657+0000","last_active":"2024-09-13T14:51:53.315582+0000","last_peered":"2024-09-13T14:51:53.315582+0000","last_clean":"2024-09-13T14:51:53.315582+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:51:53.315582+0000","last_undegraded":"2024-09-13T14:51:53.315582+0000","last_fullsized":"2024-09-13T14:51:53.315582+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:35.387452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:35.387452+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":280,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.230210+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:51:51.230210+0000","last_peered":"2024-09-13T14:51:51.230210+0000","last_clean":"2024-09-13T14:51:51.230210+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:51:51.230210+0000","last_undegraded":"2024-09-13T14:51:51.230210+0000","last_fullsized":"2024-09-13T14:51:51.230210+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":381,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.441067+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:51:51.441067+0000","last_peered":"2024-09-13T14:51:51.441067+0000","last_clean":"2024-09-13T14:51:51.441067+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:51:51.441067+0000","last_undegraded":"2024-09-13T14:51:51.441067+0000","last_fullsized":"2024-09-13T14:51:51.441067+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":63,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544321+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:51:58.544321+0000","last_peered":"2024-09-13T14:51:58.544321+0000","last_clean":"2024-09-13T14:51:58.544321+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:51:58.544321+0000","last_undegraded":"2024-09-13T14:51:58.544321+0000","last_fullsized":"2024-09-13T14:51:58.544321+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":226,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315384+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:51:53.315384+0000","last_peered":"2024-09-13T14:51:53.315384+0000","last_clean":"2024-09-13T14:51:53.315384+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:51:53.315384+0000","last_undegraded":"2024-09-13T14:51:53.315384+0000","last_fullsized":"2024-09-13T14:51:53.315384+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":429,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.967294+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:51:59.967294+0000","last_peered":"2024-09-13T14:51:59.967294+0000","last_clean":"2024-09-13T14:51:59.967294+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:51:59.967294+0000","last_undegraded":"2024-09-13T14:51:59.967294+0000","last_fullsized":"2024-09-13T14:51:59.967294+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":96,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:51.440695+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:51:51.440695+0000","last_peered":"2024-09-13T14:51:51.440695+0000","last_clean":"2024-09-13T14:51:51.440695+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:51:51.440695+0000","last_undegraded":"2024-09-13T14:51:51.440695+0000","last_fullsized":"2024-09-13T14:51:51.440695+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":435,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.966739+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:51:59.966739+0000","last_peered":"2024-09-13T14:51:59.966739+0000","last_clean":"2024-09-13T14:51:59.966739+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:51:59.966739+0000","last_undegraded":"2024-09-13T14:51:59.966739+0000","last_fullsized":"2024-09-13T14:51:59.966739+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544423+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:51:58.544423+0000","last_peered":"2024-09-13T14:51:58.544423+0000","last_clean":"2024-09-13T14:51:58.544423+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:51:58.544423+0000","last_undegraded":"2024-09-13T14:51:58.544423+0000","last_fullsized":"2024-09-13T14:51:58.544423+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":53,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.544290+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:51:58.544290+0000","last_peered":"2024-09-13T14:51:58.544290+0000","last_clean":"2024-09-13T14:51:58.544290+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:51:58.544290+0000","last_undegraded":"2024-09-13T14:51:58.544290+0000","last_fullsized":"2024-09-13T14:51:58.544290+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":251,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:51:53.315512+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:51:53.315512+0000","last_peered":"2024-09-13T14:51:53.315512+0000","last_clean":"2024-09-13T14:51:53.315512+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:51:53.315512+0000","last_undegraded":"2024-09-13T14:51:53.315512+0000","last_fullsized":"2024-09-13T14:51:53.315512+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":94,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.966830+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:51:59.966830+0000","last_peered":"2024-09-13T14:51:59.966830+0000","last_clean":"2024-09-13T14:51:59.966830+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:51:59.966830+0000","last_undegraded":"2024-09-13T14:51:59.966830+0000","last_fullsized":"2024-09-13T14:51:59.966830+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":393,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.966691+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:51:59.966691+0000","last_peered":"2024-09-13T14:51:59.966691+0000","last_clean":"2024-09-13T14:51:59.966691+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:51:59.966691+0000","last_undegraded":"2024-09-13T14:51:59.966691+0000","last_fullsized":"2024-09-13T14:51:59.966691+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365187,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908596,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":905214,"kb_avail":92836556,"statfs":{"total":95995035648,"available":95064633344,"internally_reserved":0,"allocated":3448832,"data_stored":2013810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":926939480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561417,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893036,"kb_used_data":3680,"kb_used_omap":1,"kb_used_meta":889342,"kb_avail":92852116,"statfs":{"total":95995035648,"available":95080566784,"internally_reserved":0,"allocated":3768320,"data_stored":2341313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910686552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790349,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":904712,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":92840440,"statfs":{"total":95995035648,"available":95068610560,"internally_reserved":0,"allocated":3727360,"data_stored":2289311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986577,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1459964,"kb_used_data":553076,"kb_used_omap":1,"kb_used_meta":906878,"kb_avail":92285188,"statfs":{"total":95995035648,"available":94500032512,"internally_reserved":0,"allocated":566349824,"data_stored":564911553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":928643416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519732,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560424,"kb_used_data":3928,"kb_used_omap":4,"kb_used_meta":556475,"kb_avail":93184728,"statfs":{"total":95995035648,"available":95421161472,"internally_reserved":0,"allocated":4022272,"data_stored":2599585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":569830848},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304648,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":526412,"kb_used_data":3644,"kb_used_omap":6,"kb_used_meta":522745,"kb_avail":93218740,"statfs":{"total":95995035648,"available":95455989760,"internally_reserved":0,"allocated":3731456,"data_stored":2289835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":535291335},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827951,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837672,"kb_used_data":3352,"kb_used_omap":5,"kb_used_meta":834298,"kb_avail":92907480,"statfs":{"total":95995035648,"available":95137259520,"internally_reserved":0,"allocated":3432448,"data_stored":2009188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":854321388},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285816,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":737280,"kb_used_data":3700,"kb_used_omap":5,"kb_used_meta":733562,"kb_avail":93007872,"statfs":{"total":95995035648,"available":95240060928,"internally_reserved":0,"allocated":3788800,"data_stored":2353587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":751167547},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:52:03.597 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:52:03.597 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:52:03.598 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-13T14:52:03.598 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:52:03.598 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:52:03.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-13T14:52:03.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:03.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:03.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:03.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:03.836 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 9265 9249 192.506 156.812 0.00289425 0.0051856 2024-09-13T14:52:03.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:03.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:03.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:04.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:04.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:04.170 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:04.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:04.307 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:04.336 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:04 smithi019 ceph-mon[91296]: from='client.46472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:04.336 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:04 smithi019 ceph-mon[91296]: 1.d scrub starts 2024-09-13T14:52:04.336 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:04 smithi019 ceph-mon[91296]: 1.d scrub ok 2024-09-13T14:52:04.336 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:52:04.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:04.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:04 smithi003 ceph-mon[104007]: from='client.46472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:04.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:04 smithi003 ceph-mon[104007]: 1.d scrub starts 2024-09-13T14:52:04.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:04 smithi003 ceph-mon[104007]: 1.d scrub ok 2024-09-13T14:52:04.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:52:04.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:04 smithi003 ceph-mon[107278]: from='client.46472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:04.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:04 smithi003 ceph-mon[107278]: 1.d scrub starts 2024-09-13T14:52:04.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:04 smithi003 ceph-mon[107278]: 1.d scrub ok 2024-09-13T14:52:04.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:52:04.435 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:52:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:52:04] "GET /metrics HTTP/1.1" 200 36456 "" "Prometheus/2.51.0" 2024-09-13T14:52:04.445 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:04.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:04.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:04.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:04.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:04.779 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:52:04.779 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:04.828 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:52:04.828 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:04.836 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 12111 12095 188.846 177.875 0.00428491 0.00528671 2024-09-13T14:52:04.837 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:04.864 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:52:04.865 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.874 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:04.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:04.901 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:52:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:04.940 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:52:04.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:04.975 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:52:04.975 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:04.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:05.012 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:52:05.012 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:05.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:05.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:05.018 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:52:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T14:52:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T14:52:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:52:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:27860 10.a 10:51e3daa3:::benchmark_data_smithi003_197468_object27859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:52:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:47:47.999497+0000", 2024-09-13T14:52:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 257.01541958199999, 2024-09-13T14:52:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.293360695, 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 27860 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:47.999497+0000", 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:47.999497+0000", 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:47.999496+0000", 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:47.999531+0000", 2024-09-13T14:52:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4379999999999999e-05 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:47.999549+0000", 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7938e-05 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:47.999556+0000", 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.464e-06 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.001993+0000", 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024364590000000002 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002092+0000", 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.9145e-05 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.022 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002208+0000", 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000115973 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.004739+0000", 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002531541 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.005965+0000", 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0012257360000000001 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292767+0000", 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28680239000000002 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292817+0000", 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9731999999999998e-05 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292858+0000", 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1056000000000001e-05 2024-09-13T14:52:05.024 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79795 10.d 10:b33ee9a8:::benchmark_data_smithi003_197468_object79794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.371671+0000", 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 220.64324580600001, 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29841990899999998, 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.025 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79795 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.371671+0000", 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.371671+0000", 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.371669+0000", 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.371703+0000", 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3207999999999998e-05 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.371717+0000", 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.448e-05 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.371723+0000", 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3090000000000001e-06 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.372959+0000", 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00123569 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.373073+0000", 2024-09-13T14:52:05.028 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011381 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.373192+0000", 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011882100000000001 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.376058+0000", 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028662649999999998 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.383416+0000", 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0073578410000000004 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.669988+0000", 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28657213300000001 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.670046+0000", 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7569999999999999e-05 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.670091+0000", 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5549e-05 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79801 10.b 10:deb6c730:::benchmark_data_smithi003_197468_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.375580+0000", 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 220.639336851, 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29467234199999998, 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.031 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.032 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.032 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.032 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79801 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.375580+0000", 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.375580+0000", 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.375580+0000", 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.375613+0000", 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3293000000000001e-05 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.375622+0000", 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1609999999999992e-06 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.034 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.375627+0000", 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7790000000000003e-06 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.376384+0000", 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000756926 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.376486+0000", 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000102397 2024-09-13T14:52:05.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.376598+0000", 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000111996 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.379765+0000", 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0031670790000000002 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.382645+0000", 2024-09-13T14:52:05.036 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028796569999999999 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.670159+0000", 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28751441900000002 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.670208+0000", 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9178000000000002e-05 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.670252+0000", 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3999999999999999e-05 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79813 10.b 10:d595cd80:::benchmark_data_smithi003_197468_object79812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.386215+0000", 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 220.62870236000001, 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29248787199999998, 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.038 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79813 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.386215+0000", 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.386215+0000", 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.386213+0000", 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.386246+0000", 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3005e-05 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.386262+0000", 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6019999999999999e-05 2024-09-13T14:52:05.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.386268+0000", 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8449999999999999e-06 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.386381+0000", 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011288799999999999 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.386496+0000", 2024-09-13T14:52:05.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000115226 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.386638+0000", 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014139 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390080+0000", 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0034417710000000002 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390167+0000", 2024-09-13T14:52:05.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7582000000000003e-05 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.678586+0000", 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28841876300000002 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.678666+0000", 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0493000000000003e-05 2024-09-13T14:52:05.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.678702+0000", 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6109000000000003e-05 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79818 10.a 10:5d405985:::benchmark_data_smithi003_197468_object79817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.390420+0000", 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 220.624496967, 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29094614699999999, 2024-09-13T14:52:05.044 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79818 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390420+0000", 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390420+0000", 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390419+0000", 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390522+0000", 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010356 2024-09-13T14:52:05.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390541+0000", 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8887999999999999e-05 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390547+0000", 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3890000000000004e-06 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390587+0000", 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0670000000000002e-05 2024-09-13T14:52:05.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390699+0000", 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011210899999999999 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.390805+0000", 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000105367 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.393972+0000", 2024-09-13T14:52:05.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003167207 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.394218+0000", 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000246226 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.681294+0000", 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28707533299999999 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.681333+0000", 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9189e-05 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.681366+0000", 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3356999999999999e-05 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79819 10.b 10:df55a699:::benchmark_data_smithi003_197468_object79818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.391687+0000", 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 220.623229944, 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29038150499999998, 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.051 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79819 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391687+0000", 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391687+0000", 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391686+0000", 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391752+0000", 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5945999999999995e-05 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391764+0000", 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1753000000000001e-05 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391769+0000", 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0490000000000003e-06 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391813+0000", 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4036000000000003e-05 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.391931+0000", 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011754799999999999 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.392060+0000", 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012982 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.394737+0000", 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026769630000000001 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.395117+0000", 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000379117 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.681990+0000", 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28687345600000003 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.682035+0000", 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5130000000000003e-05 2024-09-13T14:52:05.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.682069+0000", 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3407e-05 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117266 10.f 10:f2aa6f81:::benchmark_data_smithi003_197468_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.720592+0000", 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 175.29432528800001, 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.327086822, 2024-09-13T14:52:05.057 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117266 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720592+0000", 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720592+0000", 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720591+0000", 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720594+0000", 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8480000000000001e-06 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720600+0000", 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4200000000000004e-06 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720604+0000", 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4860000000000002e-06 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.061 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:52:05.061 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.746485+0000", 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.025880826999999999 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.746592+0000", 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010717000000000001 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.746704+0000", 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011236800000000001 2024-09-13T14:52:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.751485+0000", 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0047811640000000001 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.759932+0000", 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0084469260000000004 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.047614+0000", 2024-09-13T14:52:05.063 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28768163600000002 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.047651+0000", 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6480000000000003e-05 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.047679+0000", 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7974999999999999e-05 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.064 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117269 10.7 10:e83f946d:::benchmark_data_smithi003_197468_object540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.722459+0000", 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 175.292458141, 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.33327838999999998, 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117269 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722459+0000", 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722459+0000", 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722458+0000", 2024-09-13T14:52:05.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722463+0000", 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2809999999999999e-06 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722475+0000", 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2201000000000001e-05 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722481+0000", 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5419999999999996e-06 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764160+0000", 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.041679714999999999 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.068 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764315+0000", 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015493000000000001 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764457+0000", 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014210500000000001 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772074+0000", 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0076163630000000001 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.069 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.788564+0000", 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016490134 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051129+0000", 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.262565304 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051204+0000", 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4845999999999995e-05 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.070 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055737+0000", 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0045331210000000002 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117274 10.f 10:fbc2e15f:::benchmark_data_smithi003_197468_object545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.735218+0000", 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 175.27969865099999, 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.316058477, 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.071 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117274 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735218+0000", 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735218+0000", 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735216+0000", 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735230+0000", 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3811999999999999e-05 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735253+0000", 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2646e-05 2024-09-13T14:52:05.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735260+0000", 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5530000000000004e-06 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764197+0000", 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.028936565000000001 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764308+0000", 2024-09-13T14:52:05.074 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000110838 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764437+0000", 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000128823 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.771774+0000", 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.007336927 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.075 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772083+0000", 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00030953699999999997 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051206+0000", 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27912247099999998 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051246+0000", 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0608000000000002e-05 2024-09-13T14:52:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051277+0000", 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0546999999999997e-05 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117285 10.f 10:fd6a389f:::benchmark_data_smithi003_197468_object556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.752229+0000", 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 175.262687957, 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30415821399999998, 2024-09-13T14:52:05.077 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117285 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752229+0000", 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.078 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752229+0000", 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752228+0000", 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752234+0000", 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8540000000000002e-06 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752247+0000", 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3555e-05 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752253+0000", 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4539999999999998e-06 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768099+0000", 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015846628000000001 2024-09-13T14:52:05.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768230+0000", 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000130617 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768448+0000", 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000218381 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772088+0000", 2024-09-13T14:52:05.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036394679999999999 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772225+0000", 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013752 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055783+0000", 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28355824600000001 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055834+0000", 2024-09-13T14:52:05.082 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0442000000000003e-05 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.056387+0000", 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00055330699999999995 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117292 10.7 10:e057952f:::benchmark_data_smithi003_197468_object563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.760602+0000", 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 175.25431519399999, 2024-09-13T14:52:05.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29983130600000002, 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117292 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760602+0000", 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760602+0000", 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760600+0000", 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760607+0000", 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2910000000000002e-06 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760622+0000", 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4978e-05 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760628+0000", 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5559999999999998e-06 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768389+0000", 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0077610559999999997 2024-09-13T14:52:05.086 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768548+0000", 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015928000000000001 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768699+0000", 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015094800000000001 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772239+0000", 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0035398029999999998 2024-09-13T14:52:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.788617+0000", 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016378344999999999 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055881+0000", 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26726359500000002 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055933+0000", 2024-09-13T14:52:05.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2432999999999998e-05 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060433+0000", 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0044996050000000003 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117298 10.7 10:e5727b6c:::benchmark_data_smithi003_197468_object569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.772611+0000", 2024-09-13T14:52:05.089 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 175.24230576299999, 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29277972899999999, 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117298 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772611+0000", 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772611+0000", 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772610+0000", 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772616+0000", 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3389999999999998e-06 2024-09-13T14:52:05.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772628+0000", 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1877e-05 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772633+0000", 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0509999999999996e-06 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772692+0000", 2024-09-13T14:52:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8433000000000001e-05 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772798+0000", 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106079 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772899+0000", 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010132999999999999 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.776210+0000", 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0033107729999999999 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.788648+0000", 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012438289999999999 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060503+0000", 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27185490499999998 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.094 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060546+0000", 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3467000000000002e-05 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.065391+0000", 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0048445220000000004 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117301 10.f 10:f4f3ddb9:::benchmark_data_smithi003_197468_object572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.779418+0000", 2024-09-13T14:52:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 175.23549914200001, 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28115621899999999, 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117301 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779418+0000", 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.097 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.097 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.097 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779418+0000", 2024-09-13T14:52:05.097 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.097 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.097 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.097 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.097 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779416+0000", 2024-09-13T14:52:05.097 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779424+0000", 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8809999999999996e-06 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779445+0000", 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1067999999999999e-05 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779453+0000", 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4630000000000004e-06 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779515+0000", 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1679000000000004e-05 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779658+0000", 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000142948 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.099 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779790+0000", 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013255799999999999 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.783333+0000", 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0035433719999999999 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.783430+0000", 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6834000000000002e-05 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.100 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060490+0000", 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.277059208 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060540+0000", 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.028e-05 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.102 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060574+0000", 2024-09-13T14:52:05.102 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4264000000000003e-05 2024-09-13T14:52:05.102 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.102 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.102 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:123351 10.f 10:f28c766d:::benchmark_data_smithi003_197468_object6622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:17.374873+0000", 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 167.640043996, 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28567656299999999, 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123351 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.103 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.104 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374873+0000", 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374873+0000", 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374872+0000", 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:52:05.105 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374877+0000", 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2849999999999996e-06 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374893+0000", 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6758000000000001e-05 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374898+0000", 2024-09-13T14:52:05.106 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0869999999999997e-06 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.393467+0000", 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.018568474000000001 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.393577+0000", 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000110605 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:52:05.107 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.393687+0000", 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000109844 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.401477+0000", 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0077901990000000003 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.403044+0000", 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001566556 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.108 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.660438+0000", 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25739424900000002 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.660513+0000", 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4787000000000001e-05 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.660550+0000", 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6458000000000002e-05 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:123372 10.7 10:e7b4d536:::benchmark_data_smithi003_197468_object6643:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:17.399201+0000", 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 167.61571627800001, 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27458023300000001, 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123372 2024-09-13T14:52:05.110 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.399201+0000", 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.399201+0000", 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.111 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.399200+0000", 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.399206+0000", 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1070000000000001e-06 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.399216+0000", 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0159999999999999e-05 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.399221+0000", 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3299999999999998e-06 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.403186+0000", 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039650199999999997 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.403322+0000", 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013564299999999999 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.113 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.403460+0000", 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013780300000000001 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.407615+0000", 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0041550700000000003 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.412729+0000", 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0051138959999999997 2024-09-13T14:52:05.114 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.668715+0000", 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25598575200000001 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.668907+0000", 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000192318 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.673781+0000", 2024-09-13T14:52:05.115 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0048741009999999996 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149388 10.7 10:ea963fcf:::benchmark_data_smithi003_197468_object32659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.303553+0000", 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 137.71136353899999, 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.34338312199999999, 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.116 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149388 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303553+0000", 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303553+0000", 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.117 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303552+0000", 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303558+0000", 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8259999999999998e-06 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303568+0000", 2024-09-13T14:52:05.118 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7389999999999994e-06 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303572+0000", 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2459999999999997e-06 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307829+0000", 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0042576009999999997 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307977+0000", 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014806699999999999 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.308129+0000", 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015176799999999999 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.312145+0000", 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0040153669999999997 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.120 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.340964+0000", 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.028819079000000001 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642410+0000", 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30144587499999997 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642464+0000", 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4509999999999998e-05 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.646937+0000", 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0044724949999999999 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149395 10.f 10:ff0ca942:::benchmark_data_smithi003_197468_object32666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.307942+0000", 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 137.70697534300001, 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.334537682, 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149395 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307942+0000", 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.123 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307942+0000", 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307941+0000", 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307946+0000", 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2129999999999999e-06 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.124 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307956+0000", 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7189999999999996e-06 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307961+0000", 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3190000000000002e-06 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.308054+0000", 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3014000000000001e-05 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.308152+0000", 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8295000000000007e-05 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.308259+0000", 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106173 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.312303+0000", 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0040445589999999997 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.126 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314659+0000", 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0023557449999999998 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642416+0000", 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.32775716700000002 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642452+0000", 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6291999999999997e-05 2024-09-13T14:52:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642479+0000", 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6988e-05 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149399 10.7 10:e7552c3b:::benchmark_data_smithi003_197468_object32670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.314862+0000", 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 137.70005531300001, 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.33632368200000001, 2024-09-13T14:52:05.128 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149399 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314862+0000", 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314862+0000", 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314861+0000", 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314866+0000", 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.977e-06 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.130 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314874+0000", 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3399999999999998e-06 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314879+0000", 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4989999999999996e-06 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314929+0000", 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0234e-05 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.131 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.315034+0000", 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010527 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.315167+0000", 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013303800000000001 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.319534+0000", 2024-09-13T14:52:05.132 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0043670159999999996 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.341004+0000", 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.021470073999999999 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.647004+0000", 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30600010599999999 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.133 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.647052+0000", 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8063999999999999e-05 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.651185+0000", 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0041329449999999998 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149422 10.f 10:f518014d:::benchmark_data_smithi003_197468_object32693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.359274+0000", 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 137.655642769, 2024-09-13T14:52:05.134 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28782790600000002, 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149422 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359274+0000", 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.135 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359274+0000", 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359274+0000", 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359277+0000", 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8639999999999998e-06 2024-09-13T14:52:05.136 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359285+0000", 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0220000000000004e-06 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359289+0000", 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3809999999999999e-06 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359319+0000", 2024-09-13T14:52:05.137 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0329e-05 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359398+0000", 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8954000000000003e-05 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359469+0000", 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1149999999999995e-05 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.138 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.362252+0000", 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027822929999999999 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.362419+0000", 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016755700000000001 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.646982+0000", 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.284562708 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.647069+0000", 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7306999999999996e-05 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.647102+0000", 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3006999999999998e-05 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149426 10.7 10:e1c5b3e0:::benchmark_data_smithi003_197468_object32697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.376151+0000", 2024-09-13T14:52:05.140 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 137.63876641900001, 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28245281700000002, 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149426 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376151+0000", 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.141 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376151+0000", 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376149+0000", 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376158+0000", 2024-09-13T14:52:05.142 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4509999999999992e-06 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376175+0000", 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7207999999999999e-05 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376181+0000", 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3679999999999998e-06 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376240+0000", 2024-09-13T14:52:05.143 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8353999999999997e-05 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376435+0000", 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000194916 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376622+0000", 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00018775000000000001 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:52:05.144 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.379581+0000", 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029589740000000001 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.382484+0000", 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029022779999999999 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.651267+0000", 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26878351900000003 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.145 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.651315+0000", 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7701999999999998e-05 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.658603+0000", 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0072886119999999999 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:05.146 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:52:05.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:05.180 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:05.282 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:05.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:05.380 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:05 smithi003 ceph-mon[107278]: pgmap v933: 109 pgs: 109 active+clean; 133 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 17 MiB/s wr, 265 op/s 2024-09-13T14:52:05.380 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3795103230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:05.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:05 smithi003 ceph-mon[104007]: pgmap v933: 109 pgs: 109 active+clean; 133 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 17 MiB/s wr, 265 op/s 2024-09-13T14:52:05.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3795103230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:05.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:05 smithi019 ceph-mon[91296]: pgmap v933: 109 pgs: 109 active+clean; 133 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 17 MiB/s wr, 265 op/s 2024-09-13T14:52:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3795103230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:05.458 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:05.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:05.597 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:05.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:05.699 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:05.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.732 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:05.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:05.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:05.839 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 14917 14901 186.15 175.375 0.00396226 0.0053661 2024-09-13T14:52:05.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:05.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:06.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:06.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:06.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:52:06.160 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:06.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:06.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:06.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.443 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:06.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:06.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:06.586 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:52:06.686 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:06.723 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:06.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:06.836 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 17693 17677 184.04 173.5 0.00373388 0.0054267 2024-09-13T14:52:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:06.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:06.936 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:06.967 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:07.000 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:07.102 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:07.140 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:07.242 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:07.275 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:07.321 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:07 smithi019 ceph-mon[91296]: pgmap v934: 109 pgs: 109 active+clean; 602 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:52:07.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:07.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:07.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:07 smithi003 ceph-mon[104007]: pgmap v934: 109 pgs: 109 active+clean; 602 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:52:07.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:07 smithi003 ceph-mon[107278]: pgmap v934: 109 pgs: 109 active+clean; 602 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:52:07.520 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.553 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:07.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:07.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:07.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:07.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.836 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 19966 19950 178.043 142.062 0.00387345 0.00560821 2024-09-13T14:52:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:07.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:07.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:08.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:08.124 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:08.226 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.260 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:08.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:08.361 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:08.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:52:08.392 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:08.403 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:08 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2328798848' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:08.404 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:08 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2328798848' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:08 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2328798848' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:08.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:08.530 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:08.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:08.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:08.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:08.792 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:08.836 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 21685 21669 169.219 107.438 0.00969889 0.00590376 2024-09-13T14:52:08.894 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:08.980 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:52:09.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:09.044 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:09.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:09.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:09.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:09.273 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:09.282 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[107278]: pgmap v935: 109 pgs: 109 active+clean; 810 MiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:52:09.282 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:52:09.282 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:52:09.282 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2328798848' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:09.283 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[107278]: osdmap e271: 8 total, 8 up, 8 in 2024-09-13T14:52:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[104007]: pgmap v935: 109 pgs: 109 active+clean; 810 MiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:52:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:52:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:52:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2328798848' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:09 smithi003 ceph-mon[104007]: osdmap e271: 8 total, 8 up, 8 in 2024-09-13T14:52:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.304 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:09.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:09.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:09 smithi019 ceph-mon[91296]: pgmap v935: 109 pgs: 109 active+clean; 810 MiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:52:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:09 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:52:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:09 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:52:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2328798848' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:09 smithi019 ceph-mon[91296]: osdmap e271: 8 total, 8 up, 8 in 2024-09-13T14:52:09.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:09.546 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.576 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:09.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:09.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.709 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:09.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:09.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:09.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:09.837 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 23181 23165 160.807 93.5 0.00263407 0.00618902 2024-09-13T14:52:09.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:09.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:10.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:10.084 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:10.123 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:52:10.123 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:52:10.123 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:23036 11.5 11:a29e1717:::benchmark_data_smithi003_229782_object23035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:52:09.649370+0000", 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.00025387599999999999, 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00028923399999999998, 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "started", 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 23036 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:10.124 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:52:09.649370+0000", 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:52:09.649370+0000", 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:52:09.649368+0000", 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:52:09.649395+0000", 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7605000000000001e-05 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:10.126 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:52:09.649408+0000", 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2896e-05 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:52:09.649414+0000", 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9669999999999998e-06 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:52:09.649474+0000", 2024-09-13T14:52:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0003000000000001e-05 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:52:09.649585+0000", 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000111081 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 1 2024-09-13T14:52:10.128 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:52:10.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:10.222 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:10.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:10.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:10.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:52:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:52:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:10.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:10.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:10.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:10.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:10.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:10.768 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:52:10.836 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 24699 24683 154.214 94.875 0.00489659 0.0064786 2024-09-13T14:52:10.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:10.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.034 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:11.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:11.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:11.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:11.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:11.204 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:52:11.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:11.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:11.346 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:11 smithi019 ceph-mon[91296]: pgmap v937: 109 pgs: 109 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-13T14:52:11.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:11 smithi003 ceph-mon[107278]: pgmap v937: 109 pgs: 109 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-13T14:52:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:11 smithi003 ceph-mon[104007]: pgmap v937: 109 pgs: 109 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-13T14:52:11.452 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:11.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:11.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:11.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:11.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:11.836 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 26981 26965 153.16 142.625 0.00595375 0.00652271 2024-09-13T14:52:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.868 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:11.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:11.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:11.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:12.093 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:12.123 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:12.225 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:12.260 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:12.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:12.390 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:12.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:12.526 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:12.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.663 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:12.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:12.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:12.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:12.837 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 29091 29075 151.38 131.875 0.00967272 0.00660048 2024-09-13T14:52:12.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:12.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:13.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:13.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:13 smithi003 ceph-mon[104007]: pgmap v938: 109 pgs: 109 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-13T14:52:13.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:13.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:13.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:13.378 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:13 smithi019 ceph-mon[91296]: pgmap v938: 109 pgs: 109 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-13T14:52:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:52:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:13.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:52:13.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:13.438 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:13 smithi003 ceph-mon[107278]: pgmap v938: 109 pgs: 109 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-13T14:52:13.438 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:52:13.438 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:13.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:13.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:13.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.663 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:13.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:13.765 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:13.802 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:13.837 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 31216 31200 149.951 132.812 0.00672221 0.00666413 2024-09-13T14:52:13.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:13.915 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:13.932 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:14.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:14.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:14.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:52:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:52:14] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-09-13T14:52:14.187 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:14.223 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:14.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:14.357 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:14.460 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:14.498 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:14.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:14.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:14.707 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:52:14.707 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:52:14.707 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:52:14.742 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:14.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:14.837 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 33237 33221 148.262 126.312 0.00469347 0.00673938 2024-09-13T14:52:14.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:14.922 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:15.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:15.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:15.165 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.208 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:15.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:15.309 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:15.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.353 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:15.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:15.357 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:52:15.357 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:52:15.357 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:52:15.357 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:52:15.357 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:52:15.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:15 smithi003 ceph-mon[104007]: pgmap v939: 109 pgs: 109 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-09-13T14:52:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:15 smithi003 ceph-mon[107278]: pgmap v939: 109 pgs: 109 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-09-13T14:52:15.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.489 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:15.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:15.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:15.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:15 smithi019 ceph-mon[91296]: pgmap v939: 109 pgs: 109 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-09-13T14:52:15.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:15.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:15.837 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 35276 35260 146.873 127.438 0.00614591 0.00680244 2024-09-13T14:52:15.870 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:15.905 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:15.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:52:16.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:16.037 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:16.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:16.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:52:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:16.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:16.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:16.317 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:16.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:16.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:16.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:16.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:16.574 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:52:16.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:16.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:16.723 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:16.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:16.837 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 15 37047 37032 144.615 110.75 0.0063401 0.00690921 2024-09-13T14:52:16.859 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:16.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:16.995 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:17.097 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:17.135 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:17.237 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:17.271 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:17.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:17.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:17 smithi019 ceph-mon[91296]: pgmap v940: 109 pgs: 109 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-13T14:52:17.466 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:17 smithi003 ceph-mon[104007]: pgmap v940: 109 pgs: 109 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-13T14:52:17.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:17 smithi003 ceph-mon[107278]: pgmap v940: 109 pgs: 109 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-13T14:52:17.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:17.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:17.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:17.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:17.791 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:17.823 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:17.837 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 38893 38877 142.891 115.312 0.00346145 0.00699275 2024-09-13T14:52:17.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:17.961 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:18.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:18.099 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:18.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:18.235 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:18.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:18.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:18.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:18.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:18.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:18.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:18.678 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:18.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:18.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:18.837 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 40746 40730 141.387 115.812 0.0106616 0.00706735 2024-09-13T14:52:18.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:18.950 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:18.985 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:19.052 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:19.083 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:19.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:19.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.220 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:19.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:19.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:19.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:19.452 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:19.485 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:19.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:19.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:19 smithi003 ceph-mon[104007]: pgmap v941: 109 pgs: 109 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-13T14:52:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:19 smithi003 ceph-mon[107278]: pgmap v941: 109 pgs: 109 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-13T14:52:19.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:19.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:19 smithi019 ceph-mon[91296]: pgmap v941: 109 pgs: 109 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-13T14:52:19.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:19.838 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 42469 42453 139.613 107.688 0.00582767 0.00715637 2024-09-13T14:52:19.877 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.901 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:19.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:20.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:20.037 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:20.139 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:20.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:20.272 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:52:20.272 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:52:20.272 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:52:20.272 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:52:20.272 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:52:20.273 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:20.273 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:20.273 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:20.273 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:20.273 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:20.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:20.294 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:52:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:52:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:20.295 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:20 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1142059101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:20.302 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:52:20.303 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:20.322 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:20.337 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:52:20.338 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:20.370 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:52:20.370 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:20.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:20 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1142059101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:20.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:20 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1142059101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:20.424 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:20.468 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:20.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:20.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:20.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:20.743 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:20.743 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.752 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:20.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:20.838 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:52:20.837744+0000 min lat: 0.00120774 max lat: 0.279703 avg lat: 0.00725968 2024-09-13T14:52:20.838 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:52:20.838 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 44076 44060 137.654 100.438 0.00744774 0.00725968 2024-09-13T14:52:20.853 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:20.882 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:20.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:21.019 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:21.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:21.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:21.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:21.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:21.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:21.395 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:21 smithi019 ceph-mon[91296]: pgmap v942: 109 pgs: 109 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-13T14:52:21.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:21.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:52:21.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:21.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:21 smithi003 ceph-mon[104007]: pgmap v942: 109 pgs: 109 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-13T14:52:21.485 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:21 smithi003 ceph-mon[107278]: pgmap v942: 109 pgs: 109 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-13T14:52:21.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:21.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:21.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:21.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:21.840 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 45819 45803 136.277 108.938 0.011549 0.00733312 2024-09-13T14:52:21.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:21.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:22.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:22.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:22.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:22.185 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:22.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:22.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:22.327 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:22.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:22.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:22.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:22.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:22.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:22.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:22.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:22.839 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 47371 47355 134.491 97 0.00442812 0.00742979 2024-09-13T14:52:22.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:22.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:23.027 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:23.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.058 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:23.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:23.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:23.188 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:23.244 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:52:23.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:23.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:23.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:23.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:23.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:23.558 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:23.588 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:23.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:23 smithi019 ceph-mon[91296]: pgmap v943: 109 pgs: 109 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-13T14:52:23.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:23 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2670559335' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:23.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:23 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:23.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:23 smithi003 ceph-mon[107278]: pgmap v943: 109 pgs: 109 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-13T14:52:23.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:23 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2670559335' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:23.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:23 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:23.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:23 smithi003 ceph-mon[104007]: pgmap v943: 109 pgs: 109 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-13T14:52:23.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:23 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2670559335' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:23.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:23 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:23.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:23.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:52:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:23.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:23.839 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 48952 48936 132.94 98.8125 0.00726618 0.00751576 2024-09-13T14:52:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.862 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:23.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:23.962 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:52:23.962 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:52:23.963 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_6 2024-09-13T14:52:23.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:52:23.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:23.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:24.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:24.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:24.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:24.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:24.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:24.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:24.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:24 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:24.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:24 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:24.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:24 smithi003 ceph-mon[104007]: osdmap e272: 8 total, 8 up, 8 in 2024-09-13T14:52:24.434 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:52:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:52:24] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-09-13T14:52:24.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:24 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:24.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:24 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:24.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:24 smithi003 ceph-mon[107278]: osdmap e272: 8 total, 8 up, 8 in 2024-09-13T14:52:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:24.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:24.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:24.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:24.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:24.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:24 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:24.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:24 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:24 smithi019 ceph-mon[91296]: osdmap e272: 8 total, 8 up, 8 in 2024-09-13T14:52:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:24.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:24.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:24.840 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 50319 50303 130.961 85.4375 0.00339447 0.00762949 2024-09-13T14:52:24.845 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:24.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:24.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:24.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:25.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:25.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:25.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:25.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:25.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.393 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:25.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:25.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:25.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:25.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:25.664 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:25.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:25 smithi019 ceph-mon[91296]: pgmap v945: 109 pgs: 109 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-13T14:52:25.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:25 smithi003 ceph-mon[107278]: pgmap v945: 109 pgs: 109 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-13T14:52:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:25 smithi003 ceph-mon[104007]: pgmap v945: 109 pgs: 109 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-13T14:52:25.766 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.796 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.840 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 51873 51857 129.607 97.125 0.013968 0.00771023 2024-09-13T14:52:25.898 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.935 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:25.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:26.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:26.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:26.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:26.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:26.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:26.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:26.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:26.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:26.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:26.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:26.785 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:26.840 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 53510 53494 128.554 102.312 0.0168184 0.00777243 2024-09-13T14:52:26.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:26.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:27.034 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:27.067 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:27.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:27.198 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:27.203 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:27.300 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:27.329 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:27.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:27 smithi003 ceph-mon[107278]: pgmap v946: 109 pgs: 109 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-09-13T14:52:27.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:27 smithi003 ceph-mon[104007]: pgmap v946: 109 pgs: 109 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-09-13T14:52:27.419 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:27.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:27.460 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:27.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:27.474 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:27.474 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:27.562 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:27.592 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:27 smithi019 ceph-mon[91296]: pgmap v946: 109 pgs: 109 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-09-13T14:52:27.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:27.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:27.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:27.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:27.840 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 55072 55056 127.409 97.625 0.0063123 0.00784288 2024-09-13T14:52:27.866 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:27.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:27.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:28.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:28.109 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:28.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:28.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:28.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:28.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:28.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:28.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:28.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:28.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:28.804 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:28.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:28.840 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 56608 56592 126.287 96 0.0131189 0.00791335 2024-09-13T14:52:28.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:52:28.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:28.971 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:29.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.108 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:29.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:52:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:28 smithi019 ceph-mon[91296]: pgmap v947: 109 pgs: 109 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-13T14:52:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:52:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:28 smithi003 ceph-mon[107278]: pgmap v947: 109 pgs: 109 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-13T14:52:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:52:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:28 smithi003 ceph-mon[104007]: pgmap v947: 109 pgs: 109 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-13T14:52:29.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:29.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:29.353 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.385 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:29.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:29.486 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.518 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:29.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:29.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:29.649 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:29.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:29.841 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 58133 58117 125.218 95.3125 0.00302925 0.0079777 2024-09-13T14:52:29.887 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:29.918 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:30.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:30.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:30.163 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:30.198 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:30.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:30.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.333 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:30.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:30.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:30.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:30.574 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:30.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:52:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:52:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:30.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:30.742 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:30.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 59632 59616 124.167 93.6875 0.00405952 0.00804817 2024-09-13T14:52:30.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.879 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:30.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:30.980 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:31.017 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:31.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:31.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:30 smithi019 ceph-mon[91296]: pgmap v948: 109 pgs: 109 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:52:31.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:30 smithi003 ceph-mon[107278]: pgmap v948: 109 pgs: 109 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:52:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:30 smithi003 ceph-mon[104007]: pgmap v948: 109 pgs: 109 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:52:31.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:31.296 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:31.398 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.488 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:32.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:32.488 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 61024 61008 122.968 87 0.00404424 0.00811963 2024-09-13T14:52:32.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:32.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:32.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:32.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:32.841 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 62527 62511 122.061 93.9375 0.00814117 0.00818733 2024-09-13T14:52:32.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:32.893 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:32.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:33.027 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:33.129 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:33.159 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:32 smithi019 ceph-mon[91296]: pgmap v949: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-13T14:52:33.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:32 smithi003 ceph-mon[107278]: pgmap v949: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-13T14:52:33.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:32 smithi003 ceph-mon[104007]: pgmap v949: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-13T14:52:33.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:33.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:33.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:33.430 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:33.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:52:33.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:33.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:33.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:33.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:33.841 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 63878 63862 120.92 84.4375 0.00883751 0.00826444 2024-09-13T14:52:33.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:33.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:34.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:34.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:34.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:34.192 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:34.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:34.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:34.425 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:52:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:52:34] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-09-13T14:52:34.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:34.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:34.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:34.739 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:34.772 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:34.841 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 65316 65300 120.007 89.875 0.00921666 0.00832704 2024-09-13T14:52:34.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:34.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:34.991 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:52:34.992 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:35.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:35.028 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:52:35.029 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:34 smithi003 ceph-mon[104007]: pgmap v950: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-13T14:52:35.029 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:34 smithi003 ceph-mon[107278]: pgmap v950: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-13T14:52:35.029 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:35.055 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:35.066 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:52:35.066 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:35.102 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:52:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:35.137 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:52:35.138 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:35.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:35.164 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:35.171 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:52:35.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:35.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:34 smithi019 ceph-mon[91296]: pgmap v950: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-13T14:52:35.213 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:52:35.213 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:35.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:35.247 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:52:35.247 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:35.262 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:35.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:35.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:35.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:35.510 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:35.612 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:35.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:35.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:35.782 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:35.841 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 66559 66543 118.798 77.6875 0.0189718 0.00841136 2024-09-13T14:52:35.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.918 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:35.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:36.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:36.058 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:36.160 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:36.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:35 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2125674419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:36.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:35 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2125674419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:36.186 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:36.288 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:36.323 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:36.425 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:35 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2125674419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:36.462 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:36.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:36.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:36.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:36.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:36.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:36.842 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 67820 67804 117.687 78.8125 0.0117221 0.00849149 2024-09-13T14:52:36.845 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:36.878 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:36.980 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:37.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:37.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:37.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:36 smithi003 ceph-mon[107278]: pgmap v951: 109 pgs: 109 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-09-13T14:52:37.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:36 smithi003 ceph-mon[104007]: pgmap v951: 109 pgs: 109 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-09-13T14:52:37.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:37.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:37.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:37.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:36 smithi019 ceph-mon[91296]: pgmap v951: 109 pgs: 109 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-09-13T14:52:37.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:37.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:37.551 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:37.585 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:37.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:37.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:37.798 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:52:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:52:37.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:37.843 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 68926 68910 116.37 69.125 0.00926494 0.0085875 2024-09-13T14:52:37.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:37.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:38.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:38.124 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:38.160 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:38.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.295 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:38.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:38.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:38.424 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:38.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:52:38.526 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:38.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:38.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:38.686 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:38.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:38.837 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":951,"stamp":"2024-09-13T14:52:35.860656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4164556749,"num_objects":63563,"num_object_clones":0,"num_object_copies":190689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63601,"num_write_kb":4066400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":853342,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65696,"ondisk_log_size":65696,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18873252,"kb_used_data":12526372,"kb_used_omap":29,"kb_used_meta":6346786,"kb_avail":731087964,"statfs":{"total":767960285184,"available":748634075136,"internally_reserved":0,"allocated":12827004928,"data_stored":12815349008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":6499109390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1348796416,"num_objects":20581,"num_object_clones":0,"num_object_copies":61743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20581,"num_write_kb":1317184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006754"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":424,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.249683+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:52:23.249683+0000","last_peered":"2024-09-13T14:52:23.249683+0000","last_clean":"2024-09-13T14:52:23.249683+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:52:23.249683+0000","last_undegraded":"2024-09-13T14:52:23.249683+0000","last_fullsized":"2024-09-13T14:52:23.249683+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":244,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.248101+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:52:08.248101+0000","last_peered":"2024-09-13T14:52:08.248101+0000","last_clean":"2024-09-13T14:52:08.248101+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:52:08.248101+0000","last_undegraded":"2024-09-13T14:52:08.248101+0000","last_fullsized":"2024-09-13T14:52:08.248101+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":35,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240300+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:52:08.240300+0000","last_peered":"2024-09-13T14:52:08.240300+0000","last_clean":"2024-09-13T14:52:08.240300+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:52:08.240300+0000","last_undegraded":"2024-09-13T14:52:08.240300+0000","last_fullsized":"2024-09-13T14:52:08.240300+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":382,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.248652+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:52:23.248652+0000","last_peered":"2024-09-13T14:52:23.248652+0000","last_clean":"2024-09-13T14:52:23.248652+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:52:23.248652+0000","last_undegraded":"2024-09-13T14:52:23.248652+0000","last_fullsized":"2024-09-13T14:52:23.248652+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":218,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240012+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:52:08.240012+0000","last_peered":"2024-09-13T14:52:08.240012+0000","last_clean":"2024-09-13T14:52:08.240012+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:52:08.240012+0000","last_undegraded":"2024-09-13T14:52:08.240012+0000","last_fullsized":"2024-09-13T14:52:08.240012+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":251,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.248375+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:52:08.248375+0000","last_peered":"2024-09-13T14:52:08.248375+0000","last_clean":"2024-09-13T14:52:08.248375+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:52:08.248375+0000","last_undegraded":"2024-09-13T14:52:08.248375+0000","last_fullsized":"2024-09-13T14:52:08.248375+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":235,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240071+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:52:08.240071+0000","last_peered":"2024-09-13T14:52:08.240071+0000","last_clean":"2024-09-13T14:52:08.240071+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:52:08.240071+0000","last_undegraded":"2024-09-13T14:52:08.240071+0000","last_fullsized":"2024-09-13T14:52:08.240071+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":281,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.248663+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:52:08.248663+0000","last_peered":"2024-09-13T14:52:08.248663+0000","last_clean":"2024-09-13T14:52:08.248663+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:52:08.248663+0000","last_undegraded":"2024-09-13T14:52:08.248663+0000","last_fullsized":"2024-09-13T14:52:08.248663+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":411,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.252361+0000","last_change":"2024-09-13T14:31:29.949329+0000","last_active":"2024-09-13T14:52:23.252361+0000","last_peered":"2024-09-13T14:52:23.252361+0000","last_clean":"2024-09-13T14:52:23.252361+0000","last_became_active":"2024-09-13T14:31:29.946345+0000","last_became_peered":"2024-09-13T14:31:29.946345+0000","last_unstale":"2024-09-13T14:52:23.252361+0000","last_undegraded":"2024-09-13T14:52:23.252361+0000","last_fullsized":"2024-09-13T14:52:23.252361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:41.836402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":406,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.247126+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:52:23.247126+0000","last_peered":"2024-09-13T14:52:23.247126+0000","last_clean":"2024-09-13T14:52:23.247126+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:52:23.247126+0000","last_undegraded":"2024-09-13T14:52:23.247126+0000","last_fullsized":"2024-09-13T14:52:23.247126+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":253,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.248466+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:52:08.248466+0000","last_peered":"2024-09-13T14:52:08.248466+0000","last_clean":"2024-09-13T14:52:08.248466+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:52:08.248466+0000","last_undegraded":"2024-09-13T14:52:08.248466+0000","last_fullsized":"2024-09-13T14:52:08.248466+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":349,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245281+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:52:23.245281+0000","last_peered":"2024-09-13T14:52:23.245281+0000","last_clean":"2024-09-13T14:52:23.245281+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:52:23.245281+0000","last_undegraded":"2024-09-13T14:52:23.245281+0000","last_fullsized":"2024-09-13T14:52:23.245281+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":63,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.246063+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:52:23.246063+0000","last_peered":"2024-09-13T14:52:23.246063+0000","last_clean":"2024-09-13T14:52:23.246063+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:52:23.246063+0000","last_undegraded":"2024-09-13T14:52:23.246063+0000","last_fullsized":"2024-09-13T14:52:23.246063+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":442,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.249751+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:52:23.249751+0000","last_peered":"2024-09-13T14:52:23.249751+0000","last_clean":"2024-09-13T14:52:23.249751+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:52:23.249751+0000","last_undegraded":"2024-09-13T14:52:23.249751+0000","last_fullsized":"2024-09-13T14:52:23.249751+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":381,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245679+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:52:23.245679+0000","last_peered":"2024-09-13T14:52:23.245679+0000","last_clean":"2024-09-13T14:52:23.245679+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:52:23.245679+0000","last_undegraded":"2024-09-13T14:52:23.245679+0000","last_fullsized":"2024-09-13T14:52:23.245679+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":234,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.247157+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:52:08.247157+0000","last_peered":"2024-09-13T14:52:08.247157+0000","last_clean":"2024-09-13T14:52:08.247157+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:52:08.247157+0000","last_undegraded":"2024-09-13T14:52:08.247157+0000","last_fullsized":"2024-09-13T14:52:08.247157+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":349,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245456+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:52:23.245456+0000","last_peered":"2024-09-13T14:52:23.245456+0000","last_clean":"2024-09-13T14:52:23.245456+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:52:23.245456+0000","last_undegraded":"2024-09-13T14:52:23.245456+0000","last_fullsized":"2024-09-13T14:52:23.245456+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":425,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.250076+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:52:23.250076+0000","last_peered":"2024-09-13T14:52:23.250076+0000","last_clean":"2024-09-13T14:52:23.250076+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:52:23.250076+0000","last_undegraded":"2024-09-13T14:52:23.250076+0000","last_fullsized":"2024-09-13T14:52:23.250076+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":354,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.246331+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:52:23.246331+0000","last_peered":"2024-09-13T14:52:23.246331+0000","last_clean":"2024-09-13T14:52:23.246331+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:52:23.246331+0000","last_undegraded":"2024-09-13T14:52:23.246331+0000","last_fullsized":"2024-09-13T14:52:23.246331+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240292+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:52:08.240292+0000","last_peered":"2024-09-13T14:52:08.240292+0000","last_clean":"2024-09-13T14:52:08.240292+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:52:08.240292+0000","last_undegraded":"2024-09-13T14:52:08.240292+0000","last_fullsized":"2024-09-13T14:52:08.240292+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":104,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.246915+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:52:23.246915+0000","last_peered":"2024-09-13T14:52:23.246915+0000","last_clean":"2024-09-13T14:52:23.246915+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:52:23.246915+0000","last_undegraded":"2024-09-13T14:52:23.246915+0000","last_fullsized":"2024-09-13T14:52:23.246915+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":35,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.243984+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:52:08.243984+0000","last_peered":"2024-09-13T14:52:08.243984+0000","last_clean":"2024-09-13T14:52:08.243984+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:52:08.243984+0000","last_undegraded":"2024-09-13T14:52:08.243984+0000","last_fullsized":"2024-09-13T14:52:08.243984+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":173,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.246700+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:52:08.246700+0000","last_peered":"2024-09-13T14:52:08.246700+0000","last_clean":"2024-09-13T14:52:08.246700+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:52:08.246700+0000","last_undegraded":"2024-09-13T14:52:08.246700+0000","last_fullsized":"2024-09-13T14:52:08.246700+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":241,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.249315+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:52:08.249315+0000","last_peered":"2024-09-13T14:52:08.249315+0000","last_clean":"2024-09-13T14:52:08.249315+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:52:08.249315+0000","last_undegraded":"2024-09-13T14:52:08.249315+0000","last_fullsized":"2024-09-13T14:52:08.249315+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":234,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.248823+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:52:08.248823+0000","last_peered":"2024-09-13T14:52:08.248823+0000","last_clean":"2024-09-13T14:52:08.248823+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:52:08.248823+0000","last_undegraded":"2024-09-13T14:52:08.248823+0000","last_fullsized":"2024-09-13T14:52:08.248823+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"272'4059","reported_seq":4073,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.208732+0000","last_change":"2024-09-13T14:51:50.160931+0000","last_active":"2024-09-13T14:52:34.208732+0000","last_peered":"2024-09-13T14:52:34.208732+0000","last_clean":"2024-09-13T14:52:34.208732+0000","last_became_active":"2024-09-13T14:51:50.160636+0000","last_became_peered":"2024-09-13T14:51:50.160636+0000","last_unstale":"2024-09-13T14:52:34.208732+0000","last_undegraded":"2024-09-13T14:52:34.208732+0000","last_fullsized":"2024-09-13T14:52:34.208732+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:05.011657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266010624,"num_objects":4059,"num_object_clones":0,"num_object_copies":12177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4059,"num_write_kb":259776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":217,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.248141+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:52:08.248141+0000","last_peered":"2024-09-13T14:52:08.248141+0000","last_clean":"2024-09-13T14:52:08.248141+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:52:08.248141+0000","last_undegraded":"2024-09-13T14:52:08.248141+0000","last_fullsized":"2024-09-13T14:52:08.248141+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":267,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.247262+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:52:08.247262+0000","last_peered":"2024-09-13T14:52:08.247262+0000","last_clean":"2024-09-13T14:52:08.247262+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:52:08.247262+0000","last_undegraded":"2024-09-13T14:52:08.247262+0000","last_fullsized":"2024-09-13T14:52:08.247262+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":423,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.249842+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:52:23.249842+0000","last_peered":"2024-09-13T14:52:23.249842+0000","last_clean":"2024-09-13T14:52:23.249842+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:52:23.249842+0000","last_undegraded":"2024-09-13T14:52:23.249842+0000","last_fullsized":"2024-09-13T14:52:23.249842+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":111,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245702+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:52:23.245702+0000","last_peered":"2024-09-13T14:52:23.245702+0000","last_clean":"2024-09-13T14:52:23.245702+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:52:23.245702+0000","last_undegraded":"2024-09-13T14:52:23.245702+0000","last_fullsized":"2024-09-13T14:52:23.245702+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"272'4084","reported_seq":4098,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.061904+0000","last_change":"2024-09-13T14:51:50.161207+0000","last_active":"2024-09-13T14:52:34.061904+0000","last_peered":"2024-09-13T14:52:34.061904+0000","last_clean":"2024-09-13T14:52:34.061904+0000","last_became_active":"2024-09-13T14:51:50.160931+0000","last_became_peered":"2024-09-13T14:51:50.160931+0000","last_unstale":"2024-09-13T14:52:34.061904+0000","last_undegraded":"2024-09-13T14:52:34.061904+0000","last_fullsized":"2024-09-13T14:52:34.061904+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":4084,"log_dups_size":0,"ondisk_log_size":4084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:28.030946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267649024,"num_objects":4084,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4084,"num_write_kb":261376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":392,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245541+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:52:23.245541+0000","last_peered":"2024-09-13T14:52:23.245541+0000","last_clean":"2024-09-13T14:52:23.245541+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:52:23.245541+0000","last_undegraded":"2024-09-13T14:52:23.245541+0000","last_fullsized":"2024-09-13T14:52:23.245541+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":108,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.247599+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:52:08.247599+0000","last_peered":"2024-09-13T14:52:08.247599+0000","last_clean":"2024-09-13T14:52:08.247599+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:52:08.247599+0000","last_undegraded":"2024-09-13T14:52:08.247599+0000","last_fullsized":"2024-09-13T14:52:08.247599+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":381,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245842+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:52:23.245842+0000","last_peered":"2024-09-13T14:52:23.245842+0000","last_clean":"2024-09-13T14:52:23.245842+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:52:23.245842+0000","last_undegraded":"2024-09-13T14:52:23.245842+0000","last_fullsized":"2024-09-13T14:52:23.245842+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":55,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.249032+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:52:08.249032+0000","last_peered":"2024-09-13T14:52:08.249032+0000","last_clean":"2024-09-13T14:52:08.249032+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:52:08.249032+0000","last_undegraded":"2024-09-13T14:52:08.249032+0000","last_fullsized":"2024-09-13T14:52:08.249032+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"272'3970","reported_seq":3993,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.482563+0000","last_change":"2024-09-13T14:51:50.159795+0000","last_active":"2024-09-13T14:52:34.482563+0000","last_peered":"2024-09-13T14:52:34.482563+0000","last_clean":"2024-09-13T14:52:34.482563+0000","last_became_active":"2024-09-13T14:51:50.159494+0000","last_became_peered":"2024-09-13T14:51:50.159494+0000","last_unstale":"2024-09-13T14:52:34.482563+0000","last_undegraded":"2024-09-13T14:52:34.482563+0000","last_fullsized":"2024-09-13T14:52:34.482563+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:22.123702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260177920,"num_objects":3970,"num_object_clones":0,"num_object_copies":11910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":254080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":410,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.251652+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:52:23.251652+0000","last_peered":"2024-09-13T14:52:23.251652+0000","last_clean":"2024-09-13T14:52:23.251652+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:52:23.251652+0000","last_undegraded":"2024-09-13T14:52:23.251652+0000","last_fullsized":"2024-09-13T14:52:23.251652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":247,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.244268+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:52:08.244268+0000","last_peered":"2024-09-13T14:52:08.244268+0000","last_clean":"2024-09-13T14:52:08.244268+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:52:08.244268+0000","last_undegraded":"2024-09-13T14:52:08.244268+0000","last_fullsized":"2024-09-13T14:52:08.244268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":365,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.247033+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:52:23.247033+0000","last_peered":"2024-09-13T14:52:23.247033+0000","last_clean":"2024-09-13T14:52:23.247033+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:52:23.247033+0000","last_undegraded":"2024-09-13T14:52:23.247033+0000","last_fullsized":"2024-09-13T14:52:23.247033+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":208,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.246849+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:52:08.246849+0000","last_peered":"2024-09-13T14:52:08.246849+0000","last_clean":"2024-09-13T14:52:08.246849+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:52:08.246849+0000","last_undegraded":"2024-09-13T14:52:08.246849+0000","last_fullsized":"2024-09-13T14:52:08.246849+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"272'3912","reported_seq":3935,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.442270+0000","last_change":"2024-09-13T14:51:50.162139+0000","last_active":"2024-09-13T14:52:34.442270+0000","last_peered":"2024-09-13T14:52:34.442270+0000","last_clean":"2024-09-13T14:52:34.442270+0000","last_became_active":"2024-09-13T14:51:50.161828+0000","last_became_peered":"2024-09-13T14:51:50.161828+0000","last_unstale":"2024-09-13T14:52:34.442270+0000","last_undegraded":"2024-09-13T14:52:34.442270+0000","last_fullsized":"2024-09-13T14:52:34.442270+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:18.448351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256376832,"num_objects":3912,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3912,"num_write_kb":250368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.239631+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:52:08.239631+0000","last_peered":"2024-09-13T14:52:08.239631+0000","last_clean":"2024-09-13T14:52:08.239631+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:52:08.239631+0000","last_undegraded":"2024-09-13T14:52:08.239631+0000","last_fullsized":"2024-09-13T14:52:08.239631+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":223,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.249725+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:52:23.249725+0000","last_peered":"2024-09-13T14:52:23.249725+0000","last_clean":"2024-09-13T14:52:23.249725+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:52:23.249725+0000","last_undegraded":"2024-09-13T14:52:23.249725+0000","last_fullsized":"2024-09-13T14:52:23.249725+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":250,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240087+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:52:08.240087+0000","last_peered":"2024-09-13T14:52:08.240087+0000","last_clean":"2024-09-13T14:52:08.240087+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:52:08.240087+0000","last_undegraded":"2024-09-13T14:52:08.240087+0000","last_fullsized":"2024-09-13T14:52:08.240087+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":228,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.246851+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:52:08.246851+0000","last_peered":"2024-09-13T14:52:08.246851+0000","last_clean":"2024-09-13T14:52:08.246851+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:52:08.246851+0000","last_undegraded":"2024-09-13T14:52:08.246851+0000","last_fullsized":"2024-09-13T14:52:08.246851+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"272'3736","reported_seq":3759,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.763621+0000","last_change":"2024-09-13T14:51:50.161719+0000","last_active":"2024-09-13T14:52:31.763621+0000","last_peered":"2024-09-13T14:52:31.763621+0000","last_clean":"2024-09-13T14:52:31.763621+0000","last_became_active":"2024-09-13T14:51:50.161426+0000","last_became_peered":"2024-09-13T14:51:50.161426+0000","last_unstale":"2024-09-13T14:52:31.763621+0000","last_undegraded":"2024-09-13T14:52:31.763621+0000","last_fullsized":"2024-09-13T14:52:31.763621+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:58.817057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244842496,"num_objects":3736,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3736,"num_write_kb":239104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":277,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.247214+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:52:23.247214+0000","last_peered":"2024-09-13T14:52:23.247214+0000","last_clean":"2024-09-13T14:52:23.247214+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:52:23.247214+0000","last_undegraded":"2024-09-13T14:52:23.247214+0000","last_fullsized":"2024-09-13T14:52:23.247214+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":235,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.248759+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:52:08.248759+0000","last_peered":"2024-09-13T14:52:08.248759+0000","last_clean":"2024-09-13T14:52:08.248759+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:52:08.248759+0000","last_undegraded":"2024-09-13T14:52:08.248759+0000","last_fullsized":"2024-09-13T14:52:08.248759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":246,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240468+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:52:08.240468+0000","last_peered":"2024-09-13T14:52:08.240468+0000","last_clean":"2024-09-13T14:52:08.240468+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:52:08.240468+0000","last_undegraded":"2024-09-13T14:52:08.240468+0000","last_fullsized":"2024-09-13T14:52:08.240468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":55,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.239681+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:52:08.239681+0000","last_peered":"2024-09-13T14:52:08.239681+0000","last_clean":"2024-09-13T14:52:08.239681+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:52:08.239681+0000","last_undegraded":"2024-09-13T14:52:08.239681+0000","last_fullsized":"2024-09-13T14:52:08.239681+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"272'3769","reported_seq":3791,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.785649+0000","last_change":"2024-09-13T14:51:50.163583+0000","last_active":"2024-09-13T14:52:31.785649+0000","last_peered":"2024-09-13T14:52:31.785649+0000","last_clean":"2024-09-13T14:52:31.785649+0000","last_became_active":"2024-09-13T14:51:50.163295+0000","last_became_peered":"2024-09-13T14:51:50.163295+0000","last_unstale":"2024-09-13T14:52:31.785649+0000","last_undegraded":"2024-09-13T14:52:31.785649+0000","last_fullsized":"2024-09-13T14:52:31.785649+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:01.494847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247005184,"num_objects":3769,"num_object_clones":0,"num_object_copies":11307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":241216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":166,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240125+0000","last_change":"2024-09-13T14:47:40.126588+0000","last_active":"2024-09-13T14:52:08.240125+0000","last_peered":"2024-09-13T14:52:08.240125+0000","last_clean":"2024-09-13T14:52:08.240125+0000","last_became_active":"2024-09-13T14:47:40.126265+0000","last_became_peered":"2024-09-13T14:47:40.126265+0000","last_unstale":"2024-09-13T14:52:08.240125+0000","last_undegraded":"2024-09-13T14:52:08.240125+0000","last_fullsized":"2024-09-13T14:52:08.240125+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:24.944233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":217,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.246060+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:52:23.246060+0000","last_peered":"2024-09-13T14:52:23.246060+0000","last_clean":"2024-09-13T14:52:23.246060+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:52:23.246060+0000","last_undegraded":"2024-09-13T14:52:23.246060+0000","last_fullsized":"2024-09-13T14:52:23.246060+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":408,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.246480+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:52:23.246480+0000","last_peered":"2024-09-13T14:52:23.246480+0000","last_clean":"2024-09-13T14:52:23.246480+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:52:23.246480+0000","last_undegraded":"2024-09-13T14:52:23.246480+0000","last_fullsized":"2024-09-13T14:52:23.246480+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":29,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.244756+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:52:08.244756+0000","last_peered":"2024-09-13T14:52:08.244756+0000","last_clean":"2024-09-13T14:52:08.244756+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:52:08.244756+0000","last_undegraded":"2024-09-13T14:52:08.244756+0000","last_fullsized":"2024-09-13T14:52:08.244756+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"272'3814","reported_seq":3837,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.776733+0000","last_change":"2024-09-13T14:51:50.162553+0000","last_active":"2024-09-13T14:52:31.776733+0000","last_peered":"2024-09-13T14:52:31.776733+0000","last_clean":"2024-09-13T14:52:31.776733+0000","last_became_active":"2024-09-13T14:51:50.162256+0000","last_became_peered":"2024-09-13T14:51:50.162256+0000","last_unstale":"2024-09-13T14:52:31.776733+0000","last_undegraded":"2024-09-13T14:52:31.776733+0000","last_fullsized":"2024-09-13T14:52:31.776733+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":3814,"log_dups_size":0,"ondisk_log_size":3814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:13.435069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249954304,"num_objects":3814,"num_object_clones":0,"num_object_copies":11442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3814,"num_write_kb":244096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":431,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.249709+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:52:23.249709+0000","last_peered":"2024-09-13T14:52:23.249709+0000","last_clean":"2024-09-13T14:52:23.249709+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:52:23.249709+0000","last_undegraded":"2024-09-13T14:52:23.249709+0000","last_fullsized":"2024-09-13T14:52:23.249709+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":441,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.251985+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:52:23.251985+0000","last_peered":"2024-09-13T14:52:23.251985+0000","last_clean":"2024-09-13T14:52:23.251985+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:52:23.251985+0000","last_undegraded":"2024-09-13T14:52:23.251985+0000","last_fullsized":"2024-09-13T14:52:23.251985+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":284,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.243349+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:52:08.243349+0000","last_peered":"2024-09-13T14:52:08.243349+0000","last_clean":"2024-09-13T14:52:08.243349+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:52:08.243349+0000","last_undegraded":"2024-09-13T14:52:08.243349+0000","last_fullsized":"2024-09-13T14:52:08.243349+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":208,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.246476+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:52:08.246476+0000","last_peered":"2024-09-13T14:52:08.246476+0000","last_clean":"2024-09-13T14:52:08.246476+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:52:08.246476+0000","last_undegraded":"2024-09-13T14:52:08.246476+0000","last_fullsized":"2024-09-13T14:52:08.246476+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"272'3800","reported_seq":3823,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.789816+0000","last_change":"2024-09-13T14:51:50.164744+0000","last_active":"2024-09-13T14:52:31.789816+0000","last_peered":"2024-09-13T14:52:31.789816+0000","last_clean":"2024-09-13T14:52:31.789816+0000","last_became_active":"2024-09-13T14:51:50.164428+0000","last_became_peered":"2024-09-13T14:51:50.164428+0000","last_unstale":"2024-09-13T14:52:31.789816+0000","last_undegraded":"2024-09-13T14:52:31.789816+0000","last_fullsized":"2024-09-13T14:52:31.789816+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":3800,"log_dups_size":0,"ondisk_log_size":3800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:59.644564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249036800,"num_objects":3800,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3800,"num_write_kb":243200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":217,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.247355+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:52:08.247355+0000","last_peered":"2024-09-13T14:52:08.247355+0000","last_clean":"2024-09-13T14:52:08.247355+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:52:08.247355+0000","last_undegraded":"2024-09-13T14:52:08.247355+0000","last_fullsized":"2024-09-13T14:52:08.247355+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":340,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.248905+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:52:23.248905+0000","last_peered":"2024-09-13T14:52:23.248905+0000","last_clean":"2024-09-13T14:52:23.248905+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:52:23.248905+0000","last_undegraded":"2024-09-13T14:52:23.248905+0000","last_fullsized":"2024-09-13T14:52:23.248905+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":245,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.239705+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:52:08.239705+0000","last_peered":"2024-09-13T14:52:08.239705+0000","last_clean":"2024-09-13T14:52:08.239705+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:52:08.239705+0000","last_undegraded":"2024-09-13T14:52:08.239705+0000","last_fullsized":"2024-09-13T14:52:08.239705+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":29,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.243099+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:52:08.243099+0000","last_peered":"2024-09-13T14:52:08.243099+0000","last_clean":"2024-09-13T14:52:08.243099+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:52:08.243099+0000","last_undegraded":"2024-09-13T14:52:08.243099+0000","last_fullsized":"2024-09-13T14:52:08.243099+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"272'3850","reported_seq":3873,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.777562+0000","last_change":"2024-09-13T14:51:50.159479+0000","last_active":"2024-09-13T14:52:31.777562+0000","last_peered":"2024-09-13T14:52:31.777562+0000","last_clean":"2024-09-13T14:52:31.777562+0000","last_became_active":"2024-09-13T14:51:50.159183+0000","last_became_peered":"2024-09-13T14:51:50.159183+0000","last_unstale":"2024-09-13T14:52:31.777562+0000","last_undegraded":"2024-09-13T14:52:31.777562+0000","last_fullsized":"2024-09-13T14:52:31.777562+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:08.537624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252313600,"num_objects":3850,"num_object_clones":0,"num_object_copies":11550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3850,"num_write_kb":246400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":63,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.247024+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:52:23.247024+0000","last_peered":"2024-09-13T14:52:23.247024+0000","last_clean":"2024-09-13T14:52:23.247024+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:52:23.247024+0000","last_undegraded":"2024-09-13T14:52:23.247024+0000","last_fullsized":"2024-09-13T14:52:23.247024+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":351,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.246062+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:52:23.246062+0000","last_peered":"2024-09-13T14:52:23.246062+0000","last_clean":"2024-09-13T14:52:23.246062+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:52:23.246062+0000","last_undegraded":"2024-09-13T14:52:23.246062+0000","last_fullsized":"2024-09-13T14:52:23.246062+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":55,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.239543+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:52:08.239543+0000","last_peered":"2024-09-13T14:52:08.239543+0000","last_clean":"2024-09-13T14:52:08.239543+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:52:08.239543+0000","last_undegraded":"2024-09-13T14:52:08.239543+0000","last_fullsized":"2024-09-13T14:52:08.239543+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"272'3994","reported_seq":4008,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.056153+0000","last_change":"2024-09-13T14:51:50.162786+0000","last_active":"2024-09-13T14:52:34.056153+0000","last_peered":"2024-09-13T14:52:34.056153+0000","last_clean":"2024-09-13T14:52:34.056153+0000","last_became_active":"2024-09-13T14:51:50.162300+0000","last_became_peered":"2024-09-13T14:51:50.162300+0000","last_unstale":"2024-09-13T14:52:34.056153+0000","last_undegraded":"2024-09-13T14:52:34.056153+0000","last_fullsized":"2024-09-13T14:52:34.056153+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":3994,"log_dups_size":0,"ondisk_log_size":3994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:16:10.631643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261750784,"num_objects":3994,"num_object_clones":0,"num_object_copies":11982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":255616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":254,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.249734+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:52:08.249734+0000","last_peered":"2024-09-13T14:52:08.249734+0000","last_clean":"2024-09-13T14:52:08.249734+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:52:08.249734+0000","last_undegraded":"2024-09-13T14:52:08.249734+0000","last_fullsized":"2024-09-13T14:52:08.249734+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":235,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.248269+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:52:08.248269+0000","last_peered":"2024-09-13T14:52:08.248269+0000","last_clean":"2024-09-13T14:52:08.248269+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:52:08.248269+0000","last_undegraded":"2024-09-13T14:52:08.248269+0000","last_fullsized":"2024-09-13T14:52:08.248269+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":401,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.251452+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:52:23.251452+0000","last_peered":"2024-09-13T14:52:23.251452+0000","last_clean":"2024-09-13T14:52:23.251452+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:52:23.251452+0000","last_undegraded":"2024-09-13T14:52:23.251452+0000","last_fullsized":"2024-09-13T14:52:23.251452+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"272'4157","reported_seq":4180,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.473616+0000","last_change":"2024-09-13T14:51:50.163483+0000","last_active":"2024-09-13T14:52:34.473616+0000","last_peered":"2024-09-13T14:52:34.473616+0000","last_clean":"2024-09-13T14:52:34.473616+0000","last_became_active":"2024-09-13T14:51:50.163168+0000","last_became_peered":"2024-09-13T14:51:50.163168+0000","last_unstale":"2024-09-13T14:52:34.473616+0000","last_undegraded":"2024-09-13T14:52:34.473616+0000","last_fullsized":"2024-09-13T14:52:34.473616+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:29.480939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272433152,"num_objects":4157,"num_object_clones":0,"num_object_copies":12471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4157,"num_write_kb":266048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":251,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.249259+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:52:08.249259+0000","last_peered":"2024-09-13T14:52:08.249259+0000","last_clean":"2024-09-13T14:52:08.249259+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:52:08.249259+0000","last_undegraded":"2024-09-13T14:52:08.249259+0000","last_fullsized":"2024-09-13T14:52:08.249259+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":255,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.247071+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:52:08.247071+0000","last_peered":"2024-09-13T14:52:08.247071+0000","last_clean":"2024-09-13T14:52:08.247071+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:52:08.247071+0000","last_undegraded":"2024-09-13T14:52:08.247071+0000","last_fullsized":"2024-09-13T14:52:08.247071+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":37,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.239985+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:52:08.239985+0000","last_peered":"2024-09-13T14:52:08.239985+0000","last_clean":"2024-09-13T14:52:08.239985+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:52:08.239985+0000","last_undegraded":"2024-09-13T14:52:08.239985+0000","last_fullsized":"2024-09-13T14:52:08.239985+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"272'4057","reported_seq":4071,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.029952+0000","last_change":"2024-09-13T14:51:50.162628+0000","last_active":"2024-09-13T14:52:34.029952+0000","last_peered":"2024-09-13T14:52:34.029952+0000","last_clean":"2024-09-13T14:52:34.029952+0000","last_became_active":"2024-09-13T14:51:50.162356+0000","last_became_peered":"2024-09-13T14:51:50.162356+0000","last_unstale":"2024-09-13T14:52:34.029952+0000","last_undegraded":"2024-09-13T14:52:34.029952+0000","last_fullsized":"2024-09-13T14:52:34.029952+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":4057,"log_dups_size":0,"ondisk_log_size":4057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:43.465982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265879552,"num_objects":4057,"num_object_clones":0,"num_object_copies":12171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4057,"num_write_kb":259648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":234,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.249173+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:52:08.249173+0000","last_peered":"2024-09-13T14:52:08.249173+0000","last_clean":"2024-09-13T14:52:08.249173+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:52:08.249173+0000","last_undegraded":"2024-09-13T14:52:08.249173+0000","last_fullsized":"2024-09-13T14:52:08.249173+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":380,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245998+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:52:23.245998+0000","last_peered":"2024-09-13T14:52:23.245998+0000","last_clean":"2024-09-13T14:52:23.245998+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:52:23.245998+0000","last_undegraded":"2024-09-13T14:52:23.245998+0000","last_fullsized":"2024-09-13T14:52:23.245998+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":55,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.243340+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:52:08.243340+0000","last_peered":"2024-09-13T14:52:08.243340+0000","last_clean":"2024-09-13T14:52:08.243340+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:52:08.243340+0000","last_undegraded":"2024-09-13T14:52:08.243340+0000","last_fullsized":"2024-09-13T14:52:08.243340+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"272'4016","reported_seq":4030,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.013795+0000","last_change":"2024-09-13T14:51:50.161603+0000","last_active":"2024-09-13T14:52:34.013795+0000","last_peered":"2024-09-13T14:52:34.013795+0000","last_clean":"2024-09-13T14:52:34.013795+0000","last_became_active":"2024-09-13T14:51:50.161319+0000","last_became_peered":"2024-09-13T14:51:50.161319+0000","last_unstale":"2024-09-13T14:52:34.013795+0000","last_undegraded":"2024-09-13T14:52:34.013795+0000","last_fullsized":"2024-09-13T14:52:34.013795+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:55.947148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263192576,"num_objects":4016,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4016,"num_write_kb":257024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":420,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.251566+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:52:23.251566+0000","last_peered":"2024-09-13T14:52:23.251566+0000","last_clean":"2024-09-13T14:52:23.251566+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:52:23.251566+0000","last_undegraded":"2024-09-13T14:52:23.251566+0000","last_fullsized":"2024-09-13T14:52:23.251566+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":233,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240204+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:52:08.240204+0000","last_peered":"2024-09-13T14:52:08.240204+0000","last_clean":"2024-09-13T14:52:08.240204+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:52:08.240204+0000","last_undegraded":"2024-09-13T14:52:08.240204+0000","last_fullsized":"2024-09-13T14:52:08.240204+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":55,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.249430+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:52:08.249430+0000","last_peered":"2024-09-13T14:52:08.249430+0000","last_clean":"2024-09-13T14:52:08.249430+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:52:08.249430+0000","last_undegraded":"2024-09-13T14:52:08.249430+0000","last_fullsized":"2024-09-13T14:52:08.249430+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"272'4057","reported_seq":4071,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.069414+0000","last_change":"2024-09-13T14:51:50.161149+0000","last_active":"2024-09-13T14:52:34.069414+0000","last_peered":"2024-09-13T14:52:34.069414+0000","last_clean":"2024-09-13T14:52:34.069414+0000","last_became_active":"2024-09-13T14:51:50.160859+0000","last_became_peered":"2024-09-13T14:51:50.160859+0000","last_unstale":"2024-09-13T14:52:34.069414+0000","last_undegraded":"2024-09-13T14:52:34.069414+0000","last_fullsized":"2024-09-13T14:52:34.069414+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":4057,"log_dups_size":0,"ondisk_log_size":4057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:40.847290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265879552,"num_objects":4057,"num_object_clones":0,"num_object_copies":12171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4057,"num_write_kb":259648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":247,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.247733+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:52:08.247733+0000","last_peered":"2024-09-13T14:52:08.247733+0000","last_clean":"2024-09-13T14:52:08.247733+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:52:08.247733+0000","last_undegraded":"2024-09-13T14:52:08.247733+0000","last_fullsized":"2024-09-13T14:52:08.247733+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":248,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.244278+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:52:08.244278+0000","last_peered":"2024-09-13T14:52:08.244278+0000","last_clean":"2024-09-13T14:52:08.244278+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:52:08.244278+0000","last_undegraded":"2024-09-13T14:52:08.244278+0000","last_fullsized":"2024-09-13T14:52:08.244278+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":208,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.246955+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:52:08.246955+0000","last_peered":"2024-09-13T14:52:08.246955+0000","last_clean":"2024-09-13T14:52:08.246955+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:52:08.246955+0000","last_undegraded":"2024-09-13T14:52:08.246955+0000","last_fullsized":"2024-09-13T14:52:08.246955+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"272'4152","reported_seq":4174,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.602653+0000","last_change":"2024-09-13T14:51:50.162266+0000","last_active":"2024-09-13T14:52:35.602653+0000","last_peered":"2024-09-13T14:52:35.602653+0000","last_clean":"2024-09-13T14:52:35.602653+0000","last_became_active":"2024-09-13T14:51:50.161674+0000","last_became_peered":"2024-09-13T14:51:50.161674+0000","last_unstale":"2024-09-13T14:52:35.602653+0000","last_undegraded":"2024-09-13T14:52:35.602653+0000","last_fullsized":"2024-09-13T14:52:35.602653+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:59.738238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272105472,"num_objects":4152,"num_object_clones":0,"num_object_copies":12456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4152,"num_write_kb":265728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":261,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.248181+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:52:08.248181+0000","last_peered":"2024-09-13T14:52:08.248181+0000","last_clean":"2024-09-13T14:52:08.248181+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:52:08.248181+0000","last_undegraded":"2024-09-13T14:52:08.248181+0000","last_fullsized":"2024-09-13T14:52:08.248181+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":238,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.244507+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:52:08.244507+0000","last_peered":"2024-09-13T14:52:08.244507+0000","last_clean":"2024-09-13T14:52:08.244507+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:52:08.244507+0000","last_undegraded":"2024-09-13T14:52:08.244507+0000","last_fullsized":"2024-09-13T14:52:08.244507+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":108,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245794+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:52:23.245794+0000","last_peered":"2024-09-13T14:52:23.245794+0000","last_clean":"2024-09-13T14:52:23.245794+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:52:23.245794+0000","last_undegraded":"2024-09-13T14:52:23.245794+0000","last_fullsized":"2024-09-13T14:52:23.245794+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"272'4110","reported_seq":4124,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.761140+0000","last_change":"2024-09-13T14:51:50.159778+0000","last_active":"2024-09-13T14:52:34.761140+0000","last_peered":"2024-09-13T14:52:34.761140+0000","last_clean":"2024-09-13T14:52:34.761140+0000","last_became_active":"2024-09-13T14:51:50.159462+0000","last_became_peered":"2024-09-13T14:51:50.159462+0000","last_unstale":"2024-09-13T14:52:34.761140+0000","last_undegraded":"2024-09-13T14:52:34.761140+0000","last_fullsized":"2024-09-13T14:52:34.761140+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:51:49.136139+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.136139+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:20.012384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269352960,"num_objects":4110,"num_object_clones":0,"num_object_copies":12330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4110,"num_write_kb":263040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":43,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240081+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:52:08.240081+0000","last_peered":"2024-09-13T14:52:08.240081+0000","last_clean":"2024-09-13T14:52:08.240081+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:52:08.240081+0000","last_undegraded":"2024-09-13T14:52:08.240081+0000","last_fullsized":"2024-09-13T14:52:08.240081+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":33,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.244648+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:52:08.244648+0000","last_peered":"2024-09-13T14:52:08.244648+0000","last_clean":"2024-09-13T14:52:08.244648+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:52:08.244648+0000","last_undegraded":"2024-09-13T14:52:08.244648+0000","last_fullsized":"2024-09-13T14:52:08.244648+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":39,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.244174+0000","last_change":"2024-09-13T14:52:02.567722+0000","last_active":"2024-09-13T14:52:08.244174+0000","last_peered":"2024-09-13T14:52:08.244174+0000","last_clean":"2024-09-13T14:52:08.244174+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:52:08.244174+0000","last_undegraded":"2024-09-13T14:52:08.244174+0000","last_fullsized":"2024-09-13T14:52:08.244174+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:52:02.567611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:52:02.567611+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:03.419129+0000","scrub_duration":0.0030819879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":283,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.247057+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:52:08.247057+0000","last_peered":"2024-09-13T14:52:08.247057+0000","last_clean":"2024-09-13T14:52:08.247057+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:52:08.247057+0000","last_undegraded":"2024-09-13T14:52:08.247057+0000","last_fullsized":"2024-09-13T14:52:08.247057+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":389,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245674+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:52:23.245674+0000","last_peered":"2024-09-13T14:52:23.245674+0000","last_clean":"2024-09-13T14:52:23.245674+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:52:23.245674+0000","last_undegraded":"2024-09-13T14:52:23.245674+0000","last_fullsized":"2024-09-13T14:52:23.245674+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":65,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.239714+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:52:08.239714+0000","last_peered":"2024-09-13T14:52:08.239714+0000","last_clean":"2024-09-13T14:52:08.239714+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:52:08.239714+0000","last_undegraded":"2024-09-13T14:52:08.239714+0000","last_fullsized":"2024-09-13T14:52:08.239714+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":229,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.243466+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:52:08.243466+0000","last_peered":"2024-09-13T14:52:08.243466+0000","last_clean":"2024-09-13T14:52:08.243466+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:52:08.243466+0000","last_undegraded":"2024-09-13T14:52:08.243466+0000","last_fullsized":"2024-09-13T14:52:08.243466+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":433,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.249811+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:52:23.249811+0000","last_peered":"2024-09-13T14:52:23.249811+0000","last_clean":"2024-09-13T14:52:23.249811+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:52:23.249811+0000","last_undegraded":"2024-09-13T14:52:23.249811+0000","last_fullsized":"2024-09-13T14:52:23.249811+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":104,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.245802+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:52:23.245802+0000","last_peered":"2024-09-13T14:52:23.245802+0000","last_clean":"2024-09-13T14:52:23.245802+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:52:23.245802+0000","last_undegraded":"2024-09-13T14:52:23.245802+0000","last_fullsized":"2024-09-13T14:52:23.245802+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":440,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.250444+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:52:23.250444+0000","last_peered":"2024-09-13T14:52:23.250444+0000","last_clean":"2024-09-13T14:52:23.250444+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:52:23.250444+0000","last_undegraded":"2024-09-13T14:52:23.250444+0000","last_fullsized":"2024-09-13T14:52:23.250444+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":33,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.239915+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:52:08.239915+0000","last_peered":"2024-09-13T14:52:08.239915+0000","last_clean":"2024-09-13T14:52:08.239915+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:52:08.239915+0000","last_undegraded":"2024-09-13T14:52:08.239915+0000","last_fullsized":"2024-09-13T14:52:08.239915+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":55,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.240160+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:52:08.240160+0000","last_peered":"2024-09-13T14:52:08.240160+0000","last_clean":"2024-09-13T14:52:08.240160+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:52:08.240160+0000","last_undegraded":"2024-09-13T14:52:08.240160+0000","last_fullsized":"2024-09-13T14:52:08.240160+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":254,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:52:08.243394+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:52:08.243394+0000","last_peered":"2024-09-13T14:52:08.243394+0000","last_clean":"2024-09-13T14:52:08.243394+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:52:08.243394+0000","last_undegraded":"2024-09-13T14:52:08.243394+0000","last_fullsized":"2024-09-13T14:52:08.243394+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":99,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.249538+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:52:23.249538+0000","last_peered":"2024-09-13T14:52:23.249538+0000","last_clean":"2024-09-13T14:52:23.249538+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:52:23.249538+0000","last_undegraded":"2024-09-13T14:52:23.249538+0000","last_fullsized":"2024-09-13T14:52:23.249538+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":397,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:52:23.252072+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:52:23.252072+0000","last_peered":"2024-09-13T14:52:23.252072+0000","last_clean":"2024-09-13T14:52:23.252072+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:52:23.252072+0000","last_undegraded":"2024-09-13T14:52:23.252072+0000","last_fullsized":"2024-09-13T14:52:23.252072+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4163960832,"num_objects":63537,"num_object_clones":0,"num_object_copies":190611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63537,"num_write_kb":4066368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12534087680,"data_stored":12534087680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63537,"ondisk_log_size":63537,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365194,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2452912,"kb_used_data":1539368,"kb_used_omap":1,"kb_used_meta":913534,"kb_avail":91292240,"statfs":{"total":95995035648,"available":93483253760,"internally_reserved":0,"allocated":1576312832,"data_stored":1574846766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":935459160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2917864,"kb_used_data":2016032,"kb_used_omap":1,"kb_used_meta":901822,"kb_avail":90827288,"statfs":{"total":95995035648,"available":93007142912,"internally_reserved":0,"allocated":2064416768,"data_stored":2062958717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923466072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790356,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1932512,"kb_used_data":1035608,"kb_used_omap":1,"kb_used_meta":896894,"kb_avail":91812640,"statfs":{"total":95995035648,"available":94016143360,"internally_reserved":0,"allocated":1060462592,"data_stored":1059008851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918419800},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986584,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2333740,"kb_used_data":1435172,"kb_used_omap":1,"kb_used_meta":898558,"kb_avail":91411412,"statfs":{"total":95995035648,"available":93605285888,"internally_reserved":0,"allocated":1469616128,"data_stored":1468154319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920123736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519739,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2259360,"kb_used_data":1680664,"kb_used_omap":4,"kb_used_meta":578683,"kb_avail":91485792,"statfs":{"total":95995035648,"available":93681451008,"internally_reserved":0,"allocated":1720999936,"data_stored":1719546205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":592571840},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304655,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2066660,"kb_used_data":1525852,"kb_used_omap":6,"kb_used_meta":540793,"kb_avail":91678492,"statfs":{"total":95995035648,"available":93878775808,"internally_reserved":0,"allocated":1562472448,"data_stored":1561015135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":553772487},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827958,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2821856,"kb_used_data":1965272,"kb_used_omap":5,"kb_used_meta":856570,"kb_avail":90923296,"statfs":{"total":95995035648,"available":93105455104,"internally_reserved":0,"allocated":2012438528,"data_stored":2010984224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":877127916},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285823,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088348,"kb_used_data":1328404,"kb_used_omap":5,"kb_used_meta":759930,"kb_avail":91656804,"statfs":{"total":95995035648,"available":93856567296,"internally_reserved":0,"allocated":1360285696,"data_stored":1358834791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":778168379},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1357578240,"data_stored":1357578240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2030698496,"data_stored":2030698496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1591803904,"data_stored":1591803904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1748238336,"data_stored":1748238336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1046020096,"data_stored":1046020096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1070006272,"data_stored":1070006272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2096234496,"data_stored":2096234496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1593507840,"data_stored":1593507840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:52:38.843 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:52:38.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_6 pg_num 26 2024-09-13T14:52:38.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 70068 70052 115.186 71.375 0.018325 0.00867512 2024-09-13T14:52:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:38.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:38.950 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:38.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:39.038 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:38 smithi003 ceph-mon[107278]: from='client.56221 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:39.038 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:38 smithi003 ceph-mon[107278]: pgmap v952: 109 pgs: 109 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:52:39.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:38 smithi003 ceph-mon[104007]: from='client.56221 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:39.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:38 smithi003 ceph-mon[104007]: pgmap v952: 109 pgs: 109 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:52:39.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:39.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:39.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:39.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:39.356 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:39.389 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:38 smithi019 ceph-mon[91296]: from='client.56221 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:38 smithi019 ceph-mon[91296]: pgmap v952: 109 pgs: 109 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:52:39.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:39.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:39.627 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:39.662 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:39.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:39.798 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:39.843 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 71326 71310 114.249 78.625 0.0181695 0.00874801 2024-09-13T14:52:39.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:39.934 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:40.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:40.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:40.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:40.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:40.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:40.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:40.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:40.611 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:40.629 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:52:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:52:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:40.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:40.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:40.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:40.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:40.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:52:40.843231+0000 min lat: 0.00120774 max lat: 0.279703 avg lat: 0.00881323 2024-09-13T14:52:40.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:52:40.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 72614 72598 113.405 80.5 0.0102821 0.00881323 2024-09-13T14:52:40.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:40.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:41.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:41.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:41.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:41.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:41.331 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:40 smithi003 ceph-mon[107278]: pgmap v953: 109 pgs: 109 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-09-13T14:52:41.332 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:40 smithi003 ceph-mon[104007]: pgmap v953: 109 pgs: 109 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-09-13T14:52:41.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:41.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:41.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:40 smithi019 ceph-mon[91296]: pgmap v953: 109 pgs: 109 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-09-13T14:52:41.488 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:41.527 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:41.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:41.661 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:41.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:41.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:41.843 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 73824 73808 112.483 75.625 0.0152864 0.00888487 2024-09-13T14:52:41.903 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:41.933 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:41.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:42.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:42.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:42.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:42.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:42.310 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:52:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:52:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:52:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:52:42.310 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:52:42.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:42.468 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:42.505 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:42.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:42.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:42.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:42.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.779 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:42.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:42.843 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 75005 74989 111.563 73.8125 0.00875879 0.00895823 2024-09-13T14:52:42.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.919 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:42.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:52:43.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:43.059 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:43.161 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.196 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:43.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:43.221 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:42 smithi019 ceph-mon[91296]: pgmap v954: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:52:43.222 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:42 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2417404323' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:43.222 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:43.298 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:43.333 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:42 smithi003 ceph-mon[104007]: pgmap v954: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:52:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:42 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2417404323' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:42 smithi003 ceph-mon[107278]: pgmap v954: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:52:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:42 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2417404323' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:43.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:43.466 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:43.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:43.601 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:43.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:43.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:43.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 76175 76159 110.669 73.125 0.0103939 0.00903122 2024-09-13T14:52:43.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:43.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:43.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:44.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:44.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:44.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:44.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:44.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:43 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:44.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2417404323' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:44.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:43 smithi003 ceph-mon[104007]: osdmap e273: 8 total, 8 up, 8 in 2024-09-13T14:52:44.289 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:43 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:44.290 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2417404323' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:44.290 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:43 smithi003 ceph-mon[107278]: osdmap e273: 8 total, 8 up, 8 in 2024-09-13T14:52:44.290 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:52:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:52:44] "GET /metrics HTTP/1.1" 200 36529 "" "Prometheus/2.51.0" 2024-09-13T14:52:44.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:44.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:52:44.409 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:43 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2417404323' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:43 smithi019 ceph-mon[91296]: osdmap e273: 8 total, 8 up, 8 in 2024-09-13T14:52:44.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.442 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:44.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:44.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:44.554 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:44.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:44.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:44.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 77368 77352 109.848 74.5625 0.00827627 0.00909827 2024-09-13T14:52:44.853 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:44.890 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:44.992 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.034 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:45.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:45.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:45.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:45.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:45.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:44 smithi003 ceph-mon[107278]: pgmap v956: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T14:52:45.270 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:44 smithi003 ceph-mon[104007]: pgmap v956: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T14:52:45.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:45.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:45.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:44 smithi019 ceph-mon[91296]: pgmap v956: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T14:52:45.442 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:45.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:45.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:45.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:45.683 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:45.719 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:45.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:45.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:45.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 78648 78632 109.185 80 0.0113013 0.00915393 2024-09-13T14:52:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:45.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:46.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:46.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.159 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:46.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:46.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:46.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:46.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:46.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:46.551 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:46.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.593 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:46.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:46.694 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:46.723 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:46.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:46.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 79872 79856 108.474 76.5 0.00846512 0.00921306 2024-09-13T14:52:46.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:46.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:47.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:47.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:46 smithi003 ceph-mon[104007]: pgmap v957: 109 pgs: 109 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:52:47.080 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:46 smithi003 ceph-mon[107278]: pgmap v957: 109 pgs: 109 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:52:47.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:47.156 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:47.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:52:47.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:47.289 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:47.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:46 smithi019 ceph-mon[91296]: pgmap v957: 109 pgs: 109 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:52:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:47.436 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:52:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:47.478 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:52:47.478 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:47.515 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:52:47.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:47.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:47.552 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:52:47.552 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:47.581 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:52:47.581 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:47.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:47.605 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:52:47.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:47.630 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:52:47.630 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:47.665 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:52:47.665 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:47.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:47.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:47.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:47.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:47.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 80977 80961 107.636 69.0625 0.0188373 0.00928552 2024-09-13T14:52:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:47.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:48.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:48.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:48.260 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:48.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1522203336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:48.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1522203336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:48.292 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:48.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:48.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1522203336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:48.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:48.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:48.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:48.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:48.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:48.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:48.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 81784 81768 106.444 50.4375 0.00968133 0.00938988 2024-09-13T14:52:48.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:48.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:49.054 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:49.095 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:49.197 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:49.219 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:49.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:49.322 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:49.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:48 smithi003 ceph-mon[107278]: pgmap v958: 109 pgs: 109 active+clean; 4.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-13T14:52:49.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:48 smithi003 ceph-mon[104007]: pgmap v958: 109 pgs: 109 active+clean; 4.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-13T14:52:49.354 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:49.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:48 smithi019 ceph-mon[91296]: pgmap v958: 109 pgs: 109 active+clean; 4.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-13T14:52:49.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:49.478 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:49.580 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:49.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:49.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:49.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:49.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:49.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 82741 82725 105.492 59.8125 0.00831998 0.00947363 2024-09-13T14:52:49.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:49.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:50.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:50.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:50.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:50.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:50.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:50.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:50.423 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:50.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:50.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:50.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:52:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:52:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:50.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:50.808 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:50.839 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:50.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 83956 83940 104.901 75.9375 0.0132619 0.00952697 2024-09-13T14:52:50.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:50.976 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:51.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:51.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:51.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:51 smithi003 ceph-mon[107278]: pgmap v959: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:52:51.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:51 smithi003 ceph-mon[104007]: pgmap v959: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:52:51.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:51.255 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:51.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:51.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:51.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:51 smithi019 ceph-mon[91296]: pgmap v959: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:52:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:51.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:51.514 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:51.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:51.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:51.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:51.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:51.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:51.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 85054 85038 104.19 68.625 0.0312881 0.00959165 2024-09-13T14:52:51.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:51.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:52.007 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 11 pg_num to 26 2024-09-13T14:52:52.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:52 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/867232373' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:52:52.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:52 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:52:52.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:52 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/867232373' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:52:52.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:52 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:52:52.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:52.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:52.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:52.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:52.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:52.317 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:52 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/867232373' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:52:52.317 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:52 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:52:52.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:52.402 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:52.511 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:52.556 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:52.658 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:52.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.690 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:52.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:52.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:52.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 86206 86190 103.571 72 0.006716 0.0096501 2024-09-13T14:52:52.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:52.969 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:53.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.112 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:53.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:53 smithi003 ceph-mon[107278]: pgmap v960: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:52:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:53 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:52:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:53 smithi003 ceph-mon[107278]: osdmap e274: 8 total, 8 up, 8 in 2024-09-13T14:52:53.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:53 smithi003 ceph-mon[104007]: pgmap v960: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:52:53.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:53 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:52:53.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:53 smithi003 ceph-mon[104007]: osdmap e274: 8 total, 8 up, 8 in 2024-09-13T14:52:53.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:53.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:52:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:53.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:53.312 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:53 smithi019 ceph-mon[91296]: pgmap v960: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:52:53.312 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:53 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:52:53.313 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:53 smithi019 ceph-mon[91296]: osdmap e274: 8 total, 8 up, 8 in 2024-09-13T14:52:53.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:53.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:53.503 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:53.534 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:53.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:53.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:53.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:53.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:53.844 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 87341 87325 102.955 70.9375 0.00982513 0.00970777 2024-09-13T14:52:53.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.970 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:53.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:54.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:54.133 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:54.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:54.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:54.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:54.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:52:54.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:52:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:52:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:52:54] "GET /metrics HTTP/1.1" 200 36529 "" "Prometheus/2.51.0" 2024-09-13T14:52:54.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:52:54.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:54.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:54.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:54.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:54.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:54.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:54.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:54.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 87572 87556 101.316 14.4375 0.0498341 0.00972 2024-09-13T14:52:54.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.940 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:54.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:55.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:55.079 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:55.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:55 smithi019 ceph-mon[91296]: pgmap v962: 109 pgs: 109 active+clean; 5.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:52:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:52:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:55 smithi019 ceph-mon[91296]: osdmap e275: 8 total, 8 up, 8 in 2024-09-13T14:52:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:55.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:55.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:55.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:55 smithi003 ceph-mon[107278]: pgmap v962: 109 pgs: 109 active+clean; 5.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:52:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:52:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:55 smithi003 ceph-mon[107278]: osdmap e275: 8 total, 8 up, 8 in 2024-09-13T14:52:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:55.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:55 smithi003 ceph-mon[104007]: pgmap v962: 109 pgs: 109 active+clean; 5.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:52:55.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:52:55.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:55 smithi003 ceph-mon[104007]: osdmap e275: 8 total, 8 up, 8 in 2024-09-13T14:52:55.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:55.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:55.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:55.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.618 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:55.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:55.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:55.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:55.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 88501 88485 100.53 58.0625 0.0102664 0.0099421 2024-09-13T14:52:55.859 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:55.886 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:55.987 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:55.989 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:56.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:56.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:56.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:56 smithi003 ceph-mon[107278]: osdmap e276: 8 total, 8 up, 8 in 2024-09-13T14:52:56.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:56 smithi003 ceph-mon[107278]: 11.2 scrub starts 2024-09-13T14:52:56.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:56 smithi003 ceph-mon[107278]: 11.6 scrub starts 2024-09-13T14:52:56.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:56 smithi003 ceph-mon[104007]: osdmap e276: 8 total, 8 up, 8 in 2024-09-13T14:52:56.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:56 smithi003 ceph-mon[104007]: 11.2 scrub starts 2024-09-13T14:52:56.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:56 smithi003 ceph-mon[104007]: 11.6 scrub starts 2024-09-13T14:52:56.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:56.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:56.401 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:56 smithi019 ceph-mon[91296]: osdmap e276: 8 total, 8 up, 8 in 2024-09-13T14:52:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:56 smithi019 ceph-mon[91296]: 11.2 scrub starts 2024-09-13T14:52:56.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:56 smithi019 ceph-mon[91296]: 11.6 scrub starts 2024-09-13T14:52:56.446 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:56.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:56.688 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:56.723 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:56.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:56.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 89811 89795 100.196 81.875 0.00756214 0.00997483 2024-09-13T14:52:56.864 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:56.966 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:57.103 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:57.109 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:57.109 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:57.132 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:57.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:57.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:57.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:57.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:57.427 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:57 smithi019 ceph-mon[91296]: pgmap v965: 119 pgs: 6 peering, 9 unknown, 104 active+clean; 4.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail 2024-09-13T14:52:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:57 smithi019 ceph-mon[91296]: 11.9 scrub starts 2024-09-13T14:52:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:57 smithi019 ceph-mon[91296]: 11.12 scrub starts 2024-09-13T14:52:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:57 smithi019 ceph-mon[91296]: 11.16 scrub starts 2024-09-13T14:52:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:57 smithi019 ceph-mon[91296]: 11.8 scrub starts 2024-09-13T14:52:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[104007]: pgmap v965: 119 pgs: 6 peering, 9 unknown, 104 active+clean; 4.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail 2024-09-13T14:52:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[104007]: 11.9 scrub starts 2024-09-13T14:52:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[104007]: 11.12 scrub starts 2024-09-13T14:52:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[104007]: 11.16 scrub starts 2024-09-13T14:52:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[104007]: 11.8 scrub starts 2024-09-13T14:52:57.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[107278]: pgmap v965: 119 pgs: 6 peering, 9 unknown, 104 active+clean; 4.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail 2024-09-13T14:52:57.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[107278]: 11.9 scrub starts 2024-09-13T14:52:57.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[107278]: 11.12 scrub starts 2024-09-13T14:52:57.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[107278]: 11.16 scrub starts 2024-09-13T14:52:57.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:57 smithi003 ceph-mon[107278]: 11.8 scrub starts 2024-09-13T14:52:57.530 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:57.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:57.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:57.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:57.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:57.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 91340 91324 100.115 95.5625 0.00796658 0.0099837 2024-09-13T14:52:57.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:57.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:52:57.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:57.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:58.085 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:58.120 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:58.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:58.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:58.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:58.417 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:58.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:58.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:58.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:58.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:58.705 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:52:58.705 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:52:58.705 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.9476238917435175 2024-09-13T14:52:58.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 0 0.9476238917435175 2024-09-13T14:52:58.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:58.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:58.823 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:58.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 92624 92608 99.772 80.25 0.0105926 0.0100176 2024-09-13T14:52:58.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:58.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:59.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:59.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:59.134 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:59.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:59.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:59 smithi003 ceph-mon[104007]: pgmap v966: 119 pgs: 6 peering, 6 unknown, 107 active+clean; 4.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:52:59.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:52:59 smithi003 ceph-mon[104007]: 11.19 scrub starts 2024-09-13T14:52:59.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:59 smithi003 ceph-mon[107278]: pgmap v966: 119 pgs: 6 peering, 6 unknown, 107 active+clean; 4.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:52:59.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:52:59 smithi003 ceph-mon[107278]: 11.19 scrub starts 2024-09-13T14:52:59.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:59.283 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:59.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:59.430 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:59.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:59 smithi019 ceph-mon[91296]: pgmap v966: 119 pgs: 6 peering, 6 unknown, 107 active+clean; 4.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:52:59.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:52:59 smithi019 ceph-mon[91296]: 11.19 scrub starts 2024-09-13T14:52:59.473 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:59.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:59.575 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:59.604 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:52:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:59.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:59.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:59.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:52:59.831 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:59.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 93864 93848 99.3944 77.5 0.0118565 0.010056 2024-09-13T14:52:59.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:59.873 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:52:59.873 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:59.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:52:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:59.906 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:52:59.906 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:59.944 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:52:59.944 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:59.976 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:52:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:00.006 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:00.009 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:53:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:00.043 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:53:00.043 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:00.062 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:00.076 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:53:00.076 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:00.082 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[107278]: 11.0 scrub starts 2024-09-13T14:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[104007]: 11.0 scrub starts 2024-09-13T14:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4013995461' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/89232130' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:53:00.092 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:53:00.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:00.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:00.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:00.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:00.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:53:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:53:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:00 smithi019 ceph-mon[91296]: 11.0 scrub starts 2024-09-13T14:53:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:00 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4013995461' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:00 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:00 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/89232130' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:00 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:53:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4013995461' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/89232130' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:00 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:53:00.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:00.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:00.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:00.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:00.697 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.721 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:00.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:00.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.845 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:00.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:00.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:53:00.845339+0000 min lat: 0.00120774 max lat: 1.08696 avg lat: 0.0100529 2024-09-13T14:53:00.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:53:00.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 94657 94641 98.5639 49.5625 0.00325795 0.0100529 2024-09-13T14:53:00.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.971 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:00.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:01.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:01.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:01.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:01.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:01.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:01.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: pgmap v967: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-13T14:53:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: 11.10 scrub starts 2024-09-13T14:53:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:53:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: osdmap e277: 8 total, 8 up, 8 in 2024-09-13T14:53:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: 11.4 scrub starts 2024-09-13T14:53:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: 11.2 scrub starts 2024-09-13T14:53:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: osdmap e278: 8 total, 8 up, 8 in 2024-09-13T14:53:01.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:01 smithi019 ceph-mon[91296]: 11.16 scrub starts 2024-09-13T14:53:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: pgmap v967: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-13T14:53:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: 11.10 scrub starts 2024-09-13T14:53:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: osdmap e277: 8 total, 8 up, 8 in 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: 11.4 scrub starts 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: 11.2 scrub starts 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: osdmap e278: 8 total, 8 up, 8 in 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[104007]: 11.16 scrub starts 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: pgmap v967: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: 11.10 scrub starts 2024-09-13T14:53:01.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:01.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:01.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:53:01.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: osdmap e277: 8 total, 8 up, 8 in 2024-09-13T14:53:01.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:01.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:01.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: 11.4 scrub starts 2024-09-13T14:53:01.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: 11.2 scrub starts 2024-09-13T14:53:01.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: osdmap e278: 8 total, 8 up, 8 in 2024-09-13T14:53:01.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:01 smithi003 ceph-mon[107278]: 11.16 scrub starts 2024-09-13T14:53:01.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:01.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:01.575 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:01.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.599 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:01.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:01.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:01.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:01.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:01.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 94657 94641 96.9483 0 - 0.0100529 2024-09-13T14:53:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:01.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:01.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:02.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:02.114 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:02.216 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:02.245 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:02.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:02.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:02 smithi019 ceph-mon[91296]: 11.4 scrub ok 2024-09-13T14:53:02.517 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:02 smithi019 ceph-mon[91296]: 11.2 scrub ok 2024-09-13T14:53:02.517 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:02 smithi019 ceph-mon[91296]: 11.18 scrub starts 2024-09-13T14:53:02.517 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:02 smithi019 ceph-mon[91296]: 11.16 scrub ok 2024-09-13T14:53:02.517 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:02 smithi019 ceph-mon[91296]: 11.18 scrub ok 2024-09-13T14:53:02.517 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:02 smithi019 ceph-mon[91296]: 11.12 scrub starts 2024-09-13T14:53:02.517 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:02 smithi019 ceph-mon[91296]: osdmap e279: 8 total, 8 up, 8 in 2024-09-13T14:53:02.517 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:02 smithi019 ceph-mon[91296]: 11.6 scrub starts 2024-09-13T14:53:02.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:02.518 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[107278]: 11.4 scrub ok 2024-09-13T14:53:02.518 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[107278]: 11.2 scrub ok 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[107278]: 11.18 scrub starts 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[107278]: 11.16 scrub ok 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[107278]: 11.18 scrub ok 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[107278]: 11.12 scrub starts 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[107278]: osdmap e279: 8 total, 8 up, 8 in 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[107278]: 11.6 scrub starts 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[104007]: 11.4 scrub ok 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[104007]: 11.2 scrub ok 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[104007]: 11.18 scrub starts 2024-09-13T14:53:02.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[104007]: 11.16 scrub ok 2024-09-13T14:53:02.520 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[104007]: 11.18 scrub ok 2024-09-13T14:53:02.520 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[104007]: 11.12 scrub starts 2024-09-13T14:53:02.520 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[104007]: osdmap e279: 8 total, 8 up, 8 in 2024-09-13T14:53:02.520 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:02 smithi003 ceph-mon[104007]: 11.6 scrub starts 2024-09-13T14:53:02.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:02.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:02.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:02.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:02.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:02.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:02.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:02.845 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 95358 95342 96.0913 21.9062 0.00985429 0.0104013 2024-09-13T14:53:02.946 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:02.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:03.078 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:03.112 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:03.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:03.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:03.249 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:03.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[104007]: pgmap v971: 119 pgs: 1 unknown, 118 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-13T14:53:03.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[104007]: 11.19 scrub starts 2024-09-13T14:53:03.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[104007]: 11.12 scrub ok 2024-09-13T14:53:03.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[104007]: 11.6 scrub ok 2024-09-13T14:53:03.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[104007]: 11.19 scrub ok 2024-09-13T14:53:03.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[107278]: pgmap v971: 119 pgs: 1 unknown, 118 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-13T14:53:03.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[107278]: 11.19 scrub starts 2024-09-13T14:53:03.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[107278]: 11.12 scrub ok 2024-09-13T14:53:03.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[107278]: 11.6 scrub ok 2024-09-13T14:53:03.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:03 smithi003 ceph-mon[107278]: 11.19 scrub ok 2024-09-13T14:53:03.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:03.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:03 smithi019 ceph-mon[91296]: pgmap v971: 119 pgs: 1 unknown, 118 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-13T14:53:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:03 smithi019 ceph-mon[91296]: 11.19 scrub starts 2024-09-13T14:53:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:03 smithi019 ceph-mon[91296]: 11.12 scrub ok 2024-09-13T14:53:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:03 smithi019 ceph-mon[91296]: 11.6 scrub ok 2024-09-13T14:53:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:03 smithi019 ceph-mon[91296]: 11.19 scrub ok 2024-09-13T14:53:03.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:03.577 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:03.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:03.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:03.719 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:03.821 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:03.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 96498 96482 95.6969 71.25 0.00272356 0.010444 2024-09-13T14:53:03.850 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:03.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:03.981 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:03.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:04.083 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.113 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:04.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:04.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:04 smithi003 ceph-mon[107278]: 11.9 scrub starts 2024-09-13T14:53:04.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:04 smithi003 ceph-mon[107278]: 11.7 scrub starts 2024-09-13T14:53:04.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T14:53:04.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:04 smithi003 ceph-mon[107278]: 11.9 scrub ok 2024-09-13T14:53:04.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:04 smithi003 ceph-mon[104007]: 11.9 scrub starts 2024-09-13T14:53:04.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:04 smithi003 ceph-mon[104007]: 11.7 scrub starts 2024-09-13T14:53:04.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T14:53:04.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:04 smithi003 ceph-mon[104007]: 11.9 scrub ok 2024-09-13T14:53:04.184 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:53:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:53:04] "GET /metrics HTTP/1.1" 200 36529 "" "Prometheus/2.51.0" 2024-09-13T14:53:04.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:04.246 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:04.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:04.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:04 smithi019 ceph-mon[91296]: 11.9 scrub starts 2024-09-13T14:53:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:04 smithi019 ceph-mon[91296]: 11.7 scrub starts 2024-09-13T14:53:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T14:53:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:04 smithi019 ceph-mon[91296]: 11.9 scrub ok 2024-09-13T14:53:04.440 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:04.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:04.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:04.627 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:04.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.653 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:04.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:04.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:04.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 97217 97201 94.9038 44.9375 0.00207352 0.0104608 2024-09-13T14:53:04.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:04.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:05.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:05.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:05.134 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:05.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:05.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:05.280 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[107278]: pgmap v972: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 168 MiB/s wr, 2.69k op/s; 9129/281421 objects misplaced (3.244%); 32 KiB/s, 0 objects/s recovering 2024-09-13T14:53:05.280 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[107278]: 11.7 scrub ok 2024-09-13T14:53:05.280 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T14:53:05.280 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[107278]: osdmap e280: 8 total, 8 up, 8 in 2024-09-13T14:53:05.280 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[107278]: 11.8 scrub starts 2024-09-13T14:53:05.280 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[107278]: 11.8 scrub ok 2024-09-13T14:53:05.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[104007]: pgmap v972: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 168 MiB/s wr, 2.69k op/s; 9129/281421 objects misplaced (3.244%); 32 KiB/s, 0 objects/s recovering 2024-09-13T14:53:05.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[104007]: 11.7 scrub ok 2024-09-13T14:53:05.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T14:53:05.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[104007]: osdmap e280: 8 total, 8 up, 8 in 2024-09-13T14:53:05.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[104007]: 11.8 scrub starts 2024-09-13T14:53:05.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:05 smithi003 ceph-mon[104007]: 11.8 scrub ok 2024-09-13T14:53:05.283 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:05.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:53:05.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:05 smithi019 ceph-mon[91296]: pgmap v972: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 168 MiB/s wr, 2.69k op/s; 9129/281421 objects misplaced (3.244%); 32 KiB/s, 0 objects/s recovering 2024-09-13T14:53:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:05 smithi019 ceph-mon[91296]: 11.7 scrub ok 2024-09-13T14:53:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:05 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T14:53:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:05 smithi019 ceph-mon[91296]: osdmap e280: 8 total, 8 up, 8 in 2024-09-13T14:53:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:05 smithi019 ceph-mon[91296]: 11.8 scrub starts 2024-09-13T14:53:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:05 smithi019 ceph-mon[91296]: 11.8 scrub ok 2024-09-13T14:53:05.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:05.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:05.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:05.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:05.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:05.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:05.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.814 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:05.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:05.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 97217 97201 93.4439 0 - 0.0104608 2024-09-13T14:53:05.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:05.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:05.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:53:06.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:06.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:06.121 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:06 smithi003 ceph-mon[104007]: osdmap e281: 8 total, 8 up, 8 in 2024-09-13T14:53:06.121 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:06 smithi003 ceph-mon[104007]: 11.0 scrub starts 2024-09-13T14:53:06.121 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:06 smithi003 ceph-mon[104007]: 11.0 scrub ok 2024-09-13T14:53:06.121 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:06 smithi003 ceph-mon[104007]: osdmap e282: 8 total, 8 up, 8 in 2024-09-13T14:53:06.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:06.195 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:06.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:06.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:06.422 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:06 smithi019 ceph-mon[91296]: osdmap e281: 8 total, 8 up, 8 in 2024-09-13T14:53:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:06 smithi019 ceph-mon[91296]: 11.0 scrub starts 2024-09-13T14:53:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:06 smithi019 ceph-mon[91296]: 11.0 scrub ok 2024-09-13T14:53:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:06 smithi019 ceph-mon[91296]: osdmap e282: 8 total, 8 up, 8 in 2024-09-13T14:53:06.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:06 smithi003 ceph-mon[107278]: osdmap e281: 8 total, 8 up, 8 in 2024-09-13T14:53:06.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:06 smithi003 ceph-mon[107278]: 11.0 scrub starts 2024-09-13T14:53:06.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:06 smithi003 ceph-mon[107278]: 11.0 scrub ok 2024-09-13T14:53:06.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:06 smithi003 ceph-mon[107278]: osdmap e282: 8 total, 8 up, 8 in 2024-09-13T14:53:06.458 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:06.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:06.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.594 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:06.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:06.696 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:06.718 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:06.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:06.847 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 97704 97688 92.4893 15.2188 0.0139326 0.0108072 2024-09-13T14:53:06.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:06.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:06.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.002 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:07.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:07.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:07.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:07.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:07.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:07.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:07.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:07 smithi019 ceph-mon[91296]: pgmap v975: 119 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 117 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 9129/288924 objects misplaced (3.160%); 37 KiB/s, 0 objects/s recovering 2024-09-13T14:53:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:07 smithi019 ceph-mon[91296]: 11.17 scrub starts 2024-09-13T14:53:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:07 smithi019 ceph-mon[91296]: 11.5 scrub starts 2024-09-13T14:53:07.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:07 smithi003 ceph-mon[104007]: pgmap v975: 119 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 117 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 9129/288924 objects misplaced (3.160%); 37 KiB/s, 0 objects/s recovering 2024-09-13T14:53:07.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:07 smithi003 ceph-mon[104007]: 11.17 scrub starts 2024-09-13T14:53:07.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:07 smithi003 ceph-mon[104007]: 11.5 scrub starts 2024-09-13T14:53:07.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:07 smithi003 ceph-mon[107278]: pgmap v975: 119 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 117 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 9129/288924 objects misplaced (3.160%); 37 KiB/s, 0 objects/s recovering 2024-09-13T14:53:07.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:07 smithi003 ceph-mon[107278]: 11.17 scrub starts 2024-09-13T14:53:07.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:07 smithi003 ceph-mon[107278]: 11.5 scrub starts 2024-09-13T14:53:07.524 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.549 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:07.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:07.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:07.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:07.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:07.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:07.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 98717 98701 92.0538 63.3125 0.00584941 0.0108581 2024-09-13T14:53:07.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:07.967 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:08.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:08.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:08.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:08.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:08.338 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.366 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:08.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:08.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:08 smithi019 ceph-mon[91296]: 11.17 scrub ok 2024-09-13T14:53:08.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:08 smithi003 ceph-mon[104007]: 11.17 scrub ok 2024-09-13T14:53:08.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:08 smithi003 ceph-mon[107278]: 11.17 scrub ok 2024-09-13T14:53:08.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:08.495 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:08.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:08.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:08.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:08.761 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:08.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 99724 99708 91.6256 62.9375 0.0147305 0.0109085 2024-09-13T14:53:08.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:08.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:08.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:08.988 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:09.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:09.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:09.274 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:09.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:09.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:09.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:09.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:09.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:09 smithi019 ceph-mon[91296]: pgmap v977: 119 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 117 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 8630/291510 objects misplaced (2.960%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:53:09.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:09.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:09 smithi019 ceph-mon[91296]: 11.15 scrub starts 2024-09-13T14:53:09.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:09 smithi003 ceph-mon[104007]: pgmap v977: 119 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 117 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 8630/291510 objects misplaced (2.960%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:53:09.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:09.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:09 smithi003 ceph-mon[104007]: 11.15 scrub starts 2024-09-13T14:53:09.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:09 smithi003 ceph-mon[107278]: pgmap v977: 119 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 117 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 8630/291510 objects misplaced (2.960%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:53:09.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:09.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:09 smithi003 ceph-mon[107278]: 11.15 scrub starts 2024-09-13T14:53:09.519 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:09.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:09.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:09.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:09.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:09.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:09.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 100741 100725 91.2188 63.5625 0.00714763 0.0109577 2024-09-13T14:53:09.936 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:09.972 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:10.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:10.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:10.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:10.235 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:10.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:10.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:10.476 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.500 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:10.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:10.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:10.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:10.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:10.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:53:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:53:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:10.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:10.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:10.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 101657 101641 90.7335 57.25 0.0155112 0.0110161 2024-09-13T14:53:10.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:10.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:10.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:11.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:11.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:11.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:11.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:11.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:11.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:11 smithi019 ceph-mon[91296]: pgmap v978: 119 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 117 active+clean; 6.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 8630/297420 objects misplaced (2.902%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:53:11.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:11 smithi003 ceph-mon[104007]: pgmap v978: 119 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 117 active+clean; 6.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 8630/297420 objects misplaced (2.902%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:53:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:11 smithi003 ceph-mon[107278]: pgmap v978: 119 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 117 active+clean; 6.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 8630/297420 objects misplaced (2.902%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:53:11.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:11.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:11.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:11.643 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:11.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:11.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:11.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 102709 102693 90.3816 65.75 0.0132363 0.0110593 2024-09-13T14:53:11.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:12.027 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:12.054 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:12.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:12.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:12.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:12.324 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:12.426 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:12 smithi003 ceph-mon[104007]: 11.14 scrub starts 2024-09-13T14:53:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:12 smithi003 ceph-mon[107278]: 11.14 scrub starts 2024-09-13T14:53:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.468 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:12.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:12.570 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:12.605 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:12 smithi019 ceph-mon[91296]: 11.14 scrub starts 2024-09-13T14:53:12.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:12.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:12.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 103751 103735 90.0307 65.125 0.0171986 0.0111019 2024-09-13T14:53:12.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:12.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:12.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:13.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:13.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:13.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:13.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:13.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:13.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:13 smithi003 ceph-mon[104007]: pgmap v979: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3014/302820 objects degraded (0.995%); 11644/302820 objects misplaced (3.845%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:53:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:13 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 3014/302820 objects degraded (0.995%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:13 smithi003 ceph-mon[107278]: pgmap v979: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3014/302820 objects degraded (0.995%); 11644/302820 objects misplaced (3.845%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:53:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:13 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 3014/302820 objects degraded (0.995%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:13.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:13.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:13.601 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:13 smithi019 ceph-mon[91296]: pgmap v979: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3014/302820 objects degraded (0.995%); 11644/302820 objects misplaced (3.845%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:53:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:13 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 3014/302820 objects degraded (0.995%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:13.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.731 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:13.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:13.742 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:53:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:53:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36438.0:104704 11.b e282/266 11:db2e5e9f:::benchmark_data_smithi003_229782_object104703:head v 282'6440, mlcod=282'6437)", 2024-09-13T14:53:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:13.731506+0000", 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0083919149999999998, 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084201299999999996, 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.731506+0000", 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.731506+0000", 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.731507+0000", 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6000000000000002e-07 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.731530+0000", 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3774e-05 2024-09-13T14:53:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.731544+0000", 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3776e-05 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.731556+0000", 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1511999999999999e-05 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.731743+0000", 2024-09-13T14:53:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018707700000000001 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.731793+0000", 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0377999999999998e-05 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36438.0:104705 11.b e282/266 11:d8b725da:::benchmark_data_smithi003_229782_object104704:head v 282'6441, mlcod=282'6437)", 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:13.735116+0000", 2024-09-13T14:53:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0047814600000000004, 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048646879999999998, 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.735116+0000", 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.735116+0000", 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.735117+0000", 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1650000000000001e-06 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.735156+0000", 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8365000000000003e-05 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.735173+0000", 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7756000000000001e-05 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.735181+0000", 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0199999999999994e-06 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.735997+0000", 2024-09-13T14:53:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00081553400000000003 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.736017+0000", 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9508e-05 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36438.0:104712 11.7 e282/275 11:e50fb0db:::benchmark_data_smithi003_229782_object104711:head v 282'6053, mlcod=282'6051)", 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:13.738355+0000", 2024-09-13T14:53:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0015422089999999999, 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001675288, 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738355+0000", 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738355+0000", 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738357+0000", 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.178e-06 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738460+0000", 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103479 2024-09-13T14:53:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738481+0000", 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.073e-05 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738491+0000", 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8980000000000008e-06 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738597+0000", 2024-09-13T14:53:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105844 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738619+0000", 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2685999999999999e-05 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:104715 11.10 11:0dde012b:::benchmark_data_smithi003_229782_object104714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:13.738810+0000", 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0010871870000000001, 2024-09-13T14:53:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012767080000000001, 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 104715 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738810+0000", 2024-09-13T14:53:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738810+0000", 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738812+0000", 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8640000000000001e-06 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738835+0000", 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2255e-05 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738848+0000", 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3335e-05 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738870+0000", 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1858e-05 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.738929+0000", 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9175000000000002e-05 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.739088+0000", 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015941800000000001 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.739276+0000", 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018808400000000001 2024-09-13T14:53:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPushReply(11.10 282/278 [PushReplyOp(11:081d638a:::benchmark_data_smithi003_229782_object8498:head)])", 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:13.739792+0000", 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.000106078, 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035816599999999999, 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.739792+0000", 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.739792+0000", 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.739792+0000", 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9500000000000001e-07 2024-09-13T14:53:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.739799+0000", 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8990000000000004e-06 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.739817+0000", 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7348000000000001e-05 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:13.739830+0000", 2024-09-13T14:53:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2866e-05 2024-09-13T14:53:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-13T14:53:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 5 2024-09-13T14:53:13.762 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:53:13.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:13.846 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 104828 104812 89.7195 67.3125 0.0213101 0.0111398 2024-09-13T14:53:13.857 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:13.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:14.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:14.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:14.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:14.275 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:14.357 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:14 smithi003 ceph-mon[107278]: 11.5 scrub starts 2024-09-13T14:53:14.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:14 smithi003 ceph-mon[104007]: 11.5 scrub starts 2024-09-13T14:53:14.358 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:53:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:53:14] "GET /metrics HTTP/1.1" 200 36554 "" "Prometheus/2.51.0" 2024-09-13T14:53:14.378 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:14.405 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:14.507 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:14.537 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:14.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:14 smithi019 ceph-mon[91296]: 11.5 scrub starts 2024-09-13T14:53:14.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:14.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:14.847 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 105901 105885 89.4133 67.0625 0.00932114 0.0111793 2024-09-13T14:53:14.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:14.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:15.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:15.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:15.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:15.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:15.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:15.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:15.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:15 smithi003 ceph-mon[107278]: pgmap v980: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3014/303225 objects degraded (0.994%); 11354/303225 objects misplaced (3.744%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:53:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:15 smithi003 ceph-mon[107278]: 11.15 scrub starts 2024-09-13T14:53:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:15 smithi003 ceph-mon[104007]: pgmap v980: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3014/303225 objects degraded (0.994%); 11354/303225 objects misplaced (3.744%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:53:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:15 smithi003 ceph-mon[104007]: 11.15 scrub starts 2024-09-13T14:53:15.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:15.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:15.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:15.655 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:15.682 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:15 smithi019 ceph-mon[91296]: pgmap v980: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3014/303225 objects degraded (0.994%); 11354/303225 objects misplaced (3.744%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:53:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:15 smithi019 ceph-mon[91296]: 11.15 scrub starts 2024-09-13T14:53:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:15.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:53:15.783 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:15.814 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:15.847 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 106942 106926 89.0885 65.0625 0.0209205 0.011219 2024-09-13T14:53:15.916 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:15.946 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:16.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:16.072 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:16.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:16.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:16.237 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 0.947624 (f297) 2024-09-13T14:53:16.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:16.363 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:16 smithi003 ceph-mon[107278]: 11.14 scrub starts 2024-09-13T14:53:16.363 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:16 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1606975339' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.9476238917435175} : dispatch 2024-09-13T14:53:16.363 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:16 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.9476238917435175} : dispatch 2024-09-13T14:53:16.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:16 smithi003 ceph-mon[104007]: 11.14 scrub starts 2024-09-13T14:53:16.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:16 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1606975339' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.9476238917435175} : dispatch 2024-09-13T14:53:16.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:16 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.9476238917435175} : dispatch 2024-09-13T14:53:16.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:16.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:16.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:16.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:16.633 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:16 smithi019 ceph-mon[91296]: 11.14 scrub starts 2024-09-13T14:53:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:16 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1606975339' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.9476238917435175} : dispatch 2024-09-13T14:53:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:16 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.9476238917435175} : dispatch 2024-09-13T14:53:16.735 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:16.768 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:16.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:16.850 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 107996 107980 88.7795 65.875 0.0157934 0.0112582 2024-09-13T14:53:16.870 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:16.898 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:17.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:17.034 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:17.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:17.168 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:17.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:17.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:17.398 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:17.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:17.478 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:17.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:17.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:17.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:17 smithi003 ceph-mon[104007]: pgmap v981: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3014/313884 objects degraded (0.960%); 11354/313884 objects misplaced (3.617%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:53:17.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:17 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.9476238917435175}]': finished 2024-09-13T14:53:17.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:17 smithi003 ceph-mon[104007]: osdmap e283: 8 total, 8 up, 8 in 2024-09-13T14:53:17.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:17 smithi003 ceph-mon[104007]: osdmap e284: 8 total, 8 up, 8 in 2024-09-13T14:53:17.562 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:17 smithi003 ceph-mon[107278]: pgmap v981: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3014/313884 objects degraded (0.960%); 11354/313884 objects misplaced (3.617%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:53:17.562 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:17 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.9476238917435175}]': finished 2024-09-13T14:53:17.562 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:17 smithi003 ceph-mon[107278]: osdmap e283: 8 total, 8 up, 8 in 2024-09-13T14:53:17.562 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:17 smithi003 ceph-mon[107278]: osdmap e284: 8 total, 8 up, 8 in 2024-09-13T14:53:17.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:17.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:17 smithi019 ceph-mon[91296]: pgmap v981: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3014/313884 objects degraded (0.960%); 11354/313884 objects misplaced (3.617%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:53:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:17 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.9476238917435175}]': finished 2024-09-13T14:53:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:17 smithi019 ceph-mon[91296]: osdmap e283: 8 total, 8 up, 8 in 2024-09-13T14:53:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:17 smithi019 ceph-mon[91296]: osdmap e284: 8 total, 8 up, 8 in 2024-09-13T14:53:17.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:17.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:17.852 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 109046 109030 88.4776 65.625 0.00840892 0.0112967 2024-09-13T14:53:17.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:17.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:18.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:18.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:18.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:18.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:18.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:18.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:18.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:18.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:18.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:18.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:18 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/318816 objects degraded (0.945%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:18.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:18 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/318816 objects degraded (0.945%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:18.644 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:18.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.679 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:18.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:18.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:18 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/318816 objects degraded (0.945%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:18.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:18.810 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:18.851 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 110134 110118 88.215 68 0.0053984 0.0113296 2024-09-13T14:53:18.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:18.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:19.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:19.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:19.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:19.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:19.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:19.383 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:19.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:19.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:19.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:19 smithi019 ceph-mon[91296]: pgmap v984: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s; 3014/318816 objects degraded (0.945%); 11063/318816 objects misplaced (3.470%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:19.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:19 smithi003 ceph-mon[104007]: pgmap v984: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s; 3014/318816 objects degraded (0.945%); 11063/318816 objects misplaced (3.470%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:19.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:19 smithi003 ceph-mon[107278]: pgmap v984: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s; 3014/318816 objects degraded (0.945%); 11063/318816 objects misplaced (3.470%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:19.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:19.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:19.851 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 111213 111197 87.952 67.4375 0.0101748 0.0113643 2024-09-13T14:53:19.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:19.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:20.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:20.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:20.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:20.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:20.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:20.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:20.633 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:53:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:53:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:20.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:20.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:20.851 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:53:20.851202+0000 min lat: 0.00120774 max lat: 2.1123 avg lat: 0.0114085 2024-09-13T14:53:20.851 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:53:20.851 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 112190 112174 87.6159 61.0625 0.00955204 0.0114085 2024-09-13T14:53:20.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:21.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:21.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:21.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:21.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:21.300 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:21.332 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:21.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:21.468 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:21.570 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:21.597 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:21 smithi019 ceph-mon[91296]: pgmap v985: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 3014/328269 objects degraded (0.918%); 11063/328269 objects misplaced (3.370%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:21.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:21 smithi003 ceph-mon[107278]: pgmap v985: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 3014/328269 objects degraded (0.918%); 11063/328269 objects misplaced (3.370%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:21 smithi003 ceph-mon[104007]: pgmap v985: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 3014/328269 objects degraded (0.918%); 11063/328269 objects misplaced (3.370%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:21.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:21.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:21.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:21.851 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 113300 113284 87.3906 69.375 0.00293552 0.0114348 2024-09-13T14:53:21.864 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:21.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:22.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:22.112 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:22.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.139 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:22.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:22.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:22.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:22.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:22.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:22.443 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:53:22.443 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:53:22.444 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T14:53:22.444 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.183891 2024-09-13T14:53:22.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 0.18389129667963322 2024-09-13T14:53:22.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:22.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:22.692 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:22.721 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:22.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:22.851 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 114297 114281 87.0847 62.3125 0.0197308 0.0114782 2024-09-13T14:53:22.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:22.981 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:23.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:23.116 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:23.150 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:23.252 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:23.282 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:23.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:23.535 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:23.563 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:23.664 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:23.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:23 smithi019 ceph-mon[91296]: pgmap v986: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 3014/329658 objects degraded (0.914%); 11063/329658 objects misplaced (3.356%); 776 KiB/s, 12 objects/s recovering 2024-09-13T14:53:23.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:23 smithi003 ceph-mon[104007]: pgmap v986: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 3014/329658 objects degraded (0.914%); 11063/329658 objects misplaced (3.356%); 776 KiB/s, 12 objects/s recovering 2024-09-13T14:53:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:23 smithi003 ceph-mon[107278]: pgmap v986: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 3014/329658 objects degraded (0.914%); 11063/329658 objects misplaced (3.356%); 776 KiB/s, 12 objects/s recovering 2024-09-13T14:53:23.695 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:23.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.828 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:23.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:23.852 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 115440 115424 86.8962 71.4375 0.0186905 0.0115029 2024-09-13T14:53:23.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:23.958 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:24.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:24.089 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:24.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:24.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:24.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:24.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:24.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:24 smithi003 ceph-mon[107278]: 11.5 scrub starts 2024-09-13T14:53:24.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:24 smithi003 ceph-mon[104007]: 11.5 scrub starts 2024-09-13T14:53:24.411 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:53:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:53:24] "GET /metrics HTTP/1.1" 200 36554 "" "Prometheus/2.51.0" 2024-09-13T14:53:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:24.512 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:24.543 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:24.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:24.676 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:24.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:24.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:24 smithi019 ceph-mon[91296]: 11.5 scrub starts 2024-09-13T14:53:24.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:24.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:24.851 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 116481 116465 86.6362 65.0625 0.0225309 0.0115373 2024-09-13T14:53:24.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:24.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:25.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:25.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:25.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:25.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:25.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:25.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:25.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:25.506 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:25.537 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:53:25.537 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:25.576 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:53:25.577 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:25.621 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:25 smithi003 ceph-mon[107278]: pgmap v987: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 MiB/s wr, 872 op/s; 3014/334824 objects degraded (0.900%); 10724/334824 objects misplaced (3.203%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:53:25.621 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/334824 objects degraded (0.900%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:25.622 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:53:25.622 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:25.622 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:25 smithi003 ceph-mon[104007]: pgmap v987: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 MiB/s wr, 872 op/s; 3014/334824 objects degraded (0.900%); 10724/334824 objects misplaced (3.203%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:53:25.622 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/334824 objects degraded (0.900%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:25.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:25.659 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:53:25.659 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:25.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:25 smithi019 ceph-mon[91296]: pgmap v987: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 MiB/s wr, 872 op/s; 3014/334824 objects degraded (0.900%); 10724/334824 objects misplaced (3.203%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:53:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/334824 objects degraded (0.900%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:25.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:25.704 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:53:25.705 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:25.751 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:53:25.751 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:25.784 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:53:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:25.807 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:25.818 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:53:25.818 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:25.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:25.851 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:25.851 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:25.851 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 117464 117448 86.3397 61.4375 0.0222282 0.011577 2024-09-13T14:53:25.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:26.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:26.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:26.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:26.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:26.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:26.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:26.508 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:26.543 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:26.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3626804033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:26 smithi019 ceph-mon[91296]: pgmap v988: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 3014/345840 objects degraded (0.872%); 10724/345840 objects misplaced (3.101%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:53:26.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3626804033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:26.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:26 smithi003 ceph-mon[104007]: pgmap v988: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 3014/345840 objects degraded (0.872%); 10724/345840 objects misplaced (3.101%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:53:26.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3626804033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:26.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:26 smithi003 ceph-mon[107278]: pgmap v988: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 3014/345840 objects degraded (0.872%); 10724/345840 objects misplaced (3.101%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:53:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:26.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:26.788 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:26.823 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:26.852 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 15 118511 118496 86.0973 65.5 0.00489942 0.0116087 2024-09-13T14:53:26.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:26.961 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:26.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:53:27.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:27.088 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:27.190 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.226 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:27.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:27.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:27.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:27.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:27.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:27.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:27.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.649 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:27.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:27.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:27.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:27.852 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 119425 119409 85.7636 57.0625 0.0211851 0.0116546 2024-09-13T14:53:27.879 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:27.909 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:28.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:28.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:28.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:28.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:28.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:28.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:28.439 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:28.465 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:53:28.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:28.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:28 smithi019 ceph-mon[91296]: pgmap v989: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 3014/350697 objects degraded (0.859%); 10414/350697 objects misplaced (2.970%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:53:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/870851054' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:28 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:28 smithi019 ceph-mon[91296]: 11.15 scrub starts 2024-09-13T14:53:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[104007]: pgmap v989: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 3014/350697 objects degraded (0.859%); 10414/350697 objects misplaced (2.970%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:53:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/870851054' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[104007]: 11.15 scrub starts 2024-09-13T14:53:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[107278]: pgmap v989: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 3014/350697 objects degraded (0.859%); 10414/350697 objects misplaced (2.970%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:53:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/870851054' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:28.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:28 smithi003 ceph-mon[107278]: 11.15 scrub starts 2024-09-13T14:53:28.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:28.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:28.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:28.852 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 120416 120400 85.4925 61.9375 0.0102094 0.0116909 2024-09-13T14:53:28.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:29.006 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:29.033 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:29.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:29.316 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:29.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.563 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:29.593 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:29.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:29.638 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:29.639 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:29.639 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[107278]: osdmap e285: 8 total, 8 up, 8 in 2024-09-13T14:53:29.639 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:53:29.639 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:53:29.639 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:29.639 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[107278]: 11.14 deep-scrub starts 2024-09-13T14:53:29.639 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:29.639 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:29.639 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[104007]: osdmap e285: 8 total, 8 up, 8 in 2024-09-13T14:53:29.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:53:29.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:53:29.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:29.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:29 smithi003 ceph-mon[104007]: 11.14 deep-scrub starts 2024-09-13T14:53:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:29 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:29 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:29 smithi019 ceph-mon[91296]: osdmap e285: 8 total, 8 up, 8 in 2024-09-13T14:53:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:53:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:53:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:29 smithi019 ceph-mon[91296]: 11.14 deep-scrub starts 2024-09-13T14:53:29.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.701 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:29.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:29.741 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:29.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:29.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 121357 121341 85.1911 58.8125 0.0123673 0.0117331 2024-09-13T14:53:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.872 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:29.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:29.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:30.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:30.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:30.172 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:30.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:30.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:30.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:30.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:30.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:30.580 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:30 smithi003 ceph-mon[104007]: pgmap v991: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.3 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 3014/359373 objects degraded (0.839%); 10414/359373 objects misplaced (2.898%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:53:30.580 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:30 smithi003 ceph-mon[104007]: 11.5 deep-scrub starts 2024-09-13T14:53:30.580 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:30 smithi003 ceph-mon[107278]: pgmap v991: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.3 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 3014/359373 objects degraded (0.839%); 10414/359373 objects misplaced (2.898%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:53:30.580 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:30 smithi003 ceph-mon[107278]: 11.5 deep-scrub starts 2024-09-13T14:53:30.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:30.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:30 smithi019 ceph-mon[91296]: pgmap v991: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.3 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 3014/359373 objects degraded (0.839%); 10414/359373 objects misplaced (2.898%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:53:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:30 smithi019 ceph-mon[91296]: 11.5 deep-scrub starts 2024-09-13T14:53:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:53:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:53:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:30.709 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:30.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:30.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:30.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0153 2024-09-13T14:53:30.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 122377 2024-09-13T14:53:30.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T14:53:30.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T14:53:30.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 84.9696 2024-09-13T14:53:30.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 38.2526 2024-09-13T14:53:30.854 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 227.938 2024-09-13T14:53:30.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:53:30.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1359 2024-09-13T14:53:30.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 612.062 2024-09-13T14:53:30.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3647 2024-09-13T14:53:30.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-09-13T14:53:30.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0117637 2024-09-13T14:53:30.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0345925 2024-09-13T14:53:30.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.1123 2024-09-13T14:53:30.855 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00120774 2024-09-13T14:53:30.862 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:53:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:30.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:31.009 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:31.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:31.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:31.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:53:31.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:31.301 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:31.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:31.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:31.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:31.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:31.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/359373 objects degraded (0.839%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:31.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/359373 objects degraded (0.839%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:31.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/359373 objects degraded (0.839%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:31.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:53:31.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:31.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:31.881 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:31.911 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:32.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:32.042 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:32.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:32.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:32.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:32.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:32.425 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:32.453 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:32.555 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.582 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:32.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:32 smithi019 ceph-mon[91296]: pgmap v992: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.3 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 3014/360801 objects degraded (0.835%); 10414/360801 objects misplaced (2.886%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:53:32.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:32 smithi003 ceph-mon[104007]: pgmap v992: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.3 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 3014/360801 objects degraded (0.835%); 10414/360801 objects misplaced (2.886%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:53:32.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:32 smithi003 ceph-mon[107278]: pgmap v992: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.3 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 3014/360801 objects degraded (0.835%); 10414/360801 objects misplaced (2.886%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:53:32.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:32.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:32.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:32.846 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:32.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:32.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:33.092 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:33.121 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:33.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:33.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:33.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:33.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:33.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:33.491 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:33.518 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:33.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:33.649 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:33.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:33.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:33.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:34.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:34.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:34.113 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:34.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.137 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:34.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:34.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:53:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:53:34] "GET /metrics HTTP/1.1" 200 36561 "" "Prometheus/2.51.0" 2024-09-13T14:53:34.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:34.260 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:34.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:34.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:34.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:34.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:34.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:34.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:34.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:34.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:34.933 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:35.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:35.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:35.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:35.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:35.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:35.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:35.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:34 smithi019 ceph-mon[91296]: pgmap v993: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 62 MiB/s wr, 1.02k op/s; 3014/363387 objects degraded (0.829%); 10067/363387 objects misplaced (2.770%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:53:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:34 smithi003 ceph-mon[107278]: pgmap v993: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 62 MiB/s wr, 1.02k op/s; 3014/363387 objects degraded (0.829%); 10067/363387 objects misplaced (2.770%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:53:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:34 smithi003 ceph-mon[104007]: pgmap v993: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 62 MiB/s wr, 1.02k op/s; 3014/363387 objects degraded (0.829%); 10067/363387 objects misplaced (2.770%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:53:35.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.489 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:35.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:35.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:35.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:35.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:35.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:35.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:35.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:35.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:36.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:36.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:36.148 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:36.174 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:36.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:36.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:36.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:35 smithi003 ceph-mon[107278]: 11.15 scrub starts 2024-09-13T14:53:36.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/363387 objects degraded (0.829%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:36.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:35 smithi003 ceph-mon[104007]: 11.15 scrub starts 2024-09-13T14:53:36.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/363387 objects degraded (0.829%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:36.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:35 smithi019 ceph-mon[91296]: 11.15 scrub starts 2024-09-13T14:53:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/363387 objects degraded (0.829%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:36.439 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:36.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:36.569 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:36.671 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:36.697 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:36.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:36.828 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:36.930 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:36.960 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:37.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:37.088 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:37.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:36 smithi003 ceph-mon[107278]: pgmap v994: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.3 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 102 B/s rd, 44 MiB/s wr, 983 op/s; 3014/358491 objects degraded (0.841%); 10067/358491 objects misplaced (2.808%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:53:37.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:36 smithi003 ceph-mon[107278]: 11.14 scrub starts 2024-09-13T14:53:37.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:36 smithi003 ceph-mon[104007]: pgmap v994: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.3 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 102 B/s rd, 44 MiB/s wr, 983 op/s; 3014/358491 objects degraded (0.841%); 10067/358491 objects misplaced (2.808%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:53:37.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:36 smithi003 ceph-mon[104007]: 11.14 scrub starts 2024-09-13T14:53:37.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:37.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:37.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:37.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:36 smithi019 ceph-mon[91296]: pgmap v994: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.3 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 102 B/s rd, 44 MiB/s wr, 983 op/s; 3014/358491 objects degraded (0.841%); 10067/358491 objects misplaced (2.808%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:53:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:36 smithi019 ceph-mon[91296]: 11.14 scrub starts 2024-09-13T14:53:37.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:37.507 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:37.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:37.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:37.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:37.752 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:37.781 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:37.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:37.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:38.032 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:38.058 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:38.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:38.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:38.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:38.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:38.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:38.482 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:38.589 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:38.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:38.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:38.747 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:38.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:38.849 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:38.874 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:38.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:38.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:39.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:39.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:39.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:39.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:39.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:38 smithi019 ceph-mon[91296]: pgmap v995: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.2 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 975 op/s; 3014/353877 objects degraded (0.852%); 9746/353877 objects misplaced (2.754%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:53:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:38 smithi019 ceph-mon[91296]: 11.5 scrub starts 2024-09-13T14:53:39.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:38 smithi003 ceph-mon[104007]: pgmap v995: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.2 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 975 op/s; 3014/353877 objects degraded (0.852%); 9746/353877 objects misplaced (2.754%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:53:39.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:38 smithi003 ceph-mon[104007]: 11.5 scrub starts 2024-09-13T14:53:39.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:38 smithi003 ceph-mon[107278]: pgmap v995: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.2 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 975 op/s; 3014/353877 objects degraded (0.852%); 9746/353877 objects misplaced (2.754%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:53:39.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:38 smithi003 ceph-mon[107278]: 11.5 scrub starts 2024-09-13T14:53:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:39.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:39.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:39.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:39.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:39.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:39.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:39.993 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:40.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:40.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:40.239 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.271 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:40.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:40.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:40.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:40.500 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:40.500 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:40.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:40.557 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:40.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:40.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:53:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:53:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:40.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:40.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:40.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:40.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:40.971 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.1 primary-affinity to 0.183891 (8120512) 2024-09-13T14:53:40.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:41.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:41.108 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:40 smithi003 ceph-mon[104007]: pgmap v996: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 89 B/s rd, 29 MiB/s wr, 1.11k op/s; 3014/344658 objects degraded (0.874%); 9746/344658 objects misplaced (2.828%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:53:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/344658 objects degraded (0.874%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:40 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1353170498' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.18389129667963322} : dispatch 2024-09-13T14:53:41.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:40 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.18389129667963322} : dispatch 2024-09-13T14:53:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:40 smithi003 ceph-mon[107278]: pgmap v996: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 89 B/s rd, 29 MiB/s wr, 1.11k op/s; 3014/344658 objects degraded (0.874%); 9746/344658 objects misplaced (2.828%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:53:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/344658 objects degraded (0.874%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:40 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1353170498' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.18389129667963322} : dispatch 2024-09-13T14:53:41.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:40 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.18389129667963322} : dispatch 2024-09-13T14:53:41.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:41.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:41.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:41.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:41.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:40 smithi019 ceph-mon[91296]: pgmap v996: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 89 B/s rd, 29 MiB/s wr, 1.11k op/s; 3014/344658 objects degraded (0.874%); 9746/344658 objects misplaced (2.828%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:53:41.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/344658 objects degraded (0.874%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:41.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:40 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1353170498' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.18389129667963322} : dispatch 2024-09-13T14:53:41.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:40 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.18389129667963322} : dispatch 2024-09-13T14:53:41.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:41.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:41.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:53:41.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:41.661 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:41.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:41.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:41.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:41.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:42.083 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:42.114 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:42.216 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:42.247 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:42.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:42.348 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:42.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:41 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.18389129667963322}]': finished 2024-09-13T14:53:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:41 smithi019 ceph-mon[91296]: osdmap e286: 8 total, 8 up, 8 in 2024-09-13T14:53:42.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:41 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.18389129667963322}]': finished 2024-09-13T14:53:42.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:41 smithi003 ceph-mon[107278]: osdmap e286: 8 total, 8 up, 8 in 2024-09-13T14:53:42.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:41 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.18389129667963322}]': finished 2024-09-13T14:53:42.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:41 smithi003 ceph-mon[104007]: osdmap e286: 8 total, 8 up, 8 in 2024-09-13T14:53:42.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:42.515 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:42.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:42.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:42.555 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:42.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:42.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:42.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:42.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:43.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:43.100 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:43.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:42 smithi003 ceph-mon[104007]: pgmap v998: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 13 MiB/s wr, 992 op/s; 3014/343245 objects degraded (0.878%); 9746/343245 objects misplaced (2.839%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:53:43.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:42 smithi003 ceph-mon[104007]: osdmap e287: 8 total, 8 up, 8 in 2024-09-13T14:53:43.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:43.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:42 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:43.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:42 smithi003 ceph-mon[107278]: pgmap v998: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 13 MiB/s wr, 992 op/s; 3014/343245 objects degraded (0.878%); 9746/343245 objects misplaced (2.839%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:53:43.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:42 smithi003 ceph-mon[107278]: osdmap e287: 8 total, 8 up, 8 in 2024-09-13T14:53:43.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:43.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:42 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:43.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:43.232 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:43.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:43.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:42 smithi019 ceph-mon[91296]: pgmap v998: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 13 MiB/s wr, 992 op/s; 3014/343245 objects degraded (0.878%); 9746/343245 objects misplaced (2.839%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:53:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:42 smithi019 ceph-mon[91296]: osdmap e287: 8 total, 8 up, 8 in 2024-09-13T14:53:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:42 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:43.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:43.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:43.606 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:43.641 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:43.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:43.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:43.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:44.033 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:44.067 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:44.074 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:44.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:44.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:44.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:44.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:44 smithi019 ceph-mon[91296]: 11.15 scrub starts 2024-09-13T14:53:44.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:44 smithi003 ceph-mon[104007]: 11.15 scrub starts 2024-09-13T14:53:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:44 smithi003 ceph-mon[107278]: 11.15 scrub starts 2024-09-13T14:53:44.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:53:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:53:44] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-09-13T14:53:44.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:44.502 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:44.604 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:44.632 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:44.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:44.762 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:44.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:44.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:45.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:45.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:45.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:45.182 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:45.284 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:45.317 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:45.418 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:45 smithi019 ceph-mon[91296]: pgmap v1000: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 834 op/s; 3014/338445 objects degraded (0.891%); 9404/338445 objects misplaced (2.779%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:53:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:45 smithi003 ceph-mon[104007]: pgmap v1000: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 834 op/s; 3014/338445 objects degraded (0.891%); 9404/338445 objects misplaced (2.779%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:53:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:45 smithi003 ceph-mon[107278]: pgmap v1000: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 834 op/s; 3014/338445 objects degraded (0.891%); 9404/338445 objects misplaced (2.779%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:53:45.450 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:45.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:45.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:45.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:45.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:45.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:45.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:45.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:46.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:46.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:46.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/338445 objects degraded (0.891%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:46.168 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/338445 objects degraded (0.891%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:46.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:46.287 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.324 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:46.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:46.424 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:46.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/338445 objects degraded (0.891%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:46.455 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:46.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:46.589 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:46.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:46.719 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:46.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:46.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:46.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:47.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:47.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:47.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:47 smithi003 ceph-mon[107278]: pgmap v1001: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 3014/327747 objects degraded (0.920%); 9404/327747 objects misplaced (2.869%); 912 KiB/s, 14 objects/s recovering 2024-09-13T14:53:47.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:47 smithi003 ceph-mon[104007]: pgmap v1001: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 3014/327747 objects degraded (0.920%); 9404/327747 objects misplaced (2.869%); 912 KiB/s, 14 objects/s recovering 2024-09-13T14:53:47.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:47.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:47.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:47.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:47.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:47.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:47 smithi019 ceph-mon[91296]: pgmap v1001: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 3014/327747 objects degraded (0.920%); 9404/327747 objects misplaced (2.869%); 912 KiB/s, 14 objects/s recovering 2024-09-13T14:53:47.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:47.558 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:47.568 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:53:47.568 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:53:47.569 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T14:53:47.569 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-09-13T14:53:47.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-13T14:53:47.659 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:47.688 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:47.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:47.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:47.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:47.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:48.068 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:53:48.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:48.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:48.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:48.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:48.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:48.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:48.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3416028522' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:48 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:48.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3416028522' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:48 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3416028522' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:48 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:48.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:48.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:48.674 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:48.700 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:48.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:48.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:48.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:48.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:49.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:49.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[104007]: pgmap v1002: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 3014/322707 objects degraded (0.934%); 9044/322707 objects misplaced (2.803%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:53:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[104007]: 11.14 scrub starts 2024-09-13T14:53:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[104007]: osdmap e288: 8 total, 8 up, 8 in 2024-09-13T14:53:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2687327052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[107278]: pgmap v1002: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 3014/322707 objects degraded (0.934%); 9044/322707 objects misplaced (2.803%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:53:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[107278]: 11.14 scrub starts 2024-09-13T14:53:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[107278]: osdmap e288: 8 total, 8 up, 8 in 2024-09-13T14:53:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2687327052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:49.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:49.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:49.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:49.397 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:49.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:49.423 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:53:49.423 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:53:49.423 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:53:49.423 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:53:49.423 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:53:49.423 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:49 smithi019 ceph-mon[91296]: pgmap v1002: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 3014/322707 objects degraded (0.934%); 9044/322707 objects misplaced (2.803%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:53:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:49 smithi019 ceph-mon[91296]: 11.14 scrub starts 2024-09-13T14:53:49.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:49 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:49.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:49 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:49.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:49 smithi019 ceph-mon[91296]: osdmap e288: 8 total, 8 up, 8 in 2024-09-13T14:53:49.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2687327052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:49.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.524 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:49.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:49.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:49.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:49.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:49.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:49.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:49.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:50.052 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:50.080 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:50.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:50.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:50.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:50.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:50.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:50.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:50.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:50.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:50.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:53:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:53:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:50.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:50.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:50.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:50.950 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:51.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:51.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:51.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:51.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:51.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:51 smithi003 ceph-mon[104007]: pgmap v1004: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s; 3014/313743 objects degraded (0.961%); 9044/313743 objects misplaced (2.883%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:53:51.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/313743 objects degraded (0.961%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:51.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:51 smithi003 ceph-mon[107278]: pgmap v1004: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s; 3014/313743 objects degraded (0.961%); 9044/313743 objects misplaced (2.883%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:53:51.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/313743 objects degraded (0.961%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:51.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:51 smithi019 ceph-mon[91296]: pgmap v1004: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s; 3014/313743 objects degraded (0.961%); 9044/313743 objects misplaced (2.883%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:53:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/313743 objects degraded (0.961%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:51.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:51.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:51.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:51.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:51.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:51.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:51.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:51.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:51.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:52.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:52.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:52.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:52.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:52.306 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:52.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:52.354 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:144129 11.d 11:bf569b9b:::benchmark_data_smithi003_229782_object21749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.350031+0000", 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00085866999999999996, 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00088522500000000001, 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144129 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350031+0000", 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350031+0000", 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350033+0000", 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.328e-06 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350036+0000", 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4130000000000002e-06 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350046+0000", 2024-09-13T14:53:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5999999999999996e-06 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350057+0000", 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1623999999999999e-05 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350229+0000", 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000171173 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350347+0000", 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118307 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350460+0000", 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113176 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:144132 11.2 11.cafa42 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.350359+0000", 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00053108300000000001, 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00062041199999999998, 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144132 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350359+0000", 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350359+0000", 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350360+0000", 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4600000000000003e-07 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350364+0000", 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2719999999999999e-06 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350374+0000", 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8360000000000004e-06 2024-09-13T14:53:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.350381+0000", 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1420000000000004e-06 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:144106 11.c 11:3c92d9e9:::benchmark_data_smithi003_229782_object21726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.324428+0000", 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.026462396999999999, 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026600960999999999, 2024-09-13T14:53:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144106 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.324428+0000", 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.324428+0000", 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.324429+0000", 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6249999999999999e-06 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.324436+0000", 2024-09-13T14:53:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1829999999999999e-06 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.324462+0000", 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6803000000000001e-05 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.324471+0000", 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7479999999999999e-06 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.324570+0000", 2024-09-13T14:53:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.857e-05 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.324676+0000", 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105814 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.324780+0000", 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104722 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.334431+0000", 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096503979999999993 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.335318+0000", 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00088729800000000001 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:144115 11.c 11:36cb543b:::benchmark_data_smithi003_229782_object21735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.337636+0000", 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.013254528, 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013469325000000001, 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144115 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337636+0000", 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337636+0000", 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337637+0000", 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.761e-06 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337641+0000", 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9949999999999999e-06 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337651+0000", 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3789999999999995e-06 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337657+0000", 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9510000000000003e-06 2024-09-13T14:53:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.338589+0000", 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00093192500000000001 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.338956+0000", 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036744499999999999 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.339385+0000", 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042865200000000002 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:144117 11.c 11:342e5f98:::benchmark_data_smithi003_229782_object21737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.337669+0000", 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.013221278, 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013506884, 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144117 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337669+0000", 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337669+0000", 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337669+0000", 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.08e-07 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337671+0000", 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0190000000000001e-06 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337677+0000", 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7320000000000003e-06 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337682+0000", 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4769999999999997e-06 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.348948+0000", 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011266272000000001 2024-09-13T14:53:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349196+0000", 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000247841 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349313+0000", 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011743600000000001 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:144118 11.2 11:47f0452a:::benchmark_data_smithi003_229782_object21738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.337687+0000", 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.013202716999999999, 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013556688000000001, 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144118 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337687+0000", 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337687+0000", 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337688+0000", 2024-09-13T14:53:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0600000000000001e-07 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337689+0000", 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.72e-06 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337697+0000", 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3629999999999996e-06 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337701+0000", 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2710000000000003e-06 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337795+0000", 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3721999999999995e-05 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.337942+0000", 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014744500000000001 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.338076+0000", 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133573 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349565+0000", 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011489597000000001 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349577+0000", 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1635e-05 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:144121 11.c 11.320f988c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:53:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.338022+0000", 2024-09-13T14:53:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.012868053000000001, 2024-09-13T14:53:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013295329999999999, 2024-09-13T14:53:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144121 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.338022+0000", 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.338022+0000", 2024-09-13T14:53:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.338023+0000", 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8000000000000004e-07 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.338027+0000", 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6399999999999999e-06 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.338039+0000", 2024-09-13T14:53:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2632e-05 2024-09-13T14:53:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.338047+0000", 2024-09-13T14:53:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.225e-06 2024-09-13T14:53:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36438.0:144106 11.c e288/266)", 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.341251+0000", 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0096387450000000006, 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010116612000000001, 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341251+0000", 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341251+0000", 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341252+0000", 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3200000000000004e-07 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341256+0000", 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.444e-06 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341271+0000", 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5036e-05 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341279+0000", 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2530000000000004e-06 2024-09-13T14:53:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:144125 11.c 11.763ef00c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.341340+0000", 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0095502030000000002, 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010069517, 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144125 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341340+0000", 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341340+0000", 2024-09-13T14:53:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341341+0000", 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.31e-06 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341345+0000", 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4230000000000001e-06 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341358+0000", 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3148e-05 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.341365+0000", 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1670000000000002e-06 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:144127 11.12 11:4d7829d2:::benchmark_data_smithi003_229782_object21747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.342949+0000", 2024-09-13T14:53:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0079407669999999996, 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0085131270000000005, 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144127 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.342949+0000", 2024-09-13T14:53:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.342949+0000", 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.342951+0000", 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.319e-06 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.342955+0000", 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6179999999999997e-06 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.342968+0000", 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2781e-05 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.342976+0000", 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9209999999999992e-06 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.343077+0000", 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100906 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.343228+0000", 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015093100000000001 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.343417+0000", 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000189273 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349783+0000", 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063654970000000003 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36438.0:144115 11.c e288/266)", 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.343267+0000", 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0076236530000000002, 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082613140000000005, 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.343267+0000", 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.343267+0000", 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.343267+0000", 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2799999999999995e-07 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.343272+0000", 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.639e-06 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.343282+0000", 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0453e-05 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.343290+0000", 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1659999999999997e-06 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36438.0:144118 11.2 e288/275)", 2024-09-13T14:53:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.349894+0000", 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00099635700000000002, 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016729150000000001, 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349894+0000", 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349894+0000", 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349895+0000", 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6599999999999995e-07 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349898+0000", 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1990000000000002e-06 2024-09-13T14:53:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349914+0000", 2024-09-13T14:53:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6285000000000001e-05 2024-09-13T14:53:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349921+0000", 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0870000000000001e-06 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36438.0:144115 11.c e288/266)", 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:52.349949+0000", 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00094096000000000002, 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016558020000000001, 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:53:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349949+0000", 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349949+0000", 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349950+0000", 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7899999999999999e-07 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349952+0000", 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.779e-06 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349960+0000", 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7500000000000003e-06 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:52.349965+0000", 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1739999999999999e-06 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-13T14:53:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 13 2024-09-13T14:53:52.410 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:53:52.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.410 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:52.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:52.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:52.652 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:52.683 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:52.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:52.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:52.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:52.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:52.946 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:53.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:53.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:53.191 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:53.223 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:53.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:53:53.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:53 smithi003 ceph-mon[104007]: pgmap v1005: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 897 op/s; 3014/312345 objects degraded (0.965%); 9044/312345 objects misplaced (2.896%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:53:53.325 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:53 smithi003 ceph-mon[107278]: pgmap v1005: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 897 op/s; 3014/312345 objects degraded (0.965%); 9044/312345 objects misplaced (2.896%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:53:53.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:53.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:53.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:53.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:53 smithi019 ceph-mon[91296]: pgmap v1005: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 897 op/s; 3014/312345 objects degraded (0.965%); 9044/312345 objects misplaced (2.896%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:53:53.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:53.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:53.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:53.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:53.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:53.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:53:53.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:53.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:53.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:53.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:54.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:54.097 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:53:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:53:54] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-09-13T14:53:54.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:54.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:54.259 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:54.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:54.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:54.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:54.539 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:54.641 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:54.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:54.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:54.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:54.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:54.948 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:55.050 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:55.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:55.178 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:55.206 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:55.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:55.308 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:55.340 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:55 smithi019 ceph-mon[91296]: pgmap v1006: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 3014/307779 objects degraded (0.979%); 8663/307779 objects misplaced (2.815%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:55.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:55 smithi003 ceph-mon[107278]: pgmap v1006: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 3014/307779 objects degraded (0.979%); 8663/307779 objects misplaced (2.815%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:55.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:55 smithi003 ceph-mon[104007]: pgmap v1006: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 3014/307779 objects degraded (0.979%); 8663/307779 objects misplaced (2.815%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:55.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:55.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:55.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:55.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:55.610 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:55.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:55.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:55.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:55.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:55.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:55.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:55.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:56.011 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:56.113 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:56.150 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:56.252 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:56.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:56.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:56.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:56.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/307779 objects degraded (0.979%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:56.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/307779 objects degraded (0.979%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/307779 objects degraded (0.979%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:53:56.524 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:56.553 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:56.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:56.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:56.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:56.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:56.956 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:56.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:57.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.116 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:57.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:57.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:57 smithi003 ceph-mon[107278]: pgmap v1007: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3014/297372 objects degraded (1.014%); 8663/297372 objects misplaced (2.913%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:57 smithi003 ceph-mon[104007]: pgmap v1007: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3014/297372 objects degraded (1.014%); 8663/297372 objects misplaced (2.913%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:57.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:57.246 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:57.348 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:57.378 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:57 smithi019 ceph-mon[91296]: pgmap v1007: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3014/297372 objects degraded (1.014%); 8663/297372 objects misplaced (2.913%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:53:57.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.513 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:57.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:57.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:53:57.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:57.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:57.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:57.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:57.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:57.974 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:58.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:58.158 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:58.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:58.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:58.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:58.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:58.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:58.532 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.559 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:58.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:58.660 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.691 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:58.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:58.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:58.819 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:58.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:58.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:58.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:53:58 smithi003 ceph-mon[107278]: pgmap v1008: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3014/292509 objects degraded (1.030%); 8258/292509 objects misplaced (2.823%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:53:58.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:58.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:58.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:53:58 smithi003 ceph-mon[104007]: pgmap v1008: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3014/292509 objects degraded (1.030%); 8258/292509 objects misplaced (2.823%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:53:58.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:58.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:59.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:59.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:53:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:53:58 smithi019 ceph-mon[91296]: pgmap v1008: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3014/292509 objects degraded (1.030%); 8258/292509 objects misplaced (2.823%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:53:59.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:59.234 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:59.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.370 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:59.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:59.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:59.505 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:59.606 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:59.632 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:59.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:59.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:53:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:59.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:59.909 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:53:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:00.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:00.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:00.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:00.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:00.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:00.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:00.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:00.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:00.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:54:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:54:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:00.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:00.738 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:00.822 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:00.822 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:00.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:00.867 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:00.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:01.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:01.131 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:01.161 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:01.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:01.304 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:00 smithi003 ceph-mon[107278]: pgmap v1009: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 3014/283458 objects degraded (1.063%); 8258/283458 objects misplaced (2.913%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:54:01.304 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/283458 objects degraded (1.063%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:54:01.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:00 smithi003 ceph-mon[104007]: pgmap v1009: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 3014/283458 objects degraded (1.063%); 8258/283458 objects misplaced (2.913%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:54:01.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/283458 objects degraded (1.063%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:54:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:01.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:01.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:01.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:00 smithi019 ceph-mon[91296]: pgmap v1009: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 3014/283458 objects degraded (1.063%); 8258/283458 objects misplaced (2.913%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:54:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/283458 objects degraded (1.063%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:54:01.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:01.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:01.590 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:01.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:01.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.734 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:01.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:01.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:01.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T14:54:01.977 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:02.006 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:02.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:02.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:02.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:02.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:02.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:01 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2714710810' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:54:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:01 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:54:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:01 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2714710810' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:54:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:01 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:54:02.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:01 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2714710810' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:54:02.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:01 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:54:02.444 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:02.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:02.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:02.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:02.688 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:02.721 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:02.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:02.823 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:02.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:02.953 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:03.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:03.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:02 smithi003 ceph-mon[107278]: pgmap v1010: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 3014/281988 objects degraded (1.069%); 8258/281988 objects misplaced (2.928%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:54:03.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:02 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:54:03.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:02 smithi003 ceph-mon[107278]: osdmap e289: 8 total, 8 up, 8 in 2024-09-13T14:54:03.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:02 smithi003 ceph-mon[104007]: pgmap v1010: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 3014/281988 objects degraded (1.069%); 8258/281988 objects misplaced (2.928%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:54:03.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:02 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:54:03.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:02 smithi003 ceph-mon[104007]: osdmap e289: 8 total, 8 up, 8 in 2024-09-13T14:54:03.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:03.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:03.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:03.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:03.397 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:03.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:02 smithi019 ceph-mon[91296]: pgmap v1010: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 3014/281988 objects degraded (1.069%); 8258/281988 objects misplaced (2.928%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:54:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:02 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:54:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:02 smithi019 ceph-mon[91296]: osdmap e289: 8 total, 8 up, 8 in 2024-09-13T14:54:03.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:03.526 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:03.587 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:54:03.588 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:03.626 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:54:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:03.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:03.664 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:54:03.665 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:03.704 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:54:03.704 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:03.746 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:54:03.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:03.765 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:03.787 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:54:03.787 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:03.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:03.823 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:03.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:03.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:03.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:04.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.074 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:04.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:04.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:04.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:04.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:04.353 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:04.371 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:04 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1979866601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:04.372 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:54:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:54:04] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-09-13T14:54:04.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:04 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1979866601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:04 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1979866601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:04.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:04.484 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:04.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:04.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:04.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:04.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:04.767 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:04.870 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.899 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:04.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:05.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:05.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:05.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:05.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:05.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:05.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:05 smithi019 ceph-mon[91296]: pgmap v1012: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3014/277539 objects degraded (1.086%); 7822/277539 objects misplaced (2.818%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:54:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:05 smithi003 ceph-mon[107278]: pgmap v1012: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3014/277539 objects degraded (1.086%); 7822/277539 objects misplaced (2.818%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:54:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:05 smithi003 ceph-mon[104007]: pgmap v1012: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3014/277539 objects degraded (1.086%); 7822/277539 objects misplaced (2.818%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:54:05.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:05.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:05.676 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:05.711 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:05.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:05.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:05.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:05.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:06.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:06.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:06.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:06.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:06.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/277539 objects degraded (1.086%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:06.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/277539 objects degraded (1.086%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:06.379 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:06.411 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/277539 objects degraded (1.086%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:06.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:06.546 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:06.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.676 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:06.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:06.777 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:06.803 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:06.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:06.937 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:07.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:07.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.071 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:07.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:07.172 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:07 smithi003 ceph-mon[104007]: pgmap v1013: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 3014/267231 objects degraded (1.128%); 7822/267231 objects misplaced (2.927%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:54:07.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:07 smithi003 ceph-mon[107278]: pgmap v1013: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 3014/267231 objects degraded (1.128%); 7822/267231 objects misplaced (2.927%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:54:07.199 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:07.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:07.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:07 smithi019 ceph-mon[91296]: pgmap v1013: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 3014/267231 objects degraded (1.128%); 7822/267231 objects misplaced (2.927%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:54:07.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:07.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:07.606 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:07.637 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:07.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:07.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:07.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:07.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:07.930 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:54:07.930 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:54:07.930 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-13T14:54:07.930 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:54:07.930 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:54:07.930 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-13T14:54:08.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:08.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:08.155 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:08.198 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:08.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:08.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:08.308 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.332 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:08.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:08.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:08.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:08.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:08.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:08.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:08.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:08.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:08.917 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:09.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:09.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:09.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.201 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:09.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:09.245 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:09 smithi003 ceph-mon[104007]: pgmap v1014: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 999 op/s; 3014/262518 objects degraded (1.148%); 7415/262518 objects misplaced (2.825%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:09.245 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:09 smithi003 ceph-mon[107278]: pgmap v1014: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 999 op/s; 3014/262518 objects degraded (1.148%); 7415/262518 objects misplaced (2.825%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:09.302 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:09.330 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:09 smithi019 ceph-mon[91296]: pgmap v1014: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 999 op/s; 3014/262518 objects degraded (1.148%); 7415/262518 objects misplaced (2.825%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:09.432 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:09.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:09.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:54:09.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:09.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:09.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:09.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:09.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:10.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:10.048 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:54:10.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:10.162 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:10.291 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:10.372 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:10.372 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:10.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/269197272' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:10.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:10 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:10.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/269197272' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:10.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:10 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:10.426 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:10.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:54:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:54:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/269197272' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:10 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:10.456 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:10.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:10.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:10.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:10.705 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:10.736 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:10.838 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:10.875 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:10.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:11.118 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:11 smithi019 ceph-mon[91296]: pgmap v1015: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 3014/254100 objects degraded (1.186%); 7415/254100 objects misplaced (2.918%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:11.118 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:11 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:11.118 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:11 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:11.119 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:11 smithi019 ceph-mon[91296]: osdmap e290: 8 total, 8 up, 8 in 2024-09-13T14:54:11.119 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/254100 objects degraded (1.186%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:11.125 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:11.157 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:54:11.157 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "" 2024-09-13T14:54:11.157 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:54:11.159 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:11.261 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.303 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:11.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:11.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[104007]: pgmap v1015: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 3014/254100 objects degraded (1.186%); 7415/254100 objects misplaced (2.918%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:11.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:11.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:11.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[104007]: osdmap e290: 8 total, 8 up, 8 in 2024-09-13T14:54:11.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/254100 objects degraded (1.186%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:11.364 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[107278]: pgmap v1015: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 3014/254100 objects degraded (1.186%); 7415/254100 objects misplaced (2.918%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:11.364 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:11.364 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:11.364 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[107278]: osdmap e290: 8 total, 8 up, 8 in 2024-09-13T14:54:11.364 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/254100 objects degraded (1.186%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:11.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:11.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:54:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:11.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.631 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:11.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:11.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:11.765 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:11.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:11.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:11.893 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:11.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:11.994 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:12.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:12.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:12.248 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:12.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:12.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:12.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:12.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:12.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:12.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:12.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:12.877 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:12.912 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:13.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.046 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:13.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:13.093 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:13.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:13.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:13.307 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:13.342 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:13.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:13 smithi003 ceph-mon[104007]: pgmap v1017: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 977 op/s; 3014/252840 objects degraded (1.192%); 7415/252840 objects misplaced (2.933%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:13.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:13.397 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:13 smithi003 ceph-mon[107278]: pgmap v1017: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 977 op/s; 3014/252840 objects degraded (1.192%); 7415/252840 objects misplaced (2.933%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:13.397 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:13 smithi019 ceph-mon[91296]: pgmap v1017: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 977 op/s; 3014/252840 objects degraded (1.192%); 7415/252840 objects misplaced (2.933%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:13.444 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:13.472 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:13.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:13.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:13.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:13.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:13.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:13.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:14.034 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:14.136 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:14.144 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:54:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:54:14] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-09-13T14:54:14.165 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:14.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:54:14.266 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:14.298 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:14.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:14.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:14.549 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:14.579 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:14.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.717 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:14.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:14.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:14.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:14.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:14.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:15.099 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.136 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:15.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:15.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:15 smithi003 ceph-mon[104007]: pgmap v1018: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 3014/248496 objects degraded (1.213%); 6971/248496 objects misplaced (2.805%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:15.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:15 smithi003 ceph-mon[107278]: pgmap v1018: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 3014/248496 objects degraded (1.213%); 6971/248496 objects misplaced (2.805%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:15.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:15.261 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:15.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:15.395 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:15 smithi019 ceph-mon[91296]: pgmap v1018: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 3014/248496 objects degraded (1.213%); 6971/248496 objects misplaced (2.805%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:15.501 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:15.530 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:15.632 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.664 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:15.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:15.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:15.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:15.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:15.948 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:16.050 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:16.089 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:16.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:16.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:16.333 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:16.364 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:16.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/248496 objects degraded (1.213%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:16.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/248496 objects degraded (1.213%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/248496 objects degraded (1.213%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:16.466 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:16.495 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:16.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:16.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:16.626 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:16.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:16.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:16.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:16.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:17.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:17.040 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:17.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:17.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:17.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:17.207 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:54:17.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:54:17.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:17.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:17 smithi019 ceph-mon[91296]: pgmap v1019: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 3014/238284 objects degraded (1.265%); 6971/238284 objects misplaced (2.926%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:17 smithi003 ceph-mon[107278]: pgmap v1019: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 3014/238284 objects degraded (1.265%); 6971/238284 objects misplaced (2.926%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:17 smithi003 ceph-mon[104007]: pgmap v1019: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 3014/238284 objects degraded (1.265%); 6971/238284 objects misplaced (2.926%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:17.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:17.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:17.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:17.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:17.755 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:17.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.787 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:17.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:17.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:17.913 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:18.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:18.048 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:18.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:18.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:18.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:18.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:18.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:18.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:18.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:18.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:18.598 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:18.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:18.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:18.759 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:54:18.759 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:54:18.759 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:54:18.759 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:54:18.760 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:18.760 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:18.760 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:18.760 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:18.760 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:54:18.760 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:54:18.760 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:54:18.760 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:54:18.761 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:18.761 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:18.761 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:18.761 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:18.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:18.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:18.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:19.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:19.150 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:19.252 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:19.282 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:19.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:19.428 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:19 smithi019 ceph-mon[91296]: pgmap v1020: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 937 op/s; 3014/234393 objects degraded (1.286%); 6971/234393 objects misplaced (2.974%); 947 KiB/s, 14 objects/s recovering 2024-09-13T14:54:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:19 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3934188707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:19.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:19 smithi003 ceph-mon[104007]: pgmap v1020: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 937 op/s; 3014/234393 objects degraded (1.286%); 6971/234393 objects misplaced (2.974%); 947 KiB/s, 14 objects/s recovering 2024-09-13T14:54:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:19 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3934188707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:19 smithi003 ceph-mon[107278]: pgmap v1020: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 937 op/s; 3014/234393 objects degraded (1.286%); 6971/234393 objects misplaced (2.974%); 947 KiB/s, 14 objects/s recovering 2024-09-13T14:54:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:19 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3934188707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:19.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:19.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:19.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:19.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:19.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:19.845 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:19.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:19.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:19.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:20.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:20.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:20.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:20.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:20.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:20.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:20.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:20.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:54:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:54:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:20.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:20.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:20.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:20.906 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:21.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:21.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:21.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:21 smithi003 ceph-mon[107278]: pgmap v1021: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 3014/225888 objects degraded (1.334%); 6515/225888 objects misplaced (2.884%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:54:21.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/225888 objects degraded (1.334%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:21 smithi003 ceph-mon[104007]: pgmap v1021: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 3014/225888 objects degraded (1.334%); 6515/225888 objects misplaced (2.884%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:54:21.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/225888 objects degraded (1.334%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:21.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:21.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:21.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:21 smithi019 ceph-mon[91296]: pgmap v1021: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 3014/225888 objects degraded (1.334%); 6515/225888 objects misplaced (2.884%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T14:54:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/225888 objects degraded (1.334%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:21.463 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:21.494 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:21.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:21.741 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:54:21.741 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:54:21.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:54:21.741 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:54:21.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.741 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149380 10.3 10:c6822046:::benchmark_data_smithi003_197468_object32651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:54:21.741 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.296643+0000", 2024-09-13T14:54:21.741 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 274.44089814400002, 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34575353399999997, 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149380 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.742 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296643+0000", 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296643+0000", 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296645+0000", 2024-09-13T14:54:21.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.761e-06 2024-09-13T14:54:21.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296651+0000", 2024-09-13T14:54:21.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5180000000000001e-06 2024-09-13T14:54:21.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296665+0000", 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4066000000000001e-05 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296674+0000", 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6840000000000002e-06 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296741+0000", 2024-09-13T14:54:21.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7305999999999996e-05 2024-09-13T14:54:21.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:54:21.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296904+0000", 2024-09-13T14:54:21.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016293199999999999 2024-09-13T14:54:21.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297060+0000", 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015613700000000001 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.310299+0000", 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013238409 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.313242+0000", 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029432899999999999 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.640267+0000", 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32702447200000001 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.640356+0000", 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9588999999999999e-05 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642397+0000", 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020403700000000001 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149384 10.3 10:c87a8b94:::benchmark_data_smithi003_197468_object32655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.299373+0000", 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 274.43816822500003, 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34479961399999998, 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149384 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.750 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299373+0000", 2024-09-13T14:54:21.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299373+0000", 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299374+0000", 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3060000000000001e-06 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299379+0000", 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0159999999999999e-06 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299420+0000", 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0426999999999998e-05 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299427+0000", 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0639999999999996e-06 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.304870+0000", 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054428130000000003 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305026+0000", 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156138 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:54:21.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305177+0000", 2024-09-13T14:54:21.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015097900000000001 2024-09-13T14:54:21.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:21.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.310738+0000", 2024-09-13T14:54:21.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055609120000000003 2024-09-13T14:54:21.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:54:21.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.313250+0000", 2024-09-13T14:54:21.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025120889999999999 2024-09-13T14:54:21.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642486+0000", 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.329236468 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642558+0000", 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1736000000000006e-05 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644173+0000", 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016146660000000001 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149387 10.5 10:a0bfa116:::benchmark_data_smithi003_197468_object32658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.303456+0000", 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 274.43408530099998, 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34075042799999999, 2024-09-13T14:54:21.758 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.759 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.759 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:21.759 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.759 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:54:21.759 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:54:21.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:21.760 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149387 2024-09-13T14:54:21.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.760 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303456+0000", 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303456+0000", 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303457+0000", 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.443e-06 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303463+0000", 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6219999999999997e-06 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303508+0000", 2024-09-13T14:54:21.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4771999999999998e-05 2024-09-13T14:54:21.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.303540+0000", 2024-09-13T14:54:21.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2500999999999999e-05 2024-09-13T14:54:21.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.311300+0000", 2024-09-13T14:54:21.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077601040000000003 2024-09-13T14:54:21.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:54:21.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.311420+0000", 2024-09-13T14:54:21.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119769 2024-09-13T14:54:21.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:54:21.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.311533+0000", 2024-09-13T14:54:21.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112631 2024-09-13T14:54:21.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:54:21.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316488+0000", 2024-09-13T14:54:21.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049549090000000004 2024-09-13T14:54:21.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:21.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.321528+0000", 2024-09-13T14:54:21.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050398550000000002 2024-09-13T14:54:21.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:21.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644049+0000", 2024-09-13T14:54:21.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32252128600000002 2024-09-13T14:54:21.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:54:21.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644163+0000", 2024-09-13T14:54:21.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114039 2024-09-13T14:54:21.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644206+0000", 2024-09-13T14:54:21.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3497e-05 2024-09-13T14:54:21.770 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.770 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.770 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.771 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149402 10.5 10:a23e644e:::benchmark_data_smithi003_197468_object32673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:54:21.771 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.316465+0000", 2024-09-13T14:54:21.771 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 274.42107664700001, 2024-09-13T14:54:21.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32805168899999998, 2024-09-13T14:54:21.771 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.771 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.771 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:21.771 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.772 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:54:21.772 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:54:21.772 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149402 2024-09-13T14:54:21.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.772 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316465+0000", 2024-09-13T14:54:21.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316465+0000", 2024-09-13T14:54:21.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316466+0000", 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1689999999999999e-06 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316470+0000", 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9720000000000003e-06 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316479+0000", 2024-09-13T14:54:21.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0100000000000001e-06 2024-09-13T14:54:21.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316484+0000", 2024-09-13T14:54:21.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1970000000000004e-06 2024-09-13T14:54:21.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316606+0000", 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012250100000000001 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316764+0000", 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015716499999999999 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.316876+0000", 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112608 2024-09-13T14:54:21.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:54:21.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.321155+0000", 2024-09-13T14:54:21.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042787850000000002 2024-09-13T14:54:21.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.321559+0000", 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000404254 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644242+0000", 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.322682523 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644300+0000", 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8136e-05 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.644516+0000", 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000216369 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96814 11.11 11.8c21ebb1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.160407+0000", 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.577134131999998, 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.946111904, 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.780 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96814 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160407+0000", 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160407+0000", 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160408+0000", 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.372e-06 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160432+0000", 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3487999999999999e-05 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160441+0000", 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4010000000000003e-06 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160447+0000", 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6729999999999997e-06 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160517+0000", 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0185999999999995e-05 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160525+0000", 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8199999999999997e-06 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.106508+0000", 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94598301100000004 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.106519+0000", 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0953e-05 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96867 11.11 11.57744991 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.785 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.233781+0000", 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.503760357999994, 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87324690599999999, 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96867 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233781+0000", 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233781+0000", 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233781+0000", 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4000000000000001e-07 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233802+0000", 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0288999999999998e-05 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233811+0000", 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7499999999999992e-06 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233817+0000", 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7989999999999996e-06 2024-09-13T14:54:21.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.236916+0000", 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030987860000000001 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.236923+0000", 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1210000000000001e-06 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107019+0000", 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87009556899999996 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107028+0000", 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.952e-06 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96891 11.11 11.a3b0f631 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.790 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.252748+0000", 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.484793233999994, 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.85431288100000002, 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96891 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252748+0000", 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252748+0000", 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252749+0000", 2024-09-13T14:54:21.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2470000000000001e-06 2024-09-13T14:54:21.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252769+0000", 2024-09-13T14:54:21.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9615000000000001e-05 2024-09-13T14:54:21.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252784+0000", 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4899000000000001e-05 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252792+0000", 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2400000000000007e-06 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252868+0000", 2024-09-13T14:54:21.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6359000000000005e-05 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252876+0000", 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5700000000000004e-06 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107053+0000", 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.85417744399999995 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107061+0000", 2024-09-13T14:54:21.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5070000000000003e-06 2024-09-13T14:54:21.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.796 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96906 11.11 11.416bf311 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.266083+0000", 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.471458698000006, 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.841012863, 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96906 2024-09-13T14:54:21.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266083+0000", 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266083+0000", 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266084+0000", 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.33e-06 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266134+0000", 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0283e-05 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266150+0000", 2024-09-13T14:54:21.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5828000000000002e-05 2024-09-13T14:54:21.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266158+0000", 2024-09-13T14:54:21.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6229999999999997e-06 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266967+0000", 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000809886 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266977+0000", 2024-09-13T14:54:21.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0350000000000007e-06 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107087+0000", 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.84011018999999998 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107095+0000", 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6880000000000005e-06 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.802 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.803 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.803 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96933 11.11 11.30677291 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.803 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.295082+0000", 2024-09-13T14:54:21.803 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.442459645, 2024-09-13T14:54:21.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.81204666999999997, 2024-09-13T14:54:21.803 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.803 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96933 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295082+0000", 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295082+0000", 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295083+0000", 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8190000000000001e-06 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295108+0000", 2024-09-13T14:54:21.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4278e-05 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295124+0000", 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6710000000000001e-05 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295133+0000", 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4400000000000005e-06 2024-09-13T14:54:21.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295217+0000", 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4079000000000006e-05 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295227+0000", 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0227000000000001e-05 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107121+0000", 2024-09-13T14:54:21.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.81189347099999998 2024-09-13T14:54:21.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107128+0000", 2024-09-13T14:54:21.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6459999999999993e-06 2024-09-13T14:54:21.808 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.808 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.808 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.809 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96998 11.11 11.57659511 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.809 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.356695+0000", 2024-09-13T14:54:21.809 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.380845821999998, 2024-09-13T14:54:21.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.75046507100000004, 2024-09-13T14:54:21.809 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.809 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96998 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356695+0000", 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356695+0000", 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356697+0000", 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.249e-06 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356720+0000", 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3578999999999999e-05 2024-09-13T14:54:21.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356736+0000", 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5892999999999998e-05 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356744+0000", 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6329999999999996e-06 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356901+0000", 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015711800000000001 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356910+0000", 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3139999999999993e-06 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107153+0000", 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.75024257599999999 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107160+0000", 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7090000000000002e-06 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97010 11.11 11.1257951 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.364426+0000", 2024-09-13T14:54:21.814 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.373115411000001, 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74276584000000001, 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97010 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364426+0000", 2024-09-13T14:54:21.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364426+0000", 2024-09-13T14:54:21.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364427+0000", 2024-09-13T14:54:21.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.513e-06 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364456+0000", 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8504999999999999e-05 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364481+0000", 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5275000000000001e-05 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364490+0000", 2024-09-13T14:54:21.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.217e-06 2024-09-13T14:54:21.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364555+0000", 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4296999999999998e-05 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364564+0000", 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8999999999999995e-06 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.820 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107185+0000", 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74262110199999998 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107192+0000", 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0310000000000001e-06 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.821 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97011 11.11 11.c6b96131 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.364516+0000", 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.373025311999996, 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.742706172, 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.822 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97011 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364516+0000", 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364516+0000", 2024-09-13T14:54:21.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364516+0000", 2024-09-13T14:54:21.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0999999999999999e-07 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364536+0000", 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9724e-05 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364544+0000", 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6790000000000005e-06 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364549+0000", 2024-09-13T14:54:21.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6590000000000004e-06 2024-09-13T14:54:21.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364589+0000", 2024-09-13T14:54:21.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9217000000000001e-05 2024-09-13T14:54:21.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364595+0000", 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1519999999999997e-06 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107215+0000", 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74262040900000004 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107222+0000", 2024-09-13T14:54:21.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8220000000000001e-06 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97012 11.11 11.7b2575f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.364558+0000", 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.372983312000002, 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74269332099999996, 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.831 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97012 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364558+0000", 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364558+0000", 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364558+0000", 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.58e-07 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364605+0000", 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7094e-05 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364613+0000", 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2010000000000002e-06 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364618+0000", 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0640000000000001e-06 2024-09-13T14:54:21.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.366443+0000", 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018250460000000001 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.366450+0000", 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0129999999999996e-06 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107244+0000", 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74079445600000005 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107251+0000", 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9890000000000003e-06 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.837 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97049 11.11 11.b3adac71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.837 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.417966+0000", 2024-09-13T14:54:21.837 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.319574919000004, 2024-09-13T14:54:21.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.68931568700000001, 2024-09-13T14:54:21.837 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.837 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.837 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.837 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.837 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97049 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.417966+0000", 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:21.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.417966+0000", 2024-09-13T14:54:21.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.417968+0000", 2024-09-13T14:54:21.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.001e-06 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418039+0000", 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0745000000000001e-05 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418059+0000", 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9788000000000001e-05 2024-09-13T14:54:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418068+0000", 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1009999999999998e-06 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418134+0000", 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6412e-05 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418143+0000", 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9430000000000006e-06 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107275+0000", 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.689131359 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107282+0000", 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3379999999999999e-06 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97063 11.11 11.3e7508f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.435905+0000", 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.301636297000002, 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.671408, 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.844 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.845 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.845 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.845 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.845 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.845 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97063 2024-09-13T14:54:21.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.845 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435905+0000", 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435905+0000", 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435906+0000", 2024-09-13T14:54:21.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4720000000000001e-06 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435928+0000", 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1923e-05 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435939+0000", 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1097000000000001e-05 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435946+0000", 2024-09-13T14:54:21.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0890000000000002e-06 2024-09-13T14:54:21.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.436032+0000", 2024-09-13T14:54:21.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5976999999999994e-05 2024-09-13T14:54:21.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.436041+0000", 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5110000000000003e-06 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107306+0000", 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.67126498300000004 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107313+0000", 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9480000000000002e-06 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97147 11.11 11.a734b571 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.543788+0000", 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.193753615999995, 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.56355814500000001, 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.850 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97147 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543788+0000", 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543788+0000", 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543789+0000", 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.598e-06 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543813+0000", 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4238000000000001e-05 2024-09-13T14:54:21.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543833+0000", 2024-09-13T14:54:21.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9123999999999999e-05 2024-09-13T14:54:21.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543841+0000", 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5609999999999999e-06 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543919+0000", 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7885000000000004e-05 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543926+0000", 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4619999999999999e-06 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107339+0000", 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.56341212500000004 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107346+0000", 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1520000000000003e-06 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.855 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.856 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97153 11.11 11.ac919031 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.856 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.550994+0000", 2024-09-13T14:54:21.856 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.186547352000005, 2024-09-13T14:54:21.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.55638195899999998, 2024-09-13T14:54:21.857 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.857 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.857 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.857 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.857 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.857 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97153 2024-09-13T14:54:21.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.857 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.550994+0000", 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.550994+0000", 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.550996+0000", 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.81e-06 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551017+0000", 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1404999999999999e-05 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551030+0000", 2024-09-13T14:54:21.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2978e-05 2024-09-13T14:54:21.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551038+0000", 2024-09-13T14:54:21.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3289999999999994e-06 2024-09-13T14:54:21.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551316+0000", 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027794499999999998 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551324+0000", 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5630000000000003e-06 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107369+0000", 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.55604483699999996 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107376+0000", 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.092e-06 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97192 11.11 11.216cafd1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.607681+0000", 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.129859808999996, 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.49972701600000002, 2024-09-13T14:54:21.862 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97192 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607681+0000", 2024-09-13T14:54:21.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607681+0000", 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607683+0000", 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.765e-06 2024-09-13T14:54:21.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607709+0000", 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6139000000000002e-05 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607729+0000", 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9219999999999999e-05 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607738+0000", 2024-09-13T14:54:21.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1649999999999995e-06 2024-09-13T14:54:21.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607804+0000", 2024-09-13T14:54:21.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5957999999999994e-05 2024-09-13T14:54:21.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607812+0000", 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2609999999999993e-06 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107401+0000", 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.49958935700000001 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107408+0000", 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1509999999999998e-06 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97195 11.11 11.3d344711 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.613202+0000", 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.124339036999999, 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.49423730199999999, 2024-09-13T14:54:21.868 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97195 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613202+0000", 2024-09-13T14:54:21.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613202+0000", 2024-09-13T14:54:21.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613203+0000", 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2860000000000001e-06 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613225+0000", 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1877000000000001e-05 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613241+0000", 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5241e-05 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613248+0000", 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2910000000000002e-06 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613320+0000", 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2009999999999997e-05 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613328+0000", 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2879999999999993e-06 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107432+0000", 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.49410401599999998 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107439+0000", 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2930000000000003e-06 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97217 11.11 11.4babd2f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.681896+0000", 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.055645373999994, 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.425573592, 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:54:21.875 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97217 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681896+0000", 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:54:21.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681896+0000", 2024-09-13T14:54:21.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:54:21.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:54:21.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681897+0000", 2024-09-13T14:54:21.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5999999999999999e-06 2024-09-13T14:54:21.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681921+0000", 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4025000000000001e-05 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681939+0000", 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7649999999999999e-05 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681947+0000", 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5499999999999997e-06 2024-09-13T14:54:21.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.682000+0000", 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3761000000000002e-05 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.682006+0000", 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7610000000000004e-06 2024-09-13T14:54:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107462+0000", 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.42545575899999999 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107469+0000", 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.486e-06 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.880 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.881 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:54:21.881 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:54:21.881 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:54:21.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.883 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:21.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:21.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:22.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:22.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:22.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:22.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:22.322 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:22.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:22.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:22.565 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:22.595 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:22.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:22.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:22.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:22.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:22.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:22.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:54:22.981 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:23.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:23.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:23.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:23.278 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:23.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:23.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:23 smithi019 ceph-mon[91296]: pgmap v1022: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 3014/224610 objects degraded (1.342%); 6515/224610 objects misplaced (2.901%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:54:23.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:23 smithi003 ceph-mon[104007]: pgmap v1022: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 3014/224610 objects degraded (1.342%); 6515/224610 objects misplaced (2.901%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:54:23.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:23 smithi003 ceph-mon[107278]: pgmap v1022: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 3014/224610 objects degraded (1.342%); 6515/224610 objects misplaced (2.901%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:54:23.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:23.540 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:23.572 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:23.675 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:23.709 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:23.811 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:23.841 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:23.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:23.974 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:24.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:24.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:24.233 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.265 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:24.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:24.366 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:54:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:54:24] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-09-13T14:54:24.500 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:24.536 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:24.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:24.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:24.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:24.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:54:24.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:24.804 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:24.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:24.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:25.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:25.099 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:25.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:25.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:25.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:25.242 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:25 smithi003 ceph-mon[107278]: pgmap v1023: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 909 op/s; 3014/220083 objects degraded (1.369%); 6038/220083 objects misplaced (2.744%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:54:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:25.243 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:25 smithi003 ceph-mon[104007]: pgmap v1023: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 909 op/s; 3014/220083 objects degraded (1.369%); 6038/220083 objects misplaced (2.744%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:54:25.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:25.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:25 smithi019 ceph-mon[91296]: pgmap v1023: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 909 op/s; 3014/220083 objects degraded (1.369%); 6038/220083 objects misplaced (2.744%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:54:25.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:25.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:25.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:25.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:25.784 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:25.812 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:25.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:25.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:26.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:26.097 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:54:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:26.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:26.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:26.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:26.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:26.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:26.397 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:26.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/220083 objects degraded (1.369%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:26.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4138241806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:54:26.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:26 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:54:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/220083 objects degraded (1.369%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4138241806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:54:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:26 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:54:26.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/220083 objects degraded (1.369%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:26.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4138241806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:54:26.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:26 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:54:26.500 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:26.527 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:26.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:26.658 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:26.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:26.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:26.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:26.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:26.892 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:26.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.943 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:26.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:27.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:27.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:27.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:27.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:27.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:27.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:27.366 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:27 smithi003 ceph-mon[107278]: pgmap v1024: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 3014/210144 objects degraded (1.434%); 6038/210144 objects misplaced (2.873%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:54:27.366 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:27 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:54:27.366 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:27 smithi003 ceph-mon[107278]: osdmap e291: 8 total, 8 up, 8 in 2024-09-13T14:54:27.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:27 smithi003 ceph-mon[104007]: pgmap v1024: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 3014/210144 objects degraded (1.434%); 6038/210144 objects misplaced (2.873%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:54:27.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:27 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:54:27.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:27 smithi003 ceph-mon[104007]: osdmap e291: 8 total, 8 up, 8 in 2024-09-13T14:54:27.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.368 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:27.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:27.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:54:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:27 smithi019 ceph-mon[91296]: pgmap v1024: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 3014/210144 objects degraded (1.434%); 6038/210144 objects misplaced (2.873%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:54:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:27 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:54:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:27 smithi019 ceph-mon[91296]: osdmap e291: 8 total, 8 up, 8 in 2024-09-13T14:54:27.469 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:27.500 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:27.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:27.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:27.762 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:27.787 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:27.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:27.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:27.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:28.046 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:28.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:28.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:28.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:28.262 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:28.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:28 smithi003 ceph-mon[104007]: osdmap e292: 8 total, 8 up, 8 in 2024-09-13T14:54:28.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:28.345 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:28 smithi003 ceph-mon[107278]: osdmap e292: 8 total, 8 up, 8 in 2024-09-13T14:54:28.346 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:28.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:28.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:28 smithi019 ceph-mon[91296]: osdmap e292: 8 total, 8 up, 8 in 2024-09-13T14:54:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:28.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:28.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:28.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:28.772 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:28.795 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:28.897 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:28.925 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:28.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:29.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:29.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:29.146 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:54:29.149 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:29.173 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:29.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:29.334 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:29 smithi019 ceph-mon[91296]: pgmap v1027: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 3014/206286 objects degraded (1.461%); 6038/206286 objects misplaced (2.927%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:54:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:29 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/947325343' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:29 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:29.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:29 smithi003 ceph-mon[107278]: pgmap v1027: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 3014/206286 objects degraded (1.461%); 6038/206286 objects misplaced (2.927%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:54:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:29 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/947325343' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:29 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:29 smithi003 ceph-mon[104007]: pgmap v1027: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 3014/206286 objects degraded (1.461%); 6038/206286 objects misplaced (2.927%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:54:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:29 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/947325343' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:29.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:29 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:29.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:29.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:29.590 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.638 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:29.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:29.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:29.778 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:29.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:29.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:30.016 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:30.045 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:30.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:30.181 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:30.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:30.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:30.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:54:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:54:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:30 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:30 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:30.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:30 smithi019 ceph-mon[91296]: osdmap e293: 8 total, 8 up, 8 in 2024-09-13T14:54:30.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:54:30.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:30 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:30.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:30 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:30.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:30 smithi003 ceph-mon[107278]: osdmap e293: 8 total, 8 up, 8 in 2024-09-13T14:54:30.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:54:30.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:30 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:30.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:30 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:30.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:30 smithi003 ceph-mon[104007]: osdmap e293: 8 total, 8 up, 8 in 2024-09-13T14:54:30.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:54:30.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:30.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:30.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:30.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:30.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:30.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:30.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:31.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:31.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:31.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:31.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:31.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:31 smithi003 ceph-mon[104007]: pgmap v1029: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s; 3014/197643 objects degraded (1.525%); 5516/197643 objects misplaced (2.791%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:31.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/197643 objects degraded (1.525%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:31.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:31 smithi003 ceph-mon[107278]: pgmap v1029: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s; 3014/197643 objects degraded (1.525%); 5516/197643 objects misplaced (2.791%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:31.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/197643 objects degraded (1.525%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:31.318 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:31.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:31 smithi019 ceph-mon[91296]: pgmap v1029: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s; 3014/197643 objects degraded (1.525%); 5516/197643 objects misplaced (2.791%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/197643 objects degraded (1.525%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:31.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:31.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:31.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:31.625 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:31.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:31.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:31.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:31.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:32.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:32.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:32.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:32.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:32.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:32.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.356 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:32.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:32.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:32 smithi003 ceph-mon[107278]: 11.5 scrub starts 2024-09-13T14:54:32.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:32 smithi003 ceph-mon[104007]: 11.5 scrub starts 2024-09-13T14:54:32.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:32 smithi019 ceph-mon[91296]: 11.5 scrub starts 2024-09-13T14:54:32.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:32.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:32.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:32.621 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:32.723 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:32.756 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:32.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:54:32.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:54:32.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:54:32.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:54:32.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:54:32.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:32.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:32.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:54:32.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.900 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:32.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:33.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:33.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:33.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:33.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:33.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.193 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:33.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:33.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:33.334 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:33 smithi019 ceph-mon[91296]: pgmap v1030: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 3014/196359 objects degraded (1.535%); 5516/196359 objects misplaced (2.809%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:33 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3987280094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:33.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:33 smithi003 ceph-mon[104007]: pgmap v1030: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 3014/196359 objects degraded (1.535%); 5516/196359 objects misplaced (2.809%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:33.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:33 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3987280094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:33.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:33 smithi003 ceph-mon[107278]: pgmap v1030: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 3014/196359 objects degraded (1.535%); 5516/196359 objects misplaced (2.809%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:54:33.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:33 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3987280094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:33.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:33.475 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:33.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:33.605 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:33.706 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:33.743 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:33.845 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:33.877 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:33.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:34.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:34.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:34.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:34.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:34.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.286 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:34.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:34.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:34.420 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:54:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:54:34] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:54:34.522 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.557 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:34.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:34.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:34.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:34.802 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:34.831 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:34.836 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:34.836 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:34.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:35.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:35.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:35.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:35 smithi003 ceph-mon[107278]: pgmap v1031: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 3014/192039 objects degraded (1.569%); 4979/192039 objects misplaced (2.593%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:54:35.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:35 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:54:35.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:35 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:54:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:35 smithi003 ceph-mon[104007]: pgmap v1031: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 3014/192039 objects degraded (1.569%); 4979/192039 objects misplaced (2.593%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:54:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:35 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:54:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:35 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:54:35.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:35.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:35.653 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:35 smithi019 ceph-mon[91296]: pgmap v1031: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 3014/192039 objects degraded (1.569%); 4979/192039 objects misplaced (2.593%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:54:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:35 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:54:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:35 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:54:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.683 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:35.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:35.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:54:35.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:35.930 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:35.954 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:35.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:36.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:36.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:36.422 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:54:36.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:36 smithi003 ceph-mon[104007]: 11.15 scrub starts 2024-09-13T14:54:36.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:36 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1150899710' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:54:36.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:36 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:54:36.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:36 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/192039 objects degraded (1.569%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:36.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:36 smithi003 ceph-mon[107278]: 11.15 scrub starts 2024-09-13T14:54:36.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:36 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1150899710' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:54:36.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:36 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:54:36.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:36 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/192039 objects degraded (1.569%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:36.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:36.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:54:36.550 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:36.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:36 smithi019 ceph-mon[91296]: 11.15 scrub starts 2024-09-13T14:54:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:36 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1150899710' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:54:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:36 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:54:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:36 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/192039 objects degraded (1.569%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:36.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:36.796 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:36.823 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:36.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:36.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:36.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:37.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:37.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:37.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:37.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:37.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:37 smithi003 ceph-mon[104007]: pgmap v1032: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 3014/182229 objects degraded (1.654%); 4979/182229 objects misplaced (2.732%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:54:37.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:37 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:54:37.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:37 smithi003 ceph-mon[104007]: osdmap e294: 8 total, 8 up, 8 in 2024-09-13T14:54:37.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:37 smithi003 ceph-mon[107278]: pgmap v1032: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 3014/182229 objects degraded (1.654%); 4979/182229 objects misplaced (2.732%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:54:37.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:37 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:54:37.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:37 smithi003 ceph-mon[107278]: osdmap e294: 8 total, 8 up, 8 in 2024-09-13T14:54:37.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:37.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:37.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:37.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:37.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:37.674 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:37.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:37 smithi019 ceph-mon[91296]: pgmap v1032: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 3014/182229 objects degraded (1.654%); 4979/182229 objects misplaced (2.732%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:54:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:37 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:54:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:37 smithi019 ceph-mon[91296]: osdmap e294: 8 total, 8 up, 8 in 2024-09-13T14:54:37.702 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:37.804 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:37.831 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:37.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:37.958 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:38.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:38.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:38.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:54:38.206 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:38.235 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:38.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:38.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:38.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:38.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:38.590 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:38.627 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:38.729 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:38.753 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:38.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:38.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:39.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:39.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:39.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:39.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:39.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:39.300 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:39.326 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:39.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:39.460 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:39.562 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:39.570 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:39 smithi003 ceph-mon[104007]: pgmap v1034: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3014/178524 objects degraded (1.688%); 4979/178524 objects misplaced (2.789%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:54:39.570 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:39 smithi003 ceph-mon[107278]: pgmap v1034: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3014/178524 objects degraded (1.688%); 4979/178524 objects misplaced (2.789%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:54:39.596 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:39.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:39.681 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:39.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:39 smithi019 ceph-mon[91296]: pgmap v1034: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3014/178524 objects degraded (1.688%); 4979/178524 objects misplaced (2.789%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:54:39.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:39.859 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:39.892 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:39.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:39.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:40.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:40.146 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:40.172 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:40.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:40.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:40.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:40.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:40.456 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:40.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:40.582 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:54:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:54:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:40.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:40.695 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:40.733 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:40.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:40.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:40.988 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:41.022 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:41.023 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:41.027 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:41.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:41.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:41.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:41.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:41.427 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:41 smithi003 ceph-mon[107278]: pgmap v1035: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 3014/170820 objects degraded (1.764%); 4425/170820 objects misplaced (2.590%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:54:41.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/170820 objects degraded (1.764%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:41.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:41 smithi003 ceph-mon[104007]: pgmap v1035: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 3014/170820 objects degraded (1.764%); 4425/170820 objects misplaced (2.590%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:54:41.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/170820 objects degraded (1.764%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:41.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.462 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:41.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:41.563 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:41.587 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:41 smithi019 ceph-mon[91296]: pgmap v1035: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 3014/170820 objects degraded (1.764%); 4425/170820 objects misplaced (2.590%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:54:41.587 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/170820 objects degraded (1.764%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:41.596 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:41.698 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:41.737 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:41.839 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:41.884 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:41.985 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:42.020 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:42.122 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:42.156 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:42.240 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:54:42.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:42.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:42.411 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:42.450 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:42.517 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:42 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/627583055' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:42.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:42 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:42.518 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:42 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/627583055' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:42.518 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:42 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:42.552 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:42.581 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:42 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/627583055' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:42 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:42.682 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:42.735 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:42.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:42.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:42.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:43.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:43.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:43.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:43.277 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:43.307 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:43.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[107278]: pgmap v1036: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 3014/169542 objects degraded (1.778%); 4425/169542 objects misplaced (2.610%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:54:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:54:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[107278]: osdmap e295: 8 total, 8 up, 8 in 2024-09-13T14:54:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:54:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[104007]: pgmap v1036: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 3014/169542 objects degraded (1.778%); 4425/169542 objects misplaced (2.610%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:54:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:54:43.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:43.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[104007]: osdmap e295: 8 total, 8 up, 8 in 2024-09-13T14:54:43.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:54:43.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.444 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:43.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:43.545 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:43.574 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:43.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:43.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:43 smithi019 ceph-mon[91296]: pgmap v1036: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 3014/169542 objects degraded (1.778%); 4425/169542 objects misplaced (2.610%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:54:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:43 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:54:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:43 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:43 smithi019 ceph-mon[91296]: osdmap e295: 8 total, 8 up, 8 in 2024-09-13T14:54:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:54:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:43.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:43.778 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:54:43.779 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:43.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:43.820 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:54:43.820 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:43.866 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:54:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:43.906 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:54:43.906 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:43.938 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:54:43.938 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:43.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:43.977 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:54:43.978 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:43.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:44.005 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:54:44.005 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:44.040 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:54:44.041 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:44.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:44.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:54:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:54:44] "GET /metrics HTTP/1.1" 200 36553 "" "Prometheus/2.51.0" 2024-09-13T14:54:44.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:44.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:44.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:44.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:44.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:44.542 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:44.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:44 smithi019 ceph-mon[91296]: 11.14 scrub starts 2024-09-13T14:54:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3305086472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:44.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:44 smithi003 ceph-mon[107278]: 11.14 scrub starts 2024-09-13T14:54:44.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3305086472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:44 smithi003 ceph-mon[104007]: 11.14 scrub starts 2024-09-13T14:54:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3305086472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:44.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:44.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:44.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:44.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:44.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:45.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:45.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.240 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:45.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:45.367 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:45.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:45.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:45.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:45.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:45.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:45.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:45 smithi019 ceph-mon[91296]: pgmap v1038: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 3014/165402 objects degraded (1.822%); 3767/165402 objects misplaced (2.277%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:54:45.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:45 smithi003 ceph-mon[107278]: pgmap v1038: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 3014/165402 objects degraded (1.822%); 3767/165402 objects misplaced (2.277%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:54:45.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:45 smithi003 ceph-mon[104007]: pgmap v1038: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 3014/165402 objects degraded (1.822%); 3767/165402 objects misplaced (2.277%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:54:45.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:45.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:45.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:45.913 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:46.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:46.041 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:46.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:46.182 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:46.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:46.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:46.424 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:46.461 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:46.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:46.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/165402 objects degraded (1.822%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:46.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/165402 objects degraded (1.822%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:46.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/165402 objects degraded (1.822%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:46.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:46.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:46.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:46.877 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:46.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:47.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:47.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:47.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:47.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:47.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:47 smithi003 ceph-mon[104007]: pgmap v1039: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 896 op/s; 3014/156114 objects degraded (1.931%); 3767/156114 objects misplaced (2.413%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:54:47.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:47 smithi003 ceph-mon[104007]: 11.5 scrub starts 2024-09-13T14:54:47.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:47.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:47.418 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:47.520 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:47.559 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:47.660 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:47.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:47 smithi019 ceph-mon[91296]: pgmap v1039: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 896 op/s; 3014/156114 objects degraded (1.931%); 3767/156114 objects misplaced (2.413%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:54:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:47 smithi019 ceph-mon[91296]: 11.5 scrub starts 2024-09-13T14:54:47.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:47 smithi003 ceph-mon[107278]: pgmap v1039: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 896 op/s; 3014/156114 objects degraded (1.931%); 3767/156114 objects misplaced (2.413%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:54:47.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:47 smithi003 ceph-mon[107278]: 11.5 scrub starts 2024-09-13T14:54:47.685 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:47.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:47.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:47.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:47.959 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:47.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:48.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:48.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:48.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:48.232 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:48.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:48.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:48.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:48.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:48.627 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.651 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:48.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:48.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:48 smithi019 ceph-mon[91296]: 11.15 deep-scrub starts 2024-09-13T14:54:48.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:48 smithi003 ceph-mon[107278]: 11.15 deep-scrub starts 2024-09-13T14:54:48.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:48 smithi003 ceph-mon[104007]: 11.15 deep-scrub starts 2024-09-13T14:54:48.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:48.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:48.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:48.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:49.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:49.039 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:49.140 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:49.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:49.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:49.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:49.389 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:49.416 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:49.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:49.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:49.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:49 smithi019 ceph-mon[91296]: pgmap v1040: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 3014/152229 objects degraded (1.980%); 3767/152229 objects misplaced (2.475%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:54:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:49 smithi019 ceph-mon[91296]: osdmap e296: 8 total, 8 up, 8 in 2024-09-13T14:54:49.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:49 smithi003 ceph-mon[107278]: pgmap v1040: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 3014/152229 objects degraded (1.980%); 3767/152229 objects misplaced (2.475%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:54:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:49 smithi003 ceph-mon[107278]: osdmap e296: 8 total, 8 up, 8 in 2024-09-13T14:54:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:49 smithi003 ceph-mon[104007]: pgmap v1040: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 3014/152229 objects degraded (1.980%); 3767/152229 objects misplaced (2.475%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:54:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:49 smithi003 ceph-mon[104007]: osdmap e296: 8 total, 8 up, 8 in 2024-09-13T14:54:49.693 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:49.795 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:49.829 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:49.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:49.931 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:49.960 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:50.065 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:50.093 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:50.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:50.234 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:50.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:50.364 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:50.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:50.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:50.533 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:50 smithi019 ceph-mon[91296]: osdmap e297: 8 total, 8 up, 8 in 2024-09-13T14:54:50.534 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:54:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:54:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:50.606 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:50.646 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:50.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:50 smithi003 ceph-mon[104007]: osdmap e297: 8 total, 8 up, 8 in 2024-09-13T14:54:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:50 smithi003 ceph-mon[107278]: osdmap e297: 8 total, 8 up, 8 in 2024-09-13T14:54:50.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:50.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:50.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:50.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:51.004 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:51.026 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:51.128 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.157 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:51.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:51.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:51.394 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:51.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:51.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.551 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.653 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:51 smithi019 ceph-mon[91296]: pgmap v1043: 119 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 3014/144708 objects degraded (2.083%); 3014/144708 objects misplaced (2.083%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:54:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:51 smithi019 ceph-mon[91296]: 11.10 scrub starts 2024-09-13T14:54:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:51 smithi019 ceph-mon[91296]: 11.3 scrub starts 2024-09-13T14:54:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:51 smithi019 ceph-mon[91296]: 11.10 scrub ok 2024-09-13T14:54:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3014/144708 objects degraded (2.083%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[107278]: pgmap v1043: 119 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 3014/144708 objects degraded (2.083%); 3014/144708 objects misplaced (2.083%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:54:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[107278]: 11.10 scrub starts 2024-09-13T14:54:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[107278]: 11.3 scrub starts 2024-09-13T14:54:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[107278]: 11.10 scrub ok 2024-09-13T14:54:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3014/144708 objects degraded (2.083%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[104007]: pgmap v1043: 119 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 3014/144708 objects degraded (2.083%); 3014/144708 objects misplaced (2.083%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:54:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[104007]: 11.10 scrub starts 2024-09-13T14:54:51.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[104007]: 11.3 scrub starts 2024-09-13T14:54:51.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[104007]: 11.10 scrub ok 2024-09-13T14:54:51.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3014/144708 objects degraded (2.083%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.685 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:51.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:51.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:51.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:51.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:51.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:52.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:52.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:52.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:52.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:52.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:52.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:52.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:52.529 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:52.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:52.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:52.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:52.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:52.976 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:53.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:53.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:53.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:53.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:53.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:53.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:53 smithi003 ceph-mon[104007]: pgmap v1044: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 127 op/s; 2266/143592 objects degraded (1.578%); 3536/143592 objects misplaced (2.463%) 2024-09-13T14:54:53.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:53 smithi003 ceph-mon[104007]: 11.1 scrub starts 2024-09-13T14:54:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:53 smithi003 ceph-mon[107278]: pgmap v1044: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 127 op/s; 2266/143592 objects degraded (1.578%); 3536/143592 objects misplaced (2.463%) 2024-09-13T14:54:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:53 smithi003 ceph-mon[107278]: 11.1 scrub starts 2024-09-13T14:54:53.525 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:53.562 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:53.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:53.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:53 smithi019 ceph-mon[91296]: pgmap v1044: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 127 op/s; 2266/143592 objects degraded (1.578%); 3536/143592 objects misplaced (2.463%) 2024-09-13T14:54:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:53 smithi019 ceph-mon[91296]: 11.1 scrub starts 2024-09-13T14:54:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:53.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:53.815 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:53.949 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.979 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:53.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:54.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:54.114 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:54.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:54.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:54.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:54.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:54.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:54 smithi003 ceph-mon[104007]: 11.13 scrub starts 2024-09-13T14:54:54.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:54 smithi003 ceph-mon[104007]: 11.1 scrub ok 2024-09-13T14:54:54.379 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:54:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:54:54] "GET /metrics HTTP/1.1" 200 36553 "" "Prometheus/2.51.0" 2024-09-13T14:54:54.380 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:54 smithi003 ceph-mon[107278]: 11.13 scrub starts 2024-09-13T14:54:54.380 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:54 smithi003 ceph-mon[107278]: 11.1 scrub ok 2024-09-13T14:54:54.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:54.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:54.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:54.658 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:54:54.658 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:54:54.658 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:54:54.658 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:54:54.658 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:54:54.660 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:54 smithi019 ceph-mon[91296]: 11.13 scrub starts 2024-09-13T14:54:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:54 smithi019 ceph-mon[91296]: 11.1 scrub ok 2024-09-13T14:54:54.687 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:54.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:54:54.789 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:54.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:54.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:54.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:54.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:55.083 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:55.109 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:55.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:55.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:55.346 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:54:55.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:55.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:55.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:55 smithi003 ceph-mon[107278]: pgmap v1045: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2266/140190 objects degraded (1.616%); 3536/140190 objects misplaced (2.522%) 2024-09-13T14:54:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1941334129' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:54:55.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:55 smithi003 ceph-mon[104007]: pgmap v1045: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2266/140190 objects degraded (1.616%); 3536/140190 objects misplaced (2.522%) 2024-09-13T14:54:55.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1941334129' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:54:55.501 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:55.547 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:55.649 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:55.680 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:55 smithi019 ceph-mon[91296]: pgmap v1045: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2266/140190 objects degraded (1.616%); 3536/140190 objects misplaced (2.522%) 2024-09-13T14:54:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1941334129' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:54:55.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:55.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:55.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:56.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:56.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:56.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:54:56.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:56.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:56.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:56.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:56.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:56.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:56.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:56.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:56.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1941334129' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:54:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:56 smithi019 ceph-mon[91296]: osdmap e298: 8 total, 8 up, 8 in 2024-09-13T14:54:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2266/140190 objects degraded (1.616%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:56 smithi019 ceph-mon[91296]: pgmap v1047: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2266/130749 objects degraded (1.733%); 3276/130749 objects misplaced (2.506%) 2024-09-13T14:54:56.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1941334129' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:54:56.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:56 smithi003 ceph-mon[107278]: osdmap e298: 8 total, 8 up, 8 in 2024-09-13T14:54:56.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2266/140190 objects degraded (1.616%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:56.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:56 smithi003 ceph-mon[107278]: pgmap v1047: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2266/130749 objects degraded (1.733%); 3276/130749 objects misplaced (2.506%) 2024-09-13T14:54:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1941334129' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:54:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:56 smithi003 ceph-mon[104007]: osdmap e298: 8 total, 8 up, 8 in 2024-09-13T14:54:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2266/140190 objects degraded (1.616%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:56 smithi003 ceph-mon[104007]: pgmap v1047: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2266/130749 objects degraded (1.733%); 3276/130749 objects misplaced (2.506%) 2024-09-13T14:54:56.691 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:56.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:56.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:56.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:56.950 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:57.052 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:57.089 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:57.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:57.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:57.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:57.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:57.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:57.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:57.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:57.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:57.750 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:57.772 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:57.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:57.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:58.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:58.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:58.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:58.181 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:58.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:58.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:58.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:58.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:58.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:58.429 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:58.463 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:58.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:58.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:58.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:58.735 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:58.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:58.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:58.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:59.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:59.137 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:54:58 smithi019 ceph-mon[91296]: pgmap v1048: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2266/127119 objects degraded (1.783%); 3276/127119 objects misplaced (2.577%) 2024-09-13T14:54:59.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:54:58 smithi003 ceph-mon[107278]: pgmap v1048: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2266/127119 objects degraded (1.783%); 3276/127119 objects misplaced (2.577%) 2024-09-13T14:54:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:54:58 smithi003 ceph-mon[104007]: pgmap v1048: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2266/127119 objects degraded (1.783%); 3276/127119 objects misplaced (2.577%) 2024-09-13T14:54:59.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:59.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:59.383 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:59.409 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:59.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:59.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:59.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:54:59.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:59.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:54:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:59.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:59.864 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:54:59.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:59.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:00.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:00.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:00.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:00.279 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:00.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:00.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:00.407 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:00.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:00.536 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:00.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:00.654 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:55:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:55:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:00.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:00.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:00.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:00.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:00.937 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:01.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:01.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:01.170 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:01.200 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:01.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:00 smithi019 ceph-mon[91296]: pgmap v1049: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 2266/119322 objects degraded (1.899%); 3276/119322 objects misplaced (2.746%) 2024-09-13T14:55:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2266/119322 objects degraded (1.899%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:00 smithi003 ceph-mon[107278]: pgmap v1049: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 2266/119322 objects degraded (1.899%); 3276/119322 objects misplaced (2.746%) 2024-09-13T14:55:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2266/119322 objects degraded (1.899%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:00 smithi003 ceph-mon[104007]: pgmap v1049: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 2266/119322 objects degraded (1.899%); 3276/119322 objects misplaced (2.746%) 2024-09-13T14:55:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2266/119322 objects degraded (1.899%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:01.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:01.470 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:01.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:01.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:01.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:01.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:01.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:01.905 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:02.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:02.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:02.146 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:02.187 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:02.289 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:02.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.324 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:02.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:02.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:55:02.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:02.501 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:02.506 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:02.515 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:02.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:02.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:02.752 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:02.779 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:02.881 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:02.910 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:03.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:03.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:03.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:03.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:02 smithi003 ceph-mon[104007]: pgmap v1050: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 231 op/s; 2266/117663 objects degraded (1.926%); 2944/117663 objects misplaced (2.502%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:55:03.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:02 smithi003 ceph-mon[107278]: pgmap v1050: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 231 op/s; 2266/117663 objects degraded (1.926%); 2944/117663 objects misplaced (2.502%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:55:03.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:03.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:03.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:03.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:02 smithi019 ceph-mon[91296]: pgmap v1050: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 231 op/s; 2266/117663 objects degraded (1.926%); 2944/117663 objects misplaced (2.502%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:55:03.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:03.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:03.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:03.779 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:03.808 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:03.909 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:03.940 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:04.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:04.067 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:04.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:04.200 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:04.305 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:04.333 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:04.435 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:55:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:55:04] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-13T14:55:04.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:04.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:04.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:04.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:04.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:04.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:04.854 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:04.883 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:04.985 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:05.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:05.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:05.144 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:05.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:05.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:05.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:05.419 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:05.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:04 smithi019 ceph-mon[91296]: pgmap v1051: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 240 op/s; 2266/113976 objects degraded (1.988%); 2944/113976 objects misplaced (2.583%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:55:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:04 smithi019 ceph-mon[91296]: 11.3 scrub starts 2024-09-13T14:55:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:04 smithi003 ceph-mon[104007]: pgmap v1051: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 240 op/s; 2266/113976 objects degraded (1.988%); 2944/113976 objects misplaced (2.583%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:55:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:04 smithi003 ceph-mon[104007]: 11.3 scrub starts 2024-09-13T14:55:05.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:04 smithi003 ceph-mon[107278]: pgmap v1051: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 240 op/s; 2266/113976 objects degraded (1.988%); 2944/113976 objects misplaced (2.583%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:55:05.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:04 smithi003 ceph-mon[107278]: 11.3 scrub starts 2024-09-13T14:55:05.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.557 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:05.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:05.658 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:05.687 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:05.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:05.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:05.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:05.942 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:06.043 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:06.065 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:06.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:06.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2266/113976 objects degraded (1.988%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:06.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2266/113976 objects degraded (1.988%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:06.190 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:06.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:06.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:06.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2266/113976 objects degraded (1.988%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:06.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:06.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:06.587 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:06.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:06.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:06.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:06.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:06.967 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:06.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:06.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:07.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:07.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:07.290 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:07.316 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:07.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:07 smithi003 ceph-mon[107278]: pgmap v1052: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 875 op/s; 2266/104622 objects degraded (2.166%); 2654/104622 objects misplaced (2.537%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:55:07.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:07 smithi003 ceph-mon[104007]: pgmap v1052: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 875 op/s; 2266/104622 objects degraded (2.166%); 2654/104622 objects misplaced (2.537%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:55:07.418 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:07.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:07 smithi019 ceph-mon[91296]: pgmap v1052: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 875 op/s; 2266/104622 objects degraded (2.166%); 2654/104622 objects misplaced (2.537%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:55:07.441 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:07.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:07.576 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:07.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.718 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:07.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:07.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:07.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:07.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:08.074 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:08.176 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:08.211 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:08.214 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:08.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:08.344 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:08.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:08.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:08.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[107278]: 11.13 scrub starts 2024-09-13T14:55:08.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:55:08.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:55:08.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:08.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:08.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:08.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[104007]: 11.13 scrub starts 2024-09-13T14:55:08.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:55:08.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:55:08.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:08 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:08 smithi019 ceph-mon[91296]: 11.13 scrub starts 2024-09-13T14:55:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:55:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:55:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:08 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:08.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.478 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:08.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:08.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:08.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:08.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:08.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:08.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:08.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:08.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:09.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:09.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:09.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:09.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:09.293 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:09 smithi003 ceph-mon[107278]: pgmap v1053: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 823 op/s; 2266/101079 objects degraded (2.242%); 2654/101079 objects misplaced (2.626%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:55:09.294 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:09 smithi003 ceph-mon[107278]: 11.3 scrub starts 2024-09-13T14:55:09.294 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:09 smithi003 ceph-mon[104007]: pgmap v1053: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 823 op/s; 2266/101079 objects degraded (2.242%); 2654/101079 objects misplaced (2.626%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:55:09.294 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:09 smithi003 ceph-mon[104007]: 11.3 scrub starts 2024-09-13T14:55:09.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:09.354 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:09 smithi019 ceph-mon[91296]: pgmap v1053: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 823 op/s; 2266/101079 objects degraded (2.242%); 2654/101079 objects misplaced (2.626%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:55:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:09 smithi019 ceph-mon[91296]: 11.3 scrub starts 2024-09-13T14:55:09.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:09.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:09.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:09.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:09.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:09.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:09.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:10.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:10.042 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:10.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:10.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:10.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:10.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:10 smithi019 ceph-mon[91296]: 11.13 scrub starts 2024-09-13T14:55:10.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:55:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:55:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:10.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:10 smithi003 ceph-mon[104007]: 11.13 scrub starts 2024-09-13T14:55:10.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:10 smithi003 ceph-mon[107278]: 11.13 scrub starts 2024-09-13T14:55:10.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.466 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:10.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:10.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:10.597 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:10.698 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:10.734 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:10.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:10.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:10.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:11.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:11.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:11.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:11.305 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:11.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:11 smithi019 ceph-mon[91296]: pgmap v1054: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 2266/94140 objects degraded (2.407%); 2654/94140 objects misplaced (2.819%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:55:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:11 smithi019 ceph-mon[91296]: 11.3 scrub starts 2024-09-13T14:55:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2266/94140 objects degraded (2.407%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:11 smithi003 ceph-mon[104007]: pgmap v1054: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 2266/94140 objects degraded (2.407%); 2654/94140 objects misplaced (2.819%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:55:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:11 smithi003 ceph-mon[104007]: 11.3 scrub starts 2024-09-13T14:55:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2266/94140 objects degraded (2.407%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:11 smithi003 ceph-mon[107278]: pgmap v1054: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 2266/94140 objects degraded (2.407%); 2654/94140 objects misplaced (2.819%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:55:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:11 smithi003 ceph-mon[107278]: 11.3 scrub starts 2024-09-13T14:55:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2266/94140 objects degraded (2.407%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:11.440 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:11.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:11.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:11.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:11.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:11.837 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:11.866 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:11.968 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:11.995 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:12.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:12.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:12.211 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:55:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:55:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:55:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:55:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:117266 10.f e253/229 10:f2aa6f81:::benchmark_data_smithi003_197468_object537:head v 253'7261, mlcod=253'7258)", 2024-09-13T14:55:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:09.751012+0000", 2024-09-13T14:55:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 362.26802236999998, 2024-09-13T14:55:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.29645193800000003, 2024-09-13T14:55:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751012+0000", 2024-09-13T14:55:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751012+0000", 2024-09-13T14:55:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751013+0000", 2024-09-13T14:55:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0809999999999999e-06 2024-09-13T14:55:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751020+0000", 2024-09-13T14:55:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3069999999999997e-06 2024-09-13T14:55:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751032+0000", 2024-09-13T14:55:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1841999999999999e-05 2024-09-13T14:55:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751040+0000", 2024-09-13T14:55:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1170000000000004e-06 2024-09-13T14:55:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751087+0000", 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6502000000000003e-05 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037186+0000", 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28609978600000002 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.047389+0000", 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010202304000000001 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.047464+0000", 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4999000000000005e-05 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:117289 10.3 e253/242 10:cb5621ab:::benchmark_data_smithi003_197468_object560:head v 253'7331, mlcod=238'1751)", 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:09.759160+0000", 2024-09-13T14:55:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 362.25987431900001, 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28967668699999999, 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.759160+0000", 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.759160+0000", 2024-09-13T14:55:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.759161+0000", 2024-09-13T14:55:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.443e-06 2024-09-13T14:55:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.759170+0000", 2024-09-13T14:55:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7700000000000007e-06 2024-09-13T14:55:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037164+0000", 2024-09-13T14:55:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27799370499999998 2024-09-13T14:55:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037181+0000", 2024-09-13T14:55:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6906999999999999e-05 2024-09-13T14:55:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037704+0000", 2024-09-13T14:55:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052296999999999999 2024-09-13T14:55:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037727+0000", 2024-09-13T14:55:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3213e-05 2024-09-13T14:55:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.048755+0000", 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011028675 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.048836+0000", 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1003999999999999e-05 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:123357 10.5 e253/229 10:a6ed905f:::benchmark_data_smithi003_197468_object6628:head v 253'7703, mlcod=253'7701)", 2024-09-13T14:55:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.382685+0000", 2024-09-13T14:55:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 354.63634879099999, 2024-09-13T14:55:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26448326100000003, 2024-09-13T14:55:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382685+0000", 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382685+0000", 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382686+0000", 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1310000000000001e-06 2024-09-13T14:55:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382693+0000", 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0539999999999997e-06 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382707+0000", 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3354e-05 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382715+0000", 2024-09-13T14:55:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3319999999999992e-06 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382767+0000", 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1663000000000003e-05 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382788+0000", 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1077000000000002e-05 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.386308+0000", 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035196979999999999 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647169+0000", 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26086095199999998 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:123358 10.4 e253/229 10:2c9f3a1b:::benchmark_data_smithi003_197468_object6629:head v 253'7638, mlcod=253'7636)", 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.383164+0000", 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 354.63586995700001, 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.264014205, 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383164+0000", 2024-09-13T14:55:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383164+0000", 2024-09-13T14:55:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383165+0000", 2024-09-13T14:55:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8899999999999999e-07 2024-09-13T14:55:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383171+0000", 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7130000000000002e-06 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383180+0000", 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5670000000000001e-06 2024-09-13T14:55:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383186+0000", 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9830000000000001e-06 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383257+0000", 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0498000000000002e-05 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383275+0000", 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8218000000000001e-05 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647139+0000", 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26386380999999998 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647178+0000", 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9727000000000001e-05 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:123364 10.3 e253/242 10:c110b7c7:::benchmark_data_smithi003_197468_object6635:head v 253'7744, mlcod=238'1751)", 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.389216+0000", 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 354.62981812800001, 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27996448000000002, 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389216+0000", 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389216+0000", 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389217+0000", 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1629999999999999e-06 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389225+0000", 2024-09-13T14:55:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1030000000000002e-06 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647084+0000", 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.257858898 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647102+0000", 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8372e-05 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647166+0000", 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3359000000000001e-05 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647193+0000", 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6716999999999998e-05 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.669139+0000", 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021946619000000001 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.669180+0000", 2024-09-13T14:55:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1248999999999998e-05 2024-09-13T14:55:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149380 10.3 e255/242 10:c6822046:::benchmark_data_smithi003_197468_object32651:head v 255'9391, mlcod=238'1751)", 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.297146+0000", 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 324.72188784999997, 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34267334700000002, 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297146+0000", 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297146+0000", 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297147+0000", 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3900000000000004e-07 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297152+0000", 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2170000000000002e-06 2024-09-13T14:55:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297161+0000", 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7830000000000004e-06 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297167+0000", 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2960000000000004e-06 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299041+0000", 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018741980000000001 2024-09-13T14:55:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299067+0000", 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5769e-05 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.639779+0000", 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34071200499999998 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.639820+0000", 2024-09-13T14:55:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0439999999999999e-05 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149384 10.3 e255/242 10:c87a8b94:::benchmark_data_smithi003_197468_object32655:head v 255'9392, mlcod=238'1751)", 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.305345+0000", 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 324.71368919000003, 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33450042699999999, 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305345+0000", 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305345+0000", 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305346+0000", 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.012e-06 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305353+0000", 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1930000000000004e-06 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305365+0000", 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1702000000000001e-05 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305373+0000", 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8620000000000003e-06 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305428+0000", 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5164999999999998e-05 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305450+0000", 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2300999999999999e-05 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.639825+0000", 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.334374738 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.639845+0000", 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0454000000000001e-05 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149393 10.4 e255/229 10:29621cc7:::benchmark_data_smithi003_197468_object32664:head v 255'9294, mlcod=255'9292)", 2024-09-13T14:55:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.308254+0000", 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 324.71078036099999, 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33786957400000001, 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308254+0000", 2024-09-13T14:55:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308254+0000", 2024-09-13T14:55:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308255+0000", 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7499999999999998e-07 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308261+0000", 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0929999999999999e-06 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308272+0000", 2024-09-13T14:55:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1294999999999999e-05 2024-09-13T14:55:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308280+0000", 2024-09-13T14:55:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5050000000000001e-06 2024-09-13T14:55:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308435+0000", 2024-09-13T14:55:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000155883 2024-09-13T14:55:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308455+0000", 2024-09-13T14:55:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9675000000000001e-05 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646079+0000", 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33762412800000002 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646123+0000", 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4020000000000002e-05 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149388 10.7 e255/249 10:ea963fcf:::benchmark_data_smithi003_197468_object32659:head v 255'9240, mlcod=0'0)", 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.308305+0000", 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 324.71072916700001, 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.333896784, 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308305+0000", 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308305+0000", 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308305+0000", 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.08e-07 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308309+0000", 2024-09-13T14:55:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5750000000000001e-06 2024-09-13T14:55:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308316+0000", 2024-09-13T14:55:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6579999999999996e-06 2024-09-13T14:55:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308320+0000", 2024-09-13T14:55:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3630000000000004e-06 2024-09-13T14:55:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309186+0000", 2024-09-13T14:55:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00086598100000000004 2024-09-13T14:55:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309209+0000", 2024-09-13T14:55:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2806000000000002e-05 2024-09-13T14:55:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642156+0000", 2024-09-13T14:55:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33294702599999998 2024-09-13T14:55:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642202+0000", 2024-09-13T14:55:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5967000000000002e-05 2024-09-13T14:55:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149395 10.f e255/229 10:ff0ca942:::benchmark_data_smithi003_197468_object32666:head v 255'9241, mlcod=255'9239)", 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.308370+0000", 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 324.71066373000002, 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.333858402, 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308370+0000", 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308370+0000", 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308371+0000", 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4299999999999998e-07 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308375+0000", 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2119999999999997e-06 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308383+0000", 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8269999999999998e-06 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308388+0000", 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2089999999999996e-06 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.632819+0000", 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32443107599999998 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.632844+0000", 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4610999999999999e-05 2024-09-13T14:55:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642205+0000", 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0093610589999999997 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642229+0000", 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3964999999999998e-05 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149387 10.5 e255/229 10:a0bfa116:::benchmark_data_smithi003_197468_object32658:head v 255'9349, mlcod=255'9346)", 2024-09-13T14:55:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.312818+0000", 2024-09-13T14:55:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 324.70621652599999, 2024-09-13T14:55:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.331070115, 2024-09-13T14:55:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.312818+0000", 2024-09-13T14:55:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.312818+0000", 2024-09-13T14:55:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.312818+0000", 2024-09-13T14:55:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.569999999999999e-07 2024-09-13T14:55:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.312826+0000", 2024-09-13T14:55:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3000000000000004e-06 2024-09-13T14:55:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.633848+0000", 2024-09-13T14:55:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32102176399999999 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.633866+0000", 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8142999999999999e-05 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.633934+0000", 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8182999999999994e-05 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.633950+0000", 2024-09-13T14:55:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5962999999999999e-05 2024-09-13T14:55:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.643801+0000", 2024-09-13T14:55:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098514090000000002 2024-09-13T14:55:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.643888+0000", 2024-09-13T14:55:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6396000000000004e-05 2024-09-13T14:55:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:216356 10.3 e259/242 10:cb3fb980:::benchmark_data_smithi003_197468_object99627:head v 259'13586, mlcod=238'1751)", 2024-09-13T14:55:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:51:04.655985+0000", 2024-09-13T14:55:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 247.36304887399999, 2024-09-13T14:55:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22390685099999999, 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.655985+0000", 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.655985+0000", 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.655986+0000", 2024-09-13T14:55:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.238e-06 2024-09-13T14:55:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.655996+0000", 2024-09-13T14:55:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3840000000000003e-06 2024-09-13T14:55:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.656011+0000", 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5421999999999999e-05 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.656020+0000", 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4100000000000008e-06 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.656080+0000", 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0495000000000003e-05 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.656105+0000", 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5001e-05 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.879847+0000", 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.223741987 2024-09-13T14:55:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.879892+0000", 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4913999999999997e-05 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:216349 10.7 e259/249 10:e879c7de:::benchmark_data_smithi003_197468_object99620:head v 259'13401, mlcod=0'0)", 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:51:04.657014+0000", 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 247.362019933, 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22675177799999999, 2024-09-13T14:55:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.657014+0000", 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.657014+0000", 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.657015+0000", 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4900000000000004e-07 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.657022+0000", 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.567e-06 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.657033+0000", 2024-09-13T14:55:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1518999999999999e-05 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.657040+0000", 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2409999999999998e-06 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.657114+0000", 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3650000000000001e-05 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.657138+0000", 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3620000000000001e-05 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.883720+0000", 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22658251500000001 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:51:04.883766+0000", 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5717000000000003e-05 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:23153 11.0 11:03cd4422:::benchmark_data_smithi003_229782_object23152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:52:09.695207+0000", 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 182.323826811, 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27937496499999998, 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23153 2024-09-13T14:55:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.695207+0000", 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.695207+0000", 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.695208+0000", 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4199999999999995e-07 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.695228+0000", 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0112e-05 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.806793+0000", 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.111564738 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.806804+0000", 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1537e-05 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.807428+0000", 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00062398 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.807565+0000", 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013632100000000001 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.807695+0000", 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013002000000000001 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.811115+0000", 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034201460000000002 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.811164+0000", 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8902e-05 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.974492+0000", 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16332833499999999 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.974542+0000", 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0024000000000001e-05 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.974582+0000", 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0207999999999999e-05 2024-09-13T14:55:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:101345 11.19 11:98a4befa:::benchmark_data_smithi003_229782_object101344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:10.400177+0000", 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 121.618856577, 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.224713941, 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 101345 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.400177+0000", 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.400177+0000", 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.400178+0000", 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5799999999999998e-07 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.400199+0000", 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0818999999999998e-05 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.400210+0000", 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0533000000000001e-05 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.400217+0000", 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9670000000000004e-06 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.400284+0000", 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7045000000000006e-05 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.400407+0000", 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012300899999999999 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.400527+0000", 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120584 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.402963+0000", 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002435595 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.403754+0000", 2024-09-13T14:55:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00079083300000000001 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.624772+0000", 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22101806399999999 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.624827+0000", 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4669999999999997e-05 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:10.624891+0000", 2024-09-13T14:55:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4863999999999994e-05 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:206944 11.0 11:0327db45:::benchmark_data_smithi003_229782_object84564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:00.491663+0000", 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.527370964999999, 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22873053199999999, 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:55:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 206944 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.491663+0000", 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.491663+0000", 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.491664+0000", 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1500000000000005e-07 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.491666+0000", 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.717e-06 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.491674+0000", 2024-09-13T14:55:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4440000000000002e-06 2024-09-13T14:55:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.491680+0000", 2024-09-13T14:55:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7450000000000001e-06 2024-09-13T14:55:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.495148+0000", 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034686449999999998 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.495279+0000", 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130512 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.495801+0000", 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052231900000000004 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:55:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.503403+0000", 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076022090000000004 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.510250+0000", 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006846502 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.720265+0000", 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21001497499999999 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.720345+0000", 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0365999999999996e-05 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.720394+0000", 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8582999999999999e-05 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:206950 11.0 11:06d8c19a:::benchmark_data_smithi003_229782_object84570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:00.497129+0000", 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.521905557, 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22437272, 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 206950 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.497129+0000", 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.497129+0000", 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.497130+0000", 2024-09-13T14:55:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.125e-06 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.497134+0000", 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7690000000000004e-06 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.497147+0000", 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2635e-05 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.497154+0000", 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2400000000000001e-06 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.499982+0000", 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028272499999999999 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.500114+0000", 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013280599999999999 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:55:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.500606+0000", 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049173000000000001 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.503490+0000", 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028840350000000001 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.512941+0000", 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094512810000000006 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.721379+0000", 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20843761099999999 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.721462+0000", 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3057000000000001e-05 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:00.721501+0000", 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9181000000000003e-05 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:211637 11.b 11:d48e3873:::benchmark_data_smithi003_229782_object89257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.888486+0000", 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.1305479470000002, 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.45271259000000003, 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 211637 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888486+0000", 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888486+0000", 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888488+0000", 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0719999999999998e-06 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888495+0000", 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0899999999999999e-06 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888517+0000", 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2178e-05 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888527+0000", 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5289999999999997e-06 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893270+0000", 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047428959999999999 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893401+0000", 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013092099999999999 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893523+0000", 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012244799999999999 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899806+0000", 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006282625 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899844+0000", 2024-09-13T14:55:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7957000000000003e-05 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.341048+0000", 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44120459499999998 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.341122+0000", 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3960000000000003e-05 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.341199+0000", 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6318999999999999e-05 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:211648 11.0 11:011e7020:::benchmark_data_smithi003_229782_object89268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.898155+0000", 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.1208791829999996, 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.43924299999999999, 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 211648 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898155+0000", 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898155+0000", 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898156+0000", 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1800000000000005e-07 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898159+0000", 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7100000000000001e-06 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898169+0000", 2024-09-13T14:55:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.166e-06 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898174+0000", 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1619999999999999e-06 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.905754+0000", 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075797750000000004 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.905865+0000", 2024-09-13T14:55:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011133000000000001 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.906402+0000", 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053722100000000005 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.913016+0000", 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006614389 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:55:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.913100+0000", 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3379000000000002e-05 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.337269+0000", 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.42416935700000002 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.337346+0000", 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6363000000000001e-05 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.337398+0000", 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2330000000000002e-05 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:211652 11.b 11:dd38e0a4:::benchmark_data_smithi003_229782_object89272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.899777+0000", 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.119257245, 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44725128199999997, 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 211652 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899777+0000", 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899777+0000", 2024-09-13T14:55:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899778+0000", 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1119999999999999e-06 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899783+0000", 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5310000000000001e-06 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:55:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899796+0000", 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2126999999999999e-05 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899804+0000", 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9980000000000003e-06 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900561+0000", 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00075782300000000003 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900706+0000", 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014433100000000001 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900833+0000", 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012682500000000001 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.904800+0000", 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003967567 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.909921+0000", 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051206170000000001 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.346915+0000", 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.43699400100000002 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.346981+0000", 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6519000000000005e-05 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:55:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:55:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:55:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.347028+0000", 2024-09-13T14:55:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6830999999999999e-05 2024-09-13T14:55:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:55:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:55:12.332 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:55:12.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:12.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:12.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:12.503 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:12.606 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:12.639 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:12.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:12.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:12.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:12.890 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:12.923 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:13.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:13.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:13.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:13.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:13.314 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:13.347 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:13 smithi019 ceph-mon[91296]: pgmap v1055: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 741 op/s; 2266/92631 objects degraded (2.446%); 2313/92631 objects misplaced (2.497%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:55:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:13 smithi003 ceph-mon[104007]: pgmap v1055: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 741 op/s; 2266/92631 objects degraded (2.446%); 2313/92631 objects misplaced (2.497%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:55:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:13 smithi003 ceph-mon[107278]: pgmap v1055: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 741 op/s; 2266/92631 objects degraded (2.446%); 2313/92631 objects misplaced (2.497%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:55:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:13.450 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:13.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:55:13.478 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:13.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:13.618 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:13.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:13.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:13.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:13.872 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:13.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:14.009 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:14.043 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:14.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:14.190 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:14.223 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:14 smithi003 ceph-mon[104007]: 11.13 scrub starts 2024-09-13T14:55:14.223 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:55:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:55:14] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T14:55:14.223 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:14 smithi003 ceph-mon[107278]: 11.13 scrub starts 2024-09-13T14:55:14.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:14.325 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:14.378 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:14 smithi019 ceph-mon[91296]: 11.13 scrub starts 2024-09-13T14:55:14.403 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:55:14.403 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:14.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:14.451 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:55:14.452 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:14.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:14.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:55:14.490 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:14.536 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:55:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:14.572 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:55:14.572 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:14.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:14.600 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:14.610 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:55:14.610 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:14.650 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:55:14.650 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:14.696 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:55:14.696 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:14.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:14.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:14.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:14.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:14.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:15.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:15.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:15.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:15.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:15.195 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:55:15.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:15.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:15.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:15.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:15.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:15.547 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:15.601 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:15 smithi019 ceph-mon[91296]: pgmap v1056: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 2266/88965 objects degraded (2.547%); 2313/88965 objects misplaced (2.600%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:55:15.601 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4077897530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:15.601 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1022752126' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:15.601 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:15 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:15.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:15 smithi003 ceph-mon[107278]: pgmap v1056: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 2266/88965 objects degraded (2.547%); 2313/88965 objects misplaced (2.600%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:55:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4077897530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1022752126' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:15 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:15 smithi003 ceph-mon[104007]: pgmap v1056: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 2266/88965 objects degraded (2.547%); 2313/88965 objects misplaced (2.600%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:55:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4077897530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:15.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1022752126' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:15.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:15 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:15.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:15.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:15.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:15.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:15.970 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:16.005 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:16.005 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:16.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:16.104 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:16.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:16.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:16.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:16.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:16 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:16 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:55:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:16 smithi003 ceph-mon[107278]: osdmap e299: 8 total, 8 up, 8 in 2024-09-13T14:55:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2266/88965 objects degraded (2.547%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:16 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:16 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:55:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:16 smithi003 ceph-mon[104007]: osdmap e299: 8 total, 8 up, 8 in 2024-09-13T14:55:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2266/88965 objects degraded (2.547%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:16.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:16.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:16.621 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:16 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:16.622 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:16 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:55:16.622 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:16 smithi019 ceph-mon[91296]: osdmap e299: 8 total, 8 up, 8 in 2024-09-13T14:55:16.622 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2266/88965 objects degraded (2.547%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:16.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:16.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:16.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:16.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:16.909 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:16.942 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:16.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:55:17.044 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:17.091 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:17.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:17.224 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:17.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.382 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:17.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:17.434 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:17.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:17.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:17.569 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:17 smithi019 ceph-mon[91296]: pgmap v1058: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 2101/79017 objects degraded (2.659%); 2101/79017 objects misplaced (2.659%); 1.2 MiB/s, 35 objects/s recovering 2024-09-13T14:55:17.624 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:17.654 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:17.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:17.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:17 smithi003 ceph-mon[107278]: pgmap v1058: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 2101/79017 objects degraded (2.659%); 2101/79017 objects misplaced (2.659%); 1.2 MiB/s, 35 objects/s recovering 2024-09-13T14:55:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:17 smithi003 ceph-mon[104007]: pgmap v1058: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 2101/79017 objects degraded (2.659%); 2101/79017 objects misplaced (2.659%); 1.2 MiB/s, 35 objects/s recovering 2024-09-13T14:55:17.756 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:17.790 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:17.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:17.923 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:18.025 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:18.056 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:18.158 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:18.190 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:18.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.329 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:18.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:18.430 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:18.464 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:18.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:18.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:18.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:18.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:18.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:18.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:18.881 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:18.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:19.014 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:19.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:19.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:19.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:19.287 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:19.351 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:19 smithi019 ceph-mon[91296]: pgmap v1059: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 2101/75501 objects degraded (2.783%); 2101/75501 objects misplaced (2.783%); 1.2 MiB/s, 35 objects/s recovering 2024-09-13T14:55:19.389 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:19.422 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:19.504 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:19.504 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:19.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:19.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:19.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:19 smithi003 ceph-mon[104007]: pgmap v1059: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 2101/75501 objects degraded (2.783%); 2101/75501 objects misplaced (2.783%); 1.2 MiB/s, 35 objects/s recovering 2024-09-13T14:55:19.591 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:19 smithi003 ceph-mon[107278]: pgmap v1059: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 2101/75501 objects degraded (2.783%); 2101/75501 objects misplaced (2.783%); 1.2 MiB/s, 35 objects/s recovering 2024-09-13T14:55:19.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:19.702 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:19.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:19.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:19.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:19.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:19.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:20.094 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:20.125 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:20.197 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:55:20.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:20.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:20.370 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.414 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:20.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:20.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:55:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:55:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:20.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:20.563 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:20.604 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:20.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:55:20.665 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:20.692 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:20.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:20.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:20.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:20.955 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:20.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:21.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:21.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:21.118 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:21.199 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:21.235 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:55:21.239 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:21.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:21.478 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.505 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:21.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:21.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:21.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:55:21.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:21.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:21 smithi019 ceph-mon[91296]: pgmap v1060: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 2101/67680 objects degraded (3.104%); 2101/67680 objects misplaced (3.104%); 1.2 MiB/s, 35 objects/s recovering 2024-09-13T14:55:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2606234594' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:55:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:21 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:55:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2101/67680 objects degraded (3.104%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:21 smithi003 ceph-mon[107278]: pgmap v1060: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 2101/67680 objects degraded (3.104%); 2101/67680 objects misplaced (3.104%); 1.2 MiB/s, 35 objects/s recovering 2024-09-13T14:55:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2606234594' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:55:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:21 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:55:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2101/67680 objects degraded (3.104%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:21 smithi003 ceph-mon[104007]: pgmap v1060: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 2101/67680 objects degraded (3.104%); 2101/67680 objects misplaced (3.104%); 1.2 MiB/s, 35 objects/s recovering 2024-09-13T14:55:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2606234594' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:55:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:21 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:55:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2101/67680 objects degraded (3.104%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:21.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:21.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:21.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.939 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:21.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:22.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:22.065 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:22.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:22.196 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:22.298 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.331 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:22.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:22.432 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:22.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:22 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:55:22.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:22 smithi003 ceph-mon[107278]: osdmap e300: 8 total, 8 up, 8 in 2024-09-13T14:55:22.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:22 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:55:22.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:22 smithi003 ceph-mon[104007]: osdmap e300: 8 total, 8 up, 8 in 2024-09-13T14:55:22.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:22.522 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:22 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:55:22.522 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:22 smithi019 ceph-mon[91296]: osdmap e300: 8 total, 8 up, 8 in 2024-09-13T14:55:22.571 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:22.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:22.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:22.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:55:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:22.822 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:22.873 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:22.917 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:22.995 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:22.995 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:23.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:23.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:23.173 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:23.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.219 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:23.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:23.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.352 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:23.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:23.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:23.480 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:23.582 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:23.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:23 smithi003 ceph-mon[107278]: pgmap v1062: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 1937/66009 objects degraded (2.934%); 1937/66009 objects misplaced (2.934%); 192 KiB/s, 44 objects/s recovering 2024-09-13T14:55:23.603 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:23 smithi003 ceph-mon[104007]: pgmap v1062: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 1937/66009 objects degraded (2.934%); 1937/66009 objects misplaced (2.934%); 192 KiB/s, 44 objects/s recovering 2024-09-13T14:55:23.618 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:23.680 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:23 smithi019 ceph-mon[91296]: pgmap v1062: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 1937/66009 objects degraded (2.934%); 1937/66009 objects misplaced (2.934%); 192 KiB/s, 44 objects/s recovering 2024-09-13T14:55:23.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:55:23.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:23.749 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:23.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:23.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:23.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:24.040 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:24.099 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:24.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:24.176 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:24.199 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:55:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:55:24] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T14:55:24.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:24.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:24.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:24.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:24.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:24.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:24.699 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:24.725 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:24.744 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:24.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:24.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:24.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:25.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:25.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:25.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:25.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:25.266 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:55:25.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:25.366 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:25 smithi019 ceph-mon[91296]: pgmap v1063: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 1937/62082 objects degraded (3.120%); 1937/62082 objects misplaced (3.120%); 176 KiB/s, 40 objects/s recovering 2024-09-13T14:55:25.366 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:25 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/424024598' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:25.366 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:25 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:25.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:25.447 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:25.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:25.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:25 smithi003 ceph-mon[107278]: pgmap v1063: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 1937/62082 objects degraded (3.120%); 1937/62082 objects misplaced (3.120%); 176 KiB/s, 40 objects/s recovering 2024-09-13T14:55:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:25 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/424024598' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:25 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:25 smithi003 ceph-mon[104007]: pgmap v1063: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 1937/62082 objects degraded (3.120%); 1937/62082 objects misplaced (3.120%); 176 KiB/s, 40 objects/s recovering 2024-09-13T14:55:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:25 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/424024598' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:25 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:25.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.715 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:25.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:25.816 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:25.844 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:25.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:25.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:26.100 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:26.135 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:26.237 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:26.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:26.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:26.412 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:26.497 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:26.497 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:26.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:26.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:26.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:26 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:26 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:26 smithi019 ceph-mon[91296]: osdmap e301: 8 total, 8 up, 8 in 2024-09-13T14:55:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1937/62082 objects degraded (3.120%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:26.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:26 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:26.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:26 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:26.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:26 smithi003 ceph-mon[107278]: osdmap e301: 8 total, 8 up, 8 in 2024-09-13T14:55:26.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1937/62082 objects degraded (3.120%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:26.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:26 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:26.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:26 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:26.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:26 smithi003 ceph-mon[104007]: osdmap e301: 8 total, 8 up, 8 in 2024-09-13T14:55:26.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1937/62082 objects degraded (3.120%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.685 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:26.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:26.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:26.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:26.926 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:26.963 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:27.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:27.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:27.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:27.249 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:27.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:27.379 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:27 smithi019 ceph-mon[91296]: pgmap v1065: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 894 op/s; 1937/54033 objects degraded (3.585%); 1937/54033 objects misplaced (3.585%); 0 B/s, 20 objects/s recovering 2024-09-13T14:55:27.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:27.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:27.414 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:55:27.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:27.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:27 smithi003 ceph-mon[104007]: pgmap v1065: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 894 op/s; 1937/54033 objects degraded (3.585%); 1937/54033 objects misplaced (3.585%); 0 B/s, 20 objects/s recovering 2024-09-13T14:55:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:27 smithi003 ceph-mon[107278]: pgmap v1065: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 894 op/s; 1937/54033 objects degraded (3.585%); 1937/54033 objects misplaced (3.585%); 0 B/s, 20 objects/s recovering 2024-09-13T14:55:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:27.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:27.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:27.771 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:27.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:27.827 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:27.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:27.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:28.067 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:28.107 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:28.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:55:28.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:55:28.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:55:28.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:55:28.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:55:28.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:55:28.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:55:28.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:55:28.175 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:28.176 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:28.176 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:28.176 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:28.176 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:28.176 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:28.176 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:28.176 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:28.209 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:28.244 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:28.347 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:28.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:28.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1412381271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1412381271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:28.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:28.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:28.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:28.661 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:28.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1412381271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:28.762 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:28.794 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:28.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:28.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:29.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:29.076 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:29.178 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:29.315 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:29.348 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:29.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:29 smithi019 ceph-mon[91296]: pgmap v1066: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1013 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 791 op/s; 1751/48687 objects degraded (3.596%); 1751/48687 objects misplaced (3.596%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:29.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:29.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:29.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:29.609 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:29 smithi003 ceph-mon[107278]: pgmap v1066: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1013 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 791 op/s; 1751/48687 objects degraded (3.596%); 1751/48687 objects misplaced (3.596%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:29.609 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:29 smithi003 ceph-mon[104007]: pgmap v1066: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1013 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 791 op/s; 1751/48687 objects degraded (3.596%); 1751/48687 objects misplaced (3.596%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:29.637 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:29.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:55:29.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:29.765 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:29.775 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:55:29.775 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:55:29.775 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:55:29.775 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:55:29.775 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:55:29.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:29.897 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:29.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:30.030 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:30.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:30.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:30.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:30.308 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:30.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:30.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:55:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:55:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:30.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:30.489 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:30.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:55:30.591 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.621 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:30.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:30.722 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:30.758 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:30.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:30.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:30.913 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:31.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:31.036 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:31.138 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.168 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:31.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:31.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:31.305 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:55:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:31.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:31.410 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:31.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:31.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:31.602 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:31.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:55:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:31 smithi019 ceph-mon[91296]: pgmap v1067: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 844 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 1751/40548 objects degraded (4.318%); 1751/40548 objects misplaced (4.318%); 0 B/s, 21 objects/s recovering 2024-09-13T14:55:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4063612218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:55:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:31 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:55:31.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1751/40548 objects degraded (4.318%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:31.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:31 smithi003 ceph-mon[107278]: pgmap v1067: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 844 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 1751/40548 objects degraded (4.318%); 1751/40548 objects misplaced (4.318%); 0 B/s, 21 objects/s recovering 2024-09-13T14:55:31.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4063612218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:55:31.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:31 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:55:31.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1751/40548 objects degraded (4.318%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:31.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:31 smithi003 ceph-mon[104007]: pgmap v1067: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 844 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 1751/40548 objects degraded (4.318%); 1751/40548 objects misplaced (4.318%); 0 B/s, 21 objects/s recovering 2024-09-13T14:55:31.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4063612218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:55:31.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:31 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:55:31.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1751/40548 objects degraded (4.318%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:31.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:31.737 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:31.839 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:31.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:32.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:32.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:32.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:32.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:32.293 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:32.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:32.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:32.536 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:32.564 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:32.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:32 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:55:32.584 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:32 smithi019 ceph-mon[91296]: osdmap e302: 8 total, 8 up, 8 in 2024-09-13T14:55:32.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:32 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:55:32.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:32 smithi003 ceph-mon[107278]: osdmap e302: 8 total, 8 up, 8 in 2024-09-13T14:55:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:32 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:55:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:32 smithi003 ceph-mon[104007]: osdmap e302: 8 total, 8 up, 8 in 2024-09-13T14:55:32.695 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:32.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:32.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:32.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:32.980 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:32.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:33.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:33.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:33.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:33.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:33.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:33.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:33.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:33.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:33.466 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:33.499 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:33.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:33.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:33.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:33 smithi019 ceph-mon[91296]: pgmap v1069: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 810 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 1563/38946 objects degraded (4.013%); 1563/38946 objects misplaced (4.013%); 0 B/s, 46 objects/s recovering 2024-09-13T14:55:33.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:33 smithi003 ceph-mon[107278]: pgmap v1069: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 810 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 1563/38946 objects degraded (4.013%); 1563/38946 objects misplaced (4.013%); 0 B/s, 46 objects/s recovering 2024-09-13T14:55:33.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:33 smithi003 ceph-mon[104007]: pgmap v1069: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 810 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 1563/38946 objects degraded (4.013%); 1563/38946 objects misplaced (4.013%); 0 B/s, 46 objects/s recovering 2024-09-13T14:55:33.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:33.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:33.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:34.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:34.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:34.198 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:34.232 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:34.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:34.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:55:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:55:34] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:55:34.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:34.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:34.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:34.704 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:55:34.704 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214357 9.d 9:b77cc858:::benchmark_data_smithi003_156985_object93441:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.623932+0000", 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 574.07403799400004, 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0751467219999999, 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214357 2024-09-13T14:55:34.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623932+0000", 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623932+0000", 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.706 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623930+0000", 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623937+0000", 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8650000000000003e-06 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623949+0000", 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2680999999999999e-05 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.707 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.623955+0000", 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.032e-06 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.624003+0000", 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7698999999999999e-05 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.624010+0000", 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8909999999999998e-06 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.708 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.661403+0000", 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0373926769999999 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.661489+0000", 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5983999999999995e-05 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.661566+0000", 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7749000000000004e-05 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.709 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.671062+0000", 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0094950069999999998 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698569+0000", 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.027507851 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698627+0000", 2024-09-13T14:55:34.710 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7685000000000001e-05 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698863+0000", 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00023623399999999999 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698931+0000", 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7773000000000003e-05 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699078+0000", 2024-09-13T14:55:34.711 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000147312 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214375 9.d 9:be731116:::benchmark_data_smithi003_156985_object93459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.661092+0000", 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 574.03687758900003, 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.038199892, 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.712 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214375 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661092+0000", 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661092+0000", 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661090+0000", 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661102+0000", 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2103000000000001e-05 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.714 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661127+0000", 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5117e-05 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661138+0000", 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0953e-05 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661226+0000", 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7379000000000005e-05 2024-09-13T14:55:34.715 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.661239+0000", 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3434e-05 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.663711+0000", 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.002471949 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.716 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.663792+0000", 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0826000000000002e-05 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.663877+0000", 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5036000000000004e-05 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675347+0000", 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011470173 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698670+0000", 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.023323282000000001 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698715+0000", 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5213000000000002e-05 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699124+0000", 2024-09-13T14:55:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000408985 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699157+0000", 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2138000000000003e-05 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699292+0000", 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013532799999999999 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214377 9.d 9:b52ded65:::benchmark_data_smithi003_156985_object93461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.664805+0000", 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 574.03316484100003, 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.034793217, 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:55:34.720 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214377 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664805+0000", 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664805+0000", 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.721 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664804+0000", 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664809+0000", 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.887e-06 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664819+0000", 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0817e-05 2024-09-13T14:55:34.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664827+0000", 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0759999999999997e-06 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664891+0000", 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4422000000000001e-05 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.664898+0000", 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8000000000000001e-06 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.666066+0000", 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.00116863 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.666207+0000", 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014029099999999999 2024-09-13T14:55:34.724 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.666345+0000", 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013788900000000001 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675357+0000", 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0090126630000000006 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.725 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698783+0000", 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.023425537999999999 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.698825+0000", 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1802999999999999e-05 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.726 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699411+0000", 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00058682999999999999 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699469+0000", 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7234999999999997e-05 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699598+0000", 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012936800000000001 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.727 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214394 9.d 9:b3e3fb24:::benchmark_data_smithi003_156985_object93478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.688931+0000", 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 574.00903887499999, 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.010993375, 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.728 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:55:34.729 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:55:34.729 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214394 2024-09-13T14:55:34.729 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.729 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.729 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.729 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.729 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688931+0000", 2024-09-13T14:55:34.729 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.729 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688931+0000", 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688929+0000", 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.730 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.731 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.731 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688936+0000", 2024-09-13T14:55:34.731 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3320000000000003e-06 2024-09-13T14:55:34.731 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.731 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.731 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.731 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688955+0000", 2024-09-13T14:55:34.731 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.853e-05 2024-09-13T14:55:34.732 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.732 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.732 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.732 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.688962+0000", 2024-09-13T14:55:34.732 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4590000000000001e-06 2024-09-13T14:55:34.732 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.733 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.733 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.733 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689081+0000", 2024-09-13T14:55:34.733 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000119159 2024-09-13T14:55:34.733 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.733 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.734 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:55:34.734 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689185+0000", 2024-09-13T14:55:34.734 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010335 2024-09-13T14:55:34.734 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.734 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.734 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.734 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.668846+0000", 2024-09-13T14:55:34.735 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97966176900000002 2024-09-13T14:55:34.735 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.735 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.735 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.735 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.669010+0000", 2024-09-13T14:55:34.735 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016329499999999999 2024-09-13T14:55:34.736 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.736 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.736 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:55:34.736 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.669126+0000", 2024-09-13T14:55:34.736 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011607000000000001 2024-09-13T14:55:34.736 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.737 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.737 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.737 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.679730+0000", 2024-09-13T14:55:34.737 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010604165 2024-09-13T14:55:34.737 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.737 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.739 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.739 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699355+0000", 2024-09-13T14:55:34.739 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.019625423999999999 2024-09-13T14:55:34.739 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.739 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.740 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.740 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699643+0000", 2024-09-13T14:55:34.740 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000287638 2024-09-13T14:55:34.740 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.740 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.741 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.741 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699741+0000", 2024-09-13T14:55:34.741 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7619000000000002e-05 2024-09-13T14:55:34.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.741 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.741 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.742 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699790+0000", 2024-09-13T14:55:34.742 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9363000000000002e-05 2024-09-13T14:55:34.742 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.742 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.742 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.742 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699924+0000", 2024-09-13T14:55:34.743 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013407499999999999 2024-09-13T14:55:34.743 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.743 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.743 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.743 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.744 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.744 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214396 9.d 9:bf725953:::benchmark_data_smithi003_156985_object93480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:55:34.744 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.689352+0000", 2024-09-13T14:55:34.744 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 574.00861747900001, 2024-09-13T14:55:34.744 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0107824670000001, 2024-09-13T14:55:34.744 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.745 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.745 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.745 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:55:34.745 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:55:34.745 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214396 2024-09-13T14:55:34.745 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.746 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.746 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.746 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.746 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689352+0000", 2024-09-13T14:55:34.746 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.746 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.747 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.747 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.747 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689352+0000", 2024-09-13T14:55:34.747 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.747 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.747 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.747 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.748 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689351+0000", 2024-09-13T14:55:34.748 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:34.748 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.748 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.748 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.748 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689357+0000", 2024-09-13T14:55:34.749 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3130000000000004e-06 2024-09-13T14:55:34.749 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.749 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.749 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.749 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689374+0000", 2024-09-13T14:55:34.749 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6492999999999999e-05 2024-09-13T14:55:34.749 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.749 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689380+0000", 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2149999999999998e-06 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689437+0000", 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7411999999999999e-05 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.750 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.689445+0000", 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6920000000000002e-06 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.671112+0000", 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98166737599999998 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.671244+0000", 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013205399999999999 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.751 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.671336+0000", 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1335000000000006e-05 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.679738+0000", 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0084018919999999993 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699678+0000", 2024-09-13T14:55:34.752 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.019940684 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699710+0000", 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1491000000000001e-05 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.699973+0000", 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00026296499999999999 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.753 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700009+0000", 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6374000000000001e-05 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700135+0000", 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000125384 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214400 9.d 9:b4303dac:::benchmark_data_smithi003_156985_object93484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.695445+0000", 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 574.00252492799996, 2024-09-13T14:55:34.754 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0050245659999999, 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214400 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695445+0000", 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.755 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695445+0000", 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695444+0000", 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695450+0000", 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9699999999999996e-06 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695461+0000", 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1472e-05 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695465+0000", 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.087e-06 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695517+0000", 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.189e-05 2024-09-13T14:55:34.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.695526+0000", 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.969e-06 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.673704+0000", 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97817781400000003 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.673783+0000", 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9720000000000002e-05 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.673857+0000", 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3239999999999997e-05 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.683953+0000", 2024-09-13T14:55:34.759 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010095807 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700189+0000", 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016236609999999999 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700235+0000", 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6057000000000003e-05 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.760 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700388+0000", 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015292100000000001 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700412+0000", 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4006e-05 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700469+0000", 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7154000000000002e-05 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.761 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214419 9.d 9:bc751d0e:::benchmark_data_smithi003_156985_object93503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.732552+0000", 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 573.96541729600006, 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96801581000000003, 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214419 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.762 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732552+0000", 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732552+0000", 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732551+0000", 2024-09-13T14:55:34.763 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732558+0000", 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6279999999999999e-06 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732570+0000", 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2157000000000001e-05 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.764 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732576+0000", 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9499999999999998e-06 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732623+0000", 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7587999999999999e-05 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.732629+0000", 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2160000000000003e-06 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675405+0000", 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.94277538900000002 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675536+0000", 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013147199999999999 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:55:34.766 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.675664+0000", 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012742799999999999 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.683960+0000", 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0082962709999999992 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700275+0000", 2024-09-13T14:55:34.767 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016315273000000002 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700331+0000", 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5581999999999997e-05 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700490+0000", 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015882799999999999 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.768 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700512+0000", 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2340999999999998e-05 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700568+0000", 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6063999999999997e-05 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.769 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.45869.0:214424 9.d 9:bfcad95c:::benchmark_data_smithi003_156985_object93508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:46:00.741966+0000", 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 573.95600371299997, 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.958713704, 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.45869", 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1864093703", 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214424 2024-09-13T14:55:34.770 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741966+0000", 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741966+0000", 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.771 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741964+0000", 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741973+0000", 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3589999999999997e-06 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.741994+0000", 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1138e-05 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.742002+0000", 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7829999999999999e-06 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.742058+0000", 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5889e-05 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:00.742066+0000", 2024-09-13T14:55:34.773 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.807e-06 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.677279+0000", 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93521298600000002 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.677400+0000", 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000121215 2024-09-13T14:55:34.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,6,7", 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.677530+0000", 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012972400000000001 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.688181+0000", 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.01065105 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700297+0000", 2024-09-13T14:55:34.775 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012115734 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700367+0000", 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0770000000000002e-05 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700604+0000", 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00023669699999999999 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700624+0000", 2024-09-13T14:55:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0196000000000001e-05 2024-09-13T14:55:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:46:01.700680+0000", 2024-09-13T14:55:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5381999999999999e-05 2024-09-13T14:55:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.777 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.36168.0:27860 10.a e238/229 10:51e3daa3:::benchmark_data_smithi003_197468_object27859:head v 238'1740, mlcod=238'1738)", 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:47:48.002523+0000", 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 466.69544708400002, 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29010423400000002, 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002523+0000", 2024-09-13T14:55:34.778 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002523+0000", 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002521+0000", 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.779 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002546+0000", 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4780999999999999e-05 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002561+0000", 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5078e-05 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002568+0000", 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5640000000000002e-06 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.780 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.781 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002640+0000", 2024-09-13T14:55:34.781 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1926000000000002e-05 2024-09-13T14:55:34.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.781 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.781 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002677+0000", 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7039000000000002e-05 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292596+0000", 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28991923400000003 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292627+0000", 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0796000000000001e-05 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.782 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:27870 10.6 10:694fcd1d:::benchmark_data_smithi003_197468_object27869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:47:48.002696+0000", 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 466.695273138, 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29172562200000002, 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 27870 2024-09-13T14:55:34.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002696+0000", 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002696+0000", 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.784 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002695+0000", 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002728+0000", 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3930000000000002e-05 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002743+0000", 2024-09-13T14:55:34.785 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4092e-05 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002749+0000", 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2779999999999999e-06 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002799+0000", 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0587999999999997e-05 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.786 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.002927+0000", 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012778099999999999 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.290644+0000", 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28771667299999998 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294288+0000", 2024-09-13T14:55:34.787 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036443819999999998 2024-09-13T14:55:34.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.788 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.788 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.788 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294297+0000", 2024-09-13T14:55:34.788 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5660000000000007e-06 2024-09-13T14:55:34.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.788 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.788 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.788 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294345+0000", 2024-09-13T14:55:34.789 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8038999999999999e-05 2024-09-13T14:55:34.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.789 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.789 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.789 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294387+0000", 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2382999999999997e-05 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294422+0000", 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4839999999999998e-05 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.791 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.792 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:27876 10.6 10:6f1b44f1:::benchmark_data_smithi003_197468_object27875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:55:34.792 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:47:48.004208+0000", 2024-09-13T14:55:34.792 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 466.69376153000002, 2024-09-13T14:55:34.792 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29044940699999999, 2024-09-13T14:55:34.792 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.792 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.792 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 27876 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.004208+0000", 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.004208+0000", 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.004207+0000", 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.004253+0000", 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6487999999999999e-05 2024-09-13T14:55:34.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.290612+0000", 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28635865599999999 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.290626+0000", 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4279e-05 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292002+0000", 2024-09-13T14:55:34.795 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001375874 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292154+0000", 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015235499999999999 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.292252+0000", 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7626000000000003e-05 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.796 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294492+0000", 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022405459999999999 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294514+0000", 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1203e-05 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294558+0000", 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4535e-05 2024-09-13T14:55:34.797 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.798 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.798 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.798 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294621+0000", 2024-09-13T14:55:34.798 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2793999999999995e-05 2024-09-13T14:55:34.798 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.798 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.798 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.798 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:47:48.294658+0000", 2024-09-13T14:55:34.798 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6492999999999998e-05 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:79806 10.6 10:67b6a0c3:::benchmark_data_smithi003_197468_object79805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:48:24.380081+0000", 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 430.317888525, 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29455382499999999, 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.799 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79806 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380081+0000", 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380081+0000", 2024-09-13T14:55:34.800 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380079+0000", 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380113+0000", 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3945000000000001e-05 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.801 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380125+0000", 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1814000000000001e-05 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380130+0000", 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4389999999999999e-06 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380349+0000", 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00021863699999999999 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.802 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380465+0000", 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000115486 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.380589+0000", 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012487199999999999 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385248+0000", 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0046584390000000003 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.385913+0000", 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00066478399999999997 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.674526+0000", 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28861310000000001 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.804 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.674586+0000", 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0374999999999997e-05 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:48:24.674635+0000", 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8739999999999998e-05 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:174141 10.6 10:619a9ce3:::benchmark_data_smithi003_197468_object57412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:50:16.318303+0000", 2024-09-13T14:55:34.805 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 318.37966641899999, 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.16720552799999999, 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 174141 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.318303+0000", 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.806 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.318303+0000", 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.318301+0000", 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.318309+0000", 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4109999999999999e-06 2024-09-13T14:55:34.807 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.318327+0000", 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7972999999999999e-05 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.318334+0000", 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.81e-06 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.808 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.318411+0000", 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7418999999999999e-05 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.318529+0000", 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118446 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.318633+0000", 2024-09-13T14:55:34.809 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000104054 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.324063+0000", 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.005429471 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.325741+0000", 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001678361 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.810 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.485396+0000", 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.15965506300000001 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.485460+0000", 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3633999999999994e-05 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:16.485509+0000", 2024-09-13T14:55:34.811 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8715999999999999e-05 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:206935 11.17 11:eb394a87:::benchmark_data_smithi003_229782_object84555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:00.480826+0000", 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 34.217144011000002, 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24575330300000001, 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:55:34.812 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 206935 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480826+0000", 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480826+0000", 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.813 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480824+0000", 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480830+0000", 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.19e-06 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.814 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480844+0000", 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3473e-05 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480852+0000", 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7149999999999998e-06 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480912+0000", 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0019999999999998e-05 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.815 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.481041+0000", 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012961299999999999 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.481660+0000", 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00061894800000000002 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.485143+0000", 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0034825670000000002 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.816 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.485718+0000", 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00057511300000000003 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.726493+0000", 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24077558399999999 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.726540+0000", 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.655e-05 2024-09-13T14:55:34.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.726579+0000", 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8927999999999997e-05 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:206941 11.17 11:eceecd00:::benchmark_data_smithi003_229782_object84561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:00.486544+0000", 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 34.211425235, 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25040380499999998, 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.818 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 206941 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486544+0000", 2024-09-13T14:55:34.819 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486544+0000", 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486543+0000", 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.820 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486549+0000", 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9009999999999999e-06 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486561+0000", 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1782e-05 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486568+0000", 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2189999999999998e-06 2024-09-13T14:55:34.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486612+0000", 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4319e-05 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486707+0000", 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.4728999999999994e-05 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.487203+0000", 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00049568500000000005 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.489672+0000", 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024689030000000002 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.490789+0000", 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0011174679999999999 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.823 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.736847+0000", 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24605777300000001 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.736902+0000", 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4867000000000002e-05 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.736948+0000", 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6579000000000002e-05 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:206964 11.17 11:e838aa8e:::benchmark_data_smithi003_229782_object84584:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:00.511574+0000", 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 34.186395378, 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22550974500000001, 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 206964 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.825 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511574+0000", 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511574+0000", 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511572+0000", 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511581+0000", 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8599999999999999e-06 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511597+0000", 2024-09-13T14:55:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5214e-05 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511604+0000", 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0790000000000003e-06 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511660+0000", 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5927000000000002e-05 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.829 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.829 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511792+0000", 2024-09-13T14:55:34.829 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013238400000000001 2024-09-13T14:55:34.829 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.829 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.829 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:34.829 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.512364+0000", 2024-09-13T14:55:34.829 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00057206399999999995 2024-09-13T14:55:34.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.830 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.830 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.515783+0000", 2024-09-13T14:55:34.830 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003419329 2024-09-13T14:55:34.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.830 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.830 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.516623+0000", 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00083948099999999999 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.737001+0000", 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.220378661 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.737048+0000", 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6328000000000001e-05 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.831 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.737084+0000", 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6198000000000001e-05 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:206983 11.17 11:ec4b1388:::benchmark_data_smithi003_229782_object84603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:00.546413+0000", 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 34.151556403000001, 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19765461000000001, 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.832 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 206983 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546413+0000", 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546413+0000", 2024-09-13T14:55:34.833 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546412+0000", 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546418+0000", 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4670000000000001e-06 2024-09-13T14:55:34.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546430+0000", 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.199e-05 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546435+0000", 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9849999999999997e-06 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.835 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546472+0000", 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.663e-05 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546560+0000", 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7880000000000006e-05 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.547044+0000", 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00048424699999999999 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.549255+0000", 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002211373 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.549881+0000", 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00062549199999999995 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.743981+0000", 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19410002600000001 2024-09-13T14:55:34.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.744030+0000", 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.918e-05 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.744068+0000", 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7672000000000001e-05 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.838 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211651 11.7 11:e6ef7c31:::benchmark_data_smithi003_229782_object89271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.899729+0000", 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 28.798240223000001, 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.45350560400000001, 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211651 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899729+0000", 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899729+0000", 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899728+0000", 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899736+0000", 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1620000000000008e-06 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899754+0000", 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8436000000000001e-05 2024-09-13T14:55:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899762+0000", 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4619999999999999e-06 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899813+0000", 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1476e-05 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899943+0000", 2024-09-13T14:55:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012961900000000001 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.900568+0000", 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00062517599999999997 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.903464+0000", 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028962689999999999 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.905949+0000", 2024-09-13T14:55:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024846270000000001 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353141+0000", 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.447192647 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353193+0000", 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1270999999999998e-05 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353235+0000", 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2355999999999999e-05 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211672 11.7 11:e4d90238:::benchmark_data_smithi003_229782_object89292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.933359+0000", 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 28.764611071000001, 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.420019643, 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211672 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933359+0000", 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933359+0000", 2024-09-13T14:55:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933356+0000", 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933366+0000", 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7610000000000002e-06 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933383+0000", 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7181999999999999e-05 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933390+0000", 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0990000000000001e-06 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933445+0000", 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4756999999999998e-05 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933541+0000", 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6026000000000005e-05 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.934144+0000", 2024-09-13T14:55:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00060253300000000004 2024-09-13T14:55:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.936675+0000", 2024-09-13T14:55:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002530942 2024-09-13T14:55:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.937326+0000", 2024-09-13T14:55:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00065163500000000002 2024-09-13T14:55:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353302+0000", 2024-09-13T14:55:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.415976026 2024-09-13T14:55:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353342+0000", 2024-09-13T14:55:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9235e-05 2024-09-13T14:55:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353378+0000", 2024-09-13T14:55:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6594000000000002e-05 2024-09-13T14:55:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:235437 11.7 11:e4bc60de:::benchmark_data_smithi003_229782_object113057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T14:55:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:33.150334+0000", 2024-09-13T14:55:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 1.5476359980000001, 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.256778852, 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 235437 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:55:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150334+0000", 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150334+0000", 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150332+0000", 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150339+0000", 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3730000000000004e-06 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150356+0000", 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6878e-05 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150362+0000", 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.066e-06 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150413+0000", 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1004999999999997e-05 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150560+0000", 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000146976 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.151193+0000", 2024-09-13T14:55:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00063234200000000004 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.154946+0000", 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0037530139999999998 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.155346+0000", 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00039992899999999999 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.407006+0000", 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25166077399999998 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.407065+0000", 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8998e-05 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.407112+0000", 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7182000000000003e-05 2024-09-13T14:55:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.861 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.861 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.861 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:55:34.861 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:55:34.861 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:55:34.898 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:35.000 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:35.031 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:35.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:35.175 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:35.277 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:35.320 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:35.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:35.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:35.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:35.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:35.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:35 smithi019 ceph-mon[91296]: pgmap v1070: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 729 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 1563/35022 objects degraded (4.463%); 1563/35022 objects misplaced (4.463%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:35.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:35 smithi003 ceph-mon[107278]: pgmap v1070: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 729 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 1563/35022 objects degraded (4.463%); 1563/35022 objects misplaced (4.463%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:35.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:35 smithi003 ceph-mon[104007]: pgmap v1070: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 729 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 1563/35022 objects degraded (4.463%); 1563/35022 objects misplaced (4.463%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:35.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:35.741 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:35.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:35.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:35.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:55:35.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:35.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:35.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:36.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:36.132 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:36.167 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:36.269 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:36.280 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:36.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:36.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:36.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:36.539 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:36.567 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:36.598 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:36 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1563/35022 objects degraded (4.463%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:36.598 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:36 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1563/35022 objects degraded (4.463%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:36.602 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:36 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1563/35022 objects degraded (4.463%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:36.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:36.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:36.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:36.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:36.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:36.986 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:37.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:37.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:37.233 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:37.274 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:37.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:37.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:37.511 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:37 smithi019 ceph-mon[91296]: pgmap v1071: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 575 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 879 op/s; 1563/27630 objects degraded (5.657%); 1563/27630 objects misplaced (5.657%); 0 B/s, 37 objects/s recovering 2024-09-13T14:55:37.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:37.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:37.662 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:37 smithi003 ceph-mon[107278]: pgmap v1071: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 575 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 879 op/s; 1563/27630 objects degraded (5.657%); 1563/27630 objects misplaced (5.657%); 0 B/s, 37 objects/s recovering 2024-09-13T14:55:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:37 smithi003 ceph-mon[104007]: pgmap v1071: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 575 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 879 op/s; 1563/27630 objects degraded (5.657%); 1563/27630 objects misplaced (5.657%); 0 B/s, 37 objects/s recovering 2024-09-13T14:55:37.690 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:37.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:37.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:37.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:37.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:37.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:38.085 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:38.121 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:38.127 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:55:38.127 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:55:38.127 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T14:55:38.128 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:55:38.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:38.251 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:38.353 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:38.380 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:38.482 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:38.511 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:38.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:38.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:38.760 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:38.792 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:38.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:55:38.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:38.976 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:39.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:39.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:39.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:39.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:39.288 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:39.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:39.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:39.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:39.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:39.611 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:39 smithi019 ceph-mon[91296]: pgmap v1072: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 466 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 1330/22437 objects degraded (5.928%); 1330/22437 objects misplaced (5.928%); 0 B/s, 42 objects/s recovering 2024-09-13T14:55:39.611 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:39 smithi003 ceph-mon[107278]: pgmap v1072: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 466 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 1330/22437 objects degraded (5.928%); 1330/22437 objects misplaced (5.928%); 0 B/s, 42 objects/s recovering 2024-09-13T14:55:39.612 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:39 smithi003 ceph-mon[104007]: pgmap v1072: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 466 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 1330/22437 objects degraded (5.928%); 1330/22437 objects misplaced (5.928%); 0 B/s, 42 objects/s recovering 2024-09-13T14:55:39.663 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:39.702 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:39.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:39.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:39.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:39.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:40.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:40.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:55:40.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:40.129 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:40.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:40.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:40.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:40.338 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:40.380 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:55:40.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:40 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4019964950' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:40.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:40 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:40 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4019964950' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:40 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:40.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:40.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:40.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:40.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:40.665 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:40 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4019964950' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:40.665 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:40 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:40.665 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:55:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:55:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:40.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:40.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:40.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:40.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:41.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:41.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:41.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:41.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:41.310 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:55:41.310 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:55:41.310 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:55:41.310 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:55:41.310 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:55:41.327 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:41.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:41.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:41.515 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:41.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:41.559 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[107278]: pgmap v1073: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 353 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 784 op/s; 1330/17001 objects degraded (7.823%); 1330/17001 objects misplaced (7.823%); 0 B/s, 42 objects/s recovering 2024-09-13T14:55:41.559 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:41.559 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:55:41.559 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[107278]: osdmap e303: 8 total, 8 up, 8 in 2024-09-13T14:55:41.560 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1330/17001 objects degraded (7.823%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:41.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[104007]: pgmap v1073: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 353 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 784 op/s; 1330/17001 objects degraded (7.823%); 1330/17001 objects misplaced (7.823%); 0 B/s, 42 objects/s recovering 2024-09-13T14:55:41.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:41.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:55:41.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[104007]: osdmap e303: 8 total, 8 up, 8 in 2024-09-13T14:55:41.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1330/17001 objects degraded (7.823%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:41.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:41.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:41.661 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:41 smithi019 ceph-mon[91296]: pgmap v1073: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 353 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 784 op/s; 1330/17001 objects degraded (7.823%); 1330/17001 objects misplaced (7.823%); 0 B/s, 42 objects/s recovering 2024-09-13T14:55:41.661 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:41 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:41.661 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:41 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:55:41.661 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:41 smithi019 ceph-mon[91296]: osdmap e303: 8 total, 8 up, 8 in 2024-09-13T14:55:41.661 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1330/17001 objects degraded (7.823%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:41.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:41.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:41.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:41.913 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:42.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:42.068 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:55:42.091 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:42.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:42.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:42.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:42.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:42.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:42.518 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:42.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:42.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:42.656 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:42 smithi019 ceph-mon[91296]: 11.3 scrub starts 2024-09-13T14:55:42.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:42 smithi003 ceph-mon[104007]: 11.3 scrub starts 2024-09-13T14:55:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:42 smithi003 ceph-mon[107278]: 11.3 scrub starts 2024-09-13T14:55:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:42.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:42.735 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:42.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:42.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:42.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:42.960 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:43.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:43.089 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:43.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:43.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:43.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.392 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:43.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:43.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:43.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:43.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:55:43.618 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:55:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:43.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:43.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:43 smithi003 ceph-mon[104007]: pgmap v1075: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 269 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 1123/12966 objects degraded (8.661%); 1123/12966 objects misplaced (8.661%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:43.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:43.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:43 smithi003 ceph-mon[107278]: pgmap v1075: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 269 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 1123/12966 objects degraded (8.661%); 1123/12966 objects misplaced (8.661%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:43.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:43.656 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:55:43.656 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:43 smithi019 ceph-mon[91296]: pgmap v1075: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 269 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 1123/12966 objects degraded (8.661%); 1123/12966 objects misplaced (8.661%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.694 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:43.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:43.699 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:55:43.699 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:43.746 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:55:43.746 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:43.773 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:55:43.773 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:43.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:43.805 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:55:43.805 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:43.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:43.839 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:55:43.840 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:43.870 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:55:43.870 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:43.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:43.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:44.088 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:44.109 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 122377 objects 2024-09-13T14:55:44.110 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :133.247 2024-09-13T14:55:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.121 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:44.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:44.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:44.244 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:44.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:44.354 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:55:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:55:44] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:55:44.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:44 smithi019 ceph-mon[91296]: 11.13 scrub starts 2024-09-13T14:55:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3917964062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:44 smithi019 ceph-mon[91296]: pgmap v1076: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 194 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 1123/9372 objects degraded (11.983%); 1123/9372 objects misplaced (11.983%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:44.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:44.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:44.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:44 smithi003 ceph-mon[107278]: 11.13 scrub starts 2024-09-13T14:55:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3917964062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:44 smithi003 ceph-mon[107278]: pgmap v1076: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 194 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 1123/9372 objects degraded (11.983%); 1123/9372 objects misplaced (11.983%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:44 smithi003 ceph-mon[104007]: 11.13 scrub starts 2024-09-13T14:55:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3917964062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:44 smithi003 ceph-mon[104007]: pgmap v1076: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 194 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 1123/9372 objects degraded (11.983%); 1123/9372 objects misplaced (11.983%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:44.715 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:44.715 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:44.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:44.756 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:44.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:44.881 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:44.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:45.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:45.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.131 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:45.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:45.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:45.420 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:55:45.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:45.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:45.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:45.632 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:45.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:45.687 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:45.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.757 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.842 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:45.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.884 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:45 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1123/9372 objects degraded (11.983%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:45.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:45 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1123/9372 objects degraded (11.983%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:45.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:46.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:46.107 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:45 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1123/9372 objects degraded (11.983%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:46.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:46.239 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:46.261 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:46.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:46.489 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:46.495 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-09-13T14:55:46.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-09-13T14:55:46.511 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:46.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:46.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:46.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:46.762 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:46.806 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:55:46.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:46.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:46.912 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:46 smithi019 ceph-mon[91296]: pgmap v1077: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 56 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 834 op/s; 1123/2742 objects degraded (40.956%); 1123/2742 objects misplaced (40.956%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:46.912 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:46 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1301745362' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:55:46.912 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:46 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:55:46.990 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:47.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:47.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:46 smithi003 ceph-mon[104007]: pgmap v1077: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 56 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 834 op/s; 1123/2742 objects degraded (40.956%); 1123/2742 objects misplaced (40.956%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:47.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:46 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1301745362' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:55:47.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:46 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:55:47.028 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:46 smithi003 ceph-mon[107278]: pgmap v1077: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 56 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 834 op/s; 1123/2742 objects degraded (40.956%); 1123/2742 objects misplaced (40.956%); 0 B/s, 43 objects/s recovering 2024-09-13T14:55:47.028 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:46 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1301745362' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:55:47.029 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:46 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:55:47.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:47.138 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:47.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:47.265 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:47.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:55:47.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:47.392 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:47.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:47.499 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:47.499 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:47.517 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:47.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:47.641 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:47.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:47.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:47.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:47.992 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:48.014 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:48.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:55:48.056 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:47 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:55:48.056 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:47 smithi003 ceph-mon[107278]: osdmap e304: 8 total, 8 up, 8 in 2024-09-13T14:55:48.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:47 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:55:48.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:47 smithi003 ceph-mon[104007]: osdmap e304: 8 total, 8 up, 8 in 2024-09-13T14:55:48.116 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:48.120 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:47 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:55:48.120 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:47 smithi019 ceph-mon[91296]: osdmap e304: 8 total, 8 up, 8 in 2024-09-13T14:55:48.140 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:48.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:55:48.242 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:48.264 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:48.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:48.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:48.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:48.442 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:48.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:48.513 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:48.525 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:48.567 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:48.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:48.638 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:48.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:48.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:48.814 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:55:48.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:48.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:48.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[104007]: osdmap e305: 8 total, 8 up, 8 in 2024-09-13T14:55:48.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[104007]: 11.14 scrub starts 2024-09-13T14:55:48.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[104007]: 11.14 scrub ok 2024-09-13T14:55:48.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[104007]: pgmap v1080: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.9 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 635 op/s; 588/285 objects degraded (206.316%); 588/285 objects misplaced (206.316%); 0 B/s, 97 objects/s recovering 2024-09-13T14:55:48.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/875975076' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:48.922 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[107278]: osdmap e305: 8 total, 8 up, 8 in 2024-09-13T14:55:48.922 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[107278]: 11.14 scrub starts 2024-09-13T14:55:48.922 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[107278]: 11.14 scrub ok 2024-09-13T14:55:48.922 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[107278]: pgmap v1080: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.9 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 635 op/s; 588/285 objects degraded (206.316%); 588/285 objects misplaced (206.316%); 0 B/s, 97 objects/s recovering 2024-09-13T14:55:48.922 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/875975076' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:48.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:49.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:49.113 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:49.137 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:49.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:48 smithi019 ceph-mon[91296]: osdmap e305: 8 total, 8 up, 8 in 2024-09-13T14:55:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:48 smithi019 ceph-mon[91296]: 11.14 scrub starts 2024-09-13T14:55:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:48 smithi019 ceph-mon[91296]: 11.14 scrub ok 2024-09-13T14:55:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:48 smithi019 ceph-mon[91296]: pgmap v1080: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.9 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 635 op/s; 588/285 objects degraded (206.316%); 588/285 objects misplaced (206.316%); 0 B/s, 97 objects/s recovering 2024-09-13T14:55:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/875975076' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:49.239 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:49.261 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:49.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:49.385 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:49.487 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:49.510 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:49.612 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:49.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:49.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:49.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:49.863 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:49.886 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:49.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:49.916 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:49 smithi019 ceph-mon[91296]: 11.3 scrub starts 2024-09-13T14:55:49.916 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:49 smithi019 ceph-mon[91296]: 11.3 scrub ok 2024-09-13T14:55:49.916 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:49 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:55:49.917 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/875975076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:49.917 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:49 smithi019 ceph-mon[91296]: osdmap e306: 8 total, 8 up, 8 in 2024-09-13T14:55:49.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[107278]: 11.3 scrub starts 2024-09-13T14:55:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[107278]: 11.3 scrub ok 2024-09-13T14:55:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:55:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/875975076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[107278]: osdmap e306: 8 total, 8 up, 8 in 2024-09-13T14:55:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[104007]: 11.3 scrub starts 2024-09-13T14:55:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[104007]: 11.3 scrub ok 2024-09-13T14:55:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:55:49.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/875975076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:49.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:49 smithi003 ceph-mon[104007]: osdmap e306: 8 total, 8 up, 8 in 2024-09-13T14:55:49.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:50.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:50.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:50.140 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:50.140 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:50.143 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:50.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:50.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:50.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:50.394 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:50.487 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:55:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:55:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:50.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:50.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:50.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:50.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:50.643 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:50.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:50.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:50.803 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:55:50.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:50.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[107278]: 11.11 scrub starts 2024-09-13T14:55:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[107278]: 11.11 scrub ok 2024-09-13T14:55:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[107278]: 11.13 scrub starts 2024-09-13T14:55:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[107278]: 11.13 scrub ok 2024-09-13T14:55:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[107278]: pgmap v1082: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.2 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 588/156 objects degraded (376.923%); 588/156 objects misplaced (376.923%); 0 B/s, 89 objects/s recovering 2024-09-13T14:55:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 588/285 objects degraded (206.316%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[104007]: 11.11 scrub starts 2024-09-13T14:55:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[104007]: 11.11 scrub ok 2024-09-13T14:55:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[104007]: 11.13 scrub starts 2024-09-13T14:55:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[104007]: 11.13 scrub ok 2024-09-13T14:55:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[104007]: pgmap v1082: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.2 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 588/156 objects degraded (376.923%); 588/156 objects misplaced (376.923%); 0 B/s, 89 objects/s recovering 2024-09-13T14:55:50.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 588/285 objects degraded (206.316%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:50.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:51.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:51.064 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:51.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:51.142 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:50 smithi019 ceph-mon[91296]: 11.11 scrub starts 2024-09-13T14:55:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:50 smithi019 ceph-mon[91296]: 11.11 scrub ok 2024-09-13T14:55:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:50 smithi019 ceph-mon[91296]: 11.13 scrub starts 2024-09-13T14:55:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:50 smithi019 ceph-mon[91296]: 11.13 scrub ok 2024-09-13T14:55:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:50 smithi019 ceph-mon[91296]: pgmap v1082: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.2 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 588/156 objects degraded (376.923%); 588/156 objects misplaced (376.923%); 0 B/s, 89 objects/s recovering 2024-09-13T14:55:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 588/285 objects degraded (206.316%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:55:51.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:51.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:51.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.393 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:51.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:51.494 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:51.517 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:51.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:51.641 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:51.744 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.768 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:51.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:51.821 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:51 smithi019 ceph-mon[91296]: 11.5 scrub starts 2024-09-13T14:55:51.821 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:51 smithi019 ceph-mon[91296]: 11.5 scrub ok 2024-09-13T14:55:51.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:51.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:51.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:52.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:52.126 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:51 smithi003 ceph-mon[107278]: 11.5 scrub starts 2024-09-13T14:55:52.126 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:51 smithi003 ceph-mon[107278]: 11.5 scrub ok 2024-09-13T14:55:52.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:51 smithi003 ceph-mon[104007]: 11.5 scrub starts 2024-09-13T14:55:52.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:51 smithi003 ceph-mon[104007]: 11.5 scrub ok 2024-09-13T14:55:52.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:52.150 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:52.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:52.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:52.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:52.406 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:52.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:52.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:55:52.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:52.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:52.572 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:52.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.659 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:52.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:52.760 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:52.784 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:52.875 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:52.875 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:52.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.910 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:52.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:52.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:52 smithi019 ceph-mon[91296]: pgmap v1083: 119 pgs: 119 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:52.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:52 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T14:55:53.010 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:53.033 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:53.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:53.139 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:52 smithi003 ceph-mon[107278]: pgmap v1083: 119 pgs: 119 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:53.139 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:52 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T14:55:53.139 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:52 smithi003 ceph-mon[104007]: pgmap v1083: 119 pgs: 119 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:53.139 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:52 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T14:55:53.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:53.261 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:53.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:53.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:53.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:53.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:55:53.512 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:53.534 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:53.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:53.658 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:53.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:53.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:53.831 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:55:53.887 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:53.910 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:54.011 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:54.034 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:54.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 588/156 objects degraded (376.923%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:55:54.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:55:54.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T14:55:54.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[104007]: osdmap e307: 8 total, 8 up, 8 in 2024-09-13T14:55:54.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3254407227' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:55:54.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 588/156 objects degraded (376.923%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:55:54.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:55:54.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T14:55:54.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[107278]: osdmap e307: 8 total, 8 up, 8 in 2024-09-13T14:55:54.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3254407227' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:55:54.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:54.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:54.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:53 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 588/156 objects degraded (376.923%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:55:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:53 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:55:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T14:55:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:53 smithi019 ceph-mon[91296]: osdmap e307: 8 total, 8 up, 8 in 2024-09-13T14:55:54.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3254407227' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:55:54.261 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:54.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:54.385 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:54.408 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:55:54] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:55:54.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:54.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:54.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:54.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:54.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:54.841 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_6' removed 2024-09-13T14:55:54.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:54.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:55.088 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:55.110 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:54 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3254407227' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:55:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:54 smithi019 ceph-mon[91296]: osdmap e308: 8 total, 8 up, 8 in 2024-09-13T14:55:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:54 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3559768451' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:55:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:54 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:55:55.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:54 smithi019 ceph-mon[91296]: pgmap v1086: 119 pgs: 119 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:55.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3254407227' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:55:55.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[107278]: osdmap e308: 8 total, 8 up, 8 in 2024-09-13T14:55:55.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3559768451' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:55:55.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:55:55.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[107278]: pgmap v1086: 119 pgs: 119 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:55.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3254407227' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:55:55.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[104007]: osdmap e308: 8 total, 8 up, 8 in 2024-09-13T14:55:55.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3559768451' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:55:55.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:55:55.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:54 smithi003 ceph-mon[104007]: pgmap v1086: 119 pgs: 119 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:55.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:55.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:55.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:55.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:55.462 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.487 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:55.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:55.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:55.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:55.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:55.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:55.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:55.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:55.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:55.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:55.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:55.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:55.999 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:55:56.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:56.005 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:56.024 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:55:56.024 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:56.042 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:55:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:56.059 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:55:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:56.075 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:55:56.075 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:56.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:56.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:55:56.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:56.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:55 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:55:56.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:55 smithi003 ceph-mon[107278]: osdmap e309: 8 total, 8 up, 8 in 2024-09-13T14:55:56.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:55 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:55:56.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:55 smithi003 ceph-mon[104007]: osdmap e309: 8 total, 8 up, 8 in 2024-09-13T14:55:56.107 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:55:56.107 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:56.117 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:56.123 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:55:56.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:56.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:55 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:55:56.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:55 smithi019 ceph-mon[91296]: osdmap e309: 8 total, 8 up, 8 in 2024-09-13T14:55:56.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:56.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:56.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:56.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:56.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:56.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:56.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:56.626 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:56.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 14:55:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=cleanup t=2024-09-13T14:55:56.418540619Z level=info msg="Completed cleanup jobs" duration=12.84846ms 2024-09-13T14:55:56.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:56.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.879 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:56.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:56.981 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:57.006 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:57.030 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:56 smithi003 ceph-mon[107278]: pgmap v1088: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:57.030 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2068142109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:57.030 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:56 smithi003 ceph-mon[104007]: pgmap v1088: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:57.030 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2068142109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:57.055 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:55:57.056 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:55:57.056 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-09-13T14:55:57.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-09-13T14:55:57.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:57.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:56 smithi019 ceph-mon[91296]: pgmap v1088: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2068142109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:57.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:57.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:57.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:57.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:57.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:57.485 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:57.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:55:57.512 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:57.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:57.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:57.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:57.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:57.863 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:57.890 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:57.919 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:57.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:58.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:58.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:58.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:58.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:58.267 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:58.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:58.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:58.395 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:58.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:58.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:58.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:58.649 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:58.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:58.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:58.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:58.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:59.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:59.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:59.131 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:59.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:59.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:58 smithi019 ceph-mon[91296]: pgmap v1089: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:55:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:58 smithi003 ceph-mon[104007]: pgmap v1089: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:55:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:58 smithi003 ceph-mon[107278]: pgmap v1089: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:55:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:55:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:55:59.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:59.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:59.385 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:59.412 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:59.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:59.541 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:59.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:59.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:59.768 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:59.795 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:55:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:59.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:59.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:55:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:00.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:00.049 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:00.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:00.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:00.260 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:00.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:00.405 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:00.432 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:00.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:56:00.534 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:00.560 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:00.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:56:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:56:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:00.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:00.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:00.813 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:00.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:00.917 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:00.940 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:01.008 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:56:01.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:01.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:01.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:01 smithi003 ceph-mon[107278]: pgmap v1090: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:01 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3111701059' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:01 smithi003 ceph-mon[104007]: pgmap v1090: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:01 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3111701059' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:01.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:01.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:01.318 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:01.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:01 smithi019 ceph-mon[91296]: pgmap v1090: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:01 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3111701059' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:01.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:01.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:01.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:01.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:01.673 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:01.699 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:01.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:01.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:01.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:02.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:02.086 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:02.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:02.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:02.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:02.344 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:02 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:02 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3111701059' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:02 smithi019 ceph-mon[91296]: osdmap e310: 8 total, 8 up, 8 in 2024-09-13T14:56:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:02 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:02 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3111701059' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:02 smithi003 ceph-mon[107278]: osdmap e310: 8 total, 8 up, 8 in 2024-09-13T14:56:02.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:02 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:02.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:02 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3111701059' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:02.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:02 smithi003 ceph-mon[104007]: osdmap e310: 8 total, 8 up, 8 in 2024-09-13T14:56:02.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:02.471 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:02.574 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:02.600 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:02.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:02.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:02.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:02.829 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:02.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:02.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:02.984 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:02.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:03.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:03.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:03.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:03.241 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:03.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:03.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:03.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:03 smithi019 ceph-mon[91296]: pgmap v1092: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:03.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:03 smithi003 ceph-mon[107278]: pgmap v1092: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:03.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:03 smithi003 ceph-mon[104007]: pgmap v1092: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:03.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:03.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:03.602 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:03.628 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:03.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:03.756 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:03.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:03.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:03.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:04.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:04.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:04.182 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:56:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:56:04] "GET /metrics HTTP/1.1" 200 34114 "" "Prometheus/2.51.0" 2024-09-13T14:56:04.237 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:04.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.266 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:04.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:04.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:04.393 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:04.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:04.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:04.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:04.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:04.778 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:04.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:04.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:05.009 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:05.036 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:05.138 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:05.165 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:05.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:05.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:05.294 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:05.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:05.331 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:05 smithi003 ceph-mon[107278]: pgmap v1093: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:05.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:05 smithi003 ceph-mon[104007]: pgmap v1093: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:05.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:05.423 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:05 smithi019 ceph-mon[91296]: pgmap v1093: 93 pgs: 93 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:05.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:05.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:05.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:05.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:05.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:05.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:05.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:05.928 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:06.029 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:06.056 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:06.158 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:06.184 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:06.286 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:06.313 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:06.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:06.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:06.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:06.568 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:06.670 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:06.697 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:06.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:56:06.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:06.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:06.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.953 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:06.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:07.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:07.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:07 smithi003 ceph-mon[107278]: pgmap v1094: 93 pgs: 93 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:07.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:07 smithi003 ceph-mon[104007]: pgmap v1094: 93 pgs: 93 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:07.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:07.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:07.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:07.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:07.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:07.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:07 smithi019 ceph-mon[91296]: pgmap v1094: 93 pgs: 93 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:07.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:07.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:07.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:07.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:07.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:07.955 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:08.021 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_7' created 2024-09-13T14:56:08.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:08.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:08.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:08 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3770987111' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-13T14:56:08.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:08 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3770987111' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-13T14:56:08.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:08.210 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:08.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:08.339 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:08.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:08 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3770987111' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-13T14:56:08.441 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:08.464 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:08.566 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:08.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:08.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:08.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:08.841 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:08.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.967 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:08.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:09.028 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T14:56:09.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:09.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:09.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-09-13T14:56:09.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:09.215 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.232 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:56:09.232 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:56:09.232 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-13T14:56:09.232 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:56:09.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:09.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:09.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:09 smithi019 ceph-mon[91296]: pgmap v1095: 93 pgs: 93 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3770987111' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-13T14:56:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:09 smithi019 ceph-mon[91296]: osdmap e311: 8 total, 8 up, 8 in 2024-09-13T14:56:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:56:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2371439883' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:56:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:09 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:56:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[107278]: pgmap v1095: 93 pgs: 93 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3770987111' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-13T14:56:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[107278]: osdmap e311: 8 total, 8 up, 8 in 2024-09-13T14:56:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:56:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2371439883' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:56:09.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:56:09.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[104007]: pgmap v1095: 93 pgs: 93 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:09.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3770987111' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-13T14:56:09.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[104007]: osdmap e311: 8 total, 8 up, 8 in 2024-09-13T14:56:09.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:56:09.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2371439883' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:56:09.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:09 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:56:09.441 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:09.464 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.566 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:09.587 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:09.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:09.711 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:09.835 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:09.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:56:09.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:09.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:10.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:10.084 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:10.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:10 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:56:10.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:10 smithi003 ceph-mon[107278]: osdmap e312: 8 total, 8 up, 8 in 2024-09-13T14:56:10.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:10 smithi003 ceph-mon[107278]: osdmap e313: 8 total, 8 up, 8 in 2024-09-13T14:56:10.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:10 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:56:10.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:10 smithi003 ceph-mon[104007]: osdmap e312: 8 total, 8 up, 8 in 2024-09-13T14:56:10.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:10 smithi003 ceph-mon[104007]: osdmap e313: 8 total, 8 up, 8 in 2024-09-13T14:56:10.112 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:56:10.113 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:56:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:10.126 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:56:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:10.144 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:56:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:10.162 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:56:10.162 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:10.178 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:56:10.178 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:10.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:10 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:56:10.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:10 smithi019 ceph-mon[91296]: osdmap e312: 8 total, 8 up, 8 in 2024-09-13T14:56:10.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:10 smithi019 ceph-mon[91296]: osdmap e313: 8 total, 8 up, 8 in 2024-09-13T14:56:10.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:10.193 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:56:10.194 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:10.208 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:10.210 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:56:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:10.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:10.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:10.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:10.458 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:10.493 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:56:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:56:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:10.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:10.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.583 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:10.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:10.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:10.707 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:10.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:10.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:10.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:10.956 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:11.058 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.081 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:11.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.182 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.205 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:11.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:11.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:11.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:11 smithi019 ceph-mon[91296]: pgmap v1098: 109 pgs: 1 creating+peering, 7 unknown, 101 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:11 smithi019 ceph-mon[91296]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:56:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3020755114' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:11 smithi003 ceph-mon[104007]: pgmap v1098: 109 pgs: 1 creating+peering, 7 unknown, 101 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:11 smithi003 ceph-mon[104007]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:56:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3020755114' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:11 smithi003 ceph-mon[107278]: pgmap v1098: 109 pgs: 1 creating+peering, 7 unknown, 101 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:11 smithi003 ceph-mon[107278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:56:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3020755114' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:11.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:11.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:11.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:11.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:11.652 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:56:11.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:56:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:11.676 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:11.677 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:11.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:11.717 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.820 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:11.842 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:11.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:11.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:12.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:12.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:12.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:12.216 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:12.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:12.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:12.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:12.464 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:12.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:12.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:12.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:12.816 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:12.840 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:12.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:12.964 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:12.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:13.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:56:13.065 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:13.088 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:13.190 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:13.212 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:13.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:13 smithi019 ceph-mon[91296]: pgmap v1100: 109 pgs: 1 creating+peering, 2 unknown, 106 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[107278]: pgmap v1100: 109 pgs: 1 creating+peering, 2 unknown, 106 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[104007]: pgmap v1100: 109 pgs: 1 creating+peering, 2 unknown, 106 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:13.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:13.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:13.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:13.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:13.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:13.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:13.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:13.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:13.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:13.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:13.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:13.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:13.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:13.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:14.060 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:14.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:56:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:56:14] "GET /metrics HTTP/1.1" 200 36461 "" "Prometheus/2.51.0" 2024-09-13T14:56:14.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:14.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:14.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:14.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:14.588 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:14.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:14.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:14.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:14.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:14.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:14.969 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:15.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:15.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:15.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:15.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:15.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:15.346 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:15 smithi019 ceph-mon[91296]: pgmap v1101: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:15 smithi003 ceph-mon[104007]: pgmap v1101: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:15 smithi003 ceph-mon[107278]: pgmap v1101: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:15.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:15.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:15.573 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:15.599 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:15.701 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:15.727 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:15.829 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:15.856 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:15.949 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:15.950 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:15.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:15.984 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:16.086 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:16.112 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:16.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:16.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:16.339 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:16.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:16.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:16.494 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:16.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:16.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:16.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:16.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:16.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:16.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:16.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:16.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:16.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:17.100 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:17.126 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:17.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:17.288 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:17 smithi003 ceph-mon[107278]: pgmap v1102: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:17.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:17 smithi003 ceph-mon[104007]: pgmap v1102: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:17.355 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:17.382 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:17 smithi019 ceph-mon[91296]: pgmap v1102: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:17.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:17.510 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:17.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:17.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:17.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:17.737 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:17.763 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:17.824 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:17.825 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:17.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:17.891 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:17.993 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:18.019 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:18.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:18.058 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:56:18.058 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-09-13T14:56:18.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:18.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1625572222' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:18.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/388360601' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:56:18.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1625572222' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:18.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/388360601' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:56:18.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:18.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:18.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:18.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:18.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1625572222' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:18.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/388360601' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:56:18.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:18.521 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:18.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:56:18.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:18.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:18.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:18.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:18.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:18.903 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:19.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:19.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:19.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:19.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:19.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:19.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:19.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.409 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:19.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:19 smithi019 ceph-mon[91296]: pgmap v1103: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:19 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:19 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1625572222' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:19 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/388360601' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:56:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:19 smithi019 ceph-mon[91296]: osdmap e314: 8 total, 8 up, 8 in 2024-09-13T14:56:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:19 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[107278]: pgmap v1103: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1625572222' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/388360601' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:56:19.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[107278]: osdmap e314: 8 total, 8 up, 8 in 2024-09-13T14:56:19.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[104007]: pgmap v1103: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1625572222' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/388360601' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:56:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[104007]: osdmap e314: 8 total, 8 up, 8 in 2024-09-13T14:56:19.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:19 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:19.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:19.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:19.638 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:19.664 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:19.766 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.792 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:19.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:19.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:19.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:20.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:20.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:20.172 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:20.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:20 smithi003 ceph-mon[107278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:56:20.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:20 smithi003 ceph-mon[104007]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:56:20.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:20.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:20.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:20.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:20.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:20.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:20 smithi019 ceph-mon[91296]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:56:20.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:56:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:56:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:20.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:20.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:20.649 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:20.677 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:20.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:20.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:20.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:21.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:21.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:21.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:21.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:21.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:21.340 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:21 smithi003 ceph-mon[107278]: pgmap v1105: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:21.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:21 smithi003 ceph-mon[104007]: pgmap v1105: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:21.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:21.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:21 smithi019 ceph-mon[91296]: pgmap v1105: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:21.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:21.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:21.661 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:21.686 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:21.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:21.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:21.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:21.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:21.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:22.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:22.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:22.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:22.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:22.326 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:22.364 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-09-13T14:56:22.367 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:56:22.367 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:56:22.415 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T14:56:22.415 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:56:22.415 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_259359 2024-09-13T14:56:22.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:56:22.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:56:22.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:22.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:22.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.618 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:22.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:22.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:22.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:22.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:22.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:22.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:56:23.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:23.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:23.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:23.185 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:23.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:23 smithi003 ceph-mon[104007]: pgmap v1106: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:23.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:23 smithi003 ceph-mon[107278]: pgmap v1106: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:23.287 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:23.318 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:23.337 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:56:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:56:23.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3174 3158 197.272 197.375 0.00385145 0.00505086 2024-09-13T14:56:23.421 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:23 smithi019 ceph-mon[91296]: pgmap v1106: 109 pgs: 109 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-13T14:56:23.452 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:23.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:23.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:23.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:23.613 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:56:23.613 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:23.653 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:56:23.653 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:23.688 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:56:23.689 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:23.698 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:23.724 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:56:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:23.730 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:23.756 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:56:23.756 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:23.785 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:56:23.785 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:23.804 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:56:23.805 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:23.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:23.833 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:56:23.833 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:23.864 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:23.966 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:24.000 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:24.016 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:24.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:24.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:24.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:24.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:24.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:24.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:24 smithi003 ceph-mon[104007]: from='client.46709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:24.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:24 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/728463005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:24.352 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:56:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:56:24] "GET /metrics HTTP/1.1" 200 36461 "" "Prometheus/2.51.0" 2024-09-13T14:56:24.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:24 smithi003 ceph-mon[107278]: from='client.46709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:24.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:24 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/728463005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:24.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:24.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 6410 6394 199.752 202.25 0.0045248 0.00499429 2024-09-13T14:56:24.422 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:24.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:24 smithi019 ceph-mon[91296]: from='client.46709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:24.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:24 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/728463005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:24.524 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:24.559 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:24.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:24.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:24.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:24.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:24.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:25.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:25.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:25.149 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:25.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:25.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:25.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:25.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:25 smithi003 ceph-mon[104007]: pgmap v1107: 109 pgs: 109 active+clean; 9.2 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 883 KiB/s wr, 13 op/s 2024-09-13T14:56:25.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:25 smithi003 ceph-mon[107278]: pgmap v1107: 109 pgs: 109 active+clean; 9.2 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 883 KiB/s wr, 13 op/s 2024-09-13T14:56:25.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 9645 9629 200.557 202.188 0.0075282 0.00497678 2024-09-13T14:56:25.419 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:25.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:25 smithi019 ceph-mon[91296]: pgmap v1107: 109 pgs: 109 active+clean; 9.2 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 883 KiB/s wr, 13 op/s 2024-09-13T14:56:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:25.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:25.467 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1106,"stamp":"2024-09-13T14:56:21.953545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":853342,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2159,"ondisk_log_size":2159,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":330,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7511552,"kb_used_data":220260,"kb_used_omap":29,"kb_used_meta":7291170,"kb_avail":742449664,"statfs":{"total":767960285184,"available":760268455936,"internally_reserved":0,"allocated":225546240,"data_stored":211339916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":7466158606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003032"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":509,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183637+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:56:18.183637+0000","last_peered":"2024-09-13T14:56:18.183637+0000","last_clean":"2024-09-13T14:56:18.183637+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:56:18.183637+0000","last_undegraded":"2024-09-13T14:56:18.183637+0000","last_fullsized":"2024-09-13T14:56:18.183637+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":286,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340711+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:56:10.340711+0000","last_peered":"2024-09-13T14:56:10.340711+0000","last_clean":"2024-09-13T14:56:10.340711+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:56:10.340711+0000","last_undegraded":"2024-09-13T14:56:10.340711+0000","last_fullsized":"2024-09-13T14:56:10.340711+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":77,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340100+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:56:10.340100+0000","last_peered":"2024-09-13T14:56:10.340100+0000","last_clean":"2024-09-13T14:56:10.340100+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:56:10.340100+0000","last_undegraded":"2024-09-13T14:56:10.340100+0000","last_fullsized":"2024-09-13T14:56:10.340100+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":465,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339442+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:56:10.339442+0000","last_peered":"2024-09-13T14:56:10.339442+0000","last_clean":"2024-09-13T14:56:10.339442+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:56:10.339442+0000","last_undegraded":"2024-09-13T14:56:10.339442+0000","last_fullsized":"2024-09-13T14:56:10.339442+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":260,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339839+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:56:10.339839+0000","last_peered":"2024-09-13T14:56:10.339839+0000","last_clean":"2024-09-13T14:56:10.339839+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:56:10.339839+0000","last_undegraded":"2024-09-13T14:56:10.339839+0000","last_fullsized":"2024-09-13T14:56:10.339839+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":293,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341369+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:56:10.341369+0000","last_peered":"2024-09-13T14:56:10.341369+0000","last_clean":"2024-09-13T14:56:10.341369+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:56:10.341369+0000","last_undegraded":"2024-09-13T14:56:10.341369+0000","last_fullsized":"2024-09-13T14:56:10.341369+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":277,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340205+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:56:10.340205+0000","last_peered":"2024-09-13T14:56:10.340205+0000","last_clean":"2024-09-13T14:56:10.340205+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:56:10.340205+0000","last_undegraded":"2024-09-13T14:56:10.340205+0000","last_fullsized":"2024-09-13T14:56:10.340205+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":323,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341622+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:56:10.341622+0000","last_peered":"2024-09-13T14:56:10.341622+0000","last_clean":"2024-09-13T14:56:10.341622+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:56:10.341622+0000","last_undegraded":"2024-09-13T14:56:10.341622+0000","last_fullsized":"2024-09-13T14:56:10.341622+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":489,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185097+0000","last_change":"2024-09-13T14:54:27.142651+0000","last_active":"2024-09-13T14:56:18.185097+0000","last_peered":"2024-09-13T14:56:18.185097+0000","last_clean":"2024-09-13T14:56:18.185097+0000","last_became_active":"2024-09-13T14:54:27.142239+0000","last_became_peered":"2024-09-13T14:54:27.142239+0000","last_unstale":"2024-09-13T14:56:18.185097+0000","last_undegraded":"2024-09-13T14:56:18.185097+0000","last_fullsized":"2024-09-13T14:56:18.185097+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:41.308930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":492,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183717+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:56:18.183717+0000","last_peered":"2024-09-13T14:56:18.183717+0000","last_clean":"2024-09-13T14:56:18.183717+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:56:18.183717+0000","last_undegraded":"2024-09-13T14:56:18.183717+0000","last_fullsized":"2024-09-13T14:56:18.183717+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":295,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341456+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:56:10.341456+0000","last_peered":"2024-09-13T14:56:10.341456+0000","last_clean":"2024-09-13T14:56:10.341456+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:56:10.341456+0000","last_undegraded":"2024-09-13T14:56:10.341456+0000","last_fullsized":"2024-09-13T14:56:10.341456+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":432,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339505+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:56:10.339505+0000","last_peered":"2024-09-13T14:56:10.339505+0000","last_clean":"2024-09-13T14:56:10.339505+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:56:10.339505+0000","last_undegraded":"2024-09-13T14:56:10.339505+0000","last_fullsized":"2024-09-13T14:56:10.339505+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":149,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183840+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:56:18.183840+0000","last_peered":"2024-09-13T14:56:18.183840+0000","last_clean":"2024-09-13T14:56:18.183840+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:56:18.183840+0000","last_undegraded":"2024-09-13T14:56:18.183840+0000","last_fullsized":"2024-09-13T14:56:18.183840+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":528,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184057+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:56:18.184057+0000","last_peered":"2024-09-13T14:56:18.184057+0000","last_clean":"2024-09-13T14:56:18.184057+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:56:18.184057+0000","last_undegraded":"2024-09-13T14:56:18.184057+0000","last_fullsized":"2024-09-13T14:56:18.184057+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":464,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339042+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:56:10.339042+0000","last_peered":"2024-09-13T14:56:10.339042+0000","last_clean":"2024-09-13T14:56:10.339042+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:56:10.339042+0000","last_undegraded":"2024-09-13T14:56:10.339042+0000","last_fullsized":"2024-09-13T14:56:10.339042+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":276,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.343146+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:56:10.343146+0000","last_peered":"2024-09-13T14:56:10.343146+0000","last_clean":"2024-09-13T14:56:10.343146+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:56:10.343146+0000","last_undegraded":"2024-09-13T14:56:10.343146+0000","last_fullsized":"2024-09-13T14:56:10.343146+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":433,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.338720+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:56:10.338720+0000","last_peered":"2024-09-13T14:56:10.338720+0000","last_clean":"2024-09-13T14:56:10.338720+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:56:10.338720+0000","last_undegraded":"2024-09-13T14:56:10.338720+0000","last_fullsized":"2024-09-13T14:56:10.338720+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":511,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185051+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:56:18.185051+0000","last_peered":"2024-09-13T14:56:18.185051+0000","last_clean":"2024-09-13T14:56:18.185051+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:56:18.185051+0000","last_undegraded":"2024-09-13T14:56:18.185051+0000","last_fullsized":"2024-09-13T14:56:18.185051+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":438,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339035+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:56:10.339035+0000","last_peered":"2024-09-13T14:56:10.339035+0000","last_clean":"2024-09-13T14:56:10.339035+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:56:10.339035+0000","last_undegraded":"2024-09-13T14:56:10.339035+0000","last_fullsized":"2024-09-13T14:56:10.339035+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":75,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340019+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:56:10.340019+0000","last_peered":"2024-09-13T14:56:10.340019+0000","last_clean":"2024-09-13T14:56:10.340019+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:56:10.340019+0000","last_undegraded":"2024-09-13T14:56:10.340019+0000","last_fullsized":"2024-09-13T14:56:10.340019+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":189,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183818+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:56:18.183818+0000","last_peered":"2024-09-13T14:56:18.183818+0000","last_clean":"2024-09-13T14:56:18.183818+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:56:18.183818+0000","last_undegraded":"2024-09-13T14:56:18.183818+0000","last_fullsized":"2024-09-13T14:56:18.183818+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":77,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340258+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:56:10.340258+0000","last_peered":"2024-09-13T14:56:10.340258+0000","last_clean":"2024-09-13T14:56:10.340258+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:56:10.340258+0000","last_undegraded":"2024-09-13T14:56:10.340258+0000","last_fullsized":"2024-09-13T14:56:10.340258+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":215,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342749+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:56:10.342749+0000","last_peered":"2024-09-13T14:56:10.342749+0000","last_clean":"2024-09-13T14:56:10.342749+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:56:10.342749+0000","last_undegraded":"2024-09-13T14:56:10.342749+0000","last_fullsized":"2024-09-13T14:56:10.342749+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":283,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339672+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:56:10.339672+0000","last_peered":"2024-09-13T14:56:10.339672+0000","last_clean":"2024-09-13T14:56:10.339672+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:56:10.339672+0000","last_undegraded":"2024-09-13T14:56:10.339672+0000","last_fullsized":"2024-09-13T14:56:10.339672+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":276,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341701+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:56:10.341701+0000","last_peered":"2024-09-13T14:56:10.341701+0000","last_clean":"2024-09-13T14:56:10.341701+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:56:10.341701+0000","last_undegraded":"2024-09-13T14:56:10.341701+0000","last_fullsized":"2024-09-13T14:56:10.341701+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":17,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183689+0000","last_change":"2024-09-13T14:56:09.033158+0000","last_active":"2024-09-13T14:56:18.183689+0000","last_peered":"2024-09-13T14:56:18.183689+0000","last_clean":"2024-09-13T14:56:18.183689+0000","last_became_active":"2024-09-13T14:56:09.032864+0000","last_became_peered":"2024-09-13T14:56:09.032864+0000","last_unstale":"2024-09-13T14:56:18.183689+0000","last_undegraded":"2024-09-13T14:56:18.183689+0000","last_fullsized":"2024-09-13T14:56:18.183689+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:36.421513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":259,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340344+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:56:10.340344+0000","last_peered":"2024-09-13T14:56:10.340344+0000","last_clean":"2024-09-13T14:56:10.340344+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:56:10.340344+0000","last_undegraded":"2024-09-13T14:56:10.340344+0000","last_fullsized":"2024-09-13T14:56:10.340344+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":309,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.343190+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:56:10.343190+0000","last_peered":"2024-09-13T14:56:10.343190+0000","last_clean":"2024-09-13T14:56:10.343190+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:56:10.343190+0000","last_undegraded":"2024-09-13T14:56:10.343190+0000","last_fullsized":"2024-09-13T14:56:10.343190+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":509,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184231+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:56:18.184231+0000","last_peered":"2024-09-13T14:56:18.184231+0000","last_clean":"2024-09-13T14:56:18.184231+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:56:18.184231+0000","last_undegraded":"2024-09-13T14:56:18.184231+0000","last_fullsized":"2024-09-13T14:56:18.184231+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":195,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339540+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:56:10.339540+0000","last_peered":"2024-09-13T14:56:10.339540+0000","last_clean":"2024-09-13T14:56:10.339540+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:56:10.339540+0000","last_undegraded":"2024-09-13T14:56:10.339540+0000","last_fullsized":"2024-09-13T14:56:10.339540+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.338979+0000","last_change":"2024-09-13T14:56:09.034412+0000","last_active":"2024-09-13T14:56:10.338979+0000","last_peered":"2024-09-13T14:56:10.338979+0000","last_clean":"2024-09-13T14:56:10.338979+0000","last_became_active":"2024-09-13T14:56:09.033995+0000","last_became_peered":"2024-09-13T14:56:09.033995+0000","last_unstale":"2024-09-13T14:56:10.338979+0000","last_undegraded":"2024-09-13T14:56:10.338979+0000","last_fullsized":"2024-09-13T14:56:10.338979+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:12.854403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":476,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.338934+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:56:10.338934+0000","last_peered":"2024-09-13T14:56:10.338934+0000","last_clean":"2024-09-13T14:56:10.338934+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:56:10.338934+0000","last_undegraded":"2024-09-13T14:56:10.338934+0000","last_fullsized":"2024-09-13T14:56:10.338934+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":150,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342995+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:56:10.342995+0000","last_peered":"2024-09-13T14:56:10.342995+0000","last_clean":"2024-09-13T14:56:10.342995+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:56:10.342995+0000","last_undegraded":"2024-09-13T14:56:10.342995+0000","last_fullsized":"2024-09-13T14:56:10.342995+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":464,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339646+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:56:10.339646+0000","last_peered":"2024-09-13T14:56:10.339646+0000","last_clean":"2024-09-13T14:56:10.339646+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:56:10.339646+0000","last_undegraded":"2024-09-13T14:56:10.339646+0000","last_fullsized":"2024-09-13T14:56:10.339646+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":97,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340969+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:56:10.340969+0000","last_peered":"2024-09-13T14:56:10.340969+0000","last_clean":"2024-09-13T14:56:10.340969+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:56:10.340969+0000","last_undegraded":"2024-09-13T14:56:10.340969+0000","last_fullsized":"2024-09-13T14:56:10.340969+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":14,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339342+0000","last_change":"2024-09-13T14:56:09.034509+0000","last_active":"2024-09-13T14:56:10.339342+0000","last_peered":"2024-09-13T14:56:10.339342+0000","last_clean":"2024-09-13T14:56:10.339342+0000","last_became_active":"2024-09-13T14:56:09.034213+0000","last_became_peered":"2024-09-13T14:56:09.034213+0000","last_unstale":"2024-09-13T14:56:10.339342+0000","last_undegraded":"2024-09-13T14:56:10.339342+0000","last_fullsized":"2024-09-13T14:56:10.339342+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:02.978647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":495,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185037+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:56:18.185037+0000","last_peered":"2024-09-13T14:56:18.185037+0000","last_clean":"2024-09-13T14:56:18.185037+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:56:18.185037+0000","last_undegraded":"2024-09-13T14:56:18.185037+0000","last_fullsized":"2024-09-13T14:56:18.185037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":289,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340147+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:56:10.340147+0000","last_peered":"2024-09-13T14:56:10.340147+0000","last_clean":"2024-09-13T14:56:10.340147+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:56:10.340147+0000","last_undegraded":"2024-09-13T14:56:10.340147+0000","last_fullsized":"2024-09-13T14:56:10.340147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":451,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183610+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:56:18.183610+0000","last_peered":"2024-09-13T14:56:18.183610+0000","last_clean":"2024-09-13T14:56:18.183610+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:56:18.183610+0000","last_undegraded":"2024-09-13T14:56:18.183610+0000","last_fullsized":"2024-09-13T14:56:18.183610+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":250,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342862+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:56:10.342862+0000","last_peered":"2024-09-13T14:56:10.342862+0000","last_clean":"2024-09-13T14:56:10.342862+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:56:10.342862+0000","last_undegraded":"2024-09-13T14:56:10.342862+0000","last_fullsized":"2024-09-13T14:56:10.342862+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":17,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184304+0000","last_change":"2024-09-13T14:56:09.034272+0000","last_active":"2024-09-13T14:56:18.184304+0000","last_peered":"2024-09-13T14:56:18.184304+0000","last_clean":"2024-09-13T14:56:18.184304+0000","last_became_active":"2024-09-13T14:56:09.033933+0000","last_became_peered":"2024-09-13T14:56:09.033933+0000","last_unstale":"2024-09-13T14:56:18.184304+0000","last_undegraded":"2024-09-13T14:56:18.184304+0000","last_fullsized":"2024-09-13T14:56:18.184304+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:40.177740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":75,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339382+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:56:10.339382+0000","last_peered":"2024-09-13T14:56:10.339382+0000","last_clean":"2024-09-13T14:56:10.339382+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:56:10.339382+0000","last_undegraded":"2024-09-13T14:56:10.339382+0000","last_fullsized":"2024-09-13T14:56:10.339382+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":309,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184485+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:56:18.184485+0000","last_peered":"2024-09-13T14:56:18.184485+0000","last_clean":"2024-09-13T14:56:18.184485+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:56:18.184485+0000","last_undegraded":"2024-09-13T14:56:18.184485+0000","last_fullsized":"2024-09-13T14:56:18.184485+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":292,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339924+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:56:10.339924+0000","last_peered":"2024-09-13T14:56:10.339924+0000","last_clean":"2024-09-13T14:56:10.339924+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:56:10.339924+0000","last_undegraded":"2024-09-13T14:56:10.339924+0000","last_fullsized":"2024-09-13T14:56:10.339924+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":270,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.343138+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:56:10.343138+0000","last_peered":"2024-09-13T14:56:10.343138+0000","last_clean":"2024-09-13T14:56:10.343138+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:56:10.343138+0000","last_undegraded":"2024-09-13T14:56:10.343138+0000","last_fullsized":"2024-09-13T14:56:10.343138+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":17,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184282+0000","last_change":"2024-09-13T14:56:09.101249+0000","last_active":"2024-09-13T14:56:18.184282+0000","last_peered":"2024-09-13T14:56:18.184282+0000","last_clean":"2024-09-13T14:56:18.184282+0000","last_became_active":"2024-09-13T14:56:09.100796+0000","last_became_peered":"2024-09-13T14:56:09.100796+0000","last_unstale":"2024-09-13T14:56:18.184282+0000","last_undegraded":"2024-09-13T14:56:18.184282+0000","last_fullsized":"2024-09-13T14:56:18.184282+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:37.036972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":363,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183881+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:56:18.183881+0000","last_peered":"2024-09-13T14:56:18.183881+0000","last_clean":"2024-09-13T14:56:18.183881+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:56:18.183881+0000","last_undegraded":"2024-09-13T14:56:18.183881+0000","last_fullsized":"2024-09-13T14:56:18.183881+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":277,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341780+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:56:10.341780+0000","last_peered":"2024-09-13T14:56:10.341780+0000","last_clean":"2024-09-13T14:56:10.341780+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:56:10.341780+0000","last_undegraded":"2024-09-13T14:56:10.341780+0000","last_fullsized":"2024-09-13T14:56:10.341780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":288,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339744+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:56:10.339744+0000","last_peered":"2024-09-13T14:56:10.339744+0000","last_clean":"2024-09-13T14:56:10.339744+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:56:10.339744+0000","last_undegraded":"2024-09-13T14:56:10.339744+0000","last_fullsized":"2024-09-13T14:56:10.339744+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":97,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339340+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:56:10.339340+0000","last_peered":"2024-09-13T14:56:10.339340+0000","last_clean":"2024-09-13T14:56:10.339340+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:56:10.339340+0000","last_undegraded":"2024-09-13T14:56:10.339340+0000","last_fullsized":"2024-09-13T14:56:10.339340+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":11,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342511+0000","last_change":"2024-09-13T14:56:09.101282+0000","last_active":"2024-09-13T14:56:10.342511+0000","last_peered":"2024-09-13T14:56:10.342511+0000","last_clean":"2024-09-13T14:56:10.342511+0000","last_became_active":"2024-09-13T14:56:09.100956+0000","last_became_peered":"2024-09-13T14:56:09.100956+0000","last_unstale":"2024-09-13T14:56:10.342511+0000","last_undegraded":"2024-09-13T14:56:10.342511+0000","last_fullsized":"2024-09-13T14:56:10.342511+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:39.057985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":190,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.059699+0000","last_change":"2024-09-13T14:53:42.008298+0000","last_active":"2024-09-13T14:56:18.059699+0000","last_peered":"2024-09-13T14:56:18.059699+0000","last_clean":"2024-09-13T14:56:18.059699+0000","last_became_active":"2024-09-13T14:53:42.007766+0000","last_became_peered":"2024-09-13T14:53:42.007766+0000","last_unstale":"2024-09-13T14:56:18.059699+0000","last_undegraded":"2024-09-13T14:56:18.059699+0000","last_fullsized":"2024-09-13T14:56:18.059699+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:29.235755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":303,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183517+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:56:18.183517+0000","last_peered":"2024-09-13T14:56:18.183517+0000","last_clean":"2024-09-13T14:56:18.183517+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:56:18.183517+0000","last_undegraded":"2024-09-13T14:56:18.183517+0000","last_fullsized":"2024-09-13T14:56:18.183517+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":494,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184197+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:56:18.184197+0000","last_peered":"2024-09-13T14:56:18.184197+0000","last_clean":"2024-09-13T14:56:18.184197+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:56:18.184197+0000","last_undegraded":"2024-09-13T14:56:18.184197+0000","last_fullsized":"2024-09-13T14:56:18.184197+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":71,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339708+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:56:10.339708+0000","last_peered":"2024-09-13T14:56:10.339708+0000","last_clean":"2024-09-13T14:56:10.339708+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:56:10.339708+0000","last_undegraded":"2024-09-13T14:56:10.339708+0000","last_fullsized":"2024-09-13T14:56:10.339708+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":17,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183824+0000","last_change":"2024-09-13T14:56:09.032714+0000","last_active":"2024-09-13T14:56:18.183824+0000","last_peered":"2024-09-13T14:56:18.183824+0000","last_clean":"2024-09-13T14:56:18.183824+0000","last_became_active":"2024-09-13T14:56:09.032114+0000","last_became_peered":"2024-09-13T14:56:09.032114+0000","last_unstale":"2024-09-13T14:56:18.183824+0000","last_undegraded":"2024-09-13T14:56:18.183824+0000","last_fullsized":"2024-09-13T14:56:18.183824+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:53.695180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":517,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183973+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:56:18.183973+0000","last_peered":"2024-09-13T14:56:18.183973+0000","last_clean":"2024-09-13T14:56:18.183973+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:56:18.183973+0000","last_undegraded":"2024-09-13T14:56:18.183973+0000","last_fullsized":"2024-09-13T14:56:18.183973+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":526,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183819+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:56:18.183819+0000","last_peered":"2024-09-13T14:56:18.183819+0000","last_clean":"2024-09-13T14:56:18.183819+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:56:18.183819+0000","last_undegraded":"2024-09-13T14:56:18.183819+0000","last_fullsized":"2024-09-13T14:56:18.183819+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":326,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341267+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:56:10.341267+0000","last_peered":"2024-09-13T14:56:10.341267+0000","last_clean":"2024-09-13T14:56:10.341267+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:56:10.341267+0000","last_undegraded":"2024-09-13T14:56:10.341267+0000","last_fullsized":"2024-09-13T14:56:10.341267+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":250,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342941+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:56:10.342941+0000","last_peered":"2024-09-13T14:56:10.342941+0000","last_clean":"2024-09-13T14:56:10.342941+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:56:10.342941+0000","last_undegraded":"2024-09-13T14:56:10.342941+0000","last_fullsized":"2024-09-13T14:56:10.342941+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":11,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340486+0000","last_change":"2024-09-13T14:56:09.035158+0000","last_active":"2024-09-13T14:56:10.340486+0000","last_peered":"2024-09-13T14:56:10.340486+0000","last_clean":"2024-09-13T14:56:10.340486+0000","last_became_active":"2024-09-13T14:56:09.034847+0000","last_became_peered":"2024-09-13T14:56:09.034847+0000","last_unstale":"2024-09-13T14:56:10.340486+0000","last_undegraded":"2024-09-13T14:56:10.340486+0000","last_fullsized":"2024-09-13T14:56:10.340486+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:55.768082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":259,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.343082+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:56:10.343082+0000","last_peered":"2024-09-13T14:56:10.343082+0000","last_clean":"2024-09-13T14:56:10.343082+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:56:10.343082+0000","last_undegraded":"2024-09-13T14:56:10.343082+0000","last_fullsized":"2024-09-13T14:56:10.343082+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":424,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339221+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:56:10.339221+0000","last_peered":"2024-09-13T14:56:10.339221+0000","last_clean":"2024-09-13T14:56:10.339221+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:56:10.339221+0000","last_undegraded":"2024-09-13T14:56:10.339221+0000","last_fullsized":"2024-09-13T14:56:10.339221+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":287,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339974+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:56:10.339974+0000","last_peered":"2024-09-13T14:56:10.339974+0000","last_clean":"2024-09-13T14:56:10.339974+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:56:10.339974+0000","last_undegraded":"2024-09-13T14:56:10.339974+0000","last_fullsized":"2024-09-13T14:56:10.339974+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":71,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341033+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:56:10.341033+0000","last_peered":"2024-09-13T14:56:10.341033+0000","last_clean":"2024-09-13T14:56:10.341033+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:56:10.341033+0000","last_undegraded":"2024-09-13T14:56:10.341033+0000","last_fullsized":"2024-09-13T14:56:10.341033+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":11,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341004+0000","last_change":"2024-09-13T14:56:09.035403+0000","last_active":"2024-09-13T14:56:10.341004+0000","last_peered":"2024-09-13T14:56:10.341004+0000","last_clean":"2024-09-13T14:56:10.341004+0000","last_became_active":"2024-09-13T14:56:09.035064+0000","last_became_peered":"2024-09-13T14:56:09.035064+0000","last_unstale":"2024-09-13T14:56:10.341004+0000","last_undegraded":"2024-09-13T14:56:10.341004+0000","last_fullsized":"2024-09-13T14:56:10.341004+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:45:44.521747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":149,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184916+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:56:18.184916+0000","last_peered":"2024-09-13T14:56:18.184916+0000","last_clean":"2024-09-13T14:56:18.184916+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:56:18.184916+0000","last_undegraded":"2024-09-13T14:56:18.184916+0000","last_fullsized":"2024-09-13T14:56:18.184916+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":434,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339076+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:56:10.339076+0000","last_peered":"2024-09-13T14:56:10.339076+0000","last_clean":"2024-09-13T14:56:10.339076+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:56:10.339076+0000","last_undegraded":"2024-09-13T14:56:10.339076+0000","last_fullsized":"2024-09-13T14:56:10.339076+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":97,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339276+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:56:10.339276+0000","last_peered":"2024-09-13T14:56:10.339276+0000","last_clean":"2024-09-13T14:56:10.339276+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:56:10.339276+0000","last_undegraded":"2024-09-13T14:56:10.339276+0000","last_fullsized":"2024-09-13T14:56:10.339276+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":17,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.059672+0000","last_change":"2024-09-13T14:56:09.037715+0000","last_active":"2024-09-13T14:56:18.059672+0000","last_peered":"2024-09-13T14:56:18.059672+0000","last_clean":"2024-09-13T14:56:18.059672+0000","last_became_active":"2024-09-13T14:56:09.037383+0000","last_became_peered":"2024-09-13T14:56:09.037383+0000","last_unstale":"2024-09-13T14:56:18.059672+0000","last_undegraded":"2024-09-13T14:56:18.059672+0000","last_fullsized":"2024-09-13T14:56:18.059672+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:33.806063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":296,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340025+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:56:10.340025+0000","last_peered":"2024-09-13T14:56:10.340025+0000","last_clean":"2024-09-13T14:56:10.340025+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:56:10.340025+0000","last_undegraded":"2024-09-13T14:56:10.340025+0000","last_fullsized":"2024-09-13T14:56:10.340025+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":277,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340542+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:56:10.340542+0000","last_peered":"2024-09-13T14:56:10.340542+0000","last_clean":"2024-09-13T14:56:10.340542+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:56:10.340542+0000","last_undegraded":"2024-09-13T14:56:10.340542+0000","last_fullsized":"2024-09-13T14:56:10.340542+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":487,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184805+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:56:18.184805+0000","last_peered":"2024-09-13T14:56:18.184805+0000","last_clean":"2024-09-13T14:56:18.184805+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:56:18.184805+0000","last_undegraded":"2024-09-13T14:56:18.184805+0000","last_fullsized":"2024-09-13T14:56:18.184805+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342811+0000","last_change":"2024-09-13T14:56:09.035196+0000","last_active":"2024-09-13T14:56:10.342811+0000","last_peered":"2024-09-13T14:56:10.342811+0000","last_clean":"2024-09-13T14:56:10.342811+0000","last_became_active":"2024-09-13T14:56:09.034906+0000","last_became_peered":"2024-09-13T14:56:09.034906+0000","last_unstale":"2024-09-13T14:56:10.342811+0000","last_undegraded":"2024-09-13T14:56:10.342811+0000","last_fullsized":"2024-09-13T14:56:10.342811+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:43.214619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":293,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341171+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:56:10.341171+0000","last_peered":"2024-09-13T14:56:10.341171+0000","last_clean":"2024-09-13T14:56:10.341171+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:56:10.341171+0000","last_undegraded":"2024-09-13T14:56:10.341171+0000","last_fullsized":"2024-09-13T14:56:10.341171+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":297,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342749+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:56:10.342749+0000","last_peered":"2024-09-13T14:56:10.342749+0000","last_clean":"2024-09-13T14:56:10.342749+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:56:10.342749+0000","last_undegraded":"2024-09-13T14:56:10.342749+0000","last_fullsized":"2024-09-13T14:56:10.342749+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":79,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339304+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:56:10.339304+0000","last_peered":"2024-09-13T14:56:10.339304+0000","last_clean":"2024-09-13T14:56:10.339304+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:56:10.339304+0000","last_undegraded":"2024-09-13T14:56:10.339304+0000","last_fullsized":"2024-09-13T14:56:10.339304+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342588+0000","last_change":"2024-09-13T14:56:09.035205+0000","last_active":"2024-09-13T14:56:10.342588+0000","last_peered":"2024-09-13T14:56:10.342588+0000","last_clean":"2024-09-13T14:56:10.342588+0000","last_became_active":"2024-09-13T14:56:09.034915+0000","last_became_peered":"2024-09-13T14:56:09.034915+0000","last_unstale":"2024-09-13T14:56:10.342588+0000","last_undegraded":"2024-09-13T14:56:10.342588+0000","last_fullsized":"2024-09-13T14:56:10.342588+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.408509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":276,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339526+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:56:10.339526+0000","last_peered":"2024-09-13T14:56:10.339526+0000","last_clean":"2024-09-13T14:56:10.339526+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:56:10.339526+0000","last_undegraded":"2024-09-13T14:56:10.339526+0000","last_fullsized":"2024-09-13T14:56:10.339526+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":464,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339649+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:56:10.339649+0000","last_peered":"2024-09-13T14:56:10.339649+0000","last_clean":"2024-09-13T14:56:10.339649+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:56:10.339649+0000","last_undegraded":"2024-09-13T14:56:10.339649+0000","last_fullsized":"2024-09-13T14:56:10.339649+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":97,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340724+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:56:10.340724+0000","last_peered":"2024-09-13T14:56:10.340724+0000","last_clean":"2024-09-13T14:56:10.340724+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:56:10.340724+0000","last_undegraded":"2024-09-13T14:56:10.340724+0000","last_fullsized":"2024-09-13T14:56:10.340724+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":17,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184942+0000","last_change":"2024-09-13T14:56:09.033530+0000","last_active":"2024-09-13T14:56:18.184942+0000","last_peered":"2024-09-13T14:56:18.184942+0000","last_clean":"2024-09-13T14:56:18.184942+0000","last_became_active":"2024-09-13T14:56:09.032938+0000","last_became_peered":"2024-09-13T14:56:09.032938+0000","last_unstale":"2024-09-13T14:56:18.184942+0000","last_undegraded":"2024-09-13T14:56:18.184942+0000","last_fullsized":"2024-09-13T14:56:18.184942+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:58:50.158201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":506,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184923+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:56:18.184923+0000","last_peered":"2024-09-13T14:56:18.184923+0000","last_clean":"2024-09-13T14:56:18.184923+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:56:18.184923+0000","last_undegraded":"2024-09-13T14:56:18.184923+0000","last_fullsized":"2024-09-13T14:56:18.184923+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":275,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340251+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:56:10.340251+0000","last_peered":"2024-09-13T14:56:10.340251+0000","last_clean":"2024-09-13T14:56:10.340251+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:56:10.340251+0000","last_undegraded":"2024-09-13T14:56:10.340251+0000","last_fullsized":"2024-09-13T14:56:10.340251+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":97,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339820+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:56:10.339820+0000","last_peered":"2024-09-13T14:56:10.339820+0000","last_clean":"2024-09-13T14:56:10.339820+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:56:10.339820+0000","last_undegraded":"2024-09-13T14:56:10.339820+0000","last_fullsized":"2024-09-13T14:56:10.339820+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341240+0000","last_change":"2024-09-13T14:56:09.033553+0000","last_active":"2024-09-13T14:56:10.341240+0000","last_peered":"2024-09-13T14:56:10.341240+0000","last_clean":"2024-09-13T14:56:10.341240+0000","last_became_active":"2024-09-13T14:56:09.033287+0000","last_became_peered":"2024-09-13T14:56:09.033287+0000","last_unstale":"2024-09-13T14:56:10.341240+0000","last_undegraded":"2024-09-13T14:56:10.341240+0000","last_fullsized":"2024-09-13T14:56:10.341240+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:54.220513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":289,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342670+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:56:10.342670+0000","last_peered":"2024-09-13T14:56:10.342670+0000","last_clean":"2024-09-13T14:56:10.342670+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:56:10.342670+0000","last_undegraded":"2024-09-13T14:56:10.342670+0000","last_fullsized":"2024-09-13T14:56:10.342670+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":290,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340620+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:56:10.340620+0000","last_peered":"2024-09-13T14:56:10.340620+0000","last_clean":"2024-09-13T14:56:10.340620+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:56:10.340620+0000","last_undegraded":"2024-09-13T14:56:10.340620+0000","last_fullsized":"2024-09-13T14:56:10.340620+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":250,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.343066+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:56:10.343066+0000","last_peered":"2024-09-13T14:56:10.343066+0000","last_clean":"2024-09-13T14:56:10.343066+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:56:10.343066+0000","last_undegraded":"2024-09-13T14:56:10.343066+0000","last_fullsized":"2024-09-13T14:56:10.343066+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342330+0000","last_change":"2024-09-13T14:56:09.032953+0000","last_active":"2024-09-13T14:56:10.342330+0000","last_peered":"2024-09-13T14:56:10.342330+0000","last_clean":"2024-09-13T14:56:10.342330+0000","last_became_active":"2024-09-13T14:56:09.032634+0000","last_became_peered":"2024-09-13T14:56:09.032634+0000","last_unstale":"2024-09-13T14:56:10.342330+0000","last_undegraded":"2024-09-13T14:56:10.342330+0000","last_fullsized":"2024-09-13T14:56:10.342330+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:45.162779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":303,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340415+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:56:10.340415+0000","last_peered":"2024-09-13T14:56:10.340415+0000","last_clean":"2024-09-13T14:56:10.340415+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:56:10.340415+0000","last_undegraded":"2024-09-13T14:56:10.340415+0000","last_fullsized":"2024-09-13T14:56:10.340415+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":280,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341558+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:56:10.341558+0000","last_peered":"2024-09-13T14:56:10.341558+0000","last_clean":"2024-09-13T14:56:10.341558+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:56:10.341558+0000","last_undegraded":"2024-09-13T14:56:10.341558+0000","last_fullsized":"2024-09-13T14:56:10.341558+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":192,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339776+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:56:10.339776+0000","last_peered":"2024-09-13T14:56:10.339776+0000","last_clean":"2024-09-13T14:56:10.339776+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:56:10.339776+0000","last_undegraded":"2024-09-13T14:56:10.339776+0000","last_fullsized":"2024-09-13T14:56:10.339776+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340641+0000","last_change":"2024-09-13T14:56:09.031898+0000","last_active":"2024-09-13T14:56:10.340641+0000","last_peered":"2024-09-13T14:56:10.340641+0000","last_clean":"2024-09-13T14:56:10.340641+0000","last_became_active":"2024-09-13T14:56:09.031546+0000","last_became_peered":"2024-09-13T14:56:09.031546+0000","last_unstale":"2024-09-13T14:56:10.340641+0000","last_undegraded":"2024-09-13T14:56:10.340641+0000","last_fullsized":"2024-09-13T14:56:10.340641+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:18.018588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":85,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339442+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:56:10.339442+0000","last_peered":"2024-09-13T14:56:10.339442+0000","last_clean":"2024-09-13T14:56:10.339442+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:56:10.339442+0000","last_undegraded":"2024-09-13T14:56:10.339442+0000","last_fullsized":"2024-09-13T14:56:10.339442+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":75,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.341284+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:56:10.341284+0000","last_peered":"2024-09-13T14:56:10.341284+0000","last_clean":"2024-09-13T14:56:10.341284+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:56:10.341284+0000","last_undegraded":"2024-09-13T14:56:10.341284+0000","last_fullsized":"2024-09-13T14:56:10.341284+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":81,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340342+0000","last_change":"2024-09-13T14:52:02.567722+0000","last_active":"2024-09-13T14:56:10.340342+0000","last_peered":"2024-09-13T14:56:10.340342+0000","last_clean":"2024-09-13T14:56:10.340342+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:56:10.340342+0000","last_undegraded":"2024-09-13T14:56:10.340342+0000","last_fullsized":"2024-09-13T14:56:10.340342+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:52:02.567611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:52:02.567611+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:03.419129+0000","scrub_duration":0.0030819879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":325,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.342543+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:56:10.342543+0000","last_peered":"2024-09-13T14:56:10.342543+0000","last_clean":"2024-09-13T14:56:10.342543+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:56:10.342543+0000","last_undegraded":"2024-09-13T14:56:10.342543+0000","last_fullsized":"2024-09-13T14:56:10.342543+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":472,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339489+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:56:10.339489+0000","last_peered":"2024-09-13T14:56:10.339489+0000","last_clean":"2024-09-13T14:56:10.339489+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:56:10.339489+0000","last_undegraded":"2024-09-13T14:56:10.339489+0000","last_fullsized":"2024-09-13T14:56:10.339489+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":107,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339547+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:56:10.339547+0000","last_peered":"2024-09-13T14:56:10.339547+0000","last_clean":"2024-09-13T14:56:10.339547+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:56:10.339547+0000","last_undegraded":"2024-09-13T14:56:10.339547+0000","last_fullsized":"2024-09-13T14:56:10.339547+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":271,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340868+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:56:10.340868+0000","last_peered":"2024-09-13T14:56:10.340868+0000","last_clean":"2024-09-13T14:56:10.340868+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:56:10.340868+0000","last_undegraded":"2024-09-13T14:56:10.340868+0000","last_fullsized":"2024-09-13T14:56:10.340868+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":519,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184143+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:56:18.184143+0000","last_peered":"2024-09-13T14:56:18.184143+0000","last_clean":"2024-09-13T14:56:18.184143+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:56:18.184143+0000","last_undegraded":"2024-09-13T14:56:18.184143+0000","last_fullsized":"2024-09-13T14:56:18.184143+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":187,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339485+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:56:10.339485+0000","last_peered":"2024-09-13T14:56:10.339485+0000","last_clean":"2024-09-13T14:56:10.339485+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:56:10.339485+0000","last_undegraded":"2024-09-13T14:56:10.339485+0000","last_fullsized":"2024-09-13T14:56:10.339485+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":525,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184829+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:56:18.184829+0000","last_peered":"2024-09-13T14:56:18.184829+0000","last_clean":"2024-09-13T14:56:18.184829+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:56:18.184829+0000","last_undegraded":"2024-09-13T14:56:18.184829+0000","last_fullsized":"2024-09-13T14:56:18.184829+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":75,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339754+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:56:10.339754+0000","last_peered":"2024-09-13T14:56:10.339754+0000","last_clean":"2024-09-13T14:56:10.339754+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:56:10.339754+0000","last_undegraded":"2024-09-13T14:56:10.339754+0000","last_fullsized":"2024-09-13T14:56:10.339754+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":97,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.339400+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:56:10.339400+0000","last_peered":"2024-09-13T14:56:10.339400+0000","last_clean":"2024-09-13T14:56:10.339400+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:56:10.339400+0000","last_undegraded":"2024-09-13T14:56:10.339400+0000","last_fullsized":"2024-09-13T14:56:10.339400+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":296,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:56:10.340358+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:56:10.340358+0000","last_peered":"2024-09-13T14:56:10.340358+0000","last_clean":"2024-09-13T14:56:10.340358+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:56:10.340358+0000","last_undegraded":"2024-09-13T14:56:10.340358+0000","last_fullsized":"2024-09-13T14:56:10.340358+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":184,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184137+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:56:18.184137+0000","last_peered":"2024-09-13T14:56:18.184137+0000","last_clean":"2024-09-13T14:56:18.184137+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:56:18.184137+0000","last_undegraded":"2024-09-13T14:56:18.184137+0000","last_fullsized":"2024-09-13T14:56:18.184137+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":483,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183926+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:56:18.183926+0000","last_peered":"2024-09-13T14:56:18.183926+0000","last_clean":"2024-09-13T14:56:18.183926+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:56:18.183926+0000","last_undegraded":"2024-09-13T14:56:18.183926+0000","last_fullsized":"2024-09-13T14:56:18.183926+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365239,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939984,"kb_used_data":26436,"kb_used_omap":1,"kb_used_meta":913534,"kb_avail":92805168,"statfs":{"total":95995035648,"available":95032492032,"internally_reserved":0,"allocated":27070464,"data_stored":25283721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":935459160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561469,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037512,"kb_used_data":139836,"kb_used_omap":1,"kb_used_meta":897662,"kb_avail":92707640,"statfs":{"total":95995035648,"available":94932623360,"internally_reserved":0,"allocated":143192064,"data_stored":141413336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919206232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790401,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1202112,"kb_used_data":4340,"kb_used_omap":1,"kb_used_meta":1197758,"kb_avail":92543040,"statfs":{"total":95995035648,"available":94764072960,"internally_reserved":0,"allocated":4444160,"data_stored":2669742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1226504536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986629,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902668,"kb_used_data":4096,"kb_used_omap":1,"kb_used_meta":898558,"kb_avail":92842484,"statfs":{"total":95995035648,"available":95070703616,"internally_reserved":0,"allocated":4194304,"data_stored":2411818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920123736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519784,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776172,"kb_used_data":32740,"kb_used_omap":4,"kb_used_meta":743419,"kb_avail":92968980,"statfs":{"total":95995035648,"available":95200235520,"internally_reserved":0,"allocated":33525760,"data_stored":31758858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":761261504},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304700,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":711876,"kb_used_data":4344,"kb_used_omap":6,"kb_used_meta":707513,"kb_avail":93033276,"statfs":{"total":95995035648,"available":95266074624,"internally_reserved":0,"allocated":4448256,"data_stored":2670266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":724493767},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828003,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1035568,"kb_used_data":4068,"kb_used_omap":5,"kb_used_meta":1031482,"kb_avail":92709584,"statfs":{"total":95995035648,"available":94934614016,"internally_reserved":0,"allocated":4165632,"data_stored":2398157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1056237804},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285868,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905660,"kb_used_data":4400,"kb_used_omap":5,"kb_used_meta":901242,"kb_avail":92839492,"statfs":{"total":95995035648,"available":95067639808,"internally_reserved":0,"allocated":4505600,"data_stored":2734018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":922871867},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:56:25.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:56:25.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:25.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:25.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:25.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:25.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:25.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:25.977 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:26.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:26.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:26.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:26.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:26.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:26.287 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:26.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:26.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 12305 12289 191.977 166.25 0.00312099 0.00520335 2024-09-13T14:56:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:26.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:26.532 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:26.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:26.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:26.700 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:26.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:26.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:26.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:56:26.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:26.963 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:27.065 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:27.096 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:27.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:27 smithi003 ceph-mon[104007]: pgmap v1108: 109 pgs: 109 active+clean; 265 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 26 MiB/s wr, 423 op/s 2024-09-13T14:56:27.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:27 smithi003 ceph-mon[107278]: pgmap v1108: 109 pgs: 109 active+clean; 265 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 26 MiB/s wr, 423 op/s 2024-09-13T14:56:27.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:27.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:27.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.379 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:27.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:27.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 14924 14908 186.316 163.688 0.00245033 0.00536016 2024-09-13T14:56:27.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:27 smithi019 ceph-mon[91296]: pgmap v1108: 109 pgs: 109 active+clean; 265 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 26 MiB/s wr, 423 op/s 2024-09-13T14:56:27.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:27.510 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:27.612 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:27.647 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:27.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.785 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:27.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:27.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:27.921 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:28.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:28.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:28.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:28.197 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:28.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:28.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:28.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 17603 17587 183.167 167.438 0.00775731 0.00545121 2024-09-13T14:56:28.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:28.491 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:28.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:28.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:28.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:28.768 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:28.870 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:28.903 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:29.005 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:29.037 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:29.139 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:29.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:28 smithi019 ceph-mon[91296]: pgmap v1109: 109 pgs: 109 active+clean; 508 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 51 MiB/s wr, 811 op/s 2024-09-13T14:56:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:28 smithi003 ceph-mon[107278]: pgmap v1109: 109 pgs: 109 active+clean; 508 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 51 MiB/s wr, 811 op/s 2024-09-13T14:56:29.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:29.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:29.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:28 smithi003 ceph-mon[104007]: pgmap v1109: 109 pgs: 109 active+clean; 508 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 51 MiB/s wr, 811 op/s 2024-09-13T14:56:29.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:29.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:29.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 20104 20088 179.329 156.312 0.00544239 0.0055705 2024-09-13T14:56:29.501 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:29.532 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:29.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:29.669 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:29.708 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:29.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:29.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:29.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:29.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:29.958 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:30.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:30.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:30.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:30.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:30.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:30.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:30.380 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:30.416 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 22251 22235 173.685 134.188 0.0104696 0.00575162 2024-09-13T14:56:30.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:30.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:30.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:30.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:56:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:56:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:30.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:30.782 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:30.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:30.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:31.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.053 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:31.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:31.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:31.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:31.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:31.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:31.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:31.417 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 23859 23843 165.547 100.5 0.00671859 0.00603491 2024-09-13T14:56:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:31 smithi019 ceph-mon[91296]: pgmap v1110: 109 pgs: 109 active+clean; 1003 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-13T14:56:31.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:31 smithi003 ceph-mon[107278]: pgmap v1110: 109 pgs: 109 active+clean; 1003 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-13T14:56:31.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:31 smithi003 ceph-mon[104007]: pgmap v1110: 109 pgs: 109 active+clean; 1003 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-13T14:56:31.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:31.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:31.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:31.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:31.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:31.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:31.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:31.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:31.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:32.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:32.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:32.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:32.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:32.264 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:32.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:32.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:32.417 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 25824 25808 161.272 122.812 0.00322877 0.00619466 2024-09-13T14:56:32.508 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:32.542 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:32.644 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:32.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:32.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:32.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:32.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:32.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:32.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:33.069 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:33.100 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:33.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:33.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:33.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:33.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:33.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:33.417 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 27701 27685 157.275 117.312 0.00324483 0.00635208 2024-09-13T14:56:33.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:33 smithi019 ceph-mon[91296]: pgmap v1111: 109 pgs: 109 active+clean; 1.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-13T14:56:33.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:33 smithi003 ceph-mon[107278]: pgmap v1111: 109 pgs: 109 active+clean; 1.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-13T14:56:33.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:33 smithi003 ceph-mon[104007]: pgmap v1111: 109 pgs: 109 active+clean; 1.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-13T14:56:33.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:33.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:33.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:33.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:33.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:33.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:33.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:33.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:33.916 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:34.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:34.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:34.144 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:34.176 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:34.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:34.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:34.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:34.417 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 29454 29438 153.299 109.562 0.00321936 0.00651629 2024-09-13T14:56:34.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:56:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:56:34] "GET /metrics HTTP/1.1" 200 36516 "" "Prometheus/2.51.0" 2024-09-13T14:56:34.446 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:34.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:34.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:34.707 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:34.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:34.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:34.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:34.973 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:34.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:35.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:35.106 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:35.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:35.208 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.234 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:35.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:35.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:35.367 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:35.417 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 31048 31032 149.17 99.625 0.00435428 0.00669837 2024-09-13T14:56:35.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:35 smithi019 ceph-mon[91296]: pgmap v1112: 109 pgs: 109 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:56:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:35 smithi003 ceph-mon[107278]: pgmap v1112: 109 pgs: 109 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:56:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:35 smithi003 ceph-mon[104007]: pgmap v1112: 109 pgs: 109 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:56:35.469 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.502 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:35.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:35.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:35.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:35.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:35.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:35.866 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:35.904 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:35.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:36.006 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.042 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:36.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:36.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:36.176 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:36.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:36.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:36.417 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 32877 32861 146.679 114.312 0.011583 0.00681204 2024-09-13T14:56:36.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:36.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:36.568 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:36.603 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:36.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:36.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:36.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:36.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:36.988 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:37.024 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:37.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:37.129 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:37 smithi003 ceph-mon[107278]: pgmap v1113: 109 pgs: 109 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 144 MiB/s wr, 2.31k op/s 2024-09-13T14:56:37.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:37 smithi003 ceph-mon[104007]: pgmap v1113: 109 pgs: 109 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 144 MiB/s wr, 2.31k op/s 2024-09-13T14:56:37.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:37.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:37.303 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:37.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:37.418 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 34916 34900 145.391 127.438 0.00486593 0.00687293 2024-09-13T14:56:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:37 smithi019 ceph-mon[91296]: pgmap v1113: 109 pgs: 109 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 144 MiB/s wr, 2.31k op/s 2024-09-13T14:56:37.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:37.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:37.573 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:37.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:37.709 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:37.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:37.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:37.860 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:37.962 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:37.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:38.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:38.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:38.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:38.353 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:56:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:56:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:56:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:56:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96867 11.11 11:89922eea:::benchmark_data_smithi003_229782_object96866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.108653+0000", 2024-09-13T14:56:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.237021204, 2024-09-13T14:56:38.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.238741235, 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96867 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108653+0000", 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108653+0000", 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108654+0000", 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1600000000000002e-07 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108680+0000", 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6115e-05 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108688+0000", 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.551e-06 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108694+0000", 2024-09-13T14:56:38.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0800000000000002e-06 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108988+0000", 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029367500000000002 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108992+0000", 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3379999999999998e-06 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.265624+0000", 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.15663201 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.265731+0000", 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106443 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.266008+0000", 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027724899999999997 2024-09-13T14:56:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.270633+0000", 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046251069999999998 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.270772+0000", 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139266 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.273210+0000", 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024377140000000001 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.345752+0000", 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.072541319000000007 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.345806+0000", 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4181000000000002e-05 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.347394+0000", 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015887710000000001 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96891 11.11 11:8c6f0dc5:::benchmark_data_smithi003_229782_object96890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.108704+0000", 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23697049399999, 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.241508066, 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.361 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96891 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108704+0000", 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108704+0000", 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108704+0000", 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9200000000000001e-07 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108730+0000", 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5718999999999999e-05 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108739+0000", 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8030000000000003e-06 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108745+0000", 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0979999999999999e-06 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109020+0000", 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027457499999999999 2024-09-13T14:56:38.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109024+0000", 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9210000000000002e-06 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.267630+0000", 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1586063040000001 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.267786+0000", 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015642499999999999 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.267893+0000", 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106619 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.273255+0000", 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053624079999999999 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277163+0000", 2024-09-13T14:56:38.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039079859999999996 2024-09-13T14:56:38.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277221+0000", 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7834999999999998e-05 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.347517+0000", 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070295618000000004 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.347556+0000", 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8967999999999997e-05 2024-09-13T14:56:38.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.350212+0000", 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026562949999999999 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96906 11.11 11:88cfd682:::benchmark_data_smithi003_229782_object96905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.108755+0000", 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.236919903, 2024-09-13T14:56:38.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.243610758, 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96906 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108755+0000", 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108755+0000", 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108755+0000", 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2200000000000004e-07 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108784+0000", 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9030000000000002e-05 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108793+0000", 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0299999999999999e-06 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108799+0000", 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1959999999999996e-06 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109054+0000", 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025438799999999998 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109058+0000", 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3100000000000002e-06 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.269153+0000", 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.160095409 2024-09-13T14:56:38.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.269283+0000", 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012944200000000001 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.269377+0000", 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3849000000000002e-05 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.273302+0000", 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039251420000000004 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277279+0000", 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039773029999999997 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277293+0000", 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.399e-05 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.350469+0000", 2024-09-13T14:56:38.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073176137000000002 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.350559+0000", 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9900000000000003e-05 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.352365+0000", 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018061100000000001 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.376 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96933 11.11 11:894ee60c:::benchmark_data_smithi003_229782_object96932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.108809+0000", 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23686581499999, 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.245549164, 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96933 2024-09-13T14:56:38.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108809+0000", 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108809+0000", 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108809+0000", 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6000000000000004e-07 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108891+0000", 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1736999999999993e-05 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108901+0000", 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8959999999999998e-06 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108907+0000", 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5100000000000004e-06 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109085+0000", 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178157 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109089+0000", 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8399999999999997e-06 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.273335+0000", 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.16424547 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.273488+0000", 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015278899999999999 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.273660+0000", 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017264799999999999 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277300+0000", 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036397299999999999 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277325+0000", 2024-09-13T14:56:38.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5468000000000001e-05 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.347462+0000", 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070136108000000003 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.352548+0000", 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050867150000000003 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.352751+0000", 2024-09-13T14:56:38.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000202809 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.354358+0000", 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016067270000000001 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96998 11.11 11:88a9a6ea:::benchmark_data_smithi003_229782_object96997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.108935+0000", 2024-09-13T14:56:38.384 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23673911500001, 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2474018819999999, 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96998 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108935+0000", 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108935+0000", 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108936+0000", 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4400000000000001e-07 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108966+0000", 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0009999999999999e-05 2024-09-13T14:56:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108974+0000", 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5790000000000004e-06 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108979+0000", 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.408e-06 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109119+0000", 2024-09-13T14:56:38.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014053700000000001 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109123+0000", 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7440000000000001e-06 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.275411+0000", 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1662875879999999 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.275551+0000", 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014086200000000001 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.275706+0000", 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000154365 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277372+0000", 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016658 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277385+0000", 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3817e-05 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.350397+0000", 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073011106000000006 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.354699+0000", 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043028850000000002 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.354759+0000", 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9416000000000002e-05 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.356337+0000", 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015783209999999999 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.391 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97010 11.11 11:8a9ea480:::benchmark_data_smithi003_229782_object97009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.108985+0000", 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23668968699999, 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.249070382, 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97010 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108985+0000", 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108985+0000", 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.108985+0000", 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2300000000000002e-07 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109006+0000", 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1101e-05 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109011+0000", 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8740000000000003e-06 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109015+0000", 2024-09-13T14:56:38.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4759999999999998e-06 2024-09-13T14:56:38.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109150+0000", 2024-09-13T14:56:38.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:38.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013587899999999999 2024-09-13T14:56:38.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109154+0000", 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7100000000000001e-06 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277410+0000", 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.168256014 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277504+0000", 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3942000000000003e-05 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.277594+0000", 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0388000000000003e-05 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.347409+0000", 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.069814284000000004 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.350272+0000", 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028628439999999998 2024-09-13T14:56:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.352497+0000", 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022258970000000001 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.356398+0000", 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039009119999999999 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.356465+0000", 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7099000000000003e-05 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.358055+0000", 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015896389999999999 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97011 11.11 11:8c869d63:::benchmark_data_smithi003_229782_object97010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.400 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.109019+0000", 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23665519900001, 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2512365919999999, 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97011 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109019+0000", 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109019+0000", 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109020+0000", 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.22e-07 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109041+0000", 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1460000000000001e-05 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109046+0000", 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8640000000000004e-06 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109049+0000", 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3610000000000001e-06 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109205+0000", 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015583099999999999 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109216+0000", 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1082e-05 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.278677+0000", 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1694608520000001 2024-09-13T14:56:38.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.278772+0000", 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5328999999999995e-05 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.343834+0000", 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.065061315999999994 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.350289+0000", 2024-09-13T14:56:38.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006455756 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.352443+0000", 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021538460000000001 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.354612+0000", 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002169177 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.358282+0000", 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003669482 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.358347+0000", 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5006999999999996e-05 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.360256+0000", 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019089070000000001 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97012 11.11 11:8faea4de:::benchmark_data_smithi003_229782_object97011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.109055+0000", 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.236619709, 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2650873869999999, 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.408 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97012 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109055+0000", 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109055+0000", 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109055+0000", 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8599999999999998e-07 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109075+0000", 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9918000000000001e-05 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109083+0000", 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4200000000000001e-06 2024-09-13T14:56:38.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109088+0000", 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1800000000000004e-06 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109261+0000", 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017374 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109271+0000", 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1179999999999998e-06 2024-09-13T14:56:38.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.360295+0000", 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.251024143 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.360542+0000", 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024711600000000002 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.360708+0000", 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016603599999999999 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.367780+0000", 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070721610000000004 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.367828+0000", 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8068999999999997e-05 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.367901+0000", 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3009999999999994e-05 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.372060+0000", 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041584079999999997 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.372127+0000", 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7849999999999996e-05 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.374142+0000", 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002014732 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97049 11.11 11:8e35b5cd:::benchmark_data_smithi003_229782_object97048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.109105+0000", 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.236569879, 2024-09-13T14:56:38.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.285740495, 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97049 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109105+0000", 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109105+0000", 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109105+0000", 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5200000000000002e-07 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109126+0000", 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0701000000000001e-05 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109132+0000", 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0159999999999997e-06 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109135+0000", 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4709999999999999e-06 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109322+0000", 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018693299999999999 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109330+0000", 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3440000000000001e-06 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.363570+0000", 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.254239248 2024-09-13T14:56:38.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.363731+0000", 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016165 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.363900+0000", 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000168732 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.367845+0000", 2024-09-13T14:56:38.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039450450000000003 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.374216+0000", 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063704010000000004 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.381184+0000", 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006968251 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.392764+0000", 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011580564999999999 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.392824+0000", 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9592000000000001e-05 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.394845+0000", 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020210940000000002 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97063 11.11 11:8f10ae7c:::benchmark_data_smithi003_229782_object97062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.109149+0000", 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23652540399999, 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.28834308, 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.424 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97063 2024-09-13T14:56:38.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.424 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109149+0000", 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109149+0000", 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109150+0000", 2024-09-13T14:56:38.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.89e-07 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109170+0000", 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0693999999999999e-05 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109181+0000", 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0560999999999999e-05 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109187+0000", 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1e-06 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109384+0000", 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019716800000000001 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109392+0000", 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0120000000000005e-06 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.365702+0000", 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2563104489999999 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.365886+0000", 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018386800000000001 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.366054+0000", 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000167784 2024-09-13T14:56:38.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.372011+0000", 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059565219999999997 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.381266+0000", 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092550229999999994 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.381325+0000", 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9113000000000001e-05 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.395389+0000", 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01406381 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.395448+0000", 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9039999999999997e-05 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.397492+0000", 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002044447 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97147 11.11 11:8ead2ce5:::benchmark_data_smithi003_229782_object97146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.109195+0000", 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23647920600001, 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2915676899999999, 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.431 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97147 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109195+0000", 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109195+0000", 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109196+0000", 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5699999999999998e-07 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109221+0000", 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5378e-05 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109229+0000", 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9300000000000003e-06 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109235+0000", 2024-09-13T14:56:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7860000000000002e-06 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109431+0000", 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019607700000000001 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109439+0000", 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0150000000000003e-06 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.367945+0000", 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.258506312 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.368118+0000", 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017287699999999999 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.368319+0000", 2024-09-13T14:56:38.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020141899999999999 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.374161+0000", 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058419919999999998 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.381384+0000", 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007222366 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.383124+0000", 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017401960000000001 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.397545+0000", 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014421399 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.398813+0000", 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001267595 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.400763+0000", 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001949891 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97153 11.11 11:8c098935:::benchmark_data_smithi003_229782_object97152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.109251+0000", 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23642336699999, 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2966711909999999, 2024-09-13T14:56:38.439 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97153 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109251+0000", 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109251+0000", 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109251+0000", 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6199999999999999e-07 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109268+0000", 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6733000000000001e-05 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109273+0000", 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.972e-06 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109277+0000", 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.228e-06 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109496+0000", 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000218641 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109504+0000", 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999999e-06 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.370056+0000", 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2605519359999999 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.370196+0000", 2024-09-13T14:56:38.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013981200000000001 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.370346+0000", 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015060400000000001 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.378737+0000", 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0083911100000000002 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.383212+0000", 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004474998 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.392134+0000", 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089213499999999998 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.400823+0000", 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0086891480000000007 2024-09-13T14:56:38.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.402027+0000", 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001203916 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.405922+0000", 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038954110000000001 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97192 11.11 11:8bf53684:::benchmark_data_smithi003_229782_object97191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.109287+0000", 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23638782800001, 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2984745310000001, 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97192 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109287+0000", 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109287+0000", 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109287+0000", 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.32e-07 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109307+0000", 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9806e-05 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109314+0000", 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3180000000000001e-06 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109320+0000", 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.942e-06 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109538+0000", 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000217679 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109545+0000", 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.233e-06 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.374251+0000", 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2647064729999999 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.374405+0000", 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153485 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.374569+0000", 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000163871 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.381139+0000", 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065703749999999998 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.383708+0000", 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025689599999999999 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.392212+0000", 2024-09-13T14:56:38.453 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 36289 36273 141.667 85.8125 0.00749672 0.00705078 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0085037259999999996 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.405982+0000", 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01377009 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.406100+0000", 2024-09-13T14:56:38.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117858 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.407761+0000", 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016612829999999999 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97195 11.11 11:88e22cbc:::benchmark_data_smithi003_229782_object97194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.109328+0000", 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23634615899999, 2024-09-13T14:56:38.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.300331801, 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97195 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109328+0000", 2024-09-13T14:56:38.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109328+0000", 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109329+0000", 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2099999999999997e-07 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109349+0000", 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9916999999999999e-05 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109356+0000", 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2339999999999997e-06 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109361+0000", 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4489999999999998e-06 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109573+0000", 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021151399999999999 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109586+0000", 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2855e-05 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.376685+0000", 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2670996059999999 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.376932+0000", 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024697800000000002 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.377131+0000", 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000198834 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.381201+0000", 2024-09-13T14:56:38.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040699320000000001 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.386253+0000", 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050521239999999999 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.392273+0000", 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006020213 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.407912+0000", 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015638932000000001 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.408021+0000", 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108761 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.409660+0000", 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016389309999999999 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.462 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97217 11.11 11:8f4bd5d2:::benchmark_data_smithi003_229782_object97216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:05.109382+0000", 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 213.23629222700001, 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.301983058, 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97217 2024-09-13T14:56:38.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.464 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109382+0000", 2024-09-13T14:56:38.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109382+0000", 2024-09-13T14:56:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109383+0000", 2024-09-13T14:56:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8699999999999995e-07 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109406+0000", 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3703e-05 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109416+0000", 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6549999999999999e-06 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109422+0000", 2024-09-13T14:56:38.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1299999999999998e-06 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109651+0000", 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000228593 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.109658+0000", 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4070000000000004e-06 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.378771+0000", 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.269112617 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.378915+0000", 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014376600000000001 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.379075+0000", 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016023799999999999 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.381397+0000", 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023224949999999999 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.386298+0000", 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049007749999999996 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.392339+0000", 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060411509999999998 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.409745+0000", 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017406274999999999 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.409816+0000", 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0370000000000006e-05 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:06.411365+0000", 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001549396 2024-09-13T14:56:38.470 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196647 11.10 11:0f02dab9:::benchmark_data_smithi003_229782_object74267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.362025+0000", 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 109.983649533, 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96367963199999995, 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.471 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196647 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362025+0000", 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362025+0000", 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362026+0000", 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.575e-06 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362032+0000", 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5759999999999996e-06 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362050+0000", 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7706999999999998e-05 2024-09-13T14:56:38.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362058+0000", 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1510000000000004e-06 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362184+0000", 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012584599999999999 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362194+0000", 2024-09-13T14:56:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7780000000000002e-06 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.316597+0000", 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95440383100000004 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.316735+0000", 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013800100000000001 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:56:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.317290+0000", 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00055507099999999997 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325534+0000", 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082431039999999994 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325589+0000", 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5556000000000001e-05 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325643+0000", 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3569999999999997e-05 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325680+0000", 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7002999999999998e-05 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325705+0000", 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4862999999999999e-05 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.477 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196656 11.10 11:0bb599c9:::benchmark_data_smithi003_229782_object74276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.372318+0000", 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 109.973356186, 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95649141100000001, 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.478 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196656 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372318+0000", 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372318+0000", 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372320+0000", 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4610000000000001e-06 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372324+0000", 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9949999999999999e-06 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372334+0000", 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0366999999999999e-05 2024-09-13T14:56:38.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372340+0000", 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6620000000000002e-06 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372429+0000", 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9018999999999996e-05 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372440+0000", 2024-09-13T14:56:38.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1080000000000001e-05 2024-09-13T14:56:38.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.320065+0000", 2024-09-13T14:56:38.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94762556499999995 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.320193+0000", 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012741900000000001 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.320554+0000", 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036128799999999997 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325909+0000", 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053550749999999999 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325950+0000", 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0527e-05 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328705+0000", 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027551720000000002 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328766+0000", 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1408000000000006e-05 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328810+0000", 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3373e-05 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.485 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196725 11.10 11:0a049c51:::benchmark_data_smithi003_229782_object74345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.454725+0000", 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 109.890949611, 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87791044399999996, 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196725 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454725+0000", 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454725+0000", 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454726+0000", 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5850000000000001e-06 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454732+0000", 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1869999999999996e-06 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454748+0000", 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6378000000000001e-05 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454757+0000", 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5699999999999993e-06 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454827+0000", 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0030000000000005e-05 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454837+0000", 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0190999999999999e-05 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.322756+0000", 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86791948200000002 2024-09-13T14:56:38.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.322884+0000", 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012783 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.323283+0000", 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039857400000000001 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325986+0000", 2024-09-13T14:56:38.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027037799999999998 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328880+0000", 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028937870000000001 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332304+0000", 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003423929 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332354+0000", 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9883999999999999e-05 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332635+0000", 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028123699999999999 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196757 11.10 11:0c19dfc2:::benchmark_data_smithi003_229782_object74377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:56:38.492 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.484186+0000", 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 109.861488126, 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.85628883899999997, 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196757 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484186+0000", 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484186+0000", 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484188+0000", 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2580000000000001e-06 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484192+0000", 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7929999999999997e-06 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484203+0000", 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0227000000000001e-05 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484209+0000", 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4999999999999996e-06 2024-09-13T14:56:38.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484269+0000", 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0273999999999999e-05 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484279+0000", 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0049e-05 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.326015+0000", 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.84173542999999995 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.326152+0000", 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013716800000000001 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.326568+0000", 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041612499999999999 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332646+0000", 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006077743 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332686+0000", 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0349000000000003e-05 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340381+0000", 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076943139999999998 2024-09-13T14:56:38.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340431+0000", 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0798999999999999e-05 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340475+0000", 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3810000000000002e-05 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196769 11.10 11:086d7a4f:::benchmark_data_smithi003_229782_object74389:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.495957+0000", 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 109.849717043, 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.84475226599999997, 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:56:38.500 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196769 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495957+0000", 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495957+0000", 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495958+0000", 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2900000000000002e-07 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495964+0000", 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3850000000000001e-06 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495975+0000", 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1039999999999999e-05 2024-09-13T14:56:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495982+0000", 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.605e-06 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.496055+0000", 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2558999999999994e-05 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.496066+0000", 2024-09-13T14:56:38.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0643e-05 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328942+0000", 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.83287688400000004 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.329070+0000", 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012754500000000001 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:56:38.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.329470+0000", 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039961100000000002 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332697+0000", 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032277400000000002 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332735+0000", 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7824000000000002e-05 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340569+0000", 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078336359999999997 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340669+0000", 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100529 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340710+0000", 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0336000000000001e-05 2024-09-13T14:56:38.506 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.507 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.507 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.507 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:56:38.507 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:56:38.507 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:56:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:38.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:38.535 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:56:38.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:38.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:38.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[104007]: pgmap v1114: 109 pgs: 109 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-09-13T14:56:38.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1684131944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:38.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[107278]: pgmap v1114: 109 pgs: 109 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1684131944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:56:38.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:56:38.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:38 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:38.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:38.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:38.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:38.902 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:38 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:38 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:38 smithi019 ceph-mon[91296]: pgmap v1114: 109 pgs: 109 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-09-13T14:56:38.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1684131944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:38.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:38 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:38.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:38 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:56:38.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:38 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:56:38.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:38 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:38.935 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:39.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:39.068 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:39.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:39.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:39.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:39.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:39.418 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 37808 37792 138.918 94.9375 0.010155 0.00719234 2024-09-13T14:56:39.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:39.482 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:39.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:39.613 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:39.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:39 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:39 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:56:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:39 smithi003 ceph-mon[107278]: osdmap e315: 8 total, 8 up, 8 in 2024-09-13T14:56:39.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:39 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:39.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:39 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:56:39.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:39 smithi003 ceph-mon[104007]: osdmap e315: 8 total, 8 up, 8 in 2024-09-13T14:56:39.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:39.746 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:39.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:39.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:39 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:39 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:56:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:39 smithi019 ceph-mon[91296]: osdmap e315: 8 total, 8 up, 8 in 2024-09-13T14:56:39.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:40.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:40.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:40.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:40.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:40.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:40.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:40.420 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 39479 39463 136.992 104.438 0.00893872 0.00729356 2024-09-13T14:56:40.447 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:40.527 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:56:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:56:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:40.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:40.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:40.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:40.737 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:40.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:40.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:40 smithi019 ceph-mon[91296]: pgmap v1116: 109 pgs: 109 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:56:40.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:40 smithi003 ceph-mon[107278]: pgmap v1116: 109 pgs: 109 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:56:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:40 smithi003 ceph-mon[104007]: pgmap v1116: 109 pgs: 109 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:56:40.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:41.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:41.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:41.156 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:41.258 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:41.295 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:41.397 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:41.420 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 41201 41185 135.446 107.625 0.00564787 0.00737716 2024-09-13T14:56:41.434 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:41.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:41.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:41.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:41.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:41.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:41.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:41.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:41.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:42.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:56:42.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:42.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:42.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:42.263 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:42.292 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:42.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:42.420 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:56:42.419845+0000 min lat: 0.00131314 max lat: 0.143271 avg lat: 0.00746215 2024-09-13T14:56:42.420 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:56:42.420 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 42873 42857 133.897 104.5 0.00795788 0.00746215 2024-09-13T14:56:42.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:42.547 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:42.583 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:42.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:42.714 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:42.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:42.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:42.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:43.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:43.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:43 smithi003 ceph-mon[104007]: pgmap v1117: 109 pgs: 109 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-13T14:56:43.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:43.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:43 smithi003 ceph-mon[107278]: pgmap v1117: 109 pgs: 109 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-13T14:56:43.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:43.130 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:43.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:56:43.164 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:43.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.294 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:43.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:43.395 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:43.420 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 44393 44377 132.045 95 0.00644919 0.00756656 2024-09-13T14:56:43.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:43 smithi019 ceph-mon[91296]: pgmap v1117: 109 pgs: 109 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-13T14:56:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:43.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:56:43.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:56:43.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:56:43.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:43.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:43.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:43.508 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:56:43.508 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:43.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:43.550 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:56:43.550 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:43.594 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:56:43.594 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:43.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:43.633 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:56:43.634 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:43.641 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:43.648 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:43.677 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:56:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:43.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:43.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:43.784 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:56:43.786 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:56:43.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:43.881 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:43.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:44.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:44.123 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:44.154 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:44.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:44.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/828650418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:44.271 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:56:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:56:44] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-13T14:56:44.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/828650418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:44.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:44.400 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:44.420 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 45867 45851 130.23 92.125 0.00598955 0.00767337 2024-09-13T14:56:44.431 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:44.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/828650418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:44.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:44.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:44.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:44.676 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:44.706 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:44.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.767 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1117,"stamp":"2024-09-13T14:56:41.961178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2480674765,"num_objects":37869,"num_object_clones":0,"num_object_copies":113607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37907,"num_write_kb":2421984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":853342,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40002,"ondisk_log_size":40002,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14524064,"kb_used_data":7175764,"kb_used_omap":29,"kb_used_meta":7348130,"kb_avail":735437152,"statfs":{"total":767960285184,"available":753087643648,"internally_reserved":0,"allocated":7347982336,"data_stored":7333721770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":7524485646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,6,5,3,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1180827648,"num_objects":18018,"num_object_clones":0,"num_object_copies":54054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18018,"num_write_kb":1153152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004656"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":512,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555060+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:56:38.555060+0000","last_peered":"2024-09-13T14:56:38.555060+0000","last_clean":"2024-09-13T14:56:38.555060+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:56:38.555060+0000","last_undegraded":"2024-09-13T14:56:38.555060+0000","last_fullsized":"2024-09-13T14:56:38.555060+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":287,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183071+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:56:18.183071+0000","last_peered":"2024-09-13T14:56:18.183071+0000","last_clean":"2024-09-13T14:56:18.183071+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:56:18.183071+0000","last_undegraded":"2024-09-13T14:56:18.183071+0000","last_fullsized":"2024-09-13T14:56:18.183071+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":78,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060596+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:56:18.060596+0000","last_peered":"2024-09-13T14:56:18.060596+0000","last_clean":"2024-09-13T14:56:18.060596+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:56:18.060596+0000","last_undegraded":"2024-09-13T14:56:18.060596+0000","last_fullsized":"2024-09-13T14:56:18.060596+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":468,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183964+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:56:18.183964+0000","last_peered":"2024-09-13T14:56:18.183964+0000","last_clean":"2024-09-13T14:56:18.183964+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:56:18.183964+0000","last_undegraded":"2024-09-13T14:56:18.183964+0000","last_fullsized":"2024-09-13T14:56:18.183964+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":261,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060527+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:56:18.060527+0000","last_peered":"2024-09-13T14:56:18.060527+0000","last_clean":"2024-09-13T14:56:18.060527+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:56:18.060527+0000","last_undegraded":"2024-09-13T14:56:18.060527+0000","last_fullsized":"2024-09-13T14:56:18.060527+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":294,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183472+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:56:18.183472+0000","last_peered":"2024-09-13T14:56:18.183472+0000","last_clean":"2024-09-13T14:56:18.183472+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:56:18.183472+0000","last_undegraded":"2024-09-13T14:56:18.183472+0000","last_fullsized":"2024-09-13T14:56:18.183472+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":278,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060410+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:56:18.060410+0000","last_peered":"2024-09-13T14:56:18.060410+0000","last_clean":"2024-09-13T14:56:18.060410+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:56:18.060410+0000","last_undegraded":"2024-09-13T14:56:18.060410+0000","last_fullsized":"2024-09-13T14:56:18.060410+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":324,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185358+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:56:18.185358+0000","last_peered":"2024-09-13T14:56:18.185358+0000","last_clean":"2024-09-13T14:56:18.185358+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:56:18.185358+0000","last_undegraded":"2024-09-13T14:56:18.185358+0000","last_fullsized":"2024-09-13T14:56:18.185358+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":491,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557823+0000","last_change":"2024-09-13T14:54:27.142651+0000","last_active":"2024-09-13T14:56:38.557823+0000","last_peered":"2024-09-13T14:56:38.557823+0000","last_clean":"2024-09-13T14:56:38.557823+0000","last_became_active":"2024-09-13T14:54:27.142239+0000","last_became_peered":"2024-09-13T14:54:27.142239+0000","last_unstale":"2024-09-13T14:56:38.557823+0000","last_undegraded":"2024-09-13T14:56:38.557823+0000","last_fullsized":"2024-09-13T14:56:38.557823+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:41.308930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":494,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.550244+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:56:38.550244+0000","last_peered":"2024-09-13T14:56:38.550244+0000","last_clean":"2024-09-13T14:56:38.550244+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:56:38.550244+0000","last_undegraded":"2024-09-13T14:56:38.550244+0000","last_fullsized":"2024-09-13T14:56:38.550244+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":296,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183567+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:56:18.183567+0000","last_peered":"2024-09-13T14:56:18.183567+0000","last_clean":"2024-09-13T14:56:18.183567+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:56:18.183567+0000","last_undegraded":"2024-09-13T14:56:18.183567+0000","last_fullsized":"2024-09-13T14:56:18.183567+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":434,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184176+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:56:18.184176+0000","last_peered":"2024-09-13T14:56:18.184176+0000","last_clean":"2024-09-13T14:56:18.184176+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:56:18.184176+0000","last_undegraded":"2024-09-13T14:56:18.184176+0000","last_fullsized":"2024-09-13T14:56:18.184176+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":151,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.549455+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:56:38.549455+0000","last_peered":"2024-09-13T14:56:38.549455+0000","last_clean":"2024-09-13T14:56:38.549455+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:56:38.549455+0000","last_undegraded":"2024-09-13T14:56:38.549455+0000","last_fullsized":"2024-09-13T14:56:38.549455+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":530,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.556585+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:56:38.556585+0000","last_peered":"2024-09-13T14:56:38.556585+0000","last_clean":"2024-09-13T14:56:38.556585+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:56:38.556585+0000","last_undegraded":"2024-09-13T14:56:38.556585+0000","last_fullsized":"2024-09-13T14:56:38.556585+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":467,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183509+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:56:18.183509+0000","last_peered":"2024-09-13T14:56:18.183509+0000","last_clean":"2024-09-13T14:56:18.183509+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:56:18.183509+0000","last_undegraded":"2024-09-13T14:56:18.183509+0000","last_fullsized":"2024-09-13T14:56:18.183509+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":277,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185299+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:56:18.185299+0000","last_peered":"2024-09-13T14:56:18.185299+0000","last_clean":"2024-09-13T14:56:18.185299+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:56:18.185299+0000","last_undegraded":"2024-09-13T14:56:18.185299+0000","last_fullsized":"2024-09-13T14:56:18.185299+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":435,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183194+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:56:18.183194+0000","last_peered":"2024-09-13T14:56:18.183194+0000","last_clean":"2024-09-13T14:56:18.183194+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:56:18.183194+0000","last_undegraded":"2024-09-13T14:56:18.183194+0000","last_fullsized":"2024-09-13T14:56:18.183194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":513,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555333+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:56:38.555333+0000","last_peered":"2024-09-13T14:56:38.555333+0000","last_clean":"2024-09-13T14:56:38.555333+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:56:38.555333+0000","last_undegraded":"2024-09-13T14:56:38.555333+0000","last_fullsized":"2024-09-13T14:56:38.555333+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":440,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183526+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:56:18.183526+0000","last_peered":"2024-09-13T14:56:18.183526+0000","last_clean":"2024-09-13T14:56:18.183526+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:56:18.183526+0000","last_undegraded":"2024-09-13T14:56:18.183526+0000","last_fullsized":"2024-09-13T14:56:18.183526+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":76,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060568+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:56:18.060568+0000","last_peered":"2024-09-13T14:56:18.060568+0000","last_clean":"2024-09-13T14:56:18.060568+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:56:18.060568+0000","last_undegraded":"2024-09-13T14:56:18.060568+0000","last_fullsized":"2024-09-13T14:56:18.060568+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":192,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.550065+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:56:38.550065+0000","last_peered":"2024-09-13T14:56:38.550065+0000","last_clean":"2024-09-13T14:56:38.550065+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:56:38.550065+0000","last_undegraded":"2024-09-13T14:56:38.550065+0000","last_fullsized":"2024-09-13T14:56:38.550065+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":78,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184758+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:56:18.184758+0000","last_peered":"2024-09-13T14:56:18.184758+0000","last_clean":"2024-09-13T14:56:18.184758+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:56:18.184758+0000","last_undegraded":"2024-09-13T14:56:18.184758+0000","last_fullsized":"2024-09-13T14:56:18.184758+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":216,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184277+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:56:18.184277+0000","last_peered":"2024-09-13T14:56:18.184277+0000","last_clean":"2024-09-13T14:56:18.184277+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:56:18.184277+0000","last_undegraded":"2024-09-13T14:56:18.184277+0000","last_fullsized":"2024-09-13T14:56:18.184277+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":284,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184141+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:56:18.184141+0000","last_peered":"2024-09-13T14:56:18.184141+0000","last_clean":"2024-09-13T14:56:18.184141+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:56:18.184141+0000","last_undegraded":"2024-09-13T14:56:18.184141+0000","last_fullsized":"2024-09-13T14:56:18.184141+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":277,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185391+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:56:18.185391+0000","last_peered":"2024-09-13T14:56:18.185391+0000","last_clean":"2024-09-13T14:56:18.185391+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:56:18.185391+0000","last_undegraded":"2024-09-13T14:56:18.185391+0000","last_fullsized":"2024-09-13T14:56:18.185391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"315'2411","reported_seq":2430,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:40.011268+0000","last_change":"2024-09-13T14:56:09.033158+0000","last_active":"2024-09-13T14:56:40.011268+0000","last_peered":"2024-09-13T14:56:40.011268+0000","last_clean":"2024-09-13T14:56:40.011268+0000","last_became_active":"2024-09-13T14:56:09.032864+0000","last_became_peered":"2024-09-13T14:56:09.032864+0000","last_unstale":"2024-09-13T14:56:40.011268+0000","last_undegraded":"2024-09-13T14:56:40.011268+0000","last_fullsized":"2024-09-13T14:56:40.011268+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:36.421513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158007296,"num_objects":2411,"num_object_clones":0,"num_object_copies":7233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2411,"num_write_kb":154304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":260,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183068+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:56:18.183068+0000","last_peered":"2024-09-13T14:56:18.183068+0000","last_clean":"2024-09-13T14:56:18.183068+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:56:18.183068+0000","last_undegraded":"2024-09-13T14:56:18.183068+0000","last_fullsized":"2024-09-13T14:56:18.183068+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":310,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185293+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:56:18.185293+0000","last_peered":"2024-09-13T14:56:18.185293+0000","last_clean":"2024-09-13T14:56:18.185293+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:56:18.185293+0000","last_undegraded":"2024-09-13T14:56:18.185293+0000","last_fullsized":"2024-09-13T14:56:18.185293+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":511,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.556737+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:56:38.556737+0000","last_peered":"2024-09-13T14:56:38.556737+0000","last_clean":"2024-09-13T14:56:38.556737+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:56:38.556737+0000","last_undegraded":"2024-09-13T14:56:38.556737+0000","last_fullsized":"2024-09-13T14:56:38.556737+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":197,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184046+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:56:18.184046+0000","last_peered":"2024-09-13T14:56:18.184046+0000","last_clean":"2024-09-13T14:56:18.184046+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:56:18.184046+0000","last_undegraded":"2024-09-13T14:56:18.184046+0000","last_fullsized":"2024-09-13T14:56:18.184046+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"314'2103","reported_seq":2120,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:37.279752+0000","last_change":"2024-09-13T14:56:09.034412+0000","last_active":"2024-09-13T14:56:37.279752+0000","last_peered":"2024-09-13T14:56:37.279752+0000","last_clean":"2024-09-13T14:56:37.279752+0000","last_became_active":"2024-09-13T14:56:09.033995+0000","last_became_peered":"2024-09-13T14:56:09.033995+0000","last_unstale":"2024-09-13T14:56:37.279752+0000","last_undegraded":"2024-09-13T14:56:37.279752+0000","last_fullsized":"2024-09-13T14:56:37.279752+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:28.253264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137822208,"num_objects":2103,"num_object_clones":0,"num_object_copies":6309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":134592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":478,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183306+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:56:18.183306+0000","last_peered":"2024-09-13T14:56:18.183306+0000","last_clean":"2024-09-13T14:56:18.183306+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:56:18.183306+0000","last_undegraded":"2024-09-13T14:56:18.183306+0000","last_fullsized":"2024-09-13T14:56:18.183306+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":151,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185086+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:56:18.185086+0000","last_peered":"2024-09-13T14:56:18.185086+0000","last_clean":"2024-09-13T14:56:18.185086+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:56:18.185086+0000","last_undegraded":"2024-09-13T14:56:18.185086+0000","last_fullsized":"2024-09-13T14:56:18.185086+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":467,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184652+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:56:18.184652+0000","last_peered":"2024-09-13T14:56:18.184652+0000","last_clean":"2024-09-13T14:56:18.184652+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:56:18.184652+0000","last_undegraded":"2024-09-13T14:56:18.184652+0000","last_fullsized":"2024-09-13T14:56:18.184652+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":98,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184403+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:56:18.184403+0000","last_peered":"2024-09-13T14:56:18.184403+0000","last_clean":"2024-09-13T14:56:18.184403+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:56:18.184403+0000","last_undegraded":"2024-09-13T14:56:18.184403+0000","last_fullsized":"2024-09-13T14:56:18.184403+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"314'2220","reported_seq":2237,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:37.285028+0000","last_change":"2024-09-13T14:56:09.034509+0000","last_active":"2024-09-13T14:56:37.285028+0000","last_peered":"2024-09-13T14:56:37.285028+0000","last_clean":"2024-09-13T14:56:37.285028+0000","last_became_active":"2024-09-13T14:56:09.034213+0000","last_became_peered":"2024-09-13T14:56:09.034213+0000","last_unstale":"2024-09-13T14:56:37.285028+0000","last_undegraded":"2024-09-13T14:56:37.285028+0000","last_fullsized":"2024-09-13T14:56:37.285028+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2220,"log_dups_size":0,"ondisk_log_size":2220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:26.274871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145489920,"num_objects":2220,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2220,"num_write_kb":142080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":498,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557739+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:56:38.557739+0000","last_peered":"2024-09-13T14:56:38.557739+0000","last_clean":"2024-09-13T14:56:38.557739+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:56:38.557739+0000","last_undegraded":"2024-09-13T14:56:38.557739+0000","last_fullsized":"2024-09-13T14:56:38.557739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":290,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184947+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:56:18.184947+0000","last_peered":"2024-09-13T14:56:18.184947+0000","last_clean":"2024-09-13T14:56:18.184947+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:56:18.184947+0000","last_undegraded":"2024-09-13T14:56:18.184947+0000","last_fullsized":"2024-09-13T14:56:18.184947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":453,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.550163+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:56:38.550163+0000","last_peered":"2024-09-13T14:56:38.550163+0000","last_clean":"2024-09-13T14:56:38.550163+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:56:38.550163+0000","last_undegraded":"2024-09-13T14:56:38.550163+0000","last_fullsized":"2024-09-13T14:56:38.550163+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":251,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184879+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:56:18.184879+0000","last_peered":"2024-09-13T14:56:18.184879+0000","last_clean":"2024-09-13T14:56:18.184879+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:56:18.184879+0000","last_undegraded":"2024-09-13T14:56:18.184879+0000","last_fullsized":"2024-09-13T14:56:18.184879+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"315'2317","reported_seq":2335,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:40.027025+0000","last_change":"2024-09-13T14:56:09.034272+0000","last_active":"2024-09-13T14:56:40.027025+0000","last_peered":"2024-09-13T14:56:40.027025+0000","last_clean":"2024-09-13T14:56:40.027025+0000","last_became_active":"2024-09-13T14:56:09.033933+0000","last_became_peered":"2024-09-13T14:56:09.033933+0000","last_unstale":"2024-09-13T14:56:40.027025+0000","last_undegraded":"2024-09-13T14:56:40.027025+0000","last_fullsized":"2024-09-13T14:56:40.027025+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2317,"log_dups_size":0,"ondisk_log_size":2317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:40.177740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151846912,"num_objects":2317,"num_object_clones":0,"num_object_copies":6951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2317,"num_write_kb":148288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":76,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060208+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:56:18.060208+0000","last_peered":"2024-09-13T14:56:18.060208+0000","last_clean":"2024-09-13T14:56:18.060208+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:56:18.060208+0000","last_undegraded":"2024-09-13T14:56:18.060208+0000","last_fullsized":"2024-09-13T14:56:18.060208+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":311,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555656+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:56:38.555656+0000","last_peered":"2024-09-13T14:56:38.555656+0000","last_clean":"2024-09-13T14:56:38.555656+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:56:38.555656+0000","last_undegraded":"2024-09-13T14:56:38.555656+0000","last_fullsized":"2024-09-13T14:56:38.555656+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":293,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060558+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:56:18.060558+0000","last_peered":"2024-09-13T14:56:18.060558+0000","last_clean":"2024-09-13T14:56:18.060558+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:56:18.060558+0000","last_undegraded":"2024-09-13T14:56:18.060558+0000","last_fullsized":"2024-09-13T14:56:18.060558+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":271,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185207+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:56:18.185207+0000","last_peered":"2024-09-13T14:56:18.185207+0000","last_clean":"2024-09-13T14:56:18.185207+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:56:18.185207+0000","last_undegraded":"2024-09-13T14:56:18.185207+0000","last_fullsized":"2024-09-13T14:56:18.185207+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"315'2525","reported_seq":2544,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:40.996903+0000","last_change":"2024-09-13T14:56:09.101249+0000","last_active":"2024-09-13T14:56:40.996903+0000","last_peered":"2024-09-13T14:56:40.996903+0000","last_clean":"2024-09-13T14:56:40.996903+0000","last_became_active":"2024-09-13T14:56:09.100796+0000","last_became_peered":"2024-09-13T14:56:09.100796+0000","last_unstale":"2024-09-13T14:56:40.996903+0000","last_undegraded":"2024-09-13T14:56:40.996903+0000","last_fullsized":"2024-09-13T14:56:40.996903+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2525,"log_dups_size":0,"ondisk_log_size":2525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:37.036972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165478400,"num_objects":2525,"num_object_clones":0,"num_object_copies":7575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":161600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":365,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.550372+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:56:38.550372+0000","last_peered":"2024-09-13T14:56:38.550372+0000","last_clean":"2024-09-13T14:56:38.550372+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:56:38.550372+0000","last_undegraded":"2024-09-13T14:56:38.550372+0000","last_fullsized":"2024-09-13T14:56:38.550372+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":278,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185418+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:56:18.185418+0000","last_peered":"2024-09-13T14:56:18.185418+0000","last_clean":"2024-09-13T14:56:18.185418+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:56:18.185418+0000","last_undegraded":"2024-09-13T14:56:18.185418+0000","last_fullsized":"2024-09-13T14:56:18.185418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":289,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060379+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:56:18.060379+0000","last_peered":"2024-09-13T14:56:18.060379+0000","last_clean":"2024-09-13T14:56:18.060379+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:56:18.060379+0000","last_undegraded":"2024-09-13T14:56:18.060379+0000","last_fullsized":"2024-09-13T14:56:18.060379+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":98,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060077+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:56:18.060077+0000","last_peered":"2024-09-13T14:56:18.060077+0000","last_clean":"2024-09-13T14:56:18.060077+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:56:18.060077+0000","last_undegraded":"2024-09-13T14:56:18.060077+0000","last_fullsized":"2024-09-13T14:56:18.060077+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"315'2425","reported_seq":2437,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:39.345370+0000","last_change":"2024-09-13T14:56:09.101282+0000","last_active":"2024-09-13T14:56:39.345370+0000","last_peered":"2024-09-13T14:56:39.345370+0000","last_clean":"2024-09-13T14:56:39.345370+0000","last_became_active":"2024-09-13T14:56:09.100956+0000","last_became_peered":"2024-09-13T14:56:09.100956+0000","last_unstale":"2024-09-13T14:56:39.345370+0000","last_undegraded":"2024-09-13T14:56:39.345370+0000","last_fullsized":"2024-09-13T14:56:39.345370+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2425,"log_dups_size":0,"ondisk_log_size":2425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:39.057985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158924800,"num_objects":2425,"num_object_clones":0,"num_object_copies":7275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2425,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":190,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.059699+0000","last_change":"2024-09-13T14:53:42.008298+0000","last_active":"2024-09-13T14:56:18.059699+0000","last_peered":"2024-09-13T14:56:18.059699+0000","last_clean":"2024-09-13T14:56:18.059699+0000","last_became_active":"2024-09-13T14:53:42.007766+0000","last_became_peered":"2024-09-13T14:53:42.007766+0000","last_unstale":"2024-09-13T14:56:18.059699+0000","last_undegraded":"2024-09-13T14:56:18.059699+0000","last_fullsized":"2024-09-13T14:56:18.059699+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:29.235755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":305,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.548557+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:56:38.548557+0000","last_peered":"2024-09-13T14:56:38.548557+0000","last_clean":"2024-09-13T14:56:38.548557+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:56:38.548557+0000","last_undegraded":"2024-09-13T14:56:38.548557+0000","last_fullsized":"2024-09-13T14:56:38.548557+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":496,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.549806+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:56:38.549806+0000","last_peered":"2024-09-13T14:56:38.549806+0000","last_clean":"2024-09-13T14:56:38.549806+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:56:38.549806+0000","last_undegraded":"2024-09-13T14:56:38.549806+0000","last_fullsized":"2024-09-13T14:56:38.549806+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":72,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184548+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:56:18.184548+0000","last_peered":"2024-09-13T14:56:18.184548+0000","last_clean":"2024-09-13T14:56:18.184548+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:56:18.184548+0000","last_undegraded":"2024-09-13T14:56:18.184548+0000","last_fullsized":"2024-09-13T14:56:18.184548+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"315'2470","reported_seq":2489,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:40.980304+0000","last_change":"2024-09-13T14:56:09.032714+0000","last_active":"2024-09-13T14:56:40.980304+0000","last_peered":"2024-09-13T14:56:40.980304+0000","last_clean":"2024-09-13T14:56:40.980304+0000","last_became_active":"2024-09-13T14:56:09.032114+0000","last_became_peered":"2024-09-13T14:56:09.032114+0000","last_unstale":"2024-09-13T14:56:40.980304+0000","last_undegraded":"2024-09-13T14:56:40.980304+0000","last_fullsized":"2024-09-13T14:56:40.980304+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2470,"log_dups_size":0,"ondisk_log_size":2470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:53.695180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161873920,"num_objects":2470,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2470,"num_write_kb":158080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":519,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.556517+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:56:38.556517+0000","last_peered":"2024-09-13T14:56:38.556517+0000","last_clean":"2024-09-13T14:56:38.556517+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:56:38.556517+0000","last_undegraded":"2024-09-13T14:56:38.556517+0000","last_fullsized":"2024-09-13T14:56:38.556517+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":529,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.554640+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:56:38.554640+0000","last_peered":"2024-09-13T14:56:38.554640+0000","last_clean":"2024-09-13T14:56:38.554640+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:56:38.554640+0000","last_undegraded":"2024-09-13T14:56:38.554640+0000","last_fullsized":"2024-09-13T14:56:38.554640+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184190+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:56:18.184190+0000","last_peered":"2024-09-13T14:56:18.184190+0000","last_clean":"2024-09-13T14:56:18.184190+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:56:18.184190+0000","last_undegraded":"2024-09-13T14:56:18.184190+0000","last_fullsized":"2024-09-13T14:56:18.184190+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":251,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184853+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:56:18.184853+0000","last_peered":"2024-09-13T14:56:18.184853+0000","last_clean":"2024-09-13T14:56:18.184853+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:56:18.184853+0000","last_undegraded":"2024-09-13T14:56:18.184853+0000","last_fullsized":"2024-09-13T14:56:18.184853+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"315'2363","reported_seq":2374,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:39.272046+0000","last_change":"2024-09-13T14:56:09.035158+0000","last_active":"2024-09-13T14:56:39.272046+0000","last_peered":"2024-09-13T14:56:39.272046+0000","last_clean":"2024-09-13T14:56:39.272046+0000","last_became_active":"2024-09-13T14:56:09.034847+0000","last_became_peered":"2024-09-13T14:56:09.034847+0000","last_unstale":"2024-09-13T14:56:39.272046+0000","last_undegraded":"2024-09-13T14:56:39.272046+0000","last_fullsized":"2024-09-13T14:56:39.272046+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2363,"log_dups_size":0,"ondisk_log_size":2363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:55.768082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154861568,"num_objects":2363,"num_object_clones":0,"num_object_copies":7089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2363,"num_write_kb":151232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":260,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184789+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:56:18.184789+0000","last_peered":"2024-09-13T14:56:18.184789+0000","last_clean":"2024-09-13T14:56:18.184789+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:56:18.184789+0000","last_undegraded":"2024-09-13T14:56:18.184789+0000","last_fullsized":"2024-09-13T14:56:18.184789+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":426,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183632+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:56:18.183632+0000","last_peered":"2024-09-13T14:56:18.183632+0000","last_clean":"2024-09-13T14:56:18.183632+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:56:18.183632+0000","last_undegraded":"2024-09-13T14:56:18.183632+0000","last_fullsized":"2024-09-13T14:56:18.183632+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":288,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060351+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:56:18.060351+0000","last_peered":"2024-09-13T14:56:18.060351+0000","last_clean":"2024-09-13T14:56:18.060351+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:56:18.060351+0000","last_undegraded":"2024-09-13T14:56:18.060351+0000","last_fullsized":"2024-09-13T14:56:18.060351+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":72,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183533+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:56:18.183533+0000","last_peered":"2024-09-13T14:56:18.183533+0000","last_clean":"2024-09-13T14:56:18.183533+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:56:18.183533+0000","last_undegraded":"2024-09-13T14:56:18.183533+0000","last_fullsized":"2024-09-13T14:56:18.183533+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.7","version":"315'2319","reported_seq":2331,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:39.289475+0000","last_change":"2024-09-13T14:56:09.035403+0000","last_active":"2024-09-13T14:56:39.289475+0000","last_peered":"2024-09-13T14:56:39.289475+0000","last_clean":"2024-09-13T14:56:39.289475+0000","last_became_active":"2024-09-13T14:56:09.035064+0000","last_became_peered":"2024-09-13T14:56:09.035064+0000","last_unstale":"2024-09-13T14:56:39.289475+0000","last_undegraded":"2024-09-13T14:56:39.289475+0000","last_fullsized":"2024-09-13T14:56:39.289475+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2319,"log_dups_size":0,"ondisk_log_size":2319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:45:44.521747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151977984,"num_objects":2319,"num_object_clones":0,"num_object_copies":6957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2319,"num_write_kb":148416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":151,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.550715+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:56:38.550715+0000","last_peered":"2024-09-13T14:56:38.550715+0000","last_clean":"2024-09-13T14:56:38.550715+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:56:38.550715+0000","last_undegraded":"2024-09-13T14:56:38.550715+0000","last_fullsized":"2024-09-13T14:56:38.550715+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":437,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183536+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:56:18.183536+0000","last_peered":"2024-09-13T14:56:18.183536+0000","last_clean":"2024-09-13T14:56:18.183536+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:56:18.183536+0000","last_undegraded":"2024-09-13T14:56:18.183536+0000","last_fullsized":"2024-09-13T14:56:18.183536+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":98,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060133+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:56:18.060133+0000","last_peered":"2024-09-13T14:56:18.060133+0000","last_clean":"2024-09-13T14:56:18.060133+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:56:18.060133+0000","last_undegraded":"2024-09-13T14:56:18.060133+0000","last_fullsized":"2024-09-13T14:56:18.060133+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"314'2222","reported_seq":2239,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.098783+0000","last_change":"2024-09-13T14:56:09.037715+0000","last_active":"2024-09-13T14:56:38.098783+0000","last_peered":"2024-09-13T14:56:38.098783+0000","last_clean":"2024-09-13T14:56:38.098783+0000","last_became_active":"2024-09-13T14:56:09.037383+0000","last_became_peered":"2024-09-13T14:56:09.037383+0000","last_unstale":"2024-09-13T14:56:38.098783+0000","last_undegraded":"2024-09-13T14:56:38.098783+0000","last_fullsized":"2024-09-13T14:56:38.098783+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:33.806063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145620992,"num_objects":2222,"num_object_clones":0,"num_object_copies":6666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":142208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":297,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184214+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:56:18.184214+0000","last_peered":"2024-09-13T14:56:18.184214+0000","last_clean":"2024-09-13T14:56:18.184214+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:56:18.184214+0000","last_undegraded":"2024-09-13T14:56:18.184214+0000","last_fullsized":"2024-09-13T14:56:18.184214+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":278,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183205+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:56:18.183205+0000","last_peered":"2024-09-13T14:56:18.183205+0000","last_clean":"2024-09-13T14:56:18.183205+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:56:18.183205+0000","last_undegraded":"2024-09-13T14:56:18.183205+0000","last_fullsized":"2024-09-13T14:56:18.183205+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":489,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557455+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:56:38.557455+0000","last_peered":"2024-09-13T14:56:38.557455+0000","last_clean":"2024-09-13T14:56:38.557455+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:56:38.557455+0000","last_undegraded":"2024-09-13T14:56:38.557455+0000","last_fullsized":"2024-09-13T14:56:38.557455+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.5","version":"315'2354","reported_seq":2365,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:39.372017+0000","last_change":"2024-09-13T14:56:09.035196+0000","last_active":"2024-09-13T14:56:39.372017+0000","last_peered":"2024-09-13T14:56:39.372017+0000","last_clean":"2024-09-13T14:56:39.372017+0000","last_became_active":"2024-09-13T14:56:09.034906+0000","last_became_peered":"2024-09-13T14:56:09.034906+0000","last_unstale":"2024-09-13T14:56:39.372017+0000","last_undegraded":"2024-09-13T14:56:39.372017+0000","last_fullsized":"2024-09-13T14:56:39.372017+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:43.214619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154271744,"num_objects":2354,"num_object_clones":0,"num_object_copies":7062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2354,"num_write_kb":150656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":294,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184524+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:56:18.184524+0000","last_peered":"2024-09-13T14:56:18.184524+0000","last_clean":"2024-09-13T14:56:18.184524+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:56:18.184524+0000","last_undegraded":"2024-09-13T14:56:18.184524+0000","last_fullsized":"2024-09-13T14:56:18.184524+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":298,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184376+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:56:18.184376+0000","last_peered":"2024-09-13T14:56:18.184376+0000","last_clean":"2024-09-13T14:56:18.184376+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:56:18.184376+0000","last_undegraded":"2024-09-13T14:56:18.184376+0000","last_fullsized":"2024-09-13T14:56:18.184376+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":80,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060067+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:56:18.060067+0000","last_peered":"2024-09-13T14:56:18.060067+0000","last_clean":"2024-09-13T14:56:18.060067+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:56:18.060067+0000","last_undegraded":"2024-09-13T14:56:18.060067+0000","last_fullsized":"2024-09-13T14:56:18.060067+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"315'2344","reported_seq":2356,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:39.363460+0000","last_change":"2024-09-13T14:56:09.035205+0000","last_active":"2024-09-13T14:56:39.363460+0000","last_peered":"2024-09-13T14:56:39.363460+0000","last_clean":"2024-09-13T14:56:39.363460+0000","last_became_active":"2024-09-13T14:56:09.034915+0000","last_became_peered":"2024-09-13T14:56:09.034915+0000","last_unstale":"2024-09-13T14:56:39.363460+0000","last_undegraded":"2024-09-13T14:56:39.363460+0000","last_fullsized":"2024-09-13T14:56:39.363460+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.408509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153616384,"num_objects":2344,"num_object_clones":0,"num_object_copies":7032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2344,"num_write_kb":150016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":277,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183964+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:56:18.183964+0000","last_peered":"2024-09-13T14:56:18.183964+0000","last_clean":"2024-09-13T14:56:18.183964+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:56:18.183964+0000","last_undegraded":"2024-09-13T14:56:18.183964+0000","last_fullsized":"2024-09-13T14:56:18.183964+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":466,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184162+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:56:18.184162+0000","last_peered":"2024-09-13T14:56:18.184162+0000","last_clean":"2024-09-13T14:56:18.184162+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:56:18.184162+0000","last_undegraded":"2024-09-13T14:56:18.184162+0000","last_fullsized":"2024-09-13T14:56:18.184162+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":98,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183860+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:56:18.183860+0000","last_peered":"2024-09-13T14:56:18.183860+0000","last_clean":"2024-09-13T14:56:18.183860+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:56:18.183860+0000","last_undegraded":"2024-09-13T14:56:18.183860+0000","last_fullsized":"2024-09-13T14:56:18.183860+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"315'2512","reported_seq":2531,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:40.984000+0000","last_change":"2024-09-13T14:56:09.033530+0000","last_active":"2024-09-13T14:56:40.984000+0000","last_peered":"2024-09-13T14:56:40.984000+0000","last_clean":"2024-09-13T14:56:40.984000+0000","last_became_active":"2024-09-13T14:56:09.032938+0000","last_became_peered":"2024-09-13T14:56:09.032938+0000","last_unstale":"2024-09-13T14:56:40.984000+0000","last_undegraded":"2024-09-13T14:56:40.984000+0000","last_fullsized":"2024-09-13T14:56:40.984000+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2512,"log_dups_size":0,"ondisk_log_size":2512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:58:50.158201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164626432,"num_objects":2512,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2512,"num_write_kb":160768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":508,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557661+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:56:38.557661+0000","last_peered":"2024-09-13T14:56:38.557661+0000","last_clean":"2024-09-13T14:56:38.557661+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:56:38.557661+0000","last_undegraded":"2024-09-13T14:56:38.557661+0000","last_fullsized":"2024-09-13T14:56:38.557661+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":276,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060452+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:56:18.060452+0000","last_peered":"2024-09-13T14:56:18.060452+0000","last_clean":"2024-09-13T14:56:18.060452+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:56:18.060452+0000","last_undegraded":"2024-09-13T14:56:18.060452+0000","last_fullsized":"2024-09-13T14:56:18.060452+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":98,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183986+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:56:18.183986+0000","last_peered":"2024-09-13T14:56:18.183986+0000","last_clean":"2024-09-13T14:56:18.183986+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:56:18.183986+0000","last_undegraded":"2024-09-13T14:56:18.183986+0000","last_fullsized":"2024-09-13T14:56:18.183986+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"315'2428","reported_seq":2440,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:39.299225+0000","last_change":"2024-09-13T14:56:09.033553+0000","last_active":"2024-09-13T14:56:39.299225+0000","last_peered":"2024-09-13T14:56:39.299225+0000","last_clean":"2024-09-13T14:56:39.299225+0000","last_became_active":"2024-09-13T14:56:09.033287+0000","last_became_peered":"2024-09-13T14:56:09.033287+0000","last_unstale":"2024-09-13T14:56:39.299225+0000","last_undegraded":"2024-09-13T14:56:39.299225+0000","last_fullsized":"2024-09-13T14:56:39.299225+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2428,"log_dups_size":0,"ondisk_log_size":2428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:54.220513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159121408,"num_objects":2428,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2428,"num_write_kb":155392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":290,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184194+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:56:18.184194+0000","last_peered":"2024-09-13T14:56:18.184194+0000","last_clean":"2024-09-13T14:56:18.184194+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:56:18.184194+0000","last_undegraded":"2024-09-13T14:56:18.184194+0000","last_fullsized":"2024-09-13T14:56:18.184194+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":291,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184959+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:56:18.184959+0000","last_peered":"2024-09-13T14:56:18.184959+0000","last_clean":"2024-09-13T14:56:18.184959+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:56:18.184959+0000","last_undegraded":"2024-09-13T14:56:18.184959+0000","last_fullsized":"2024-09-13T14:56:18.184959+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":251,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185212+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:56:18.185212+0000","last_peered":"2024-09-13T14:56:18.185212+0000","last_clean":"2024-09-13T14:56:18.185212+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:56:18.185212+0000","last_undegraded":"2024-09-13T14:56:18.185212+0000","last_fullsized":"2024-09-13T14:56:18.185212+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"315'2401","reported_seq":2413,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:39.371582+0000","last_change":"2024-09-13T14:56:09.032953+0000","last_active":"2024-09-13T14:56:39.371582+0000","last_peered":"2024-09-13T14:56:39.371582+0000","last_clean":"2024-09-13T14:56:39.371582+0000","last_became_active":"2024-09-13T14:56:09.032634+0000","last_became_peered":"2024-09-13T14:56:09.032634+0000","last_unstale":"2024-09-13T14:56:39.371582+0000","last_undegraded":"2024-09-13T14:56:39.371582+0000","last_fullsized":"2024-09-13T14:56:39.371582+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2401,"log_dups_size":0,"ondisk_log_size":2401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:45.162779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157351936,"num_objects":2401,"num_object_clones":0,"num_object_copies":7203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2401,"num_write_kb":153664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":304,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183160+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:56:18.183160+0000","last_peered":"2024-09-13T14:56:18.183160+0000","last_clean":"2024-09-13T14:56:18.183160+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:56:18.183160+0000","last_undegraded":"2024-09-13T14:56:18.183160+0000","last_fullsized":"2024-09-13T14:56:18.183160+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.185039+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:56:18.185039+0000","last_peered":"2024-09-13T14:56:18.185039+0000","last_clean":"2024-09-13T14:56:18.185039+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:56:18.185039+0000","last_undegraded":"2024-09-13T14:56:18.185039+0000","last_fullsized":"2024-09-13T14:56:18.185039+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":194,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184158+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:56:18.184158+0000","last_peered":"2024-09-13T14:56:18.184158+0000","last_clean":"2024-09-13T14:56:18.184158+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:56:18.184158+0000","last_undegraded":"2024-09-13T14:56:18.184158+0000","last_fullsized":"2024-09-13T14:56:18.184158+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"315'2429","reported_seq":2440,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:39.293592+0000","last_change":"2024-09-13T14:56:09.031898+0000","last_active":"2024-09-13T14:56:39.293592+0000","last_peered":"2024-09-13T14:56:39.293592+0000","last_clean":"2024-09-13T14:56:39.293592+0000","last_became_active":"2024-09-13T14:56:09.031546+0000","last_became_peered":"2024-09-13T14:56:09.031546+0000","last_unstale":"2024-09-13T14:56:39.293592+0000","last_undegraded":"2024-09-13T14:56:39.293592+0000","last_fullsized":"2024-09-13T14:56:39.293592+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":2429,"log_dups_size":0,"ondisk_log_size":2429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:18.018588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159186944,"num_objects":2429,"num_object_clones":0,"num_object_copies":7287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":155456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":86,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060151+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:56:18.060151+0000","last_peered":"2024-09-13T14:56:18.060151+0000","last_clean":"2024-09-13T14:56:18.060151+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:56:18.060151+0000","last_undegraded":"2024-09-13T14:56:18.060151+0000","last_fullsized":"2024-09-13T14:56:18.060151+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":76,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184284+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:56:18.184284+0000","last_peered":"2024-09-13T14:56:18.184284+0000","last_clean":"2024-09-13T14:56:18.184284+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:56:18.184284+0000","last_undegraded":"2024-09-13T14:56:18.184284+0000","last_fullsized":"2024-09-13T14:56:18.184284+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":82,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184698+0000","last_change":"2024-09-13T14:52:02.567722+0000","last_active":"2024-09-13T14:56:18.184698+0000","last_peered":"2024-09-13T14:56:18.184698+0000","last_clean":"2024-09-13T14:56:18.184698+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:56:18.184698+0000","last_undegraded":"2024-09-13T14:56:18.184698+0000","last_fullsized":"2024-09-13T14:56:18.184698+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:52:02.567611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:52:02.567611+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:03.419129+0000","scrub_duration":0.0030819879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":326,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.183998+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:56:18.183998+0000","last_peered":"2024-09-13T14:56:18.183998+0000","last_clean":"2024-09-13T14:56:18.183998+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:56:18.183998+0000","last_undegraded":"2024-09-13T14:56:18.183998+0000","last_fullsized":"2024-09-13T14:56:18.183998+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":475,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184548+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:56:18.184548+0000","last_peered":"2024-09-13T14:56:18.184548+0000","last_clean":"2024-09-13T14:56:18.184548+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:56:18.184548+0000","last_undegraded":"2024-09-13T14:56:18.184548+0000","last_fullsized":"2024-09-13T14:56:18.184548+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":108,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060258+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:56:18.060258+0000","last_peered":"2024-09-13T14:56:18.060258+0000","last_clean":"2024-09-13T14:56:18.060258+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:56:18.060258+0000","last_undegraded":"2024-09-13T14:56:18.060258+0000","last_fullsized":"2024-09-13T14:56:18.060258+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":272,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184037+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:56:18.184037+0000","last_peered":"2024-09-13T14:56:18.184037+0000","last_clean":"2024-09-13T14:56:18.184037+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:56:18.184037+0000","last_undegraded":"2024-09-13T14:56:18.184037+0000","last_fullsized":"2024-09-13T14:56:18.184037+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":521,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.556655+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:56:38.556655+0000","last_peered":"2024-09-13T14:56:38.556655+0000","last_clean":"2024-09-13T14:56:38.556655+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:56:38.556655+0000","last_undegraded":"2024-09-13T14:56:38.556655+0000","last_fullsized":"2024-09-13T14:56:38.556655+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":190,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184108+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:56:18.184108+0000","last_peered":"2024-09-13T14:56:18.184108+0000","last_clean":"2024-09-13T14:56:18.184108+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:56:18.184108+0000","last_undegraded":"2024-09-13T14:56:18.184108+0000","last_fullsized":"2024-09-13T14:56:18.184108+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":527,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.558466+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:56:38.558466+0000","last_peered":"2024-09-13T14:56:38.558466+0000","last_clean":"2024-09-13T14:56:38.558466+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:56:38.558466+0000","last_undegraded":"2024-09-13T14:56:38.558466+0000","last_fullsized":"2024-09-13T14:56:38.558466+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":76,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060492+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:56:18.060492+0000","last_peered":"2024-09-13T14:56:18.060492+0000","last_clean":"2024-09-13T14:56:18.060492+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:56:18.060492+0000","last_undegraded":"2024-09-13T14:56:18.060492+0000","last_fullsized":"2024-09-13T14:56:18.060492+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":98,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.060187+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:56:18.060187+0000","last_peered":"2024-09-13T14:56:18.060187+0000","last_clean":"2024-09-13T14:56:18.060187+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:56:18.060187+0000","last_undegraded":"2024-09-13T14:56:18.060187+0000","last_fullsized":"2024-09-13T14:56:18.060187+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":297,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:56:18.184655+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:56:18.184655+0000","last_peered":"2024-09-13T14:56:18.184655+0000","last_clean":"2024-09-13T14:56:18.184655+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:56:18.184655+0000","last_undegraded":"2024-09-13T14:56:18.184655+0000","last_fullsized":"2024-09-13T14:56:18.184655+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":186,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555355+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:56:38.555355+0000","last_peered":"2024-09-13T14:56:38.555355+0000","last_clean":"2024-09-13T14:56:38.555355+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:56:38.555355+0000","last_undegraded":"2024-09-13T14:56:38.555355+0000","last_fullsized":"2024-09-13T14:56:38.555355+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":485,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.554747+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:56:38.554747+0000","last_peered":"2024-09-13T14:56:38.554747+0000","last_clean":"2024-09-13T14:56:38.554747+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:56:38.554747+0000","last_undegraded":"2024-09-13T14:56:38.554747+0000","last_fullsized":"2024-09-13T14:56:38.554747+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":2480078848,"num_objects":37843,"num_object_clones":0,"num_object_copies":113529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37843,"num_write_kb":2421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7398227968,"data_stored":7398227968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37843,"ondisk_log_size":37843,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365243,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1957472,"kb_used_data":1048084,"kb_used_omap":1,"kb_used_meta":909374,"kb_avail":91787680,"statfs":{"total":95995035648,"available":93990584320,"internally_reserved":0,"allocated":1073238016,"data_stored":1071443427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931199320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561473,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2023620,"kb_used_data":1130044,"kb_used_omap":1,"kb_used_meta":893502,"kb_avail":91721532,"statfs":{"total":95995035648,"available":93922848768,"internally_reserved":0,"allocated":1157165056,"data_stored":1155386328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914946392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790405,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509840,"kb_used_data":307908,"kb_used_omap":1,"kb_used_meta":1201918,"kb_avail":92235312,"statfs":{"total":95995035648,"available":94448959488,"internally_reserved":0,"allocated":315297792,"data_stored":313515528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1230764376},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986633,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1355992,"kb_used_data":461584,"kb_used_omap":1,"kb_used_meta":894398,"kb_avail":92389160,"statfs":{"total":95995035648,"available":94606499840,"internally_reserved":0,"allocated":472662016,"data_stored":470871684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915863896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519788,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1828924,"kb_used_data":1079924,"kb_used_omap":4,"kb_used_meta":748987,"kb_avail":91916228,"statfs":{"total":95995035648,"available":94122217472,"internally_reserved":0,"allocated":1105842176,"data_stored":1104067768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":766963136},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304704,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2245012,"kb_used_data":1505544,"kb_used_omap":6,"kb_used_meta":739449,"kb_avail":91500140,"statfs":{"total":95995035648,"available":93696143360,"internally_reserved":0,"allocated":1541677056,"data_stored":1539891220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":757196231},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828007,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1736956,"kb_used_data":713780,"kb_used_omap":5,"kb_used_meta":1023162,"kb_avail":92008196,"statfs":{"total":95995035648,"available":94216392704,"internally_reserved":0,"allocated":730910720,"data_stored":729135739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1047718124},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285872,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1866248,"kb_used_data":928896,"kb_used_omap":5,"kb_used_meta":937338,"kb_avail":91878904,"statfs":{"total":95995035648,"available":94083997696,"internally_reserved":0,"allocated":951189504,"data_stored":949410076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":959834171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":975503360,"data_stored":975503360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":734003200,"data_stored":734003200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1589575680,"data_stored":1589575680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1131216896,"data_stored":1131216896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":469303296,"data_stored":469303296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":321388544,"data_stored":321388544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1096548352,"data_stored":1096548352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1080688640,"data_stored":1080688640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:56:44.772 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:56:44.772 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:56:44.772 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:56:44.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:56:44.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:44.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:44.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:45.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:45.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:45.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:45.268 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:45 smithi003 ceph-mon[107278]: from='client.56407 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:45.268 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:45 smithi003 ceph-mon[107278]: pgmap v1118: 109 pgs: 109 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:56:45.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:45 smithi003 ceph-mon[104007]: from='client.56407 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:45.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:45 smithi003 ceph-mon[104007]: pgmap v1118: 109 pgs: 109 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:56:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:45.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:45.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:45.420 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 47420 47404 128.786 97.0625 0.00798925 0.00775829 2024-09-13T14:56:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:45 smithi019 ceph-mon[91296]: from='client.56407 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:45 smithi019 ceph-mon[91296]: pgmap v1118: 109 pgs: 109 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:56:45.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:45.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:45.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:45.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:45.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:45.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:45.876 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:45.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:46.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:46.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:46.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:46.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:46.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:46.422 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 49050 49034 127.661 101.875 0.0120191 0.00782749 2024-09-13T14:56:46.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:46.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:46.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:46.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:46.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:46.711 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:46.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:46.844 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:46.946 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:46.972 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:47.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:47.103 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:47 smithi003 ceph-mon[107278]: pgmap v1119: 109 pgs: 109 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-13T14:56:47.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:47 smithi003 ceph-mon[104007]: pgmap v1119: 109 pgs: 109 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-13T14:56:47.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:47.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:47.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.377 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:47.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:47.421 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 50375 50359 125.867 82.8125 0.0143997 0.00793955 2024-09-13T14:56:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:47 smithi019 ceph-mon[91296]: pgmap v1119: 109 pgs: 109 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-13T14:56:47.478 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:47.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.513 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:47.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:47.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:47.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:47.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:47.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:47.917 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:47.944 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:48.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:48.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:48.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:48.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:48.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:48.421 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 51848 51832 124.567 92.0625 0.0101355 0.00802248 2024-09-13T14:56:48.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:48.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:48.627 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:48.654 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:48.723 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:48.724 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:48.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:48.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:48.897 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:48.929 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:48.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:56:49.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:49.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:49.182 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:49.211 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:49.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:49.340 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:49.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:49 smithi003 ceph-mon[104007]: pgmap v1120: 109 pgs: 109 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T14:56:49.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:49 smithi003 ceph-mon[107278]: pgmap v1120: 109 pgs: 109 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T14:56:49.424 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 53305 53289 123.323 91.0625 0.00985186 0.00810308 2024-09-13T14:56:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:49 smithi019 ceph-mon[91296]: pgmap v1120: 109 pgs: 109 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T14:56:49.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:49.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:49.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:56:49.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:49.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:49.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:49.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:49.911 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:49.937 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:50.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:50.072 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:50.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:50.202 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:50.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:50.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:50.422 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 54608 54592 121.827 81.4375 0.0107673 0.00820291 2024-09-13T14:56:50.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:50.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:50.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:50.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:56:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:56:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:50.728 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:50.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:50.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:50.924 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:50.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:50.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:50.990 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:51.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:51.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:51.093 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:56:51.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:51 smithi003 ceph-mon[104007]: pgmap v1121: 109 pgs: 109 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T14:56:51.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:51 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2698029669' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:51.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:51 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:51.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:51 smithi003 ceph-mon[107278]: pgmap v1121: 109 pgs: 109 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T14:56:51.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:51 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2698029669' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:51.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:51 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:51.190 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:51.221 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:51.322 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:51.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:51.422 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 56022 56006 120.674 88.375 0.00521385 0.00828228 2024-09-13T14:56:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:51 smithi019 ceph-mon[91296]: pgmap v1121: 109 pgs: 109 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T14:56:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:51 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2698029669' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:51.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:51 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:51.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:51.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:51.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:51.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:51.739 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:51.769 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:51.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:51.916 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:52.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:52.051 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:52.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:52.289 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:52.319 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:52 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:52 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:56:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:52 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:52.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:52 smithi003 ceph-mon[104007]: osdmap e316: 8 total, 8 up, 8 in 2024-09-13T14:56:52.358 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:52 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:52.358 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:52 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:56:52.359 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:52 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:52.359 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:52 smithi003 ceph-mon[107278]: osdmap e316: 8 total, 8 up, 8 in 2024-09-13T14:56:52.421 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:52.422 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 57492 57476 119.714 91.875 0.00642248 0.00834802 2024-09-13T14:56:52.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:52 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:52 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:56:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:52 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:52.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:52 smithi019 ceph-mon[91296]: osdmap e316: 8 total, 8 up, 8 in 2024-09-13T14:56:52.452 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:52.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:52.585 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:52.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:52.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:52.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:52.857 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:52.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:52.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:52.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:53.099 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:53.130 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:53.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:53 smithi019 ceph-mon[91296]: pgmap v1123: 109 pgs: 109 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-13T14:56:53.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:53.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:53.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:53.372 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:53.405 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:53.422 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 58935 58919 118.761 90.1875 0.00996134 0.00841462 2024-09-13T14:56:53.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:53 smithi003 ceph-mon[107278]: pgmap v1123: 109 pgs: 109 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-13T14:56:53.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:53 smithi003 ceph-mon[104007]: pgmap v1123: 109 pgs: 109 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-13T14:56:53.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:53.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:53.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:53.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:53.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:53.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:53.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:53.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:54.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:54.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:54.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:54.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:54.337 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:54.367 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:54.423 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 60377 60361 117.866 90.125 0.00849063 0.0084792 2024-09-13T14:56:54.432 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:56:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:56:54] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-13T14:56:54.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:54.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:54.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:54.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:54.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:54.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:54.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:54.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:54.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:55.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:55.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:55.180 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:55.212 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:55.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:55.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:55.423 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 61651 61635 116.707 79.625 0.00765815 0.00856247 2024-09-13T14:56:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:55 smithi019 ceph-mon[91296]: pgmap v1124: 109 pgs: 109 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T14:56:55.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:55 smithi003 ceph-mon[104007]: pgmap v1124: 109 pgs: 109 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T14:56:55.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:55 smithi003 ceph-mon[107278]: pgmap v1124: 109 pgs: 109 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T14:56:55.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:55.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:55.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:55.625 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:55.727 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.762 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:55.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:55.863 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:55.893 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:55.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:56.136 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:56.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:56.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:56.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:56.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.425 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 63093 63077 115.925 90.125 0.00653565 0.00862075 2024-09-13T14:56:56.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:56.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:56.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:56.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:56.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:56.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:56.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:57.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:57.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:57.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:56:57.139 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T14:56:57.141 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T14:56:57.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:57.301 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:57.331 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:57.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:57 smithi003 ceph-mon[104007]: pgmap v1125: 109 pgs: 109 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-09-13T14:56:57.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:57 smithi003 ceph-mon[107278]: pgmap v1125: 109 pgs: 109 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-09-13T14:56:57.423 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 64209 64193 114.605 69.75 0.0157806 0.00871978 2024-09-13T14:56:57.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:57 smithi019 ceph-mon[91296]: pgmap v1125: 109 pgs: 109 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-09-13T14:56:57.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:57.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:57.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:57.607 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:57.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:57.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:57.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:57.883 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:57.913 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:57.913 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:57.986 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:58.018 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:58.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:58.147 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:58.249 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:58.277 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:58.379 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:58.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:58 smithi003 ceph-mon[104007]: from='client.36684 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:58.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:58.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:58.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:58 smithi003 ceph-mon[107278]: from='client.36684 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:58.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:58.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:58.407 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:58.426 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 65570 65554 113.782 85.0625 0.0122413 0.00878289 2024-09-13T14:56:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:58 smithi019 ceph-mon[91296]: from='client.36684 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:56:58.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:58.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:58.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:58.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:58.686 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:58.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:58.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:56:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:58.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:58.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:58.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:59.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:59.118 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:59.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:59.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:59.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:59.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:59.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:56:59 smithi003 ceph-mon[107278]: pgmap v1126: 109 pgs: 109 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-13T14:56:59.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:56:59 smithi003 ceph-mon[104007]: pgmap v1126: 109 pgs: 109 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-13T14:56:59.426 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 66858 66842 112.876 80.5 0.0104595 0.00885298 2024-09-13T14:56:59.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:56:59 smithi019 ceph-mon[91296]: pgmap v1126: 109 pgs: 109 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-13T14:56:59.457 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1125,"stamp":"2024-09-13T14:56:55.966519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3865778125,"num_objects":59004,"num_object_clones":0,"num_object_copies":177012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59042,"num_write_kb":3774624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":853342,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61137,"ondisk_log_size":61137,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18798004,"kb_used_data":11360948,"kb_used_omap":29,"kb_used_meta":7436962,"kb_avail":731163212,"statfs":{"total":767960285184,"available":748711129088,"internally_reserved":0,"allocated":11633610752,"data_stored":11619271726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":7615449614},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":959971328,"num_objects":14648,"num_object_clones":0,"num_object_copies":43944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14648,"num_write_kb":937472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003766"},"pg_stats":[{"pgid":"3.1f","version":"65'4","reported_seq":512,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555060+0000","last_change":"2024-09-13T14:35:30.477266+0000","last_active":"2024-09-13T14:56:38.555060+0000","last_peered":"2024-09-13T14:56:38.555060+0000","last_clean":"2024-09-13T14:56:38.555060+0000","last_became_active":"2024-09-13T14:35:30.476895+0000","last_became_peered":"2024-09-13T14:35:30.476895+0000","last_unstale":"2024-09-13T14:56:38.555060+0000","last_undegraded":"2024-09-13T14:56:38.555060+0000","last_fullsized":"2024-09-13T14:56:38.555060+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:26.631867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":288,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555526+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T14:56:38.555526+0000","last_peered":"2024-09-13T14:56:38.555526+0000","last_clean":"2024-09-13T14:56:38.555526+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T14:56:38.555526+0000","last_undegraded":"2024-09-13T14:56:38.555526+0000","last_fullsized":"2024-09-13T14:56:38.555526+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":79,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560882+0000","last_change":"2024-09-13T14:48:29.129149+0000","last_active":"2024-09-13T14:56:38.560882+0000","last_peered":"2024-09-13T14:56:38.560882+0000","last_clean":"2024-09-13T14:56:38.560882+0000","last_became_active":"2024-09-13T14:48:29.128762+0000","last_became_peered":"2024-09-13T14:48:29.128762+0000","last_unstale":"2024-09-13T14:56:38.560882+0000","last_undegraded":"2024-09-13T14:56:38.560882+0000","last_fullsized":"2024-09-13T14:56:38.560882+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.680926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":472,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.085769+0000","last_change":"2024-09-13T14:37:08.753013+0000","last_active":"2024-09-13T14:56:51.085769+0000","last_peered":"2024-09-13T14:56:51.085769+0000","last_clean":"2024-09-13T14:56:51.085769+0000","last_became_active":"2024-09-13T14:37:08.749206+0000","last_became_peered":"2024-09-13T14:37:08.749206+0000","last_unstale":"2024-09-13T14:56:51.085769+0000","last_undegraded":"2024-09-13T14:56:51.085769+0000","last_fullsized":"2024-09-13T14:56:51.085769+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:12.183584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":262,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560935+0000","last_change":"2024-09-13T14:21:52.612480+0000","last_active":"2024-09-13T14:56:38.560935+0000","last_peered":"2024-09-13T14:56:38.560935+0000","last_clean":"2024-09-13T14:56:38.560935+0000","last_became_active":"2024-09-13T14:21:52.612159+0000","last_became_peered":"2024-09-13T14:21:52.612159+0000","last_unstale":"2024-09-13T14:56:38.560935+0000","last_undegraded":"2024-09-13T14:56:38.560935+0000","last_fullsized":"2024-09-13T14:56:38.560935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:14.203243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":295,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555788+0000","last_change":"2024-09-13T14:37:08.749493+0000","last_active":"2024-09-13T14:56:38.555788+0000","last_peered":"2024-09-13T14:56:38.555788+0000","last_clean":"2024-09-13T14:56:38.555788+0000","last_became_active":"2024-09-13T14:37:08.749093+0000","last_became_peered":"2024-09-13T14:37:08.749093+0000","last_unstale":"2024-09-13T14:56:38.555788+0000","last_undegraded":"2024-09-13T14:56:38.555788+0000","last_fullsized":"2024-09-13T14:56:38.555788+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:34.334570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":279,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560757+0000","last_change":"2024-09-13T14:32:09.491001+0000","last_active":"2024-09-13T14:56:38.560757+0000","last_peered":"2024-09-13T14:56:38.560757+0000","last_clean":"2024-09-13T14:56:38.560757+0000","last_became_active":"2024-09-13T14:32:09.490637+0000","last_became_peered":"2024-09-13T14:32:09.490637+0000","last_unstale":"2024-09-13T14:56:38.560757+0000","last_undegraded":"2024-09-13T14:56:38.560757+0000","last_fullsized":"2024-09-13T14:56:38.560757+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:14.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":325,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557165+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T14:56:38.557165+0000","last_peered":"2024-09-13T14:56:38.557165+0000","last_clean":"2024-09-13T14:56:38.557165+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T14:56:38.557165+0000","last_undegraded":"2024-09-13T14:56:38.557165+0000","last_fullsized":"2024-09-13T14:56:38.557165+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":491,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557823+0000","last_change":"2024-09-13T14:54:27.142651+0000","last_active":"2024-09-13T14:56:38.557823+0000","last_peered":"2024-09-13T14:56:38.557823+0000","last_clean":"2024-09-13T14:56:38.557823+0000","last_became_active":"2024-09-13T14:54:27.142239+0000","last_became_peered":"2024-09-13T14:54:27.142239+0000","last_unstale":"2024-09-13T14:56:38.557823+0000","last_undegraded":"2024-09-13T14:56:38.557823+0000","last_fullsized":"2024-09-13T14:56:38.557823+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:41.308930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":496,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.092690+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T14:56:51.092690+0000","last_peered":"2024-09-13T14:56:51.092690+0000","last_clean":"2024-09-13T14:56:51.092690+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T14:56:51.092690+0000","last_undegraded":"2024-09-13T14:56:51.092690+0000","last_fullsized":"2024-09-13T14:56:51.092690+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":297,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555857+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T14:56:38.555857+0000","last_peered":"2024-09-13T14:56:38.555857+0000","last_clean":"2024-09-13T14:56:38.555857+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T14:56:38.555857+0000","last_undegraded":"2024-09-13T14:56:38.555857+0000","last_fullsized":"2024-09-13T14:56:38.555857+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":438,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.084675+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T14:56:51.084675+0000","last_peered":"2024-09-13T14:56:51.084675+0000","last_clean":"2024-09-13T14:56:51.084675+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T14:56:51.084675+0000","last_undegraded":"2024-09-13T14:56:51.084675+0000","last_fullsized":"2024-09-13T14:56:51.084675+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":153,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.092512+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T14:56:51.092512+0000","last_peered":"2024-09-13T14:56:51.092512+0000","last_clean":"2024-09-13T14:56:51.092512+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T14:56:51.092512+0000","last_undegraded":"2024-09-13T14:56:51.092512+0000","last_fullsized":"2024-09-13T14:56:51.092512+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":530,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.556585+0000","last_change":"2024-09-13T14:49:05.847920+0000","last_active":"2024-09-13T14:56:38.556585+0000","last_peered":"2024-09-13T14:56:38.556585+0000","last_clean":"2024-09-13T14:56:38.556585+0000","last_became_active":"2024-09-13T14:49:05.842712+0000","last_became_peered":"2024-09-13T14:49:05.842712+0000","last_unstale":"2024-09-13T14:56:38.556585+0000","last_undegraded":"2024-09-13T14:56:38.556585+0000","last_fullsized":"2024-09-13T14:56:38.556585+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:32.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":471,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.085712+0000","last_change":"2024-09-13T14:47:40.126225+0000","last_active":"2024-09-13T14:56:51.085712+0000","last_peered":"2024-09-13T14:56:51.085712+0000","last_clean":"2024-09-13T14:56:51.085712+0000","last_became_active":"2024-09-13T14:47:40.125879+0000","last_became_peered":"2024-09-13T14:47:40.125879+0000","last_unstale":"2024-09-13T14:56:51.085712+0000","last_undegraded":"2024-09-13T14:56:51.085712+0000","last_fullsized":"2024-09-13T14:56:51.085712+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:04.736203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":278,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.548427+0000","last_change":"2024-09-13T14:32:09.491556+0000","last_active":"2024-09-13T14:56:38.548427+0000","last_peered":"2024-09-13T14:56:38.548427+0000","last_clean":"2024-09-13T14:56:38.548427+0000","last_became_active":"2024-09-13T14:32:09.490630+0000","last_became_peered":"2024-09-13T14:32:09.490630+0000","last_unstale":"2024-09-13T14:56:38.548427+0000","last_undegraded":"2024-09-13T14:56:38.548427+0000","last_fullsized":"2024-09-13T14:56:38.548427+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:34.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":439,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.085433+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T14:56:51.085433+0000","last_peered":"2024-09-13T14:56:51.085433+0000","last_clean":"2024-09-13T14:56:51.085433+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T14:56:51.085433+0000","last_undegraded":"2024-09-13T14:56:51.085433+0000","last_fullsized":"2024-09-13T14:56:51.085433+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":513,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555333+0000","last_change":"2024-09-13T14:48:29.126725+0000","last_active":"2024-09-13T14:56:38.555333+0000","last_peered":"2024-09-13T14:56:38.555333+0000","last_clean":"2024-09-13T14:56:38.555333+0000","last_became_active":"2024-09-13T14:48:29.126397+0000","last_became_peered":"2024-09-13T14:48:29.126397+0000","last_unstale":"2024-09-13T14:56:38.555333+0000","last_undegraded":"2024-09-13T14:56:38.555333+0000","last_fullsized":"2024-09-13T14:56:38.555333+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:17.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":444,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.085259+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T14:56:51.085259+0000","last_peered":"2024-09-13T14:56:51.085259+0000","last_clean":"2024-09-13T14:56:51.085259+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T14:56:51.085259+0000","last_undegraded":"2024-09-13T14:56:51.085259+0000","last_fullsized":"2024-09-13T14:56:51.085259+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":77,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560825+0000","last_change":"2024-09-13T14:48:38.828731+0000","last_active":"2024-09-13T14:56:38.560825+0000","last_peered":"2024-09-13T14:56:38.560825+0000","last_clean":"2024-09-13T14:56:38.560825+0000","last_became_active":"2024-09-13T14:48:38.828307+0000","last_became_peered":"2024-09-13T14:48:38.828307+0000","last_unstale":"2024-09-13T14:56:38.560825+0000","last_undegraded":"2024-09-13T14:56:38.560825+0000","last_fullsized":"2024-09-13T14:56:38.560825+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:46.588296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":194,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.092769+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T14:56:51.092769+0000","last_peered":"2024-09-13T14:56:51.092769+0000","last_clean":"2024-09-13T14:56:51.092769+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T14:56:51.092769+0000","last_undegraded":"2024-09-13T14:56:51.092769+0000","last_fullsized":"2024-09-13T14:56:51.092769+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.638778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":79,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547591+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T14:56:38.547591+0000","last_peered":"2024-09-13T14:56:38.547591+0000","last_clean":"2024-09-13T14:56:38.547591+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T14:56:38.547591+0000","last_undegraded":"2024-09-13T14:56:38.547591+0000","last_fullsized":"2024-09-13T14:56:38.547591+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":217,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547155+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T14:56:38.547155+0000","last_peered":"2024-09-13T14:56:38.547155+0000","last_clean":"2024-09-13T14:56:38.547155+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T14:56:38.547155+0000","last_undegraded":"2024-09-13T14:56:38.547155+0000","last_fullsized":"2024-09-13T14:56:38.547155+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":285,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557188+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T14:56:38.557188+0000","last_peered":"2024-09-13T14:56:38.557188+0000","last_clean":"2024-09-13T14:56:38.557188+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T14:56:38.557188+0000","last_undegraded":"2024-09-13T14:56:38.557188+0000","last_fullsized":"2024-09-13T14:56:38.557188+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":278,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557226+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T14:56:38.557226+0000","last_peered":"2024-09-13T14:56:38.557226+0000","last_clean":"2024-09-13T14:56:38.557226+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T14:56:38.557226+0000","last_undegraded":"2024-09-13T14:56:38.557226+0000","last_fullsized":"2024-09-13T14:56:38.557226+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"316'3855","reported_seq":3876,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:55.022548+0000","last_change":"2024-09-13T14:56:09.033158+0000","last_active":"2024-09-13T14:56:55.022548+0000","last_peered":"2024-09-13T14:56:55.022548+0000","last_clean":"2024-09-13T14:56:55.022548+0000","last_became_active":"2024-09-13T14:56:09.032864+0000","last_became_peered":"2024-09-13T14:56:09.032864+0000","last_unstale":"2024-09-13T14:56:55.022548+0000","last_undegraded":"2024-09-13T14:56:55.022548+0000","last_fullsized":"2024-09-13T14:56:55.022548+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:36.421513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252641280,"num_objects":3855,"num_object_clones":0,"num_object_copies":11565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3855,"num_write_kb":246720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":261,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555635+0000","last_change":"2024-09-13T14:21:53.702547+0000","last_active":"2024-09-13T14:56:38.555635+0000","last_peered":"2024-09-13T14:56:38.555635+0000","last_clean":"2024-09-13T14:56:38.555635+0000","last_became_active":"2024-09-13T14:21:53.701535+0000","last_became_peered":"2024-09-13T14:21:53.701535+0000","last_unstale":"2024-09-13T14:56:38.555635+0000","last_undegraded":"2024-09-13T14:56:38.555635+0000","last_fullsized":"2024-09-13T14:56:38.555635+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:43.053260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":311,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.548471+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T14:56:38.548471+0000","last_peered":"2024-09-13T14:56:38.548471+0000","last_clean":"2024-09-13T14:56:38.548471+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T14:56:38.548471+0000","last_undegraded":"2024-09-13T14:56:38.548471+0000","last_fullsized":"2024-09-13T14:56:38.548471+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":511,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.556737+0000","last_change":"2024-09-13T14:32:09.487072+0000","last_active":"2024-09-13T14:56:38.556737+0000","last_peered":"2024-09-13T14:56:38.556737+0000","last_clean":"2024-09-13T14:56:38.556737+0000","last_became_active":"2024-09-13T14:32:09.486438+0000","last_became_peered":"2024-09-13T14:32:09.486438+0000","last_unstale":"2024-09-13T14:56:38.556737+0000","last_undegraded":"2024-09-13T14:56:38.556737+0000","last_fullsized":"2024-09-13T14:56:38.556737+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:41.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":201,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.084612+0000","last_change":"2024-09-13T14:48:38.829427+0000","last_active":"2024-09-13T14:56:51.084612+0000","last_peered":"2024-09-13T14:56:51.084612+0000","last_clean":"2024-09-13T14:56:51.084612+0000","last_became_active":"2024-09-13T14:48:38.829057+0000","last_became_peered":"2024-09-13T14:48:38.829057+0000","last_unstale":"2024-09-13T14:56:51.084612+0000","last_undegraded":"2024-09-13T14:56:51.084612+0000","last_fullsized":"2024-09-13T14:56:51.084612+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:13.658287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"316'3529","reported_seq":3549,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:52.322220+0000","last_change":"2024-09-13T14:56:09.034412+0000","last_active":"2024-09-13T14:56:52.322220+0000","last_peered":"2024-09-13T14:56:52.322220+0000","last_clean":"2024-09-13T14:56:52.322220+0000","last_became_active":"2024-09-13T14:56:09.033995+0000","last_became_peered":"2024-09-13T14:56:09.033995+0000","last_unstale":"2024-09-13T14:56:52.322220+0000","last_undegraded":"2024-09-13T14:56:52.322220+0000","last_fullsized":"2024-09-13T14:56:52.322220+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:28.253264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231276544,"num_objects":3529,"num_object_clones":0,"num_object_copies":10587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3529,"num_write_kb":225856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":482,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.085624+0000","last_change":"2024-09-13T14:49:05.859619+0000","last_active":"2024-09-13T14:56:51.085624+0000","last_peered":"2024-09-13T14:56:51.085624+0000","last_clean":"2024-09-13T14:56:51.085624+0000","last_became_active":"2024-09-13T14:49:05.859313+0000","last_became_peered":"2024-09-13T14:49:05.859313+0000","last_unstale":"2024-09-13T14:56:51.085624+0000","last_undegraded":"2024-09-13T14:56:51.085624+0000","last_fullsized":"2024-09-13T14:56:51.085624+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:05.432297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.548235+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T14:56:38.548235+0000","last_peered":"2024-09-13T14:56:38.548235+0000","last_clean":"2024-09-13T14:56:38.548235+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T14:56:38.548235+0000","last_undegraded":"2024-09-13T14:56:38.548235+0000","last_fullsized":"2024-09-13T14:56:38.548235+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":471,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.086437+0000","last_change":"2024-09-13T14:37:08.756339+0000","last_active":"2024-09-13T14:56:51.086437+0000","last_peered":"2024-09-13T14:56:51.086437+0000","last_clean":"2024-09-13T14:56:51.086437+0000","last_became_active":"2024-09-13T14:37:08.753708+0000","last_became_peered":"2024-09-13T14:37:08.753708+0000","last_unstale":"2024-09-13T14:56:51.086437+0000","last_undegraded":"2024-09-13T14:56:51.086437+0000","last_fullsized":"2024-09-13T14:56:51.086437+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:48.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":99,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.558299+0000","last_change":"2024-09-13T14:47:07.239516+0000","last_active":"2024-09-13T14:56:38.558299+0000","last_peered":"2024-09-13T14:56:38.558299+0000","last_clean":"2024-09-13T14:56:38.558299+0000","last_became_active":"2024-09-13T14:47:07.239097+0000","last_became_peered":"2024-09-13T14:47:07.239097+0000","last_unstale":"2024-09-13T14:56:38.558299+0000","last_undegraded":"2024-09-13T14:56:38.558299+0000","last_fullsized":"2024-09-13T14:56:38.558299+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:49.418974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.418974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:20.864053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"316'3633","reported_seq":3654,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:52.314428+0000","last_change":"2024-09-13T14:56:09.034509+0000","last_active":"2024-09-13T14:56:52.314428+0000","last_peered":"2024-09-13T14:56:52.314428+0000","last_clean":"2024-09-13T14:56:52.314428+0000","last_became_active":"2024-09-13T14:56:09.034213+0000","last_became_peered":"2024-09-13T14:56:09.034213+0000","last_unstale":"2024-09-13T14:56:52.314428+0000","last_undegraded":"2024-09-13T14:56:52.314428+0000","last_fullsized":"2024-09-13T14:56:52.314428+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:26.274871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238092288,"num_objects":3633,"num_object_clones":0,"num_object_copies":10899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3633,"num_write_kb":232512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":498,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557739+0000","last_change":"2024-09-13T14:31:29.946960+0000","last_active":"2024-09-13T14:56:38.557739+0000","last_peered":"2024-09-13T14:56:38.557739+0000","last_clean":"2024-09-13T14:56:38.557739+0000","last_became_active":"2024-09-13T14:31:29.946233+0000","last_became_peered":"2024-09-13T14:31:29.946233+0000","last_unstale":"2024-09-13T14:56:38.557739+0000","last_undegraded":"2024-09-13T14:56:38.557739+0000","last_fullsized":"2024-09-13T14:56:38.557739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:13.134982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.549304+0000","last_change":"2024-09-13T14:31:29.962686+0000","last_active":"2024-09-13T14:56:38.549304+0000","last_peered":"2024-09-13T14:56:38.549304+0000","last_clean":"2024-09-13T14:56:38.549304+0000","last_became_active":"2024-09-13T14:31:29.962311+0000","last_became_peered":"2024-09-13T14:31:29.962311+0000","last_unstale":"2024-09-13T14:56:38.549304+0000","last_undegraded":"2024-09-13T14:56:38.549304+0000","last_fullsized":"2024-09-13T14:56:38.549304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:57.686656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":455,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.092551+0000","last_change":"2024-09-13T14:48:38.833059+0000","last_active":"2024-09-13T14:56:51.092551+0000","last_peered":"2024-09-13T14:56:51.092551+0000","last_clean":"2024-09-13T14:56:51.092551+0000","last_became_active":"2024-09-13T14:48:38.832700+0000","last_became_peered":"2024-09-13T14:48:38.832700+0000","last_unstale":"2024-09-13T14:56:51.092551+0000","last_undegraded":"2024-09-13T14:56:51.092551+0000","last_fullsized":"2024-09-13T14:56:51.092551+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.580714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":252,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.548126+0000","last_change":"2024-09-13T14:47:45.935437+0000","last_active":"2024-09-13T14:56:38.548126+0000","last_peered":"2024-09-13T14:56:38.548126+0000","last_clean":"2024-09-13T14:56:38.548126+0000","last_became_active":"2024-09-13T14:47:07.243749+0000","last_became_peered":"2024-09-13T14:47:07.243749+0000","last_unstale":"2024-09-13T14:56:38.548126+0000","last_undegraded":"2024-09-13T14:56:38.548126+0000","last_fullsized":"2024-09-13T14:56:38.548126+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:45.935355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.935355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:05.290003+0000","scrub_duration":0.0040929859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"316'3696","reported_seq":3717,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:55.077261+0000","last_change":"2024-09-13T14:56:09.034272+0000","last_active":"2024-09-13T14:56:55.077261+0000","last_peered":"2024-09-13T14:56:55.077261+0000","last_clean":"2024-09-13T14:56:55.077261+0000","last_became_active":"2024-09-13T14:56:09.033933+0000","last_became_peered":"2024-09-13T14:56:09.033933+0000","last_unstale":"2024-09-13T14:56:55.077261+0000","last_undegraded":"2024-09-13T14:56:55.077261+0000","last_fullsized":"2024-09-13T14:56:55.077261+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3696,"log_dups_size":0,"ondisk_log_size":3696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:40.177740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242221056,"num_objects":3696,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3696,"num_write_kb":236544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":77,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560426+0000","last_change":"2024-09-13T14:48:38.828520+0000","last_active":"2024-09-13T14:56:38.560426+0000","last_peered":"2024-09-13T14:56:38.560426+0000","last_clean":"2024-09-13T14:56:38.560426+0000","last_became_active":"2024-09-13T14:48:38.828054+0000","last_became_peered":"2024-09-13T14:48:38.828054+0000","last_unstale":"2024-09-13T14:56:38.560426+0000","last_undegraded":"2024-09-13T14:56:38.560426+0000","last_fullsized":"2024-09-13T14:56:38.560426+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:48.255490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":311,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555656+0000","last_change":"2024-09-13T14:45:44.450268+0000","last_active":"2024-09-13T14:56:38.555656+0000","last_peered":"2024-09-13T14:56:38.555656+0000","last_clean":"2024-09-13T14:56:38.555656+0000","last_became_active":"2024-09-13T14:45:44.449578+0000","last_became_peered":"2024-09-13T14:45:44.449578+0000","last_unstale":"2024-09-13T14:56:38.555656+0000","last_undegraded":"2024-09-13T14:56:38.555656+0000","last_fullsized":"2024-09-13T14:56:38.555656+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:35.563171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":294,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560841+0000","last_change":"2024-09-13T14:37:08.755661+0000","last_active":"2024-09-13T14:56:38.560841+0000","last_peered":"2024-09-13T14:56:38.560841+0000","last_clean":"2024-09-13T14:56:38.560841+0000","last_became_active":"2024-09-13T14:37:08.755005+0000","last_became_peered":"2024-09-13T14:37:08.755005+0000","last_unstale":"2024-09-13T14:56:38.560841+0000","last_undegraded":"2024-09-13T14:56:38.560841+0000","last_fullsized":"2024-09-13T14:56:38.560841+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:11.728511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":272,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.548331+0000","last_change":"2024-09-13T14:49:05.839436+0000","last_active":"2024-09-13T14:56:38.548331+0000","last_peered":"2024-09-13T14:56:38.548331+0000","last_clean":"2024-09-13T14:56:38.548331+0000","last_became_active":"2024-09-13T14:49:05.836437+0000","last_became_peered":"2024-09-13T14:49:05.836437+0000","last_unstale":"2024-09-13T14:56:38.548331+0000","last_undegraded":"2024-09-13T14:56:38.548331+0000","last_fullsized":"2024-09-13T14:56:38.548331+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:50.880092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:50.880092+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.634834+0000","scrub_duration":0.0080747389999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"315'3481","reported_seq":3500,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:50.984412+0000","last_change":"2024-09-13T14:56:09.101249+0000","last_active":"2024-09-13T14:56:50.984412+0000","last_peered":"2024-09-13T14:56:50.984412+0000","last_clean":"2024-09-13T14:56:50.984412+0000","last_became_active":"2024-09-13T14:56:09.100796+0000","last_became_peered":"2024-09-13T14:56:09.100796+0000","last_unstale":"2024-09-13T14:56:50.984412+0000","last_undegraded":"2024-09-13T14:56:50.984412+0000","last_fullsized":"2024-09-13T14:56:50.984412+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:37.036972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228130816,"num_objects":3481,"num_object_clones":0,"num_object_copies":10443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3481,"num_write_kb":222784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":367,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.093027+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T14:56:51.093027+0000","last_peered":"2024-09-13T14:56:51.093027+0000","last_clean":"2024-09-13T14:56:51.093027+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T14:56:51.093027+0000","last_undegraded":"2024-09-13T14:56:51.093027+0000","last_fullsized":"2024-09-13T14:56:51.093027+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":279,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557237+0000","last_change":"2024-09-13T14:31:29.951620+0000","last_active":"2024-09-13T14:56:38.557237+0000","last_peered":"2024-09-13T14:56:38.557237+0000","last_clean":"2024-09-13T14:56:38.557237+0000","last_became_active":"2024-09-13T14:31:29.951294+0000","last_became_peered":"2024-09-13T14:31:29.951294+0000","last_unstale":"2024-09-13T14:56:38.557237+0000","last_undegraded":"2024-09-13T14:56:38.557237+0000","last_fullsized":"2024-09-13T14:56:38.557237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:54.716255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":290,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560577+0000","last_change":"2024-09-13T14:32:09.490611+0000","last_active":"2024-09-13T14:56:38.560577+0000","last_peered":"2024-09-13T14:56:38.560577+0000","last_clean":"2024-09-13T14:56:38.560577+0000","last_became_active":"2024-09-13T14:32:09.490291+0000","last_became_peered":"2024-09-13T14:32:09.490291+0000","last_unstale":"2024-09-13T14:56:38.560577+0000","last_undegraded":"2024-09-13T14:56:38.560577+0000","last_fullsized":"2024-09-13T14:56:38.560577+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:46.554907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097960800000000008,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":99,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560285+0000","last_change":"2024-09-13T14:47:07.230948+0000","last_active":"2024-09-13T14:56:38.560285+0000","last_peered":"2024-09-13T14:56:38.560285+0000","last_clean":"2024-09-13T14:56:38.560285+0000","last_became_active":"2024-09-13T14:47:07.230499+0000","last_became_peered":"2024-09-13T14:47:07.230499+0000","last_unstale":"2024-09-13T14:56:38.560285+0000","last_undegraded":"2024-09-13T14:56:38.560285+0000","last_fullsized":"2024-09-13T14:56:38.560285+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:23.323328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"316'3813","reported_seq":3826,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:54.381289+0000","last_change":"2024-09-13T14:56:09.101282+0000","last_active":"2024-09-13T14:56:54.381289+0000","last_peered":"2024-09-13T14:56:54.381289+0000","last_clean":"2024-09-13T14:56:54.381289+0000","last_became_active":"2024-09-13T14:56:09.100956+0000","last_became_peered":"2024-09-13T14:56:09.100956+0000","last_unstale":"2024-09-13T14:56:54.381289+0000","last_undegraded":"2024-09-13T14:56:54.381289+0000","last_fullsized":"2024-09-13T14:56:54.381289+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3813,"log_dups_size":0,"ondisk_log_size":3813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:39.057985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249888768,"num_objects":3813,"num_object_clones":0,"num_object_copies":11439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3813,"num_write_kb":244032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":194,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.090330+0000","last_change":"2024-09-13T14:53:42.008298+0000","last_active":"2024-09-13T14:56:51.090330+0000","last_peered":"2024-09-13T14:56:51.090330+0000","last_clean":"2024-09-13T14:56:51.090330+0000","last_became_active":"2024-09-13T14:53:42.007766+0000","last_became_peered":"2024-09-13T14:53:42.007766+0000","last_unstale":"2024-09-13T14:56:51.090330+0000","last_undegraded":"2024-09-13T14:56:51.090330+0000","last_fullsized":"2024-09-13T14:56:51.090330+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:29.235755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":307,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.094620+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T14:56:51.094620+0000","last_peered":"2024-09-13T14:56:51.094620+0000","last_clean":"2024-09-13T14:56:51.094620+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T14:56:51.094620+0000","last_undegraded":"2024-09-13T14:56:51.094620+0000","last_fullsized":"2024-09-13T14:56:51.094620+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":498,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.092815+0000","last_change":"2024-09-13T14:37:08.754464+0000","last_active":"2024-09-13T14:56:51.092815+0000","last_peered":"2024-09-13T14:56:51.092815+0000","last_clean":"2024-09-13T14:56:51.092815+0000","last_became_active":"2024-09-13T14:37:08.754014+0000","last_became_peered":"2024-09-13T14:37:08.754014+0000","last_unstale":"2024-09-13T14:56:51.092815+0000","last_undegraded":"2024-09-13T14:56:51.092815+0000","last_fullsized":"2024-09-13T14:56:51.092815+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:58.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":73,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547300+0000","last_change":"2024-09-13T14:49:05.834249+0000","last_active":"2024-09-13T14:56:38.547300+0000","last_peered":"2024-09-13T14:56:38.547300+0000","last_clean":"2024-09-13T14:56:38.547300+0000","last_became_active":"2024-09-13T14:49:05.833909+0000","last_became_peered":"2024-09-13T14:49:05.833909+0000","last_unstale":"2024-09-13T14:56:38.547300+0000","last_undegraded":"2024-09-13T14:56:38.547300+0000","last_fullsized":"2024-09-13T14:56:38.547300+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:08.680654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.e","version":"315'3388","reported_seq":3406,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.011835+0000","last_change":"2024-09-13T14:56:09.032714+0000","last_active":"2024-09-13T14:56:51.011835+0000","last_peered":"2024-09-13T14:56:51.011835+0000","last_clean":"2024-09-13T14:56:51.011835+0000","last_became_active":"2024-09-13T14:56:09.032114+0000","last_became_peered":"2024-09-13T14:56:09.032114+0000","last_unstale":"2024-09-13T14:56:51.011835+0000","last_undegraded":"2024-09-13T14:56:51.011835+0000","last_fullsized":"2024-09-13T14:56:51.011835+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:53.695180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222035968,"num_objects":3388,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":216832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":519,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.556517+0000","last_change":"2024-09-13T14:35:30.477930+0000","last_active":"2024-09-13T14:56:38.556517+0000","last_peered":"2024-09-13T14:56:38.556517+0000","last_clean":"2024-09-13T14:56:38.556517+0000","last_became_active":"2024-09-13T14:35:30.477626+0000","last_became_peered":"2024-09-13T14:35:30.477626+0000","last_unstale":"2024-09-13T14:56:38.556517+0000","last_undegraded":"2024-09-13T14:56:38.556517+0000","last_fullsized":"2024-09-13T14:56:38.556517+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:12.945450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":529,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.554640+0000","last_change":"2024-09-13T14:48:29.137045+0000","last_active":"2024-09-13T14:56:38.554640+0000","last_peered":"2024-09-13T14:56:38.554640+0000","last_clean":"2024-09-13T14:56:38.554640+0000","last_became_active":"2024-09-13T14:48:29.136728+0000","last_became_peered":"2024-09-13T14:48:29.136728+0000","last_unstale":"2024-09-13T14:56:38.554640+0000","last_undegraded":"2024-09-13T14:56:38.554640+0000","last_fullsized":"2024-09-13T14:56:38.554640+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:09.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":328,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547437+0000","last_change":"2024-09-13T14:47:40.126692+0000","last_active":"2024-09-13T14:56:38.547437+0000","last_peered":"2024-09-13T14:56:38.547437+0000","last_clean":"2024-09-13T14:56:38.547437+0000","last_became_active":"2024-09-13T14:47:40.126352+0000","last_became_peered":"2024-09-13T14:47:40.126352+0000","last_unstale":"2024-09-13T14:56:38.547437+0000","last_undegraded":"2024-09-13T14:56:38.547437+0000","last_fullsized":"2024-09-13T14:56:38.547437+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:44.677166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":252,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547203+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T14:56:38.547203+0000","last_peered":"2024-09-13T14:56:38.547203+0000","last_clean":"2024-09-13T14:56:38.547203+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T14:56:38.547203+0000","last_undegraded":"2024-09-13T14:56:38.547203+0000","last_fullsized":"2024-09-13T14:56:38.547203+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"316'3804","reported_seq":3817,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:54.299024+0000","last_change":"2024-09-13T14:56:09.035158+0000","last_active":"2024-09-13T14:56:54.299024+0000","last_peered":"2024-09-13T14:56:54.299024+0000","last_clean":"2024-09-13T14:56:54.299024+0000","last_became_active":"2024-09-13T14:56:09.034847+0000","last_became_peered":"2024-09-13T14:56:09.034847+0000","last_unstale":"2024-09-13T14:56:54.299024+0000","last_undegraded":"2024-09-13T14:56:54.299024+0000","last_fullsized":"2024-09-13T14:56:54.299024+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3804,"log_dups_size":0,"ondisk_log_size":3804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:55.768082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249298944,"num_objects":3804,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3804,"num_write_kb":243456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":261,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.548180+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T14:56:38.548180+0000","last_peered":"2024-09-13T14:56:38.548180+0000","last_clean":"2024-09-13T14:56:38.548180+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T14:56:38.548180+0000","last_undegraded":"2024-09-13T14:56:38.548180+0000","last_fullsized":"2024-09-13T14:56:38.548180+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":430,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.084884+0000","last_change":"2024-09-13T14:37:08.756189+0000","last_active":"2024-09-13T14:56:51.084884+0000","last_peered":"2024-09-13T14:56:51.084884+0000","last_clean":"2024-09-13T14:56:51.084884+0000","last_became_active":"2024-09-13T14:37:08.755890+0000","last_became_peered":"2024-09-13T14:37:08.755890+0000","last_unstale":"2024-09-13T14:56:51.084884+0000","last_undegraded":"2024-09-13T14:56:51.084884+0000","last_fullsized":"2024-09-13T14:56:51.084884+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:16.747833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":289,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560459+0000","last_change":"2024-09-13T14:35:30.477399+0000","last_active":"2024-09-13T14:56:38.560459+0000","last_peered":"2024-09-13T14:56:38.560459+0000","last_clean":"2024-09-13T14:56:38.560459+0000","last_became_active":"2024-09-13T14:35:30.476516+0000","last_became_peered":"2024-09-13T14:35:30.476516+0000","last_unstale":"2024-09-13T14:56:38.560459+0000","last_undegraded":"2024-09-13T14:56:38.560459+0000","last_fullsized":"2024-09-13T14:56:38.560459+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:30.870295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035246099999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":73,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547134+0000","last_change":"2024-09-13T14:49:05.859580+0000","last_active":"2024-09-13T14:56:38.547134+0000","last_peered":"2024-09-13T14:56:38.547134+0000","last_clean":"2024-09-13T14:56:38.547134+0000","last_became_active":"2024-09-13T14:49:05.859092+0000","last_became_peered":"2024-09-13T14:49:05.859092+0000","last_unstale":"2024-09-13T14:56:38.547134+0000","last_undegraded":"2024-09-13T14:56:38.547134+0000","last_fullsized":"2024-09-13T14:56:38.547134+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:02.340821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.7","version":"316'3744","reported_seq":3757,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:54.316801+0000","last_change":"2024-09-13T14:56:09.035403+0000","last_active":"2024-09-13T14:56:54.316801+0000","last_peered":"2024-09-13T14:56:54.316801+0000","last_clean":"2024-09-13T14:56:54.316801+0000","last_became_active":"2024-09-13T14:56:09.035064+0000","last_became_peered":"2024-09-13T14:56:09.035064+0000","last_unstale":"2024-09-13T14:56:54.316801+0000","last_undegraded":"2024-09-13T14:56:54.316801+0000","last_fullsized":"2024-09-13T14:56:54.316801+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3744,"log_dups_size":0,"ondisk_log_size":3744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:45:44.521747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245366784,"num_objects":3744,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":239616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":153,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.092683+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T14:56:51.092683+0000","last_peered":"2024-09-13T14:56:51.092683+0000","last_clean":"2024-09-13T14:56:51.092683+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T14:56:51.092683+0000","last_undegraded":"2024-09-13T14:56:51.092683+0000","last_fullsized":"2024-09-13T14:56:51.092683+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":441,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.085810+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T14:56:51.085810+0000","last_peered":"2024-09-13T14:56:51.085810+0000","last_clean":"2024-09-13T14:56:51.085810+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T14:56:51.085810+0000","last_undegraded":"2024-09-13T14:56:51.085810+0000","last_fullsized":"2024-09-13T14:56:51.085810+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":99,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560339+0000","last_change":"2024-09-13T14:47:07.236951+0000","last_active":"2024-09-13T14:56:38.560339+0000","last_peered":"2024-09-13T14:56:38.560339+0000","last_clean":"2024-09-13T14:56:38.560339+0000","last_became_active":"2024-09-13T14:47:07.236563+0000","last_became_peered":"2024-09-13T14:47:07.236563+0000","last_unstale":"2024-09-13T14:56:38.560339+0000","last_undegraded":"2024-09-13T14:56:38.560339+0000","last_fullsized":"2024-09-13T14:56:38.560339+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:42.662383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:42.662383+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:35.701724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"316'3587","reported_seq":3608,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:53.097691+0000","last_change":"2024-09-13T14:56:09.037715+0000","last_active":"2024-09-13T14:56:53.097691+0000","last_peered":"2024-09-13T14:56:53.097691+0000","last_clean":"2024-09-13T14:56:53.097691+0000","last_became_active":"2024-09-13T14:56:09.037383+0000","last_became_peered":"2024-09-13T14:56:09.037383+0000","last_unstale":"2024-09-13T14:56:53.097691+0000","last_undegraded":"2024-09-13T14:56:53.097691+0000","last_fullsized":"2024-09-13T14:56:53.097691+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3587,"log_dups_size":0,"ondisk_log_size":3587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:33.806063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235077632,"num_objects":3587,"num_object_clones":0,"num_object_copies":10761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":229568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":298,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.558484+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T14:56:38.558484+0000","last_peered":"2024-09-13T14:56:38.558484+0000","last_clean":"2024-09-13T14:56:38.558484+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T14:56:38.558484+0000","last_undegraded":"2024-09-13T14:56:38.558484+0000","last_fullsized":"2024-09-13T14:56:38.558484+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":279,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555731+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T14:56:38.555731+0000","last_peered":"2024-09-13T14:56:38.555731+0000","last_clean":"2024-09-13T14:56:38.555731+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T14:56:38.555731+0000","last_undegraded":"2024-09-13T14:56:38.555731+0000","last_fullsized":"2024-09-13T14:56:38.555731+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":489,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557455+0000","last_change":"2024-09-13T14:48:38.828385+0000","last_active":"2024-09-13T14:56:38.557455+0000","last_peered":"2024-09-13T14:56:38.557455+0000","last_clean":"2024-09-13T14:56:38.557455+0000","last_became_active":"2024-09-13T14:48:38.828013+0000","last_became_peered":"2024-09-13T14:48:38.828013+0000","last_unstale":"2024-09-13T14:56:38.557455+0000","last_undegraded":"2024-09-13T14:56:38.557455+0000","last_fullsized":"2024-09-13T14:56:38.557455+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:48.452242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:48.452242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:55.816098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.5","version":"316'3717","reported_seq":3728,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:54.381729+0000","last_change":"2024-09-13T14:56:09.035196+0000","last_active":"2024-09-13T14:56:54.381729+0000","last_peered":"2024-09-13T14:56:54.381729+0000","last_clean":"2024-09-13T14:56:54.381729+0000","last_became_active":"2024-09-13T14:56:09.034906+0000","last_became_peered":"2024-09-13T14:56:09.034906+0000","last_unstale":"2024-09-13T14:56:54.381729+0000","last_undegraded":"2024-09-13T14:56:54.381729+0000","last_fullsized":"2024-09-13T14:56:54.381729+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:43.214619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243597312,"num_objects":3717,"num_object_clones":0,"num_object_copies":11151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":237888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":295,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.558507+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T14:56:38.558507+0000","last_peered":"2024-09-13T14:56:38.558507+0000","last_clean":"2024-09-13T14:56:38.558507+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T14:56:38.558507+0000","last_undegraded":"2024-09-13T14:56:38.558507+0000","last_fullsized":"2024-09-13T14:56:38.558507+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":299,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547231+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T14:56:38.547231+0000","last_peered":"2024-09-13T14:56:38.547231+0000","last_clean":"2024-09-13T14:56:38.547231+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T14:56:38.547231+0000","last_undegraded":"2024-09-13T14:56:38.547231+0000","last_fullsized":"2024-09-13T14:56:38.547231+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":81,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560222+0000","last_change":"2024-09-13T14:48:38.843649+0000","last_active":"2024-09-13T14:56:38.560222+0000","last_peered":"2024-09-13T14:56:38.560222+0000","last_clean":"2024-09-13T14:56:38.560222+0000","last_became_active":"2024-09-13T14:48:38.830945+0000","last_became_peered":"2024-09-13T14:48:38.830945+0000","last_unstale":"2024-09-13T14:56:38.560222+0000","last_undegraded":"2024-09-13T14:56:38.560222+0000","last_fullsized":"2024-09-13T14:56:38.560222+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:39.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:39.694334+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:23.062817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"316'3764","reported_seq":3777,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:54.376072+0000","last_change":"2024-09-13T14:56:09.035205+0000","last_active":"2024-09-13T14:56:54.376072+0000","last_peered":"2024-09-13T14:56:54.376072+0000","last_clean":"2024-09-13T14:56:54.376072+0000","last_became_active":"2024-09-13T14:56:09.034915+0000","last_became_peered":"2024-09-13T14:56:09.034915+0000","last_unstale":"2024-09-13T14:56:54.376072+0000","last_undegraded":"2024-09-13T14:56:54.376072+0000","last_fullsized":"2024-09-13T14:56:54.376072+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.408509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246677504,"num_objects":3764,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3764,"num_write_kb":240896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":278,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557097+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T14:56:38.557097+0000","last_peered":"2024-09-13T14:56:38.557097+0000","last_clean":"2024-09-13T14:56:38.557097+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T14:56:38.557097+0000","last_undegraded":"2024-09-13T14:56:38.557097+0000","last_fullsized":"2024-09-13T14:56:38.557097+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":470,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.086254+0000","last_change":"2024-09-13T14:37:08.749946+0000","last_active":"2024-09-13T14:56:51.086254+0000","last_peered":"2024-09-13T14:56:51.086254+0000","last_clean":"2024-09-13T14:56:51.086254+0000","last_became_active":"2024-09-13T14:37:08.749558+0000","last_became_peered":"2024-09-13T14:37:08.749558+0000","last_unstale":"2024-09-13T14:56:51.086254+0000","last_undegraded":"2024-09-13T14:56:51.086254+0000","last_fullsized":"2024-09-13T14:56:51.086254+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:01.750861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":99,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547644+0000","last_change":"2024-09-13T14:47:07.247958+0000","last_active":"2024-09-13T14:56:38.547644+0000","last_peered":"2024-09-13T14:56:38.547644+0000","last_clean":"2024-09-13T14:56:38.547644+0000","last_became_active":"2024-09-13T14:47:07.247640+0000","last_became_peered":"2024-09-13T14:47:07.247640+0000","last_unstale":"2024-09-13T14:56:38.547644+0000","last_undegraded":"2024-09-13T14:56:38.547644+0000","last_fullsized":"2024-09-13T14:56:38.547644+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:45:40.724228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:45:40.724228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:18.892070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"315'3466","reported_seq":3485,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:50.945979+0000","last_change":"2024-09-13T14:56:09.033530+0000","last_active":"2024-09-13T14:56:50.945979+0000","last_peered":"2024-09-13T14:56:50.945979+0000","last_clean":"2024-09-13T14:56:50.945979+0000","last_became_active":"2024-09-13T14:56:09.032938+0000","last_became_peered":"2024-09-13T14:56:09.032938+0000","last_unstale":"2024-09-13T14:56:50.945979+0000","last_undegraded":"2024-09-13T14:56:50.945979+0000","last_fullsized":"2024-09-13T14:56:50.945979+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3466,"log_dups_size":0,"ondisk_log_size":3466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:58:50.158201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227147776,"num_objects":3466,"num_object_clones":0,"num_object_copies":10398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":221824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":508,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.557661+0000","last_change":"2024-09-13T14:32:09.489033+0000","last_active":"2024-09-13T14:56:38.557661+0000","last_peered":"2024-09-13T14:56:38.557661+0000","last_clean":"2024-09-13T14:56:38.557661+0000","last_became_active":"2024-09-13T14:32:09.488670+0000","last_became_peered":"2024-09-13T14:32:09.488670+0000","last_unstale":"2024-09-13T14:56:38.557661+0000","last_undegraded":"2024-09-13T14:56:38.557661+0000","last_fullsized":"2024-09-13T14:56:38.557661+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:56.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":277,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560849+0000","last_change":"2024-09-13T14:37:08.782365+0000","last_active":"2024-09-13T14:56:38.560849+0000","last_peered":"2024-09-13T14:56:38.560849+0000","last_clean":"2024-09-13T14:56:38.560849+0000","last_became_active":"2024-09-13T14:37:08.781919+0000","last_became_peered":"2024-09-13T14:37:08.781919+0000","last_unstale":"2024-09-13T14:56:38.560849+0000","last_undegraded":"2024-09-13T14:56:38.560849+0000","last_fullsized":"2024-09-13T14:56:38.560849+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:17.931695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":99,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.558338+0000","last_change":"2024-09-13T14:47:07.244360+0000","last_active":"2024-09-13T14:56:38.558338+0000","last_peered":"2024-09-13T14:56:38.558338+0000","last_clean":"2024-09-13T14:56:38.558338+0000","last_became_active":"2024-09-13T14:47:07.243883+0000","last_became_peered":"2024-09-13T14:47:07.243883+0000","last_unstale":"2024-09-13T14:56:38.558338+0000","last_undegraded":"2024-09-13T14:56:38.558338+0000","last_fullsized":"2024-09-13T14:56:38.558338+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:40.349501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:40.349501+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:44.458779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"316'3799","reported_seq":3812,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:54.296462+0000","last_change":"2024-09-13T14:56:09.033553+0000","last_active":"2024-09-13T14:56:54.296462+0000","last_peered":"2024-09-13T14:56:54.296462+0000","last_clean":"2024-09-13T14:56:54.296462+0000","last_became_active":"2024-09-13T14:56:09.033287+0000","last_became_peered":"2024-09-13T14:56:09.033287+0000","last_unstale":"2024-09-13T14:56:54.296462+0000","last_undegraded":"2024-09-13T14:56:54.296462+0000","last_fullsized":"2024-09-13T14:56:54.296462+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:54.220513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248971264,"num_objects":3799,"num_object_clones":0,"num_object_copies":11397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3799,"num_write_kb":243136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":291,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547081+0000","last_change":"2024-09-13T14:49:05.822115+0000","last_active":"2024-09-13T14:56:38.547081+0000","last_peered":"2024-09-13T14:56:38.547081+0000","last_clean":"2024-09-13T14:56:38.547081+0000","last_became_active":"2024-09-13T14:49:05.821806+0000","last_became_peered":"2024-09-13T14:49:05.821806+0000","last_unstale":"2024-09-13T14:56:38.547081+0000","last_undegraded":"2024-09-13T14:56:38.547081+0000","last_fullsized":"2024-09-13T14:56:38.547081+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:13.648400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":292,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.549319+0000","last_change":"2024-09-13T14:48:38.838133+0000","last_active":"2024-09-13T14:56:38.549319+0000","last_peered":"2024-09-13T14:56:38.549319+0000","last_clean":"2024-09-13T14:56:38.549319+0000","last_became_active":"2024-09-13T14:48:38.831079+0000","last_became_peered":"2024-09-13T14:48:38.831079+0000","last_unstale":"2024-09-13T14:56:38.549319+0000","last_undegraded":"2024-09-13T14:56:38.549319+0000","last_fullsized":"2024-09-13T14:56:38.549319+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.966369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":252,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.548400+0000","last_change":"2024-09-13T14:47:48.848231+0000","last_active":"2024-09-13T14:56:38.548400+0000","last_peered":"2024-09-13T14:56:38.548400+0000","last_clean":"2024-09-13T14:56:38.548400+0000","last_became_active":"2024-09-13T14:47:07.237796+0000","last_became_peered":"2024-09-13T14:47:07.237796+0000","last_unstale":"2024-09-13T14:56:38.548400+0000","last_undegraded":"2024-09-13T14:56:38.548400+0000","last_fullsized":"2024-09-13T14:56:38.548400+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:14.636510+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"316'3821","reported_seq":3834,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:54.380866+0000","last_change":"2024-09-13T14:56:09.032953+0000","last_active":"2024-09-13T14:56:54.380866+0000","last_peered":"2024-09-13T14:56:54.380866+0000","last_clean":"2024-09-13T14:56:54.380866+0000","last_became_active":"2024-09-13T14:56:09.032634+0000","last_became_peered":"2024-09-13T14:56:09.032634+0000","last_unstale":"2024-09-13T14:56:54.380866+0000","last_undegraded":"2024-09-13T14:56:54.380866+0000","last_fullsized":"2024-09-13T14:56:54.380866+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:45.162779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250413056,"num_objects":3821,"num_object_clones":0,"num_object_copies":11463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":244544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":305,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555537+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T14:56:38.555537+0000","last_peered":"2024-09-13T14:56:38.555537+0000","last_clean":"2024-09-13T14:56:38.555537+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T14:56:38.555537+0000","last_undegraded":"2024-09-13T14:56:38.555537+0000","last_fullsized":"2024-09-13T14:56:38.555537+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":282,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547917+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T14:56:38.547917+0000","last_peered":"2024-09-13T14:56:38.547917+0000","last_clean":"2024-09-13T14:56:38.547917+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T14:56:38.547917+0000","last_undegraded":"2024-09-13T14:56:38.547917+0000","last_fullsized":"2024-09-13T14:56:38.547917+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":198,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.084733+0000","last_change":"2024-09-13T14:47:07.258211+0000","last_active":"2024-09-13T14:56:51.084733+0000","last_peered":"2024-09-13T14:56:51.084733+0000","last_clean":"2024-09-13T14:56:51.084733+0000","last_became_active":"2024-09-13T14:47:07.247621+0000","last_became_peered":"2024-09-13T14:47:07.247621+0000","last_unstale":"2024-09-13T14:56:51.084733+0000","last_undegraded":"2024-09-13T14:56:51.084733+0000","last_fullsized":"2024-09-13T14:56:51.084733+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:41.340178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:41.340178+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:21.977203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"316'3881","reported_seq":3894,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:54.278262+0000","last_change":"2024-09-13T14:56:09.031898+0000","last_active":"2024-09-13T14:56:54.278262+0000","last_peered":"2024-09-13T14:56:54.278262+0000","last_clean":"2024-09-13T14:56:54.278262+0000","last_became_active":"2024-09-13T14:56:09.031546+0000","last_became_peered":"2024-09-13T14:56:09.031546+0000","last_unstale":"2024-09-13T14:56:54.278262+0000","last_undegraded":"2024-09-13T14:56:54.278262+0000","last_fullsized":"2024-09-13T14:56:54.278262+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:56:08.015749+0000","last_clean_scrub_stamp":"2024-09-13T14:56:08.015749+0000","objects_scrubbed":0,"log_size":3881,"log_dups_size":0,"ondisk_log_size":3881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:18.018588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254345216,"num_objects":3881,"num_object_clones":0,"num_object_copies":11643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3881,"num_write_kb":248384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":87,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560271+0000","last_change":"2024-09-13T14:49:05.820232+0000","last_active":"2024-09-13T14:56:38.560271+0000","last_peered":"2024-09-13T14:56:38.560271+0000","last_clean":"2024-09-13T14:56:38.560271+0000","last_became_active":"2024-09-13T14:49:05.819924+0000","last_became_peered":"2024-09-13T14:49:05.819924+0000","last_unstale":"2024-09-13T14:56:38.560271+0000","last_undegraded":"2024-09-13T14:56:38.560271+0000","last_fullsized":"2024-09-13T14:56:38.560271+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.050685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":77,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547596+0000","last_change":"2024-09-13T14:49:05.863693+0000","last_active":"2024-09-13T14:56:38.547596+0000","last_peered":"2024-09-13T14:56:38.547596+0000","last_clean":"2024-09-13T14:56:38.547596+0000","last_became_active":"2024-09-13T14:49:05.849750+0000","last_became_peered":"2024-09-13T14:49:05.849750+0000","last_unstale":"2024-09-13T14:56:38.547596+0000","last_undegraded":"2024-09-13T14:56:38.547596+0000","last_fullsized":"2024-09-13T14:56:38.547596+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:46.820146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":83,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547720+0000","last_change":"2024-09-13T14:52:02.567722+0000","last_active":"2024-09-13T14:56:38.547720+0000","last_peered":"2024-09-13T14:56:38.547720+0000","last_clean":"2024-09-13T14:56:38.547720+0000","last_became_active":"2024-09-13T14:49:05.826318+0000","last_became_peered":"2024-09-13T14:49:05.826318+0000","last_unstale":"2024-09-13T14:56:38.547720+0000","last_undegraded":"2024-09-13T14:56:38.547720+0000","last_fullsized":"2024-09-13T14:56:38.547720+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:52:02.567611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:52:02.567611+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:03.419129+0000","scrub_duration":0.0030819879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":327,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547161+0000","last_change":"2024-09-13T14:48:38.836263+0000","last_active":"2024-09-13T14:56:38.547161+0000","last_peered":"2024-09-13T14:56:38.547161+0000","last_clean":"2024-09-13T14:56:38.547161+0000","last_became_active":"2024-09-13T14:48:38.828839+0000","last_became_peered":"2024-09-13T14:48:38.828839+0000","last_unstale":"2024-09-13T14:56:38.547161+0000","last_undegraded":"2024-09-13T14:56:38.547161+0000","last_fullsized":"2024-09-13T14:56:38.547161+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:47.097878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":479,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.086299+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T14:56:51.086299+0000","last_peered":"2024-09-13T14:56:51.086299+0000","last_clean":"2024-09-13T14:56:51.086299+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T14:56:51.086299+0000","last_undegraded":"2024-09-13T14:56:51.086299+0000","last_fullsized":"2024-09-13T14:56:51.086299+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":109,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560440+0000","last_change":"2024-09-13T14:47:40.127818+0000","last_active":"2024-09-13T14:56:38.560440+0000","last_peered":"2024-09-13T14:56:38.560440+0000","last_clean":"2024-09-13T14:56:38.560440+0000","last_became_active":"2024-09-13T14:47:40.127477+0000","last_became_peered":"2024-09-13T14:47:40.127477+0000","last_unstale":"2024-09-13T14:56:38.560440+0000","last_undegraded":"2024-09-13T14:56:38.560440+0000","last_fullsized":"2024-09-13T14:56:38.560440+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:12.474184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":273,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547759+0000","last_change":"2024-09-13T14:48:29.122122+0000","last_active":"2024-09-13T14:56:38.547759+0000","last_peered":"2024-09-13T14:56:38.547759+0000","last_clean":"2024-09-13T14:56:38.547759+0000","last_became_active":"2024-09-13T14:48:29.121785+0000","last_became_peered":"2024-09-13T14:48:29.121785+0000","last_unstale":"2024-09-13T14:56:38.547759+0000","last_undegraded":"2024-09-13T14:56:38.547759+0000","last_fullsized":"2024-09-13T14:56:38.547759+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:44.138017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":521,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.556655+0000","last_change":"2024-09-13T14:49:05.848458+0000","last_active":"2024-09-13T14:56:38.556655+0000","last_peered":"2024-09-13T14:56:38.556655+0000","last_clean":"2024-09-13T14:56:38.556655+0000","last_became_active":"2024-09-13T14:49:05.847216+0000","last_became_peered":"2024-09-13T14:49:05.847216+0000","last_unstale":"2024-09-13T14:56:38.556655+0000","last_undegraded":"2024-09-13T14:56:38.556655+0000","last_fullsized":"2024-09-13T14:56:38.556655+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:09.832561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":194,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-13T14:56:51.086106+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T14:56:51.086106+0000","last_peered":"2024-09-13T14:56:51.086106+0000","last_clean":"2024-09-13T14:56:51.086106+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T14:56:51.086106+0000","last_undegraded":"2024-09-13T14:56:51.086106+0000","last_fullsized":"2024-09-13T14:56:51.086106+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:53.508481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":527,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.558466+0000","last_change":"2024-09-13T14:49:05.859717+0000","last_active":"2024-09-13T14:56:38.558466+0000","last_peered":"2024-09-13T14:56:38.558466+0000","last_clean":"2024-09-13T14:56:38.558466+0000","last_became_active":"2024-09-13T14:49:05.859471+0000","last_became_peered":"2024-09-13T14:49:05.859471+0000","last_unstale":"2024-09-13T14:56:38.558466+0000","last_undegraded":"2024-09-13T14:56:38.558466+0000","last_fullsized":"2024-09-13T14:56:38.558466+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:36.127926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":77,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560746+0000","last_change":"2024-09-13T14:48:38.829142+0000","last_active":"2024-09-13T14:56:38.560746+0000","last_peered":"2024-09-13T14:56:38.560746+0000","last_clean":"2024-09-13T14:56:38.560746+0000","last_became_active":"2024-09-13T14:48:38.828786+0000","last_became_peered":"2024-09-13T14:48:38.828786+0000","last_unstale":"2024-09-13T14:56:38.560746+0000","last_undegraded":"2024-09-13T14:56:38.560746+0000","last_fullsized":"2024-09-13T14:56:38.560746+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:32.622407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":99,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.560322+0000","last_change":"2024-09-13T14:47:07.237793+0000","last_active":"2024-09-13T14:56:38.560322+0000","last_peered":"2024-09-13T14:56:38.560322+0000","last_clean":"2024-09-13T14:56:38.560322+0000","last_became_active":"2024-09-13T14:47:07.237386+0000","last_became_peered":"2024-09-13T14:47:07.237386+0000","last_unstale":"2024-09-13T14:56:38.560322+0000","last_undegraded":"2024-09-13T14:56:38.560322+0000","last_fullsized":"2024-09-13T14:56:38.560322+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.236433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":298,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.547807+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T14:56:38.547807+0000","last_peered":"2024-09-13T14:56:38.547807+0000","last_clean":"2024-09-13T14:56:38.547807+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T14:56:38.547807+0000","last_undegraded":"2024-09-13T14:56:38.547807+0000","last_fullsized":"2024-09-13T14:56:38.547807+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":187,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.555407+0000","last_change":"2024-09-13T14:47:40.128245+0000","last_active":"2024-09-13T14:56:38.555407+0000","last_peered":"2024-09-13T14:56:38.555407+0000","last_clean":"2024-09-13T14:56:38.555407+0000","last_became_active":"2024-09-13T14:47:40.126463+0000","last_became_peered":"2024-09-13T14:47:40.126463+0000","last_unstale":"2024-09-13T14:56:38.555407+0000","last_undegraded":"2024-09-13T14:56:38.555407+0000","last_fullsized":"2024-09-13T14:56:38.555407+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:29.558191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":485,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-13T14:56:38.554747+0000","last_change":"2024-09-13T14:48:38.828539+0000","last_active":"2024-09-13T14:56:38.554747+0000","last_peered":"2024-09-13T14:56:38.554747+0000","last_clean":"2024-09-13T14:56:38.554747+0000","last_became_active":"2024-09-13T14:48:38.827637+0000","last_became_peered":"2024-09-13T14:48:38.827637+0000","last_unstale":"2024-09-13T14:56:38.554747+0000","last_undegraded":"2024-09-13T14:56:38.554747+0000","last_fullsized":"2024-09-13T14:56:38.554747+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:23.130677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":3865182208,"num_objects":58978,"num_object_clones":0,"num_object_copies":176934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58978,"num_write_kb":3774592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11644567552,"data_stored":11644567552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58978,"ondisk_log_size":58978,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":286,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365246,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2591468,"kb_used_data":1677924,"kb_used_omap":1,"kb_used_meta":913534,"kb_avail":91153684,"statfs":{"total":95995035648,"available":93341372416,"internally_reserved":0,"allocated":1718194176,"data_stored":1716391741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":935459160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561476,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2672420,"kb_used_data":1770588,"kb_used_omap":1,"kb_used_meta":901822,"kb_avail":91072732,"statfs":{"total":95995035648,"available":93258477568,"internally_reserved":0,"allocated":1813082112,"data_stored":1811287692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923466072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790408,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1681948,"kb_used_data":488340,"kb_used_omap":1,"kb_used_meta":1193598,"kb_avail":92063204,"statfs":{"total":95995035648,"available":94272720896,"internally_reserved":0,"allocated":500060160,"data_stored":498270050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1222244696},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986636,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1620904,"kb_used_data":726496,"kb_used_omap":1,"kb_used_meta":894398,"kb_avail":92124248,"statfs":{"total":95995035648,"available":94335229952,"internally_reserved":0,"allocated":743931904,"data_stored":742133726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915863896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519791,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2599196,"kb_used_data":1827988,"kb_used_omap":4,"kb_used_meta":771195,"kb_avail":91145956,"statfs":{"total":95995035648,"available":93333458944,"internally_reserved":0,"allocated":1871859712,"data_stored":1870069612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":789704128},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304707,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3167904,"kb_used_data":2414552,"kb_used_omap":6,"kb_used_meta":753337,"kb_avail":90577248,"statfs":{"total":95995035648,"available":92751101952,"internally_reserved":0,"allocated":2472501248,"data_stored":2470707566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":771417543},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828010,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2231324,"kb_used_data":1163732,"kb_used_omap":5,"kb_used_meta":1067578,"kb_avail":91513828,"statfs":{"total":95995035648,"available":93710159872,"internally_reserved":0,"allocated":1191661568,"data_stored":1189870895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1093200108},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285874,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2232840,"kb_used_data":1291328,"kb_used_omap":5,"kb_used_meta":941498,"kb_avail":91512312,"statfs":{"total":95995035648,"available":93708607488,"internally_reserved":0,"allocated":1322319872,"data_stored":1320540444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":964094011},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1342242816,"data_stored":1342242816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1195573248,"data_stored":1195573248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2510815232,"data_stored":2510815232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1877868544,"data_stored":1877868544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":743571456,"data_stored":743571456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":505937920,"data_stored":505937920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1744830464,"data_stored":1744830464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1723727872,"data_stored":1723727872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:56:59.462 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:56:59.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-09-13T14:56:59.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:59.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:59.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:59.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:59.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:56:59.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:59.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:59.861 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:56:59.861 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:59.876 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:56:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:59.902 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:59.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:56:59.908 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:59.955 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:56:59.956 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:59.960 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:59.978 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:00.003 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:57:00.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:00.004 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:00.031 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:57:00.032 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:00.062 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:57:00.062 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:00.094 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:57:00.095 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:00.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:00.126 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:57:00.126 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:00.134 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:00.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:00.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:00.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:00.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:00.426 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 67940 67924 111.685 67.625 0.009996 0.00894729 2024-09-13T14:57:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:00 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1594866366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:00.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:57:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:57:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:00.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:00 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1594866366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:00.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:00 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1594866366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:00.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:00.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:00.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:00.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:00.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:00.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:00.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:00.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:00.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:01.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:01.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:01.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:01.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:01.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:01.382 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:01.412 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:01.426 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 69185 69169 110.817 77.8125 0.0154857 0.00901857 2024-09-13T14:57:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:01 smithi019 ceph-mon[91296]: pgmap v1127: 109 pgs: 109 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-13T14:57:01.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:01 smithi003 ceph-mon[107278]: pgmap v1127: 109 pgs: 109 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-13T14:57:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:01 smithi003 ceph-mon[104007]: pgmap v1127: 109 pgs: 109 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-13T14:57:01.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.548 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:01.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:01.649 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:01.677 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:01.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:01.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:01.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:02.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:02.070 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:02.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:02.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:02.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:02.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:02.426 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:57:02.426263+0000 min lat: 0.00131314 max lat: 0.223735 avg lat: 0.00910833 2024-09-13T14:57:02.427 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:57:02.427 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 70259 70243 109.724 67.125 0.00916462 0.00910833 2024-09-13T14:57:02.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:02.486 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:02.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.624 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:02.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:02.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:02.753 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:02.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:02.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:03.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:03.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:03.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:03.175 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:03.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:03.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:03.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:03.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:03.427 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 71568 71552 109.044 81.8125 0.00969344 0.00916516 2024-09-13T14:57:03.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:03 smithi019 ceph-mon[91296]: pgmap v1128: 109 pgs: 109 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T14:57:03.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:03 smithi003 ceph-mon[107278]: pgmap v1128: 109 pgs: 109 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T14:57:03.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:03 smithi003 ceph-mon[104007]: pgmap v1128: 109 pgs: 109 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T14:57:03.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:03.568 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:03.594 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:03.696 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:03.730 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:03.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:03.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:03.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:04.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:04.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:04.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:04.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:04.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:04.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:04.426 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 72651 72635 108.059 67.6875 0.00850067 0.00924787 2024-09-13T14:57:04.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:57:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:57:04] "GET /metrics HTTP/1.1" 200 36527 "" "Prometheus/2.51.0" 2024-09-13T14:57:04.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:04.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:04.572 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:04.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:04.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:04.842 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:04.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:04.972 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:05.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:05.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:05.212 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:05.249 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:05.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:05.373 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:05.427 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 73801 73785 107.217 71.875 0.0941875 0.00931704 2024-09-13T14:57:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:05 smithi019 ceph-mon[91296]: pgmap v1129: 109 pgs: 109 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T14:57:05.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:05 smithi003 ceph-mon[107278]: pgmap v1129: 109 pgs: 109 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T14:57:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:05 smithi003 ceph-mon[104007]: pgmap v1129: 109 pgs: 109 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T14:57:05.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:05.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:05.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:05.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:05.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:05.770 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:05.798 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:05.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:05.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:06.044 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:06.075 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:06.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:06.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:06.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:06.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:06.427 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 75006 74990 106.492 75.3125 0.0109211 0.00938467 2024-09-13T14:57:06.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:06.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:06.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:06.662 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:06.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:06.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:06.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:06.921 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:06.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:07.023 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:07.045 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:07.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.174 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:07.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:07.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:07.304 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:07.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:07.428 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 75796 75780 105.221 49.375 0.00499887 0.0094971 2024-09-13T14:57:07.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:07 smithi019 ceph-mon[91296]: pgmap v1130: 109 pgs: 109 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:57:07.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:07 smithi003 ceph-mon[107278]: pgmap v1130: 109 pgs: 109 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:57:07.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:07 smithi003 ceph-mon[104007]: pgmap v1130: 109 pgs: 109 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:57:07.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:07.553 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:07.577 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:07.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:07.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:07.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:07.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:07.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:08.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:08.120 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:08.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:08.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:08.398 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:08.428 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 76762 76746 104.246 60.375 0.0198144 0.00958695 2024-09-13T14:57:08.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:08.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:08.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:08.664 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:08.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:08.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:08.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:08.929 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:09.031 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:09.060 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:09.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:09.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:09.305 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:09.428 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 77605 77589 103.149 52.6875 0.0150806 0.00968825 2024-09-13T14:57:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:09 smithi019 ceph-mon[91296]: pgmap v1131: 109 pgs: 109 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:57:09.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:09 smithi003 ceph-mon[107278]: pgmap v1131: 109 pgs: 109 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:57:09.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:09 smithi003 ceph-mon[104007]: pgmap v1131: 109 pgs: 109 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:57:09.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:09.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:09.580 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:09.611 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:09.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:09.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:09.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:09.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:09.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:10.018 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:10.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:10.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:10.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:10.334 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:10.428 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 78762 78746 102.506 72.3125 0.00941281 0.00974968 2024-09-13T14:57:10.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:10.472 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:10.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:10.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:10.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:57:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:57:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:10.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:10.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:10.862 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:10.887 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:10.989 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:11.017 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:11.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:11.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:11.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:11.295 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:11.318 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:57:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94200 11.10 11:0fc22623:::benchmark_data_smithi003_229782_object94199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.711793+0000", 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.574768613, 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.439892046, 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94200 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711793+0000", 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711793+0000", 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711795+0000", 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.046e-06 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711838+0000", 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2548e-05 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711861+0000", 2024-09-13T14:57:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3754e-05 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711871+0000", 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6840000000000008e-06 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712997+0000", 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011257089999999999 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713006+0000", 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4299999999999995e-06 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.125730+0000", 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4127240780000001 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.125938+0000", 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000208128 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.126154+0000", 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021603200000000001 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134250+0000", 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080953889999999997 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134267+0000", 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6864e-05 2024-09-13T14:57:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134403+0000", 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000136105 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.149934+0000", 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015531420000000001 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.149998+0000", 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4157000000000003e-05 2024-09-13T14:57:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.151685+0000", 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001686702 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94227 11.10 11:0a18fc76:::benchmark_data_smithi003_229782_object94226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.711885+0000", 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.574677001, 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4469483990000001, 2024-09-13T14:57:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94227 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711885+0000", 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711885+0000", 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711885+0000", 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.51e-07 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711932+0000", 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7104000000000002e-05 2024-09-13T14:57:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711941+0000", 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1339999999999993e-06 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711947+0000", 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0619999999999997e-06 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713030+0000", 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001082548 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713036+0000", 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3099999999999997e-06 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.129572+0000", 2024-09-13T14:57:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.416535189 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.129723+0000", 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015137399999999999 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.129885+0000", 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000161743 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134480+0000", 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045955980000000002 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134497+0000", 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6827e-05 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.151980+0000", 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017482971 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.156634+0000", 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004653857 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.156752+0000", 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117652 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.158833+0000", 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020814789999999998 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94238 11.10 11:0ada4fde:::benchmark_data_smithi003_229782_object94237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.711957+0000", 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.574604239, 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4488024209999999, 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94238 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711957+0000", 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711957+0000", 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711958+0000", 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3399999999999999e-07 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712132+0000", 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017390999999999999 2024-09-13T14:57:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712143+0000", 2024-09-13T14:57:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1022000000000001e-05 2024-09-13T14:57:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712149+0000", 2024-09-13T14:57:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5719999999999999e-06 2024-09-13T14:57:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713056+0000", 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00090699699999999999 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713062+0000", 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9259999999999997e-06 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.132064+0000", 2024-09-13T14:57:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4190016240000001 2024-09-13T14:57:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.132265+0000", 2024-09-13T14:57:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020077800000000001 2024-09-13T14:57:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.132471+0000", 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000206741 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134508+0000", 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020368769999999999 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.138832+0000", 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043240589999999999 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152136+0000", 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013303756999999999 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.159014+0000", 2024-09-13T14:57:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068779590000000003 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.159104+0000", 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9450000000000006e-05 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.160760+0000", 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016562149999999999 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94322 11.10 11:0b36970b:::benchmark_data_smithi003_229782_object94321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712159+0000", 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57440254799999, 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4504645759999999, 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94322 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712159+0000", 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712159+0000", 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712160+0000", 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8100000000000003e-07 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712179+0000", 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9034000000000001e-05 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712185+0000", 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6810000000000001e-06 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712191+0000", 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2689999999999999e-06 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713082+0000", 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00089189900000000003 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713089+0000", 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0680000000000002e-06 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134543+0000", 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.421454663 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134719+0000", 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017592199999999999 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134915+0000", 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019601699999999999 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.138916+0000", 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040005939999999997 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.151928+0000", 2024-09-13T14:57:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013012767999999999 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152184+0000", 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025554000000000001 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.160833+0000", 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0086491569999999993 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.160922+0000", 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8930000000000004e-05 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.162624+0000", 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001701553 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94324 11.10 11:0e3d6c56:::benchmark_data_smithi003_229782_object94323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712197+0000", 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.574364809, 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4524904750000001, 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94324 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712197+0000", 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712197+0000", 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712197+0000", 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6600000000000002e-07 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712243+0000", 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5791000000000003e-05 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712250+0000", 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3429999999999998e-06 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712256+0000", 2024-09-13T14:57:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8860000000000001e-06 2024-09-13T14:57:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713109+0000", 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00085227100000000004 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713115+0000", 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1469999999999998e-06 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.136848+0000", 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.423733149 2024-09-13T14:57:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.136985+0000", 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013682599999999999 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.137129+0000", 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000144647 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.141149+0000", 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040198680000000002 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152001+0000", 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010852194000000001 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152265+0000", 2024-09-13T14:57:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026327499999999999 2024-09-13T14:57:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.162693+0000", 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010428174 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.162890+0000", 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019689600000000001 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.164687+0000", 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017975420000000001 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94374 11.10 11:09485132:::benchmark_data_smithi003_229782_object94373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712266+0000", 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.574295947, 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4743467940000001, 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94374 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712266+0000", 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712266+0000", 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712266+0000", 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9100000000000004e-07 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712308+0000", 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1468999999999999e-05 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712317+0000", 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8519999999999993e-06 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712325+0000", 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4020000000000002e-06 2024-09-13T14:57:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713135+0000", 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00081022799999999997 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713141+0000", 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0499999999999997e-06 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.138952+0000", 2024-09-13T14:57:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4258111630000001 2024-09-13T14:57:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.139106+0000", 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015347599999999999 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.139274+0000", 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016841 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.143560+0000", 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004285634 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152009+0000", 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084487620000000003 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.164758+0000", 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012749442 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.181149+0000", 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016390873 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.183513+0000", 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023641220000000002 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.186612+0000", 2024-09-13T14:57:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00309932 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94402 11.10 11:0ea6a2db:::benchmark_data_smithi003_229782_object94401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712334+0000", 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57422766900001, 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.499560526, 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94402 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712334+0000", 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712334+0000", 2024-09-13T14:57:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712335+0000", 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6499999999999999e-07 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712354+0000", 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9945999999999998e-05 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712362+0000", 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4270000000000002e-06 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712368+0000", 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1809999999999998e-06 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713161+0000", 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00079314499999999998 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713167+0000", 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9920000000000004e-06 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.141180+0000", 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.428012394 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.141341+0000", 2024-09-13T14:57:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016166100000000001 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.141512+0000", 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017027700000000001 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.143609+0000", 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020970279999999999 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152018+0000", 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084095290000000007 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174707+0000", 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022689227999999999 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.210055+0000", 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035347283 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.210328+0000", 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027289600000000001 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.211894+0000", 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015669740000000001 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94404 11.10 11:0c74a415:::benchmark_data_smithi003_229782_object94403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712377+0000", 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.574184468, 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.50167396, 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94404 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712377+0000", 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712377+0000", 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712378+0000", 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4099999999999999e-07 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712399+0000", 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1491000000000001e-05 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712406+0000", 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1760000000000004e-06 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712412+0000", 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9050000000000002e-06 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713188+0000", 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00077545700000000001 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713194+0000", 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0519999999999998e-06 2024-09-13T14:57:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.143644+0000", 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4304505789999999 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.143816+0000", 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017148599999999999 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.144002+0000", 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018566200000000001 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.149883+0000", 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058818059999999998 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152024+0000", 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021409860000000001 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174825+0000", 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022800606000000001 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.211951+0000", 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.037126064 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.212009+0000", 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8159999999999999e-05 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.214051+0000", 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020419890000000001 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94466 11.10 11:0be2a9ea:::benchmark_data_smithi003_229782_object94465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712420+0000", 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57414121799999, 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.503092879, 2024-09-13T14:57:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94466 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712420+0000", 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712420+0000", 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712421+0000", 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1799999999999995e-07 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712445+0000", 2024-09-13T14:57:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4474e-05 2024-09-13T14:57:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712570+0000", 2024-09-13T14:57:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012472000000000001 2024-09-13T14:57:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712579+0000", 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3449999999999995e-06 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713214+0000", 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00063451600000000001 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713220+0000", 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1009999999999997e-06 2024-09-13T14:57:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.145696+0000", 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4324761859999999 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.145830+0000", 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013381100000000001 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.145993+0000", 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016244299999999999 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.151767+0000", 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057746070000000002 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152032+0000", 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000264966 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174928+0000", 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022896382 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.214109+0000", 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039180461999999999 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.214171+0000", 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1915999999999995e-05 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.215513+0000", 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001342432 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94472 11.10 11:0db08c66:::benchmark_data_smithi003_229782_object94471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712593+0000", 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57396897199999, 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5046174130000001, 2024-09-13T14:57:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94472 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712593+0000", 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712593+0000", 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712593+0000", 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8299999999999997e-07 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712616+0000", 2024-09-13T14:57:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2486000000000001e-05 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712624+0000", 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4940000000000003e-06 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712630+0000", 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8560000000000003e-06 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713243+0000", 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00061295200000000005 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713249+0000", 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.105e-06 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.148138+0000", 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.434889173 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.148257+0000", 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011830800000000001 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.148447+0000", 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019088699999999999 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152040+0000", 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035921659999999999 2024-09-13T14:57:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152087+0000", 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7513999999999999e-05 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.186679+0000", 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.034591898000000003 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.215562+0000", 2024-09-13T14:57:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028882642 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.215728+0000", 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000166253 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.217210+0000", 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001482096 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94521 11.10 11:0f0380e7:::benchmark_data_smithi003_229782_object94520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713089+0000", 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57347235700001, 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.510600586, 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94521 2024-09-13T14:57:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713089+0000", 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713089+0000", 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713090+0000", 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2399999999999998e-07 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713113+0000", 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2945000000000001e-05 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713124+0000", 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1613e-05 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713132+0000", 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4819999999999997e-06 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713269+0000", 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000137546 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713276+0000", 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1489999999999999e-06 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152328+0000", 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.439052405 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152513+0000", 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018483599999999999 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152708+0000", 2024-09-13T14:57:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019519599999999999 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.158865+0000", 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061571389999999998 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.158937+0000", 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1451000000000004e-05 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.186776+0000", 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027839702000000001 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.217262+0000", 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.030485763999999999 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.217324+0000", 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2100000000000005e-05 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.223690+0000", 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063656340000000002 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94522 11.10 11:0fe3a0c0:::benchmark_data_smithi003_229782_object94521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713142+0000", 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57341933500001, 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5127048080000001, 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94522 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713142+0000", 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713142+0000", 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713143+0000", 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8599999999999998e-07 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713168+0000", 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5605999999999998e-05 2024-09-13T14:57:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713177+0000", 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2979999999999992e-06 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713183+0000", 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2570000000000004e-06 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713298+0000", 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114678 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713304+0000", 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1190000000000002e-06 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.154748+0000", 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.441443987 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.154941+0000", 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019285999999999999 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.155133+0000", 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019257899999999999 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.158961+0000", 2024-09-13T14:57:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038275319999999998 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.164814+0000", 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058529740000000004 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207491+0000", 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.042677155000000001 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.223750+0000", 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016259447 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.223900+0000", 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014954599999999999 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.225847+0000", 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019472840000000001 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94575 11.10 11:0c0f3ecd:::benchmark_data_smithi003_229782_object94574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713191+0000", 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57337019799999, 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.51553099, 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94575 2024-09-13T14:57:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713191+0000", 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713191+0000", 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713192+0000", 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7399999999999998e-07 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713216+0000", 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3824000000000001e-05 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713224+0000", 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6000000000000007e-06 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713230+0000", 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9139999999999996e-06 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713324+0000", 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3720000000000004e-05 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713330+0000", 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9460000000000003e-06 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.164861+0000", 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4515310960000001 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.165011+0000", 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015026300000000001 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.165194+0000", 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018294599999999999 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174721+0000", 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095271079999999994 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174785+0000", 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4045000000000001e-05 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207535+0000", 2024-09-13T14:57:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032749160999999999 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.225911+0000", 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018376445000000002 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.225973+0000", 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2070999999999996e-05 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.228722+0000", 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027493769999999999 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94605 11.10 11:0d24cfc4:::benchmark_data_smithi003_229782_object94604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713238+0000", 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57332380400001, 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.517634414, 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94605 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713238+0000", 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713238+0000", 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713238+0000", 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8299999999999997e-07 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713260+0000", 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1301000000000002e-05 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713267+0000", 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.272e-06 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713272+0000", 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6130000000000003e-06 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713350+0000", 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7585999999999999e-05 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713358+0000", 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5100000000000001e-06 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.167319+0000", 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4539618830000001 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.167492+0000", 2024-09-13T14:57:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017285999999999999 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.167697+0000", 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000204448 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174728+0000", 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070309939999999996 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174877+0000", 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000149631 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207572+0000", 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032694873999999999 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.228787+0000", 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021214551000000002 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.228856+0000", 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9036999999999996e-05 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.230872+0000", 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020163709999999999 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94638 11.10 11:0b0578dd:::benchmark_data_smithi003_229782_object94637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713292+0000", 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57326920899999, 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5195766050000001, 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94638 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713292+0000", 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713292+0000", 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713293+0000", 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6300000000000005e-07 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713347+0000", 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4518999999999998e-05 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713357+0000", 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4709999999999996e-06 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713363+0000", 2024-09-13T14:57:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1149999999999999e-06 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713525+0000", 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000162342 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713534+0000", 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6859999999999995e-06 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.169822+0000", 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4562879689999999 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.169982+0000", 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016018799999999999 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.170142+0000", 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016017900000000001 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174733+0000", 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045909169999999999 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.179836+0000", 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051030240000000003 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207632+0000", 2024-09-13T14:57:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027795306999999998 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.230931+0000", 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023299626 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.230992+0000", 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0327e-05 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.232869+0000", 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001877372 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94657 11.10 11:0ea01dfd:::benchmark_data_smithi003_229782_object94656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713432+0000", 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.57312986900001, 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5214419449999999, 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94657 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713432+0000", 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713432+0000", 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713432+0000", 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1900000000000002e-07 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713487+0000", 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4701999999999999e-05 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713498+0000", 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1035e-05 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713505+0000", 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5729999999999996e-06 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713559+0000", 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4731999999999997e-05 2024-09-13T14:57:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713566+0000", 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.956e-06 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.172320+0000", 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.458753387 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.172629+0000", 2024-09-13T14:57:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000309019 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.172812+0000", 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000183606 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.177428+0000", 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046158479999999997 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.179899+0000", 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002470875 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207665+0000", 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027765517 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.232925+0000", 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025259884 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.232985+0000", 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0173999999999997e-05 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.234874+0000", 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018890179999999999 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36438.0:211624 11.c e298/266 11:31867109:::benchmark_data_smithi003_229782_object89244:head v 298'13060, mlcod=298'13055)", 2024-09-13T14:57:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.893384+0000", 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.393177643, 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44854644199999999, 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893384+0000", 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893384+0000", 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893385+0000", 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0249999999999999e-06 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893391+0000", 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0700000000000003e-06 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893403+0000", 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1562999999999999e-05 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893410+0000", 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6629999999999993e-06 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893492+0000", 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1903000000000005e-05 2024-09-13T14:57:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893511+0000", 2024-09-13T14:57:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8811999999999999e-05 2024-09-13T14:57:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:11 smithi003 ceph-mon[104007]: pgmap v1132: 109 pgs: 109 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T14:57:11.453 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:11 smithi003 ceph-mon[107278]: pgmap v1132: 109 pgs: 109 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T14:57:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.341882+0000", 2024-09-13T14:57:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44837057800000002 2024-09-13T14:57:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.341930+0000", 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8828000000000001e-05 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36438.0:211645 11.f e298/266 11:fd3da10e:::benchmark_data_smithi003_229782_object89265:head v 298'13455, mlcod=298'13453)", 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.896596+0000", 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.389965651, 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.456244539, 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.896596+0000", 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.896596+0000", 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.896597+0000", 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.153e-06 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.896606+0000", 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3689999999999996e-06 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.896619+0000", 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2687999999999999e-05 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333384+0000", 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.43676469400000001 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333552+0000", 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000167722 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333571+0000", 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9490000000000001e-05 2024-09-13T14:57:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.352791+0000", 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019219409999999999 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.352841+0000", 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0012999999999997e-05 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36438.0:211647 11.e e298/266 11:743fb1be:::benchmark_data_smithi003_229782_object89267:head v 298'13264, mlcod=298'13262)", 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.898458+0000", 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.388103297, 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.451011626, 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898458+0000", 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898458+0000", 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898459+0000", 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9999999999999995e-07 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898466+0000", 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.029e-06 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333387+0000", 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.434920266 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333397+0000", 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0815000000000001e-05 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333490+0000", 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2147999999999999e-05 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333511+0000", 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1135e-05 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.346115+0000", 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012603961 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.349470+0000", 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033552719999999999 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36438.0:211637 11.b e298/266 11:d48e3873:::benchmark_data_smithi003_229782_object89257:head v 298'13033, mlcod=298'13028)", 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.898692+0000", 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.387869826, 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44215423199999998, 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898692+0000", 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898692+0000", 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898693+0000", 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.23e-07 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898700+0000", 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0589999999999997e-06 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333687+0000", 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.43498704500000002 2024-09-13T14:57:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333694+0000", 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.861e-06 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333859+0000", 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016579499999999999 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.333878+0000", 2024-09-13T14:57:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8512000000000002e-05 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.340662+0000", 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067841919999999997 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.340846+0000", 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000183945 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:11.468 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:57:11.468 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 79815 79799 101.758 65.8125 0.0397518 0.0098215 2024-09-13T14:57:11.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:11.469 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:11 smithi019 ceph-mon[91296]: pgmap v1132: 109 pgs: 109 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T14:57:11.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:11.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:11.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:11.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:11.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:11.890 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:11.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:12.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.050 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:12.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:12.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:57:12.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:12.161 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:57:12.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:12.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:12.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.306 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:12.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:12.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:12.428 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 80891 80875 101.067 67.25 0.0136567 0.00988906 2024-09-13T14:57:12.431 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:12 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1110318005' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:12 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:12 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1110318005' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:12 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:12 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1110318005' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:12 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:12.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:12.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:12.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:12.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:12.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:12.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:12.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:12.878 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:12.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:13.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:13.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:13.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:13.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:13.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:13.285 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:13.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:13.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:13.428 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 82021 82005 100.47 70.625 0.0130306 0.00994774 2024-09-13T14:57:13.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:13.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:13 smithi019 ceph-mon[91296]: pgmap v1133: 109 pgs: 109 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:57:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:13 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:13 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:57:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:13 smithi019 ceph-mon[91296]: osdmap e317: 8 total, 8 up, 8 in 2024-09-13T14:57:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[104007]: pgmap v1133: 109 pgs: 109 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:57:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:57:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[104007]: osdmap e317: 8 total, 8 up, 8 in 2024-09-13T14:57:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[107278]: pgmap v1133: 109 pgs: 109 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:57:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:57:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[107278]: osdmap e317: 8 total, 8 up, 8 in 2024-09-13T14:57:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:13.531 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:13.562 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:13.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:13.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:13.807 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:13.839 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:13.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:13.969 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:14.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:14.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:14.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:14.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:14.353 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:14.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:14.429 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 83114 83098 99.852 68.3125 0.00657644 0.0100095 2024-09-13T14:57:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:57:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:57:14] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-13T14:57:14.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:14.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:14.647 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:14.675 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:14.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:14.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:14.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:14.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:15.065 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.091 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:15.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:15.192 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:15.222 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:15.295 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:15 smithi003 ceph-mon[107278]: pgmap v1135: 109 pgs: 109 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:57:15.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:15 smithi003 ceph-mon[104007]: pgmap v1135: 109 pgs: 109 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:57:15.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:15.346 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:15.429 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 84160 84144 99.2015 65.375 0.0151776 0.0100753 2024-09-13T14:57:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:15 smithi019 ceph-mon[91296]: pgmap v1135: 109 pgs: 109 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:57:15.448 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:15.480 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:15.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:15.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:15.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:57:15.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:15.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:15.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:15.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:16.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:16.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:16.177 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:16.208 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:16.208 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:16.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:16.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:16.430 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 85161 85145 98.5207 62.5625 0.0204414 0.0101447 2024-09-13T14:57:16.463 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:16.492 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:16.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:57:16.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:16.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:16.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:16.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:16.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:16.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:17.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.058 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:17.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:17.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:17.185 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pg_num to 31 2024-09-13T14:57:17.188 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:17.291 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:17.318 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:17.420 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:17.430 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 86130 86114 97.8305 60.5625 0.0167631 0.0102166 2024-09-13T14:57:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:17 smithi003 ceph-mon[104007]: pgmap v1136: 109 pgs: 109 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:57:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:17 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/856381599' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:57:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:17 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:57:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:17 smithi003 ceph-mon[107278]: pgmap v1136: 109 pgs: 109 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:57:17.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:17 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/856381599' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:57:17.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:17 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:57:17.434 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:17 smithi019 ceph-mon[91296]: pgmap v1136: 109 pgs: 109 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:57:17.435 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:17 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/856381599' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:57:17.435 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:17 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:57:17.452 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:17.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:17.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:17.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:17.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:17.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:17.868 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:17.970 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:17.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:18.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:18.128 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:18.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:18.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:18.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:18.305 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:18 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T14:57:18.305 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:18 smithi003 ceph-mon[104007]: osdmap e318: 8 total, 8 up, 8 in 2024-09-13T14:57:18.305 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:18.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:18 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T14:57:18.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:18 smithi003 ceph-mon[107278]: osdmap e318: 8 total, 8 up, 8 in 2024-09-13T14:57:18.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:18.370 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:18.399 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:18.430 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 87231 87215 97.3123 68.8125 0.0115078 0.0102711 2024-09-13T14:57:18.501 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:18.532 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:18.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:18 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T14:57:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:18 smithi019 ceph-mon[91296]: osdmap e318: 8 total, 8 up, 8 in 2024-09-13T14:57:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:18.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:18.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:18.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:18.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:18.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:19.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:19.078 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:19.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:19.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:19.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:19.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:19.363 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:19 smithi003 ceph-mon[107278]: pgmap v1138: 109 pgs: 109 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 60 MiB/s wr, 954 op/s 2024-09-13T14:57:19.363 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:19 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T14:57:19.364 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:19 smithi003 ceph-mon[107278]: osdmap e319: 8 total, 8 up, 8 in 2024-09-13T14:57:19.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:19 smithi003 ceph-mon[104007]: pgmap v1138: 109 pgs: 109 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 60 MiB/s wr, 954 op/s 2024-09-13T14:57:19.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:19 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T14:57:19.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:19 smithi003 ceph-mon[104007]: osdmap e319: 8 total, 8 up, 8 in 2024-09-13T14:57:19.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:19.430 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 88287 88271 96.763 66 0.0061415 0.0103289 2024-09-13T14:57:19.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:19.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:19.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:19.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:19 smithi019 ceph-mon[91296]: pgmap v1138: 109 pgs: 109 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 60 MiB/s wr, 954 op/s 2024-09-13T14:57:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:19 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T14:57:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:19 smithi019 ceph-mon[91296]: osdmap e319: 8 total, 8 up, 8 in 2024-09-13T14:57:19.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:19.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:19.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:19.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:20.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:20.104 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:20.180 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:20.206 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:20.235 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:20.246 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:57:20.246 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:20.286 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:57:20.286 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:20.316 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:57:20.316 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:20.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:20.353 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:57:20.354 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:20.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:20.387 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:57:20.387 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:20.425 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:57:20.425 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:20.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:20 smithi003 ceph-mon[104007]: osdmap e320: 8 total, 8 up, 8 in 2024-09-13T14:57:20.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:20 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T14:57:20.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:20 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4263998492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:20.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:20 smithi003 ceph-mon[107278]: osdmap e320: 8 total, 8 up, 8 in 2024-09-13T14:57:20.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:20 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T14:57:20.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:20 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4263998492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:20.430 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 89274 89258 96.1582 61.6875 0.0339001 0.0103937 2024-09-13T14:57:20.458 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:57:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:20.482 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:20.497 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:57:20.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:20.514 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:20.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:20.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:57:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:57:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:20 smithi019 ceph-mon[91296]: osdmap e320: 8 total, 8 up, 8 in 2024-09-13T14:57:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:20 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T14:57:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:20 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4263998492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:20.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:20.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:20.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:20.943 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:21.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:21.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:21.190 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:21.220 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:21.322 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:21.323 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:21 smithi003 ceph-mon[104007]: pgmap v1141: 119 pgs: 2 unknown, 117 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:57:21.323 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:21 smithi003 ceph-mon[104007]: 1.c scrub starts 2024-09-13T14:57:21.323 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:21 smithi003 ceph-mon[104007]: 1.1b scrub starts 2024-09-13T14:57:21.324 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:21 smithi003 ceph-mon[107278]: pgmap v1141: 119 pgs: 2 unknown, 117 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:57:21.324 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:21 smithi003 ceph-mon[107278]: 1.c scrub starts 2024-09-13T14:57:21.324 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:21 smithi003 ceph-mon[107278]: 1.1b scrub starts 2024-09-13T14:57:21.353 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:21.430 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 90336 90320 95.6534 66.375 0.0170068 0.0104496 2024-09-13T14:57:21.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:21.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:21.601 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:21.630 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:21 smithi019 ceph-mon[91296]: pgmap v1141: 119 pgs: 2 unknown, 117 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:57:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:21 smithi019 ceph-mon[91296]: 1.c scrub starts 2024-09-13T14:57:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:21 smithi019 ceph-mon[91296]: 1.1b scrub starts 2024-09-13T14:57:21.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:21.759 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:21.782 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:21.783 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:21.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:21.895 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:21.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:22.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:22.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:22.174 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:22.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:22.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:22.399 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:22 smithi003 ceph-mon[104007]: 1.1c scrub starts 2024-09-13T14:57:22.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:22 smithi003 ceph-mon[107278]: 1.1c scrub starts 2024-09-13T14:57:22.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:22.433 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:57:22.432512+0000 min lat: 0.00131314 max lat: 0.257553 avg lat: 0.0105048 2024-09-13T14:57:22.433 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:57:22.433 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 91376 91360 95.1393 65 0.0207929 0.0105048 2024-09-13T14:57:22.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:22.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:22.584 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:22 smithi019 ceph-mon[91296]: 1.1c scrub starts 2024-09-13T14:57:22.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:22.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:22.818 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:22.850 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:22.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:22.981 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:23.083 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:23.114 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:23.216 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:23.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.247 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:23.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:23.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:23.356 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:57:23.356 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:57:23.356 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:57:23.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd out 5 2024-09-13T14:57:23.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:23.433 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 92364 92348 94.5919 61.75 0.00327386 0.0105512 2024-09-13T14:57:23.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:23.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:23.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.627 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:23.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:23.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:23 smithi019 ceph-mon[91296]: pgmap v1142: 119 pgs: 1 active+clean+scrubbing, 1 unknown, 117 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:57:23.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:23 smithi003 ceph-mon[107278]: pgmap v1142: 119 pgs: 1 active+clean+scrubbing, 1 unknown, 117 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:57:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:23 smithi003 ceph-mon[104007]: pgmap v1142: 119 pgs: 1 active+clean+scrubbing, 1 unknown, 117 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:57:23.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:23.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:23.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:23.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:23.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:23.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:24.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:24.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:24.131 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:24.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:24.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:24.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:24.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:24.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:24.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:57:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:57:24] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-13T14:57:24.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:24.434 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 93103 93087 93.8113 46.1875 0.0206285 0.0106549 2024-09-13T14:57:24.443 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:24.545 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:24.577 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:24.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:24.710 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:24.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:24.844 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:24.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:24.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:25.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:25.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:25.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:25.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:25.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:25.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:25.433 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 94142 94126 93.3529 64.9375 0.0128072 0.0107065 2024-09-13T14:57:25.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:57:25.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:25.545 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:25 smithi003 ceph-mon[107278]: pgmap v1143: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 907 op/s 2024-09-13T14:57:25.545 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:25 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T14:57:25.545 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:25 smithi003 ceph-mon[107278]: osdmap e321: 8 total, 8 up, 8 in 2024-09-13T14:57:25.545 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:25 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T14:57:25.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:25 smithi003 ceph-mon[104007]: pgmap v1143: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 907 op/s 2024-09-13T14:57:25.546 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:25 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T14:57:25.546 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:25 smithi003 ceph-mon[104007]: osdmap e321: 8 total, 8 up, 8 in 2024-09-13T14:57:25.546 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:25 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T14:57:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:25.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:25.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:25 smithi019 ceph-mon[91296]: pgmap v1143: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 907 op/s 2024-09-13T14:57:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:25 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T14:57:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:25 smithi019 ceph-mon[91296]: osdmap e321: 8 total, 8 up, 8 in 2024-09-13T14:57:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:25 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T14:57:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:25.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:25.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:25.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:25.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:25.988 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:25.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:26.041 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:26.098 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:26.128 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:26.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:26.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:26.378 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:26.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:26 smithi003 ceph-mon[104007]: 1.c scrub starts 2024-09-13T14:57:26.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:26 smithi003 ceph-mon[104007]: osdmap e322: 8 total, 8 up, 8 in 2024-09-13T14:57:26.380 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:26 smithi003 ceph-mon[107278]: 1.c scrub starts 2024-09-13T14:57:26.380 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:26 smithi003 ceph-mon[107278]: osdmap e322: 8 total, 8 up, 8 in 2024-09-13T14:57:26.408 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:26.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:26.433 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 95064 95048 92.7947 57.625 0.00619276 0.0107717 2024-09-13T14:57:26.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:26.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:26.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:26 smithi019 ceph-mon[91296]: 1.c scrub starts 2024-09-13T14:57:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:26 smithi019 ceph-mon[91296]: osdmap e322: 8 total, 8 up, 8 in 2024-09-13T14:57:26.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:26.821 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.850 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:26.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:26.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:57:26.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:26.975 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:27.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:27.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:27.249 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:27.279 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:27.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:27.297 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:57:27.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:27.409 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:27.433 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 95954 95938 92.2229 55.625 0.0112536 0.0108384 2024-09-13T14:57:27.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:27.655 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:27 smithi019 ceph-mon[91296]: 1.16 scrub starts 2024-09-13T14:57:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:27 smithi019 ceph-mon[91296]: pgmap v1146: 119 pgs: 6 peering, 2 active+clean+scrubbing, 111 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T14:57:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:27 smithi019 ceph-mon[91296]: 1.1d scrub starts 2024-09-13T14:57:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:27 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:57:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:27 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/150347135' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:27 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[107278]: 1.16 scrub starts 2024-09-13T14:57:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[107278]: pgmap v1146: 119 pgs: 6 peering, 2 active+clean+scrubbing, 111 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T14:57:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[107278]: 1.1d scrub starts 2024-09-13T14:57:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:57:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/150347135' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:27.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[104007]: 1.16 scrub starts 2024-09-13T14:57:27.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[104007]: pgmap v1146: 119 pgs: 6 peering, 2 active+clean+scrubbing, 111 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T14:57:27.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[104007]: 1.1d scrub starts 2024-09-13T14:57:27.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:57:27.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/150347135' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:27.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:27 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:27.687 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:27.789 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:27.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.814 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:27.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:27.915 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:27.948 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:27.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:28.050 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:28.081 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:28.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:28.215 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:28.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:28.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T14:57:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[107278]: osdmap e323: 8 total, 8 up, 8 in 2024-09-13T14:57:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T14:57:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[104007]: osdmap e323: 8 total, 8 up, 8 in 2024-09-13T14:57:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:28.434 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 96970 96954 91.7877 63.5 0.00887641 0.0108892 2024-09-13T14:57:28.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:28.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:28.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:28 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T14:57:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:28 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:28 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:28 smithi019 ceph-mon[91296]: osdmap e323: 8 total, 8 up, 8 in 2024-09-13T14:57:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:28.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:28.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:28.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:28.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:29.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:29.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:29.181 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:29.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:29.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:29.433 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 97985 97969 91.3645 63.4375 0.00473603 0.0109401 2024-09-13T14:57:29.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:29.604 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:29.637 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:29 smithi019 ceph-mon[91296]: pgmap v1148: 119 pgs: 6 peering, 113 active+clean; 5.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 2.8 KiB/s rd, 56 MiB/s wr, 903 op/s 2024-09-13T14:57:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:29 smithi019 ceph-mon[91296]: 1.19 scrub starts 2024-09-13T14:57:29.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:29 smithi003 ceph-mon[104007]: pgmap v1148: 119 pgs: 6 peering, 113 active+clean; 5.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 2.8 KiB/s rd, 56 MiB/s wr, 903 op/s 2024-09-13T14:57:29.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:29 smithi003 ceph-mon[104007]: 1.19 scrub starts 2024-09-13T14:57:29.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:29 smithi003 ceph-mon[107278]: pgmap v1148: 119 pgs: 6 peering, 113 active+clean; 5.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 2.8 KiB/s rd, 56 MiB/s wr, 903 op/s 2024-09-13T14:57:29.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:29 smithi003 ceph-mon[107278]: 1.19 scrub starts 2024-09-13T14:57:29.739 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:29.768 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:29.870 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.902 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:29.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:30.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:30.039 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:30.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:30.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:30.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:30.273 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:30.299 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:30.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:30.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:30.433 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 98999 98983 90.9529 63.375 0.029484 0.0109894 2024-09-13T14:57:30.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:30.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:30.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:30.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:30 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T14:57:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:57:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:57:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:30.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:30 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T14:57:30.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:30 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T14:57:30.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:30.721 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:30.754 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:30.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:30.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:30.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.015 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:31.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:31.116 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:31.143 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:31.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:31.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:57:31.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:31.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:31 smithi003 ceph-mon[104007]: pgmap v1149: 119 pgs: 119 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 2.8 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 18 KiB/s, 0 objects/s recovering 2024-09-13T14:57:31.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:31.434 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 99944 99928 90.4901 59.0625 0.0192754 0.0110442 2024-09-13T14:57:31.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:31.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:31.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:31.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:31.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:31.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:31 smithi019 ceph-mon[91296]: pgmap v1149: 119 pgs: 119 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 2.8 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 18 KiB/s, 0 objects/s recovering 2024-09-13T14:57:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:31 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:57:31.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:31 smithi003 ceph-mon[107278]: pgmap v1149: 119 pgs: 119 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 2.8 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 18 KiB/s, 0 objects/s recovering 2024-09-13T14:57:31.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:31 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:57:31.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:31 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:57:31.697 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:31.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:31.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:31.831 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:31.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:31.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:31.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:32.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:32.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:32.208 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:32.239 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:32.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:32.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:32.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:32.434 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 100701 100685 89.8733 47.3125 0.0151056 0.0111214 2024-09-13T14:57:32.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:32.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:32.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:32.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:32 smithi019 ceph-mon[91296]: 1.16 scrub starts 2024-09-13T14:57:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:32 smithi019 ceph-mon[91296]: 1.c scrub starts 2024-09-13T14:57:32.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:32 smithi003 ceph-mon[104007]: 1.16 scrub starts 2024-09-13T14:57:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:32 smithi003 ceph-mon[104007]: 1.c scrub starts 2024-09-13T14:57:32.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:32 smithi003 ceph-mon[107278]: 1.16 scrub starts 2024-09-13T14:57:32.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:32 smithi003 ceph-mon[107278]: 1.c scrub starts 2024-09-13T14:57:32.745 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:32.770 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:32.872 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:32.896 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:32.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:33.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:33.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:33.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:33.312 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:33.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:33.414 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:33.434 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 101659 101643 89.4508 59.875 0.0307318 0.0111734 2024-09-13T14:57:33.439 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:33.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:33.557 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:33 smithi003 ceph-mon[104007]: pgmap v1150: 119 pgs: 1 active+clean+scrubbing, 118 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 2.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 14 KiB/s, 0 objects/s recovering 2024-09-13T14:57:33.557 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:33 smithi003 ceph-mon[104007]: 1.1d scrub starts 2024-09-13T14:57:33.557 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:33 smithi003 ceph-mon[107278]: pgmap v1150: 119 pgs: 1 active+clean+scrubbing, 118 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 2.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 14 KiB/s, 0 objects/s recovering 2024-09-13T14:57:33.558 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:33 smithi003 ceph-mon[107278]: 1.1d scrub starts 2024-09-13T14:57:33.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:33.673 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:33.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:33 smithi019 ceph-mon[91296]: pgmap v1150: 119 pgs: 1 active+clean+scrubbing, 118 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 2.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 14 KiB/s, 0 objects/s recovering 2024-09-13T14:57:33.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:33 smithi019 ceph-mon[91296]: 1.1d scrub starts 2024-09-13T14:57:33.699 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:33.801 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:33.829 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:33.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:33.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:34.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:34.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:34.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:34.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:34.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:57:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:57:34] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-09-13T14:57:34.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T14:57:34.080+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e323 definitely_dead 0 2024-09-13T14:57:34.343 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:34.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:34.434 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 102249 102233 88.7206 36.875 0.0219245 0.011263 2024-09-13T14:57:34.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:34.470 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:34.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.595 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:34.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:34.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:34 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T14:57:34.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:34 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2680146725' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:57:34.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:34 smithi019 ceph-mon[91296]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:57:34.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:34 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T14:57:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:34 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2680146725' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:57:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:34 smithi003 ceph-mon[104007]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:57:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:34 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T14:57:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:34 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2680146725' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:57:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:34 smithi003 ceph-mon[107278]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:57:34.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:34.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:34.773 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.5. 2024-09-13T14:57:34.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:34.845 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:34.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:34.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:35.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.097 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:35.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:35.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:35.226 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:35.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:35.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:35.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:35.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:35 smithi003 ceph-mon[107278]: pgmap v1151: 119 pgs: 5 active+clean+scrubbing, 114 active+clean; 6.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.9 KiB/s rd, 60 MiB/s wr, 963 op/s; 31 KiB/s, 0 objects/s recovering 2024-09-13T14:57:35.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:35 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2680146725' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:57:35.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:35 smithi003 ceph-mon[107278]: osdmap e324: 8 total, 8 up, 7 in 2024-09-13T14:57:35.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:35 smithi003 ceph-mon[107278]: 1.19 scrub starts 2024-09-13T14:57:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:35 smithi003 ceph-mon[104007]: pgmap v1151: 119 pgs: 5 active+clean+scrubbing, 114 active+clean; 6.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.9 KiB/s rd, 60 MiB/s wr, 963 op/s; 31 KiB/s, 0 objects/s recovering 2024-09-13T14:57:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:35 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2680146725' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:57:35.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:35 smithi003 ceph-mon[104007]: osdmap e324: 8 total, 8 up, 7 in 2024-09-13T14:57:35.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:35 smithi003 ceph-mon[104007]: 1.19 scrub starts 2024-09-13T14:57:35.434 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 102294 102278 87.544 2.8125 0.00374476 0.0112606 2024-09-13T14:57:35.465 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:35.491 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:35.593 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:35.616 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:35.662 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:35.662 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:35 smithi019 ceph-mon[91296]: pgmap v1151: 119 pgs: 5 active+clean+scrubbing, 114 active+clean; 6.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.9 KiB/s rd, 60 MiB/s wr, 963 op/s; 31 KiB/s, 0 objects/s recovering 2024-09-13T14:57:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:35 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2680146725' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:57:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:35 smithi019 ceph-mon[91296]: osdmap e324: 8 total, 8 up, 7 in 2024-09-13T14:57:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:35 smithi019 ceph-mon[91296]: 1.19 scrub starts 2024-09-13T14:57:35.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:35.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:35.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:35.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:35.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:35.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:35.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:36.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:36.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:36.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:36.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:36.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:36.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:36.382 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:36.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:57:36.435 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 102294 102278 86.3611 0 - 0.0112606 2024-09-13T14:57:36.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.516 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:36.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:36.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:36.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:36 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T14:57:36.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:36 smithi003 ceph-mon[107278]: osdmap e325: 8 total, 8 up, 7 in 2024-09-13T14:57:36.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:36 smithi003 ceph-mon[107278]: 1.a scrub starts 2024-09-13T14:57:36.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:36 smithi003 ceph-mon[107278]: osdmap e326: 8 total, 8 up, 7 in 2024-09-13T14:57:36.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:36 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T14:57:36.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:36 smithi003 ceph-mon[104007]: osdmap e325: 8 total, 8 up, 7 in 2024-09-13T14:57:36.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:36 smithi003 ceph-mon[104007]: 1.a scrub starts 2024-09-13T14:57:36.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:36 smithi003 ceph-mon[104007]: osdmap e326: 8 total, 8 up, 7 in 2024-09-13T14:57:36.629 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:36.651 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:57:36.651 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:36.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:36.681 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:57:36.681 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:36 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T14:57:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:36 smithi019 ceph-mon[91296]: osdmap e325: 8 total, 8 up, 7 in 2024-09-13T14:57:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:36 smithi019 ceph-mon[91296]: 1.a scrub starts 2024-09-13T14:57:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:36 smithi019 ceph-mon[91296]: osdmap e326: 8 total, 8 up, 7 in 2024-09-13T14:57:36.710 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:57:36.710 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:36.733 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:57:36.733 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:36.761 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:36.762 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:57:36.762 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:36.793 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:36.795 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:57:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:36.829 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:57:36.829 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:36.862 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:57:36.862 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:36.895 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:36.930 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:37.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:37.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:37.205 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:37.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:37.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:37.435 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 103030 103014 85.823 23 0.0101039 0.0116466 2024-09-13T14:57:37.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:37.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:37.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:37.630 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:37.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[104007]: 1.17 scrub starts 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[104007]: 1.16 scrub starts 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[104007]: pgmap v1154: 119 pgs: 5 peering, 5 active+clean+scrubbing, 109 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[104007]: 12.f starting backfill to osd.4 from (0'0,0'0] MAX to 323'6453 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[104007]: 1.c scrub starts 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[104007]: 1.8 scrub starts 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3022738663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[107278]: 1.17 scrub starts 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[107278]: 1.16 scrub starts 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[107278]: pgmap v1154: 119 pgs: 5 peering, 5 active+clean+scrubbing, 109 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T14:57:37.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[107278]: 12.f starting backfill to osd.4 from (0'0,0'0] MAX to 323'6453 2024-09-13T14:57:37.654 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:57:37.654 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[107278]: 1.c scrub starts 2024-09-13T14:57:37.654 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[107278]: 1.8 scrub starts 2024-09-13T14:57:37.654 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3022738663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:37 smithi019 ceph-mon[91296]: 1.17 scrub starts 2024-09-13T14:57:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:37 smithi019 ceph-mon[91296]: 1.16 scrub starts 2024-09-13T14:57:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:37 smithi019 ceph-mon[91296]: pgmap v1154: 119 pgs: 5 peering, 5 active+clean+scrubbing, 109 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T14:57:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:37 smithi019 ceph-mon[91296]: 12.f starting backfill to osd.4 from (0'0,0'0] MAX to 323'6453 2024-09-13T14:57:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:37 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:57:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:37 smithi019 ceph-mon[91296]: 1.c scrub starts 2024-09-13T14:57:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:37 smithi019 ceph-mon[91296]: 1.8 scrub starts 2024-09-13T14:57:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3022738663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:37.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:37.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:37.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:37.917 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:38.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:38.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:38.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:38.307 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:38.340 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:38.435 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 103968 103952 85.4651 58.625 0.061072 0.0116944 2024-09-13T14:57:38.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.482 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:38.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:38.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:38.592 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:38 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T14:57:38.592 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:38 smithi003 ceph-mon[104007]: 1.1d scrub starts 2024-09-13T14:57:38.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:38 smithi003 ceph-mon[104007]: 1.18 scrub starts 2024-09-13T14:57:38.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:38 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:38.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:38 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T14:57:38.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:38 smithi003 ceph-mon[107278]: 1.1d scrub starts 2024-09-13T14:57:38.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:38 smithi003 ceph-mon[107278]: 1.18 scrub starts 2024-09-13T14:57:38.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:38 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:38.594 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:38 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T14:57:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:38 smithi019 ceph-mon[91296]: 1.1d scrub starts 2024-09-13T14:57:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:38 smithi019 ceph-mon[91296]: 1.18 scrub starts 2024-09-13T14:57:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:38 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:38.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:38.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:38.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:38.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:38.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:39.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:39.111 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:39.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:39.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:39.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:39.436 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 104792 104776 85.0227 51.5 0.0161348 0.0117561 2024-09-13T14:57:39.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:39.561 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:39.628 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[107278]: pgmap v1156: 119 pgs: 1 remapped+peering, 5 peering, 1 active+clean+scrubbing, 112 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 47 MiB/s wr, 752 op/s 2024-09-13T14:57:39.628 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T14:57:39.628 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:57:39.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:39.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:39.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[104007]: pgmap v1156: 119 pgs: 1 remapped+peering, 5 peering, 1 active+clean+scrubbing, 112 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 47 MiB/s wr, 752 op/s 2024-09-13T14:57:39.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T14:57:39.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:57:39.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:39.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:39 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:39.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:39 smithi019 ceph-mon[91296]: pgmap v1156: 119 pgs: 1 remapped+peering, 5 peering, 1 active+clean+scrubbing, 112 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 47 MiB/s wr, 752 op/s 2024-09-13T14:57:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:39 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T14:57:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:39 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:57:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:39 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:39 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:39.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:57:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:39.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:39.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:39.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:39.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:40.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:40.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:40.170 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:40.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:40.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:40.359 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:57:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:57:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:40.412 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:57:40.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:40 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:40 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:40 smithi003 ceph-mon[104007]: 1.19 scrub starts 2024-09-13T14:57:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:40 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2773390319' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:40.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:40 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:40.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:40 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:40.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:40 smithi003 ceph-mon[107278]: 1.19 scrub starts 2024-09-13T14:57:40.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:40 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2773390319' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:40.438 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 15 105547 105532 84.5361 47.25 0.0262179 0.0118192 2024-09-13T14:57:40.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:40.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:40.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:40.587 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:57:40.587 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:57:40.587 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-13T14:57:40.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd in 5 2024-09-13T14:57:40.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:40 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:40 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:40 smithi019 ceph-mon[91296]: 1.19 scrub starts 2024-09-13T14:57:40.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:40 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2773390319' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:40.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:40.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:40.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:40.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:41.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:41.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:41.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:41.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:41.325 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:41.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:41.360 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:41.403 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[107278]: pgmap v1157: 119 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 5 peering, 112 active+clean; 6.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 26 MiB/s wr, 422 op/s; 5912/309732 objects misplaced (1.909%); 6.5 MiB/s, 106 objects/s recovering 2024-09-13T14:57:41.403 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:41.403 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2773390319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:41.403 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[107278]: osdmap e327: 8 total, 8 up, 7 in 2024-09-13T14:57:41.403 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[107278]: 1.a scrub starts 2024-09-13T14:57:41.403 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[107278]: 1.a scrub ok 2024-09-13T14:57:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:41.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:41.439 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 106320 106304 84.0765 48.25 0.0124529 0.011887 2024-09-13T14:57:41.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.549 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:41.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:41.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:41.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[104007]: pgmap v1157: 119 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 5 peering, 112 active+clean; 6.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 26 MiB/s wr, 422 op/s; 5912/309732 objects misplaced (1.909%); 6.5 MiB/s, 106 objects/s recovering 2024-09-13T14:57:41.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:41.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2773390319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:41.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[104007]: osdmap e327: 8 total, 8 up, 7 in 2024-09-13T14:57:41.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[104007]: 1.a scrub starts 2024-09-13T14:57:41.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:41 smithi003 ceph-mon[104007]: 1.a scrub ok 2024-09-13T14:57:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:41 smithi019 ceph-mon[91296]: pgmap v1157: 119 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 5 peering, 112 active+clean; 6.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 26 MiB/s wr, 422 op/s; 5912/309732 objects misplaced (1.909%); 6.5 MiB/s, 106 objects/s recovering 2024-09-13T14:57:41.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:41 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:41.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:41 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2773390319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:41.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:41 smithi019 ceph-mon[91296]: osdmap e327: 8 total, 8 up, 7 in 2024-09-13T14:57:41.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:41 smithi019 ceph-mon[91296]: 1.a scrub starts 2024-09-13T14:57:41.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:41 smithi019 ceph-mon[91296]: 1.a scrub ok 2024-09-13T14:57:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.696 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:41.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:41.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.848 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:41.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:41.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:41.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:42.092 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:42.124 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:42.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:42.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:42.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:42.416 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:42.439 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T14:57:42.438587+0000 min lat: 0.00131314 max lat: 2.39922 avg lat: 0.0119447 2024-09-13T14:57:42.439 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:57:42.439 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 107159 107143 83.6811 52.4375 0.0164171 0.0119447 2024-09-13T14:57:42.517 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:42.551 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:42.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:42 smithi019 ceph-mon[91296]: 1.16 scrub starts 2024-09-13T14:57:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:42 smithi019 ceph-mon[91296]: 1.16 scrub ok 2024-09-13T14:57:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:42 smithi019 ceph-mon[91296]: 1.17 scrub starts 2024-09-13T14:57:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:42 smithi019 ceph-mon[91296]: 1.17 scrub ok 2024-09-13T14:57:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:42 smithi019 ceph-mon[91296]: 1.c scrub starts 2024-09-13T14:57:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:42 smithi019 ceph-mon[91296]: 1.c scrub ok 2024-09-13T14:57:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:42 smithi019 ceph-mon[91296]: 1.8 scrub starts 2024-09-13T14:57:42.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:42 smithi019 ceph-mon[91296]: 1.8 scrub ok 2024-09-13T14:57:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[107278]: 1.16 scrub starts 2024-09-13T14:57:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[107278]: 1.16 scrub ok 2024-09-13T14:57:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[107278]: 1.17 scrub starts 2024-09-13T14:57:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[107278]: 1.17 scrub ok 2024-09-13T14:57:42.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[107278]: 1.c scrub starts 2024-09-13T14:57:42.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[107278]: 1.c scrub ok 2024-09-13T14:57:42.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[107278]: 1.8 scrub starts 2024-09-13T14:57:42.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[107278]: 1.8 scrub ok 2024-09-13T14:57:42.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[104007]: 1.16 scrub starts 2024-09-13T14:57:42.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[104007]: 1.16 scrub ok 2024-09-13T14:57:42.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[104007]: 1.17 scrub starts 2024-09-13T14:57:42.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[104007]: 1.17 scrub ok 2024-09-13T14:57:42.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[104007]: 1.c scrub starts 2024-09-13T14:57:42.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[104007]: 1.c scrub ok 2024-09-13T14:57:42.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[104007]: 1.8 scrub starts 2024-09-13T14:57:42.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:42 smithi003 ceph-mon[104007]: 1.8 scrub ok 2024-09-13T14:57:42.689 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:57:42.689 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:57:42.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36168.0:149380 10.3 10:c6822046:::benchmark_data_smithi003_197468_object32651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.296643+0000", 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 474.86289688300002, 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34575353399999997, 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.690 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36168", 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 149380 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296643+0000", 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296643+0000", 2024-09-13T14:57:42.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296645+0000", 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.761e-06 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296651+0000", 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5180000000000001e-06 2024-09-13T14:57:42.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296665+0000", 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4066000000000001e-05 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296674+0000", 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6840000000000002e-06 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296741+0000", 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7305999999999996e-05 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.296904+0000", 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016293199999999999 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297060+0000", 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015613700000000001 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.310299+0000", 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013238409 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.313242+0000", 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029432899999999999 2024-09-13T14:57:42.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.640267+0000", 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32702447200000001 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.640356+0000", 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9588999999999999e-05 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642397+0000", 2024-09-13T14:57:42.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020403700000000001 2024-09-13T14:57:42.697 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.697 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.697 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.697 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96814 11.11 11.8c21ebb1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.697 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.160407+0000", 2024-09-13T14:57:42.697 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.99913287099997, 2024-09-13T14:57:42.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.946111904, 2024-09-13T14:57:42.698 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.698 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.698 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.698 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96814 2024-09-13T14:57:42.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160407+0000", 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160407+0000", 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160408+0000", 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.372e-06 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160432+0000", 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3487999999999999e-05 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160441+0000", 2024-09-13T14:57:42.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4010000000000003e-06 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160447+0000", 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6729999999999997e-06 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160517+0000", 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0185999999999995e-05 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.160525+0000", 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8199999999999997e-06 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.106508+0000", 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94598301100000004 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.106519+0000", 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0953e-05 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.702 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96867 11.11 11.57744991 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.233781+0000", 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.92575909700003, 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87324690599999999, 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96867 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233781+0000", 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233781+0000", 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233781+0000", 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4000000000000001e-07 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233802+0000", 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0288999999999998e-05 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233811+0000", 2024-09-13T14:57:42.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7499999999999992e-06 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.233817+0000", 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7989999999999996e-06 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.236916+0000", 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030987860000000001 2024-09-13T14:57:42.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.236923+0000", 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1210000000000001e-06 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107019+0000", 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87009556899999996 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107028+0000", 2024-09-13T14:57:42.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.952e-06 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96891 11.11 11.a3b0f631 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.252748+0000", 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.906791973, 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.85431288100000002, 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.708 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96891 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252748+0000", 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252748+0000", 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252749+0000", 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2470000000000001e-06 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252769+0000", 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9615000000000001e-05 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252784+0000", 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4899000000000001e-05 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252792+0000", 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2400000000000007e-06 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252868+0000", 2024-09-13T14:57:42.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6359000000000005e-05 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.252876+0000", 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5700000000000004e-06 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107053+0000", 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.85417744399999995 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107061+0000", 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5070000000000003e-06 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96906 11.11 11.416bf311 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.266083+0000", 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.893457437, 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.841012863, 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.713 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.714 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.714 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.714 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.714 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96906 2024-09-13T14:57:42.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.714 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266083+0000", 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266083+0000", 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266084+0000", 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.33e-06 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266134+0000", 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0283e-05 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266150+0000", 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5828000000000002e-05 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266158+0000", 2024-09-13T14:57:42.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6229999999999997e-06 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266967+0000", 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000809886 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.266977+0000", 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0350000000000007e-06 2024-09-13T14:57:42.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107087+0000", 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.84011018999999998 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107095+0000", 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6880000000000005e-06 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96933 11.11 11.30677291 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.295082+0000", 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.86445838399999, 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.81204666999999997, 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96933 2024-09-13T14:57:42.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295082+0000", 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295082+0000", 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295083+0000", 2024-09-13T14:57:42.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8190000000000001e-06 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295108+0000", 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4278e-05 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295124+0000", 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6710000000000001e-05 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295133+0000", 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4400000000000005e-06 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295217+0000", 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4079000000000006e-05 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.295227+0000", 2024-09-13T14:57:42.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0227000000000001e-05 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107121+0000", 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.81189347099999998 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107128+0000", 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6459999999999993e-06 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:96998 11.11 11.57659511 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.356695+0000", 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.80284456099997, 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.75046507100000004, 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 96998 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.724 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356695+0000", 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356695+0000", 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356697+0000", 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.249e-06 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356720+0000", 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3578999999999999e-05 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356736+0000", 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5892999999999998e-05 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356744+0000", 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6329999999999996e-06 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356901+0000", 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015711800000000001 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.356910+0000", 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3139999999999993e-06 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107153+0000", 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.75024257599999999 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107160+0000", 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7090000000000002e-06 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.728 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97010 11.11 11.1257951 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.364426+0000", 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.79511415000002, 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74276584000000001, 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97010 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364426+0000", 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364426+0000", 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364427+0000", 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.513e-06 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364456+0000", 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8504999999999999e-05 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364481+0000", 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5275000000000001e-05 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364490+0000", 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.217e-06 2024-09-13T14:57:42.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364555+0000", 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4296999999999998e-05 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364564+0000", 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8999999999999995e-06 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107185+0000", 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74262110199999998 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107192+0000", 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0310000000000001e-06 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.733 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97011 11.11 11.c6b96131 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.364516+0000", 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.79502405099998, 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.742706172, 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97011 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364516+0000", 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364516+0000", 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364516+0000", 2024-09-13T14:57:42.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0999999999999999e-07 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364536+0000", 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9724e-05 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364544+0000", 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6790000000000005e-06 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364549+0000", 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6590000000000004e-06 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364589+0000", 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9217000000000001e-05 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364595+0000", 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1519999999999997e-06 2024-09-13T14:57:42.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107215+0000", 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74262040900000004 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107222+0000", 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8220000000000001e-06 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97012 11.11 11.7b2575f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.364558+0000", 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.79498205099998, 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74269332099999996, 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97012 2024-09-13T14:57:42.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364558+0000", 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364558+0000", 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364558+0000", 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.58e-07 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364605+0000", 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7094e-05 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364613+0000", 2024-09-13T14:57:42.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2010000000000002e-06 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.364618+0000", 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0640000000000001e-06 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.366443+0000", 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018250460000000001 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.366450+0000", 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0129999999999996e-06 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107244+0000", 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74079445600000005 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107251+0000", 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9890000000000003e-06 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.743 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97049 11.11 11.b3adac71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.417966+0000", 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.74157365799999, 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.68931568700000001, 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.744 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97049 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.417966+0000", 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.417966+0000", 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.417968+0000", 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.001e-06 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418039+0000", 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0745000000000001e-05 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418059+0000", 2024-09-13T14:57:42.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9788000000000001e-05 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418068+0000", 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1009999999999998e-06 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418134+0000", 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6412e-05 2024-09-13T14:57:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.418143+0000", 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9430000000000006e-06 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107275+0000", 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.689131359 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107282+0000", 2024-09-13T14:57:42.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3379999999999999e-06 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97063 11.11 11.3e7508f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.435905+0000", 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.72363503600002, 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.671408, 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.749 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97063 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435905+0000", 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435905+0000", 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435906+0000", 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4720000000000001e-06 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435928+0000", 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1923e-05 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435939+0000", 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1097000000000001e-05 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.435946+0000", 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0890000000000002e-06 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.436032+0000", 2024-09-13T14:57:42.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5976999999999994e-05 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.436041+0000", 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5110000000000003e-06 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107306+0000", 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.67126498300000004 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107313+0000", 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9480000000000002e-06 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97147 11.11 11.a734b571 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.543788+0000", 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.61575235499998, 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.56355814500000001, 2024-09-13T14:57:42.754 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97147 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543788+0000", 2024-09-13T14:57:42.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543788+0000", 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543789+0000", 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.598e-06 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543813+0000", 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4238000000000001e-05 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543833+0000", 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9123999999999999e-05 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543841+0000", 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5609999999999999e-06 2024-09-13T14:57:42.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543919+0000", 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7885000000000004e-05 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.543926+0000", 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4619999999999999e-06 2024-09-13T14:57:42.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107339+0000", 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.56341212500000004 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107346+0000", 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1520000000000003e-06 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.759 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97153 11.11 11.ac919031 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.550994+0000", 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.60854609099999, 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.55638195899999998, 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.760 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97153 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.550994+0000", 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.550994+0000", 2024-09-13T14:57:42.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.550996+0000", 2024-09-13T14:57:42.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.81e-06 2024-09-13T14:57:42.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551017+0000", 2024-09-13T14:57:42.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1404999999999999e-05 2024-09-13T14:57:42.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551030+0000", 2024-09-13T14:57:42.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2978e-05 2024-09-13T14:57:42.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551038+0000", 2024-09-13T14:57:42.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3289999999999994e-06 2024-09-13T14:57:42.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551316+0000", 2024-09-13T14:57:42.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027794499999999998 2024-09-13T14:57:42.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.551324+0000", 2024-09-13T14:57:42.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5630000000000003e-06 2024-09-13T14:57:42.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107369+0000", 2024-09-13T14:57:42.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.55604483699999996 2024-09-13T14:57:42.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107376+0000", 2024-09-13T14:57:42.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.092e-06 2024-09-13T14:57:42.768 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.768 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.768 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.769 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97192 11.11 11.216cafd1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.769 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.607681+0000", 2024-09-13T14:57:42.769 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.55185854799998, 2024-09-13T14:57:42.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.49972701600000002, 2024-09-13T14:57:42.770 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.770 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.770 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.770 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.770 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.770 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.771 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97192 2024-09-13T14:57:42.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.771 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607681+0000", 2024-09-13T14:57:42.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607681+0000", 2024-09-13T14:57:42.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607683+0000", 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.765e-06 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607709+0000", 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6139000000000002e-05 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607729+0000", 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9219999999999999e-05 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607738+0000", 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1649999999999995e-06 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607804+0000", 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5957999999999994e-05 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.607812+0000", 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2609999999999993e-06 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107401+0000", 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.49958935700000001 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107408+0000", 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1509999999999998e-06 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97195 11.11 11.3d344711 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.613202+0000", 2024-09-13T14:57:42.776 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.54633777599997, 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.49423730199999999, 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97195 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613202+0000", 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613202+0000", 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613203+0000", 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2860000000000001e-06 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613225+0000", 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1877000000000001e-05 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613241+0000", 2024-09-13T14:57:42.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5241e-05 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613248+0000", 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2910000000000002e-06 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613320+0000", 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2009999999999997e-05 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.613328+0000", 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2879999999999993e-06 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107432+0000", 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.49410401599999998 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107439+0000", 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2930000000000003e-06 2024-09-13T14:57:42.781 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.782 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.782 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:97217 11.11 11.4babd2f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:57:42.782 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:04.681896+0000", 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 277.477644113, 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.425573592, 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 97217 2024-09-13T14:57:42.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681896+0000", 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681896+0000", 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681897+0000", 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5999999999999999e-06 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681921+0000", 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4025000000000001e-05 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681939+0000", 2024-09-13T14:57:42.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7649999999999999e-05 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.681947+0000", 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5499999999999997e-06 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.682000+0000", 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3761000000000002e-05 2024-09-13T14:57:42.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:04.682006+0000", 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7610000000000004e-06 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107462+0000", 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.42545575899999999 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:05.107469+0000", 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.486e-06 2024-09-13T14:57:42.787 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:211624 11.c 11:31867109:::benchmark_data_smithi003_229782_object89244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.872462+0000", 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 156.28707808799999, 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.47033695800000003, 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.788 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 211624 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.872462+0000", 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.872462+0000", 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.872463+0000", 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1260000000000001e-06 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.872468+0000", 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3689999999999998e-06 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.872485+0000", 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6705e-05 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.872493+0000", 2024-09-13T14:57:42.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2279999999999998e-06 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.892355+0000", 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019861601999999999 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.892531+0000", 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017563499999999999 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893190+0000", 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000659814 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.897337+0000", 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041465130000000001 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:42.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898110+0000", 2024-09-13T14:57:42.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00077328 2024-09-13T14:57:42.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:42.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.342697+0000", 2024-09-13T14:57:42.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44458679499999998 2024-09-13T14:57:42.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:42.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.342753+0000", 2024-09-13T14:57:42.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5921000000000002e-05 2024-09-13T14:57:42.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.342799+0000", 2024-09-13T14:57:42.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5970000000000002e-05 2024-09-13T14:57:42.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.797 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.797 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:211649 11.c 11:3605ebbe:::benchmark_data_smithi003_229782_object89269:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:57:42.797 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.898195+0000", 2024-09-13T14:57:42.798 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 156.26134509299999, 2024-09-13T14:57:42.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44472306, 2024-09-13T14:57:42.798 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.798 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.798 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:42.798 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.799 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.799 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.799 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 211649 2024-09-13T14:57:42.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.799 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898195+0000", 2024-09-13T14:57:42.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898195+0000", 2024-09-13T14:57:42.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898196+0000", 2024-09-13T14:57:42.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6099999999999999e-07 2024-09-13T14:57:42.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898199+0000", 2024-09-13T14:57:42.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4759999999999998e-06 2024-09-13T14:57:42.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898210+0000", 2024-09-13T14:57:42.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0404000000000001e-05 2024-09-13T14:57:42.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898217+0000", 2024-09-13T14:57:42.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3590000000000002e-06 2024-09-13T14:57:42.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898277+0000", 2024-09-13T14:57:42.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9858999999999997e-05 2024-09-13T14:57:42.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:42.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898402+0000", 2024-09-13T14:57:42.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012514700000000001 2024-09-13T14:57:42.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:57:42.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898576+0000", 2024-09-13T14:57:42.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017413800000000001 2024-09-13T14:57:42.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:42.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.907598+0000", 2024-09-13T14:57:42.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0090222600000000007 2024-09-13T14:57:42.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:42.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.907612+0000", 2024-09-13T14:57:42.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3045e-05 2024-09-13T14:57:42.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:42.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.342835+0000", 2024-09-13T14:57:42.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.43522317500000002 2024-09-13T14:57:42.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:42.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.342886+0000", 2024-09-13T14:57:42.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1177999999999997e-05 2024-09-13T14:57:42.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.342918+0000", 2024-09-13T14:57:42.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2057999999999997e-05 2024-09-13T14:57:42.811 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.811 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.811 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.812 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:211653 11.c 11:3a9478c6:::benchmark_data_smithi003_229782_object89273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:57:42.812 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.900667+0000", 2024-09-13T14:57:42.812 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 156.25887277000001, 2024-09-13T14:57:42.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.442352725, 2024-09-13T14:57:42.812 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:57:42.812 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:57:42.812 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:42.812 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 211653 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900667+0000", 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900667+0000", 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900668+0000", 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.124e-06 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900672+0000", 2024-09-13T14:57:42.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5489999999999998e-06 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900682+0000", 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9599999999999995e-06 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900688+0000", 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4409999999999998e-06 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.902709+0000", 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020202929999999998 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.902878+0000", 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016900300000000001 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.902990+0000", 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112323 2024-09-13T14:57:42.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.910443+0000", 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074531880000000003 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.910480+0000", 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6619999999999998e-05 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.342951+0000", 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.43247102399999998 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.342990+0000", 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9124e-05 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:57:42.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:57:42.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.343020+0000", 2024-09-13T14:57:42.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0076000000000001e-05 2024-09-13T14:57:42.819 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.819 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.819 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.819 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:57:42.819 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:57:42.819 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:57:42.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:42.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:42.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:43.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:43.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:43.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:57:43.219 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:43.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:43.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:43.389 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:43.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[104007]: pgmap v1159: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+remapped+backfilling, 115 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 30 MiB/s wr, 473 op/s; 12301/312972 objects misplaced (3.930%); 6.5 MiB/s, 105 objects/s recovering 2024-09-13T14:57:43.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-09-13T14:57:43.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T14:57:43.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:43.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:43 smithi019 ceph-mon[91296]: pgmap v1159: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+remapped+backfilling, 115 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 30 MiB/s wr, 473 op/s; 12301/312972 objects misplaced (3.930%); 6.5 MiB/s, 105 objects/s recovering 2024-09-13T14:57:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:43 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-09-13T14:57:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:43 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T14:57:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:43.439 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 108038 108022 83.3262 54.9375 0.0115696 0.0119953 2024-09-13T14:57:43.491 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:43.528 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:43.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:43.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:43.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[107278]: pgmap v1159: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+remapped+backfilling, 115 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 30 MiB/s wr, 473 op/s; 12301/312972 objects misplaced (3.930%); 6.5 MiB/s, 105 objects/s recovering 2024-09-13T14:57:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-09-13T14:57:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T14:57:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:43.779 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:43.816 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:43.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:43.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:44.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:44.098 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:44.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:44.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:57:44.232 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:44.334 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.373 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:44.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:44.424 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:57:44] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-09-13T14:57:44.439 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 108883 108867 82.9541 52.8125 0.0208545 0.0120479 2024-09-13T14:57:44.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:44.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:44.578 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:44 smithi019 ceph-mon[91296]: 1.b scrub ok 2024-09-13T14:57:44.578 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:44 smithi019 ceph-mon[91296]: 1.1d scrub starts 2024-09-13T14:57:44.578 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:44 smithi019 ceph-mon[91296]: 1.1d scrub ok 2024-09-13T14:57:44.578 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:44 smithi019 ceph-mon[91296]: 1.18 scrub starts 2024-09-13T14:57:44.578 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:44 smithi019 ceph-mon[91296]: 1.18 scrub ok 2024-09-13T14:57:44.578 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:44 smithi019 ceph-mon[91296]: pgmap v1160: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 115 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 47 MiB/s wr, 746 op/s; 10956/321453 objects misplaced (3.408%); 18 MiB/s, 281 objects/s recovering 2024-09-13T14:57:44.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:44.653 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[107278]: 1.b scrub ok 2024-09-13T14:57:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[107278]: 1.1d scrub starts 2024-09-13T14:57:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[107278]: 1.1d scrub ok 2024-09-13T14:57:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[107278]: 1.18 scrub starts 2024-09-13T14:57:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[107278]: 1.18 scrub ok 2024-09-13T14:57:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[107278]: pgmap v1160: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 115 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 47 MiB/s wr, 746 op/s; 10956/321453 objects misplaced (3.408%); 18 MiB/s, 281 objects/s recovering 2024-09-13T14:57:44.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[104007]: 1.b scrub ok 2024-09-13T14:57:44.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[104007]: 1.1d scrub starts 2024-09-13T14:57:44.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[104007]: 1.1d scrub ok 2024-09-13T14:57:44.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[104007]: 1.18 scrub starts 2024-09-13T14:57:44.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[104007]: 1.18 scrub ok 2024-09-13T14:57:44.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:44 smithi003 ceph-mon[104007]: pgmap v1160: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 115 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 47 MiB/s wr, 746 op/s; 10956/321453 objects misplaced (3.408%); 18 MiB/s, 281 objects/s recovering 2024-09-13T14:57:44.755 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:44.793 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:44.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:44.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:44.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:45.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:45.074 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:45.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:45.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:45.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:45.439 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 109708 109692 82.5759 51.5625 0.0135299 0.0121036 2024-09-13T14:57:45.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:45.497 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:45.599 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:45.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:45 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T14:57:45.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:45 smithi003 ceph-mon[107278]: 1.1e scrub ok 2024-09-13T14:57:45.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:45 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:45.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:45 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:45.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:45 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T14:57:45.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:45 smithi003 ceph-mon[104007]: 1.1e scrub ok 2024-09-13T14:57:45.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:45 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:45.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:45 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:45.635 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:45.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:45 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T14:57:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:45 smithi019 ceph-mon[91296]: 1.1e scrub ok 2024-09-13T14:57:45.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:45 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:45.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:45 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:45.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:45.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:45.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:45.918 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:46.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:46.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:46.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:46.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:46.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:46.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:46.439 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 110647 110631 82.2915 58.6875 0.0184783 0.0121463 2024-09-13T14:57:46.456 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:46.494 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:46.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:46.624 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:46.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:46 smithi003 ceph-mon[104007]: 1.19 scrub starts 2024-09-13T14:57:46.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:46 smithi003 ceph-mon[104007]: 1.19 scrub ok 2024-09-13T14:57:46.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:46 smithi003 ceph-mon[104007]: pgmap v1161: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 117 active+clean; 6.6 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 39 MiB/s wr, 622 op/s; 10956/323007 objects misplaced (3.392%); 14 MiB/s, 233 objects/s recovering 2024-09-13T14:57:46.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:46 smithi003 ceph-mon[107278]: 1.19 scrub starts 2024-09-13T14:57:46.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:46 smithi003 ceph-mon[107278]: 1.19 scrub ok 2024-09-13T14:57:46.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:46 smithi003 ceph-mon[107278]: pgmap v1161: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 117 active+clean; 6.6 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 39 MiB/s wr, 622 op/s; 10956/323007 objects misplaced (3.392%); 14 MiB/s, 233 objects/s recovering 2024-09-13T14:57:46.727 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:46.760 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:46.862 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:46.889 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:46 smithi019 ceph-mon[91296]: 1.19 scrub starts 2024-09-13T14:57:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:46 smithi019 ceph-mon[91296]: 1.19 scrub ok 2024-09-13T14:57:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:46 smithi019 ceph-mon[91296]: pgmap v1161: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 117 active+clean; 6.6 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 39 MiB/s wr, 622 op/s; 10956/323007 objects misplaced (3.392%); 14 MiB/s, 233 objects/s recovering 2024-09-13T14:57:46.956 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:57:46.956 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:46.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:46.992 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:57:46.992 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:47.037 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:57:47.037 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:47.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:47.078 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:57:47.078 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:47.106 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:57:47.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:47.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:57:47.140 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:47.142 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:47.172 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:57:47.172 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:47.174 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:47.206 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:57:47.207 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:47.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:47.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:47.418 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:47.439 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 111483 111467 81.9381 52.25 0.00964227 0.0121975 2024-09-13T14:57:47.454 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:47.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:47.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:47.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:47.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:47.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:47.868 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:47.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1215466222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:47.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1215466222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:47.899 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:47.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1215466222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:48.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:48.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:48.142 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:48.173 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:48.261 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:48.267 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:48.274 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:48.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:48.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:48.440 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 112349 112333 81.6143 54.125 0.00785591 0.0122472 2024-09-13T14:57:48.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:48.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:48.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:48.723 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:48.758 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:48.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:48.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:49.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:49.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:49.146 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:49.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:49.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:49.314 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:49.416 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:49.440 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 113249 113233 81.3228 56.25 0.0575584 0.0122887 2024-09-13T14:57:49.446 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:49.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:49.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:49.673 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:49.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:49 smithi019 ceph-mon[91296]: pgmap v1162: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 117 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 44 MiB/s wr, 707 op/s; 10893/327126 objects misplaced (3.330%); 14 MiB/s, 231 objects/s recovering 2024-09-13T14:57:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:49 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:49 smithi003 ceph-mon[104007]: pgmap v1162: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 117 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 44 MiB/s wr, 707 op/s; 10893/327126 objects misplaced (3.330%); 14 MiB/s, 231 objects/s recovering 2024-09-13T14:57:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:49 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:49 smithi003 ceph-mon[107278]: pgmap v1162: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 117 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 44 MiB/s wr, 707 op/s; 10893/327126 objects misplaced (3.330%); 14 MiB/s, 231 objects/s recovering 2024-09-13T14:57:49.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:49 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:57:49.698 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:49.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:49.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:49.950 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:49.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:50.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:50.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:50.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:57:50.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:50.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:50.440 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 113875 113859 80.8433 39.125 0.0160869 0.0123639 2024-09-13T14:57:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:50.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:50.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:50.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:57:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:57:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:50.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:50.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:50.834 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:50.870 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:50.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:51.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:51.123 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:51.156 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:51.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:51.288 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:51.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:51.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:51.440 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 114746 114730 80.5466 54.4375 0.00821607 0.0124086 2024-09-13T14:57:51.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:51.553 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:51.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:51.559 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:51 smithi003 ceph-mon[107278]: pgmap v1163: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 117 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s; 9404/334335 objects misplaced (2.813%); 21 MiB/s, 329 objects/s recovering 2024-09-13T14:57:51.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:51 smithi003 ceph-mon[104007]: pgmap v1163: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 117 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s; 9404/334335 objects misplaced (2.813%); 21 MiB/s, 329 objects/s recovering 2024-09-13T14:57:51.655 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:51 smithi019 ceph-mon[91296]: pgmap v1163: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 117 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s; 9404/334335 objects misplaced (2.813%); 21 MiB/s, 329 objects/s recovering 2024-09-13T14:57:51.688 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:51.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:51.819 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:51.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:51.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:51.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T14:57:51.556+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e327 definitely_dead 0 2024-09-13T14:57:51.954 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:52.056 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:52.089 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:52.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:52.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:52.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:52.334 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.5. 2024-09-13T14:57:52.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:52.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:52.440 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 90 6 115623 115617 80.2676 55.4375 0.00962879 0.0124488 2024-09-13T14:57:52.478 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.501 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:52.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:52.602 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:52.624 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:52 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1492640590' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:57:52.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:52 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1492640590' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:57:52.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:52 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1492640590' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:57:52.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:52.749 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:52.851 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:52.874 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:52.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:53.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:53.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:53.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:53.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:53.256 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:53.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:53.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:53.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:53 smithi019 ceph-mon[91296]: pgmap v1164: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 117 active+clean; 6.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 49 MiB/s wr, 786 op/s; 9316/338073 objects misplaced (2.756%); 18 MiB/s, 291 objects/s recovering 2024-09-13T14:57:53.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1492640590' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:57:53.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:53 smithi019 ceph-mon[91296]: osdmap e328: 8 total, 8 up, 8 in 2024-09-13T14:57:53.440 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 91 6 115623 115617 79.3857 0 - 0.0124488 2024-09-13T14:57:53.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:53.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:53.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:53.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:53.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:53 smithi003 ceph-mon[104007]: pgmap v1164: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 117 active+clean; 6.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 49 MiB/s wr, 786 op/s; 9316/338073 objects misplaced (2.756%); 18 MiB/s, 291 objects/s recovering 2024-09-13T14:57:53.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1492640590' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:57:53.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:53 smithi003 ceph-mon[104007]: osdmap e328: 8 total, 8 up, 8 in 2024-09-13T14:57:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:53 smithi003 ceph-mon[107278]: pgmap v1164: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 117 active+clean; 6.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 49 MiB/s wr, 786 op/s; 9316/338073 objects misplaced (2.756%); 18 MiB/s, 291 objects/s recovering 2024-09-13T14:57:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1492640590' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:57:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:53 smithi003 ceph-mon[107278]: osdmap e328: 8 total, 8 up, 8 in 2024-09-13T14:57:53.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:53.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:53.864 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:53.887 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:53.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:54.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:54.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:54.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:54.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:57:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:57:54] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-09-13T14:57:54.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:54.260 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:54.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:54.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:54.440 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 91.9474 2024-09-13T14:57:54.440 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 115623 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 78.5932 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 38.4955 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 202.25 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1257 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 615.927 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3236 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0125204 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0307615 2024-09-13T14:57:54.441 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.39922 2024-09-13T14:57:54.442 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00131314 2024-09-13T14:57:54.446 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:57:54.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:54 smithi003 ceph-mon[107278]: osdmap e329: 8 total, 8 up, 8 in 2024-09-13T14:57:54.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:54 smithi003 ceph-mon[104007]: osdmap e329: 8 total, 8 up, 8 in 2024-09-13T14:57:54.488 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:54.515 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:54.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:54.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:54.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:54 smithi019 ceph-mon[91296]: osdmap e329: 8 total, 8 up, 8 in 2024-09-13T14:57:54.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:54.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:54.913 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:54.943 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:55.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:55.087 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:55.189 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:55.223 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:55.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:55 smithi019 ceph-mon[91296]: pgmap v1167: 119 pgs: 1 remapped+peering, 27 peering, 1 active+recovering+undersized+remapped, 90 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s; 6238/345570 objects misplaced (1.805%); 26 MiB/s, 407 objects/s recovering 2024-09-13T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:55 smithi019 ceph-mon[91296]: osdmap e330: 8 total, 8 up, 8 in 2024-09-13T14:57:55.456 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:55.486 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:55.534 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:55 smithi003 ceph-mon[107278]: pgmap v1167: 119 pgs: 1 remapped+peering, 27 peering, 1 active+recovering+undersized+remapped, 90 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s; 6238/345570 objects misplaced (1.805%); 26 MiB/s, 407 objects/s recovering 2024-09-13T14:57:55.534 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:55 smithi003 ceph-mon[107278]: osdmap e330: 8 total, 8 up, 8 in 2024-09-13T14:57:55.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:55 smithi003 ceph-mon[104007]: pgmap v1167: 119 pgs: 1 remapped+peering, 27 peering, 1 active+recovering+undersized+remapped, 90 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s; 6238/345570 objects misplaced (1.805%); 26 MiB/s, 407 objects/s recovering 2024-09-13T14:57:55.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:55 smithi003 ceph-mon[104007]: osdmap e330: 8 total, 8 up, 8 in 2024-09-13T14:57:55.589 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:55.617 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:55.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:55.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:55.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:55.890 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-13T14:57:55.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:56.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:56.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:56.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:56.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:56.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:56.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:56.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:56.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:56.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:56.625 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:56.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:56 smithi003 ceph-mon[104007]: 1.d scrub starts 2024-09-13T14:57:56.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:56 smithi003 ceph-mon[104007]: 1.d scrub ok 2024-09-13T14:57:56.642 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:56 smithi003 ceph-mon[107278]: 1.d scrub starts 2024-09-13T14:57:56.642 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:56 smithi003 ceph-mon[107278]: 1.d scrub ok 2024-09-13T14:57:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:56 smithi019 ceph-mon[91296]: 1.d scrub starts 2024-09-13T14:57:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:56 smithi019 ceph-mon[91296]: 1.d scrub ok 2024-09-13T14:57:56.727 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:56.759 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:56.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:56.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:57.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:57.037 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:57.139 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:57.167 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:57.195 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:57.196 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:57.269 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:57.297 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:57.399 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:57.429 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:57.534 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:57.570 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:57.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:57 smithi003 ceph-mon[107278]: pgmap v1169: 119 pgs: 1 remapped+peering, 27 peering, 1 active+recovering+undersized+remapped, 90 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 42 MiB/s wr, 681 op/s; 6238/346062 objects misplaced (1.803%); 17 MiB/s, 263 objects/s recovering 2024-09-13T14:57:57.644 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:57 smithi003 ceph-mon[107278]: 1.5 scrub starts 2024-09-13T14:57:57.644 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:57 smithi003 ceph-mon[107278]: 1.5 scrub ok 2024-09-13T14:57:57.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:57 smithi003 ceph-mon[104007]: pgmap v1169: 119 pgs: 1 remapped+peering, 27 peering, 1 active+recovering+undersized+remapped, 90 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 42 MiB/s wr, 681 op/s; 6238/346062 objects misplaced (1.803%); 17 MiB/s, 263 objects/s recovering 2024-09-13T14:57:57.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:57 smithi003 ceph-mon[104007]: 1.5 scrub starts 2024-09-13T14:57:57.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:57 smithi003 ceph-mon[104007]: 1.5 scrub ok 2024-09-13T14:57:57.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:57.681 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:57 smithi019 ceph-mon[91296]: pgmap v1169: 119 pgs: 1 remapped+peering, 27 peering, 1 active+recovering+undersized+remapped, 90 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 42 MiB/s wr, 681 op/s; 6238/346062 objects misplaced (1.803%); 17 MiB/s, 263 objects/s recovering 2024-09-13T14:57:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:57 smithi019 ceph-mon[91296]: 1.5 scrub starts 2024-09-13T14:57:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:57 smithi019 ceph-mon[91296]: 1.5 scrub ok 2024-09-13T14:57:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:57.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:57.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:57.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:57.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:58.095 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:58.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:58.262 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:58.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:57:58.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[104007]: 1.9 scrub starts 2024-09-13T14:57:58.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[104007]: 1.9 scrub ok 2024-09-13T14:57:58.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:58.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1682604474' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:58.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:58.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[107278]: 1.9 scrub starts 2024-09-13T14:57:58.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[107278]: 1.9 scrub ok 2024-09-13T14:57:58.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:58.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1682604474' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:58.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:58 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:58.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:58.526 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:58.553 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:58.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:58 smithi019 ceph-mon[91296]: 1.9 scrub starts 2024-09-13T14:57:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:58 smithi019 ceph-mon[91296]: 1.9 scrub ok 2024-09-13T14:57:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:58 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1682604474' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:58 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:58.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:58.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:58.961 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:58.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:59.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:59.091 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:59.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:59.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:57:59.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:59.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:59.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:59.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:59.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:59.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:59.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:59.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:59.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:59 smithi019 ceph-mon[91296]: pgmap v1170: 119 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 25 peering, 92 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 31 MiB/s wr, 630 op/s; 420/344421 objects misplaced (0.122%); 20 MiB/s, 319 objects/s recovering 2024-09-13T14:57:59.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:59 smithi019 ceph-mon[91296]: 1.1c scrub starts 2024-09-13T14:57:59.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:59 smithi019 ceph-mon[91296]: 1.1c scrub ok 2024-09-13T14:57:59.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:59 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:59.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:59 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T14:57:59.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:59 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:59.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:59 smithi019 ceph-mon[91296]: osdmap e331: 8 total, 8 up, 8 in 2024-09-13T14:57:59.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:57:59 smithi019 ceph-mon[91296]: 1.15 scrub starts 2024-09-13T14:57:59.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[104007]: pgmap v1170: 119 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 25 peering, 92 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 31 MiB/s wr, 630 op/s; 420/344421 objects misplaced (0.122%); 20 MiB/s, 319 objects/s recovering 2024-09-13T14:57:59.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[104007]: 1.1c scrub starts 2024-09-13T14:57:59.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[104007]: 1.1c scrub ok 2024-09-13T14:57:59.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:59.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T14:57:59.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:59.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[104007]: osdmap e331: 8 total, 8 up, 8 in 2024-09-13T14:57:59.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[104007]: 1.15 scrub starts 2024-09-13T14:57:59.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[107278]: pgmap v1170: 119 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 25 peering, 92 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 31 MiB/s wr, 630 op/s; 420/344421 objects misplaced (0.122%); 20 MiB/s, 319 objects/s recovering 2024-09-13T14:57:59.688 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[107278]: 1.1c scrub starts 2024-09-13T14:57:59.688 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[107278]: 1.1c scrub ok 2024-09-13T14:57:59.688 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:59.688 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T14:57:59.688 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:59.689 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[107278]: osdmap e331: 8 total, 8 up, 8 in 2024-09-13T14:57:59.689 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:57:59 smithi003 ceph-mon[107278]: 1.15 scrub starts 2024-09-13T14:57:59.708 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:57:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:59.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:59.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:59.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:59.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:57:59.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:00.055 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:00.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:00.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:00.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:00.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:00.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:00.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:00.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:00.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:00.573 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:00.675 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:58:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:58:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:00 smithi019 ceph-mon[91296]: pgmap v1172: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 153 B/s rd, 4.3 MiB/s wr, 504 op/s; 420/338253 objects misplaced (0.124%); 4.0 MiB/s, 65 objects/s recovering 2024-09-13T14:58:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:00 smithi003 ceph-mon[104007]: pgmap v1172: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 153 B/s rd, 4.3 MiB/s wr, 504 op/s; 420/338253 objects misplaced (0.124%); 4.0 MiB/s, 65 objects/s recovering 2024-09-13T14:58:00.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:00 smithi003 ceph-mon[107278]: pgmap v1172: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 153 B/s rd, 4.3 MiB/s wr, 504 op/s; 420/338253 objects misplaced (0.124%); 4.0 MiB/s, 65 objects/s recovering 2024-09-13T14:58:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.704 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:00.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:00.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:00.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:00.891 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:58:00.891 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:58:00.891 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 6, 7, 5] 2024-09-13T14:58:00.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd out 0 2024-09-13T14:58:00.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:01.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:01.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:01.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:01.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:01.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:01.442 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:01.543 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:01.573 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:01.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:01 smithi019 ceph-mon[91296]: 1.e scrub starts 2024-09-13T14:58:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:01 smithi019 ceph-mon[91296]: 1.e scrub ok 2024-09-13T14:58:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:01 smithi019 ceph-mon[91296]: 1.15 scrub starts 2024-09-13T14:58:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:01 smithi003 ceph-mon[104007]: 1.e scrub starts 2024-09-13T14:58:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:01 smithi003 ceph-mon[104007]: 1.e scrub ok 2024-09-13T14:58:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:01 smithi003 ceph-mon[104007]: 1.15 scrub starts 2024-09-13T14:58:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:01 smithi003 ceph-mon[107278]: 1.e scrub starts 2024-09-13T14:58:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:01 smithi003 ceph-mon[107278]: 1.e scrub ok 2024-09-13T14:58:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:01 smithi003 ceph-mon[107278]: 1.15 scrub starts 2024-09-13T14:58:01.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:01.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:01.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:01.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:01.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:01.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:58:01.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:02.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:02.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:02.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:02.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:02.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:02.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:02.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:02.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:02.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:02.722 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:02.730 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:02.745 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:02.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:02.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:02.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:02.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:03.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:03.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:03.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:03.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[104007]: pgmap v1173: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 127 B/s rd, 3.6 MiB/s wr, 609 op/s; 26/333717 objects misplaced (0.008%); 6.4 MiB/s, 103 objects/s recovering 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[104007]: osdmap e332: 8 total, 8 up, 8 in 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[107278]: pgmap v1173: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 127 B/s rd, 3.6 MiB/s wr, 609 op/s; 26/333717 objects misplaced (0.008%); 6.4 MiB/s, 103 objects/s recovering 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:03.249 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[107278]: osdmap e332: 8 total, 8 up, 8 in 2024-09-13T14:58:03.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:58:03.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:58:03.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:03.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:03 smithi019 ceph-mon[91296]: pgmap v1173: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 127 B/s rd, 3.6 MiB/s wr, 609 op/s; 26/333717 objects misplaced (0.008%); 6.4 MiB/s, 103 objects/s recovering 2024-09-13T14:58:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:03 smithi019 ceph-mon[91296]: osdmap e332: 8 total, 8 up, 8 in 2024-09-13T14:58:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:58:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:58:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:03.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:03.504 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:03.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:03.633 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:03.735 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:03.763 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:03.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.908 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:03.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:04.009 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:04.035 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:04.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:04.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:04.274 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:04.304 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:04.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:58:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:58:04] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:58:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:04.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.572 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:04.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:04.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:04.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:04 smithi003 ceph-mon[107278]: osdmap e333: 8 total, 8 up, 8 in 2024-09-13T14:58:04.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:04 smithi003 ceph-mon[107278]: pgmap v1176: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 170 B/s rd, 22 KiB/s wr, 1.06k op/s; 26/325311 objects misplaced (0.008%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:58:04.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:04 smithi003 ceph-mon[107278]: 1.15 scrub starts 2024-09-13T14:58:04.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:04 smithi003 ceph-mon[107278]: 1.15 scrub ok 2024-09-13T14:58:04.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:04 smithi003 ceph-mon[104007]: osdmap e333: 8 total, 8 up, 8 in 2024-09-13T14:58:04.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:04 smithi003 ceph-mon[104007]: pgmap v1176: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 170 B/s rd, 22 KiB/s wr, 1.06k op/s; 26/325311 objects misplaced (0.008%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:58:04.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:04 smithi003 ceph-mon[104007]: 1.15 scrub starts 2024-09-13T14:58:04.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:04 smithi003 ceph-mon[104007]: 1.15 scrub ok 2024-09-13T14:58:04.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:04.794 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:58:04.794 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:04.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.833 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:58:04.834 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:04.858 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:04.868 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:58:04.868 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:04.904 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:58:04.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:04 smithi019 ceph-mon[91296]: osdmap e333: 8 total, 8 up, 8 in 2024-09-13T14:58:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:04 smithi019 ceph-mon[91296]: pgmap v1176: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 170 B/s rd, 22 KiB/s wr, 1.06k op/s; 26/325311 objects misplaced (0.008%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:58:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:04 smithi019 ceph-mon[91296]: 1.15 scrub starts 2024-09-13T14:58:04.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:04 smithi019 ceph-mon[91296]: 1.15 scrub ok 2024-09-13T14:58:04.935 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:58:04.935 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:04.960 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.965 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:58:04.965 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:04.987 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:05.004 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:58:05.004 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:05.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:58:05.049 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:05.088 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:05.113 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:05.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:05.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:05.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:05.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:05.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:05.529 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:05.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:05.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:05.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:05.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:05.852 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4258342422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:05.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:05 smithi003 ceph-mon[104007]: 1.1a scrub starts 2024-09-13T14:58:05.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:05 smithi003 ceph-mon[104007]: 1.1a scrub ok 2024-09-13T14:58:05.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4258342422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:05.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:05 smithi003 ceph-mon[107278]: 1.1a scrub starts 2024-09-13T14:58:05.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:05 smithi003 ceph-mon[107278]: 1.1a scrub ok 2024-09-13T14:58:05.917 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:05.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4258342422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:05.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:05 smithi019 ceph-mon[91296]: 1.1a scrub starts 2024-09-13T14:58:05.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:05 smithi019 ceph-mon[91296]: 1.1a scrub ok 2024-09-13T14:58:05.945 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:06.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:06.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:06.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.207 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:06.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:06.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:06.331 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:58:06.331 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:58:06.331 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:58:06.331 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:58:06.331 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:58:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:06.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:06.448 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:06.476 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:06.578 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:06.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:06.767 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:06 smithi003 ceph-mon[104007]: 1.7 scrub starts 2024-09-13T14:58:06.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:06 smithi003 ceph-mon[104007]: 1.7 scrub ok 2024-09-13T14:58:06.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:06 smithi003 ceph-mon[104007]: pgmap v1177: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 26/324072 objects misplaced (0.008%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:58:06.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:06 smithi003 ceph-mon[107278]: 1.7 scrub starts 2024-09-13T14:58:06.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:06 smithi003 ceph-mon[107278]: 1.7 scrub ok 2024-09-13T14:58:06.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:06 smithi003 ceph-mon[107278]: pgmap v1177: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 26/324072 objects misplaced (0.008%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:58:06.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:06.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:06 smithi019 ceph-mon[91296]: 1.7 scrub starts 2024-09-13T14:58:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:06 smithi019 ceph-mon[91296]: 1.7 scrub ok 2024-09-13T14:58:06.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:06 smithi019 ceph-mon[91296]: pgmap v1177: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 26/324072 objects misplaced (0.008%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:58:07.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:07.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:07.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:07.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:07.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:07.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:07.322 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:07.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:07.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:07.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:58:07.574 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:07.600 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:07.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:07.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:07.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:07 smithi003 ceph-mon[104007]: 1.1b scrub starts 2024-09-13T14:58:07.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:07 smithi003 ceph-mon[104007]: 1.1b scrub ok 2024-09-13T14:58:07.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:07 smithi003 ceph-mon[104007]: 1.6 scrub starts 2024-09-13T14:58:07.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:07 smithi003 ceph-mon[104007]: 1.6 scrub ok 2024-09-13T14:58:07.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:07 smithi003 ceph-mon[107278]: 1.1b scrub starts 2024-09-13T14:58:07.731 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:07 smithi003 ceph-mon[107278]: 1.1b scrub ok 2024-09-13T14:58:07.731 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:07 smithi003 ceph-mon[107278]: 1.6 scrub starts 2024-09-13T14:58:07.731 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:07 smithi003 ceph-mon[107278]: 1.6 scrub ok 2024-09-13T14:58:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:07.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:07.857 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.890 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:07.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:07 smithi019 ceph-mon[91296]: 1.1b scrub starts 2024-09-13T14:58:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:07 smithi019 ceph-mon[91296]: 1.1b scrub ok 2024-09-13T14:58:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:07 smithi019 ceph-mon[91296]: 1.6 scrub starts 2024-09-13T14:58:07.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:07 smithi019 ceph-mon[91296]: 1.6 scrub ok 2024-09-13T14:58:07.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:08.129 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:08.155 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:08.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:08.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:08.293 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:08.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:08.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:08.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:08.433 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:08.535 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.567 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:08.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:08.617 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:08.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:08.694 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:08.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:08 smithi003 ceph-mon[104007]: pgmap v1178: 119 pgs: 119 active+clean; 6.5 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:58:08.708 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:08 smithi003 ceph-mon[107278]: pgmap v1178: 119 pgs: 119 active+clean; 6.5 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:58:08.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:08.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:08.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:08 smithi019 ceph-mon[91296]: pgmap v1178: 119 pgs: 119 active+clean; 6.5 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:58:08.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:08.971 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:08.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:08 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T14:58:08.704+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e333 definitely_dead 0 2024-09-13T14:58:09.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:09.100 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:09.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:09.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:09.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:09.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:09.489 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.0. 2024-09-13T14:58:09.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:09.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:09.626 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:09.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:09.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:09.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:09.826 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/519936952' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-13T14:58:09.826 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:09 smithi003 ceph-mon[107278]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-13T14:58:09.826 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:09 smithi003 ceph-mon[107278]: 1.18 scrub starts 2024-09-13T14:58:09.826 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:09 smithi003 ceph-mon[107278]: 1.18 scrub ok 2024-09-13T14:58:09.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/519936952' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-13T14:58:09.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:09 smithi003 ceph-mon[104007]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-13T14:58:09.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:09 smithi003 ceph-mon[104007]: 1.18 scrub starts 2024-09-13T14:58:09.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:09 smithi003 ceph-mon[104007]: 1.18 scrub ok 2024-09-13T14:58:09.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.899 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:09.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/519936952' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-13T14:58:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:09 smithi019 ceph-mon[91296]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-13T14:58:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:09 smithi019 ceph-mon[91296]: 1.18 scrub starts 2024-09-13T14:58:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:09 smithi019 ceph-mon[91296]: 1.18 scrub ok 2024-09-13T14:58:10.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:10.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:10.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:58:10.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:10.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:10.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:10.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:10.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:10.491 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:58:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:58:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:10.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:10.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:10.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:10.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:10.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:10.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:10.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.913 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:10.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/519936952' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-13T14:58:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:10 smithi019 ceph-mon[91296]: osdmap e334: 8 total, 8 up, 7 in 2024-09-13T14:58:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:10 smithi019 ceph-mon[91296]: pgmap v1180: 119 pgs: 119 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 944 op/s; 220 KiB/s, 3 objects/s recovering 2024-09-13T14:58:10.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/519936952' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-13T14:58:10.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:10 smithi003 ceph-mon[107278]: osdmap e334: 8 total, 8 up, 7 in 2024-09-13T14:58:10.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:10 smithi003 ceph-mon[107278]: pgmap v1180: 119 pgs: 119 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 944 op/s; 220 KiB/s, 3 objects/s recovering 2024-09-13T14:58:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/519936952' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-13T14:58:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:10 smithi003 ceph-mon[104007]: osdmap e334: 8 total, 8 up, 7 in 2024-09-13T14:58:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:10 smithi003 ceph-mon[104007]: pgmap v1180: 119 pgs: 119 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 944 op/s; 220 KiB/s, 3 objects/s recovering 2024-09-13T14:58:11.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:11.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:11.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:11.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:11.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.292 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:11.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:11.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:11.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:11.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:11 smithi003 ceph-mon[104007]: osdmap e335: 8 total, 8 up, 7 in 2024-09-13T14:58:11.519 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:11.548 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:11.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:11.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:11 smithi019 ceph-mon[91296]: osdmap e335: 8 total, 8 up, 7 in 2024-09-13T14:58:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.692 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:11.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:11.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:11.826 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:11.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:11.964 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:11 smithi003 ceph-mon[107278]: osdmap e335: 8 total, 8 up, 7 in 2024-09-13T14:58:12.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:12.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:12.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:12.245 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:12.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.281 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:12.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:12.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:12.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:12.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:12.560 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:12.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:12.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:12.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:12.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:12.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:12 smithi019 ceph-mon[91296]: osdmap e336: 8 total, 8 up, 7 in 2024-09-13T14:58:12.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:12 smithi019 ceph-mon[91296]: 12.c starting backfill to osd.7 from (314'995,327'7272] MIN to 333'8109 2024-09-13T14:58:12.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:12 smithi019 ceph-mon[91296]: 12.8 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7943 2024-09-13T14:58:12.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:12 smithi019 ceph-mon[91296]: 12.3 starting backfill to osd.4 from (0'0,0'0] MAX to 333'8121 2024-09-13T14:58:12.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:12 smithi019 ceph-mon[91296]: 12.e starting backfill to osd.3 from (0'0,0'0] MAX to 333'7955 2024-09-13T14:58:12.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:12 smithi019 ceph-mon[91296]: 12.6 starting backfill to osd.6 from (0'0,0'0] MAX to 333'8002 2024-09-13T14:58:12.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:12 smithi019 ceph-mon[91296]: pgmap v1183: 119 pgs: 3 peering, 116 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 734 op/s; 277 KiB/s, 4 objects/s recovering 2024-09-13T14:58:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[107278]: osdmap e336: 8 total, 8 up, 7 in 2024-09-13T14:58:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[107278]: 12.c starting backfill to osd.7 from (314'995,327'7272] MIN to 333'8109 2024-09-13T14:58:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[107278]: 12.8 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7943 2024-09-13T14:58:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[107278]: 12.3 starting backfill to osd.4 from (0'0,0'0] MAX to 333'8121 2024-09-13T14:58:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[107278]: 12.e starting backfill to osd.3 from (0'0,0'0] MAX to 333'7955 2024-09-13T14:58:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[107278]: 12.6 starting backfill to osd.6 from (0'0,0'0] MAX to 333'8002 2024-09-13T14:58:12.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[107278]: pgmap v1183: 119 pgs: 3 peering, 116 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 734 op/s; 277 KiB/s, 4 objects/s recovering 2024-09-13T14:58:12.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[104007]: osdmap e336: 8 total, 8 up, 7 in 2024-09-13T14:58:12.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[104007]: 12.c starting backfill to osd.7 from (314'995,327'7272] MIN to 333'8109 2024-09-13T14:58:12.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[104007]: 12.8 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7943 2024-09-13T14:58:12.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[104007]: 12.3 starting backfill to osd.4 from (0'0,0'0] MAX to 333'8121 2024-09-13T14:58:12.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[104007]: 12.e starting backfill to osd.3 from (0'0,0'0] MAX to 333'7955 2024-09-13T14:58:12.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[104007]: 12.6 starting backfill to osd.6 from (0'0,0'0] MAX to 333'8002 2024-09-13T14:58:12.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:12 smithi003 ceph-mon[104007]: pgmap v1183: 119 pgs: 3 peering, 116 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 734 op/s; 277 KiB/s, 4 objects/s recovering 2024-09-13T14:58:12.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:12.990 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:13.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:13.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:13.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:13.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:13.385 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:13.418 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:13.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:13.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:13.661 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:13.689 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:13.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:13.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:13.826 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:13.928 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:13 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:58:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:13.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:13 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:58:13.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:13.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:13.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:13 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:58:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:13.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:13.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:14.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:14.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:14.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:14.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:14.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.407 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:14.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:58:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:58:14] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:58:14.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:14.544 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:14.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:14.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:14.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:14.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:14 smithi019 ceph-mon[91296]: pgmap v1184: 119 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 peering, 111 active+clean; 6.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 941 op/s; 30367/303360 objects misplaced (10.010%); 9.2 MiB/s, 148 objects/s recovering 2024-09-13T14:58:14.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:14 smithi003 ceph-mon[107278]: pgmap v1184: 119 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 peering, 111 active+clean; 6.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 941 op/s; 30367/303360 objects misplaced (10.010%); 9.2 MiB/s, 148 objects/s recovering 2024-09-13T14:58:14.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:14 smithi003 ceph-mon[104007]: pgmap v1184: 119 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 peering, 111 active+clean; 6.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 941 op/s; 30367/303360 objects misplaced (10.010%); 9.2 MiB/s, 148 objects/s recovering 2024-09-13T14:58:14.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.980 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:15.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:15.122 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:15.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:15.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:15.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:15.399 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:15.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:15.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:15.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:15.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:15.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:15.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:15.930 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:15.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:16.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:16.103 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:58:16.103 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T14:58:16.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T14:58:16.103 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T14:58:16.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.103 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:117266 10.f e253/229 10:f2aa6f81:::benchmark_data_smithi003_197468_object537:head v 253'7261, mlcod=253'7258)", 2024-09-13T14:58:16.103 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:09.751012+0000", 2024-09-13T14:58:16.103 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 546.34792225800004, 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.29645193800000003, 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751012+0000", 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751012+0000", 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751013+0000", 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0809999999999999e-06 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751020+0000", 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3069999999999997e-06 2024-09-13T14:58:16.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751032+0000", 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1841999999999999e-05 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751040+0000", 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1170000000000004e-06 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.751087+0000", 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6502000000000003e-05 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037186+0000", 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28609978600000002 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.047389+0000", 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010202304000000001 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.047464+0000", 2024-09-13T14:58:16.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4999000000000005e-05 2024-09-13T14:58:16.108 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.108 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.108 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.108 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:117289 10.3 e253/242 10:cb5621ab:::benchmark_data_smithi003_197468_object560:head v 253'7331, mlcod=238'1751)", 2024-09-13T14:58:16.108 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:09.759160+0000", 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 546.339774207, 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.28967668699999999, 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.759160+0000", 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.759160+0000", 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.759161+0000", 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.443e-06 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:09.759170+0000", 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7700000000000007e-06 2024-09-13T14:58:16.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037164+0000", 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27799370499999998 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037181+0000", 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6906999999999999e-05 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037704+0000", 2024-09-13T14:58:16.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052296999999999999 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.037727+0000", 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3213e-05 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.048755+0000", 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011028675 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:10.048836+0000", 2024-09-13T14:58:16.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1003999999999999e-05 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:123357 10.5 e253/229 10:a6ed905f:::benchmark_data_smithi003_197468_object6628:head v 253'7703, mlcod=253'7701)", 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.382685+0000", 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 538.71624867900005, 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26448326100000003, 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382685+0000", 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382685+0000", 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382686+0000", 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1310000000000001e-06 2024-09-13T14:58:16.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382693+0000", 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0539999999999997e-06 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382707+0000", 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3354e-05 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382715+0000", 2024-09-13T14:58:16.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3319999999999992e-06 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382767+0000", 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1663000000000003e-05 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.382788+0000", 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1077000000000002e-05 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.386308+0000", 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035196979999999999 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647169+0000", 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26086095199999998 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.118 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:123358 10.4 e253/229 10:2c9f3a1b:::benchmark_data_smithi003_197468_object6629:head v 253'7638, mlcod=253'7636)", 2024-09-13T14:58:16.118 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.383164+0000", 2024-09-13T14:58:16.118 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 538.71576984499995, 2024-09-13T14:58:16.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.264014205, 2024-09-13T14:58:16.118 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383164+0000", 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383164+0000", 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383165+0000", 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8899999999999999e-07 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383171+0000", 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7130000000000002e-06 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383180+0000", 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5670000000000001e-06 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383186+0000", 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9830000000000001e-06 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383257+0000", 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0498000000000002e-05 2024-09-13T14:58:16.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.383275+0000", 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8218000000000001e-05 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647139+0000", 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26386380999999998 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647178+0000", 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9727000000000001e-05 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:123364 10.3 e253/242 10:c110b7c7:::benchmark_data_smithi003_197468_object6635:head v 253'7744, mlcod=238'1751)", 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:17.389216+0000", 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 538.70971801600001, 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27996448000000002, 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.123 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389216+0000", 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389216+0000", 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389217+0000", 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1629999999999999e-06 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.389225+0000", 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1030000000000002e-06 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647084+0000", 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.257858898 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647102+0000", 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8372e-05 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647166+0000", 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3359000000000001e-05 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.647193+0000", 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6716999999999998e-05 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.669139+0000", 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021946619000000001 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:17.669180+0000", 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1248999999999998e-05 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.127 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149380 10.3 e255/242 10:c6822046:::benchmark_data_smithi003_197468_object32651:head v 255'9391, mlcod=238'1751)", 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.297146+0000", 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 508.80178773799997, 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34267334700000002, 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297146+0000", 2024-09-13T14:58:16.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297146+0000", 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297147+0000", 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3900000000000004e-07 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297152+0000", 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2170000000000002e-06 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297161+0000", 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7830000000000004e-06 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.297167+0000", 2024-09-13T14:58:16.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2960000000000004e-06 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299041+0000", 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018741980000000001 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.299067+0000", 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5769e-05 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.639779+0000", 2024-09-13T14:58:16.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34071200499999998 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.639820+0000", 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0439999999999999e-05 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149384 10.3 e255/242 10:c87a8b94:::benchmark_data_smithi003_197468_object32655:head v 255'9392, mlcod=238'1751)", 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.305345+0000", 2024-09-13T14:58:16.132 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 508.79358907800002, 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33450042699999999, 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305345+0000", 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305345+0000", 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305346+0000", 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.012e-06 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305353+0000", 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1930000000000004e-06 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305365+0000", 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1702000000000001e-05 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305373+0000", 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8620000000000003e-06 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305428+0000", 2024-09-13T14:58:16.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5164999999999998e-05 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.305450+0000", 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2300999999999999e-05 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.639825+0000", 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.334374738 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.639845+0000", 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0454000000000001e-05 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149393 10.4 e255/229 10:29621cc7:::benchmark_data_smithi003_197468_object32664:head v 255'9294, mlcod=255'9292)", 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.308254+0000", 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 508.79068024899999, 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33786957400000001, 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.137 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308254+0000", 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308254+0000", 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308255+0000", 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7499999999999998e-07 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308261+0000", 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0929999999999999e-06 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308272+0000", 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1294999999999999e-05 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308280+0000", 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5050000000000001e-06 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308435+0000", 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000155883 2024-09-13T14:58:16.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308455+0000", 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9675000000000001e-05 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646079+0000", 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33762412800000002 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.646123+0000", 2024-09-13T14:58:16.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4020000000000002e-05 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149388 10.7 e255/249 10:ea963fcf:::benchmark_data_smithi003_197468_object32659:head v 255'9240, mlcod=0'0)", 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.308305+0000", 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 508.79062905500001, 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.333896784, 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308305+0000", 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308305+0000", 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308305+0000", 2024-09-13T14:58:16.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.08e-07 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308309+0000", 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5750000000000001e-06 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308316+0000", 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6579999999999996e-06 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308320+0000", 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3630000000000004e-06 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309186+0000", 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00086598100000000004 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.309209+0000", 2024-09-13T14:58:16.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2806000000000002e-05 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642156+0000", 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33294702599999998 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642202+0000", 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5967000000000002e-05 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149395 10.f e255/229 10:ff0ca942:::benchmark_data_smithi003_197468_object32666:head v 255'9241, mlcod=255'9239)", 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.308370+0000", 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 508.79056361800002, 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.333858402, 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308370+0000", 2024-09-13T14:58:16.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308370+0000", 2024-09-13T14:58:16.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:16.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308371+0000", 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4299999999999998e-07 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308375+0000", 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2119999999999997e-06 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308383+0000", 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8269999999999998e-06 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.308388+0000", 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2089999999999996e-06 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.632819+0000", 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32443107599999998 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.632844+0000", 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4610999999999999e-05 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642205+0000", 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0093610589999999997 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.642229+0000", 2024-09-13T14:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3964999999999998e-05 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36168.0:149387 10.5 e255/229 10:a0bfa116:::benchmark_data_smithi003_197468_object32658:head v 255'9349, mlcod=255'9346)", 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:49:47.312818+0000", 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 508.78611641399999, 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.331070115, 2024-09-13T14:58:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.312818+0000", 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.312818+0000", 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.312818+0000", 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.569999999999999e-07 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.312826+0000", 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3000000000000004e-06 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.633848+0000", 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.32102176399999999 2024-09-13T14:58:16.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.633866+0000", 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8142999999999999e-05 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.633934+0000", 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8182999999999994e-05 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.633950+0000", 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5962999999999999e-05 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.643801+0000", 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098514090000000002 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:49:47.643888+0000", 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6396000000000004e-05 2024-09-13T14:58:16.157 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:23153 11.0 11:03cd4422:::benchmark_data_smithi003_229782_object23152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:52:09.695207+0000", 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 366.403726699, 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27937496499999998, 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:58:16.158 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23153 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.695207+0000", 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.695207+0000", 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.695208+0000", 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4199999999999995e-07 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.695228+0000", 2024-09-13T14:58:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0112e-05 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.806793+0000", 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.111564738 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.806804+0000", 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1537e-05 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.807428+0000", 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00062398 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.807565+0000", 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013632100000000001 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.807695+0000", 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013002000000000001 2024-09-13T14:58:16.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.811115+0000", 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034201460000000002 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.811164+0000", 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8902e-05 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.974492+0000", 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16332833499999999 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.974542+0000", 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0024000000000001e-05 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:52:09.974582+0000", 2024-09-13T14:58:16.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0207999999999999e-05 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:211637 11.b 11:d48e3873:::benchmark_data_smithi003_229782_object89257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.888486+0000", 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 190.210447835, 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.45271259000000003, 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.165 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 211637 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888486+0000", 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888486+0000", 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888488+0000", 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0719999999999998e-06 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888495+0000", 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0899999999999999e-06 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888517+0000", 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2178e-05 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.888527+0000", 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5289999999999997e-06 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893270+0000", 2024-09-13T14:58:16.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047428959999999999 2024-09-13T14:58:16.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893401+0000", 2024-09-13T14:58:16.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013092099999999999 2024-09-13T14:58:16.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.893523+0000", 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012244799999999999 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899806+0000", 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006282625 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899844+0000", 2024-09-13T14:58:16.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7957000000000003e-05 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.341048+0000", 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44120459499999998 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.341122+0000", 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3960000000000003e-05 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.341199+0000", 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6318999999999999e-05 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:211648 11.0 11:011e7020:::benchmark_data_smithi003_229782_object89268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.898155+0000", 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 190.200779071, 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.43924299999999999, 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.172 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 211648 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898155+0000", 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898155+0000", 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898156+0000", 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1800000000000005e-07 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898159+0000", 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7100000000000001e-06 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898169+0000", 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.166e-06 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.898174+0000", 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1619999999999999e-06 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.905754+0000", 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075797750000000004 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.905865+0000", 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011133000000000001 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.906402+0000", 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053722100000000005 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.913016+0000", 2024-09-13T14:58:16.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006614389 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.913100+0000", 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3379000000000002e-05 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.337269+0000", 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.42416935700000002 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.337346+0000", 2024-09-13T14:58:16.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6363000000000001e-05 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.337398+0000", 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2330000000000002e-05 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:211652 11.b 11:dd38e0a4:::benchmark_data_smithi003_229782_object89272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:05.899777+0000", 2024-09-13T14:58:16.178 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 190.199157133, 2024-09-13T14:58:16.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44725128199999997, 2024-09-13T14:58:16.179 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.179 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:58:16.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:58:16.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:16.179 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 211652 2024-09-13T14:58:16.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.180 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899777+0000", 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899777+0000", 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899778+0000", 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1119999999999999e-06 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899783+0000", 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5310000000000001e-06 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899796+0000", 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2126999999999999e-05 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.899804+0000", 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9980000000000003e-06 2024-09-13T14:58:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900561+0000", 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00075782300000000003 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900706+0000", 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014433100000000001 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:58:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.900833+0000", 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012682500000000001 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.904800+0000", 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003967567 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:05.909921+0000", 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051206170000000001 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.346915+0000", 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.43699400100000002 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.346981+0000", 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6519000000000005e-05 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:06.347028+0000", 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6830999999999999e-05 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:235433 11.b 11:df5b6600:::benchmark_data_smithi003_229782_object113053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:33.147681+0000", 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 162.951253259, 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.272995561, 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:58:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 235433 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.147681+0000", 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.147681+0000", 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.147682+0000", 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.263e-06 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.147687+0000", 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3410000000000002e-06 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.147703+0000", 2024-09-13T14:58:16.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5480000000000001e-05 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.147710+0000", 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4959999999999999e-06 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.147762+0000", 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1978000000000003e-05 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.147831+0000", 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8594000000000001e-05 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.147930+0000", 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9075999999999998e-05 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.151374+0000", 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003444141 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.154765+0000", 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003390891 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.420577+0000", 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26581239000000001 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.420635+0000", 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7431e-05 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.420676+0000", 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1480000000000003e-05 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:235436 11.0 11:0041c16f:::benchmark_data_smithi003_229782_object113056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:55:33.149293+0000", 2024-09-13T14:58:16.192 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 162.94964078999999, 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.25398980999999998, 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 235436 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.149293+0000", 2024-09-13T14:58:16.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.149293+0000", 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.149294+0000", 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0669999999999999e-06 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.149299+0000", 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6859999999999997e-06 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.149315+0000", 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5659999999999999e-05 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.149322+0000", 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7729999999999999e-06 2024-09-13T14:58:16.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.149402+0000", 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9485000000000002e-05 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.149521+0000", 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119388 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.150039+0000", 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00051747700000000002 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.155806+0000", 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057673159999999998 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.157086+0000", 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012796140000000001 2024-09-13T14:58:16.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.403021+0000", 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24593572 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.403149+0000", 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012765399999999999 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:55:33.403283+0000", 2024-09-13T14:58:16.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013396999999999999 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:75469 12.8 12:1f64234b:::benchmark_data_smithi003_259359_object75468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:57:06.913947+0000", 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 69.184987105000005, 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.25449888500000001, 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.199 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75469 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.913947+0000", 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.913947+0000", 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.913949+0000", 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.688e-06 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.913972+0000", 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3074e-05 2024-09-13T14:58:16.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.913984+0000", 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2599000000000001e-05 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.913992+0000", 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7789999999999996e-06 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.914862+0000", 2024-09-13T14:58:16.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00086965100000000004 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.915022+0000", 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016050600000000001 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.915168+0000", 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014538100000000001 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.922683+0000", 2024-09-13T14:58:16.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075152559999999997 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:06.922782+0000", 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9223999999999997e-05 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:07.168292+0000", 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.245509744 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:07.168398+0000", 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106394 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:57:07.168446+0000", 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7589000000000001e-05 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129096 12.8 12:15f7dc4c:::benchmark_data_smithi003_259359_object13470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T14:58:16.205 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.516501+0000", 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.5824333939999997, 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2720113149999999, 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129096 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516501+0000", 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516501+0000", 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516502+0000", 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8470000000000001e-06 2024-09-13T14:58:16.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516508+0000", 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1420000000000001e-06 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516521+0000", 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2961e-05 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516530+0000", 2024-09-13T14:58:16.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5359999999999998e-06 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516935+0000", 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00040524499999999999 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516946+0000", 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0339999999999999e-05 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.504919+0000", 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98797316099999999 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.504934+0000", 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4715000000000001e-05 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.766100+0000", 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.261166786 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.766220+0000", 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119614 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.766708+0000", 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00048800500000000001 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.782160+0000", 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015452097 2024-09-13T14:58:16.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788271+0000", 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061114000000000003 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788310+0000", 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8896999999999997e-05 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788347+0000", 2024-09-13T14:58:16.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6692e-05 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788403+0000", 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6465000000000002e-05 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788512+0000", 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108412 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129111 12.8 12:149496a4:::benchmark_data_smithi003_259359_object13485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.535225+0000", 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.5637088019999998, 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2598992739999999, 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129111 2024-09-13T14:58:16.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535225+0000", 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535225+0000", 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T14:58:16.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535226+0000", 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3209999999999999e-06 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535233+0000", 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4169999999999997e-06 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535247+0000", 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.393e-05 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535254+0000", 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0829999999999998e-06 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535319+0000", 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5075999999999998e-05 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535327+0000", 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0660000000000004e-06 2024-09-13T14:58:16.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.506143+0000", 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97081633899999997 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.506153+0000", 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3030000000000005e-06 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.775368+0000", 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2692155810000001 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.775502+0000", 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013353799999999999 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.775635+0000", 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133282 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.785399+0000", 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0097642519999999993 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788571+0000", 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031718509999999998 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788631+0000", 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9475000000000002e-05 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.794958+0000", 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063276979999999997 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.795020+0000", 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1444000000000003e-05 2024-09-13T14:58:16.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.795124+0000", 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010461799999999999 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T14:58:16.222 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:58:16.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:16.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:16.394 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:16.427 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:16 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 131/302259 objects degraded (0.043%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:16 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:58:16.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:16 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 131/302259 objects degraded (0.043%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:16 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:58:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:16 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 131/302259 objects degraded (0.043%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:16 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:58:16.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:16.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:16.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:16.700 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:16.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:16.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:16.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:16.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:17.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:17.138 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:58:17.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:17.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:17.308 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:17.410 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:17 smithi019 ceph-mon[91296]: pgmap v1185: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 511 op/s; 131/302259 objects degraded (0.043%); 36600/302259 objects misplaced (12.109%); 8.8 MiB/s, 142 objects/s recovering 2024-09-13T14:58:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:17 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3427714636' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:17 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:17 smithi003 ceph-mon[107278]: pgmap v1185: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 511 op/s; 131/302259 objects degraded (0.043%); 36600/302259 objects misplaced (12.109%); 8.8 MiB/s, 142 objects/s recovering 2024-09-13T14:58:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:17 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3427714636' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:17 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:17 smithi003 ceph-mon[104007]: pgmap v1185: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 511 op/s; 131/302259 objects degraded (0.043%); 36600/302259 objects misplaced (12.109%); 8.8 MiB/s, 142 objects/s recovering 2024-09-13T14:58:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:17 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3427714636' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:17 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:17.443 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:17.545 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.583 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:17.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:17.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:17.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:17.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:17.862 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:17.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.994 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:17.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:18.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:18.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:18.243 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:18.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.279 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:18.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:18.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:18.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:18.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:18 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:18.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:18 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:18.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:18 smithi019 ceph-mon[91296]: osdmap e337: 8 total, 8 up, 7 in 2024-09-13T14:58:18.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:18 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:18.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:18 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:18.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:18 smithi003 ceph-mon[107278]: osdmap e337: 8 total, 8 up, 7 in 2024-09-13T14:58:18.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:18 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:18.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:18 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:18.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:18 smithi003 ceph-mon[104007]: osdmap e337: 8 total, 8 up, 7 in 2024-09-13T14:58:18.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:18.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:18.659 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.695 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:18.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:18.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:18.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:18.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:19.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:19.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:19.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:58:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:19.221 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:19.276 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:19.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:19.366 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:19 smithi003 ceph-mon[107278]: pgmap v1187: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 131/300669 objects degraded (0.044%); 35674/300669 objects misplaced (11.865%); 13 MiB/s, 217 objects/s recovering 2024-09-13T14:58:19.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:19 smithi003 ceph-mon[104007]: pgmap v1187: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 131/300669 objects degraded (0.044%); 35674/300669 objects misplaced (11.865%); 13 MiB/s, 217 objects/s recovering 2024-09-13T14:58:19.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:19.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:19.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:19.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:19 smithi019 ceph-mon[91296]: pgmap v1187: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 131/300669 objects degraded (0.044%); 35674/300669 objects misplaced (11.865%); 13 MiB/s, 217 objects/s recovering 2024-09-13T14:58:19.519 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.560 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:19.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:19.580 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:19.661 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:19.695 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:19.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:19.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:19.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:19.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:19.971 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:20.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:20.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:20.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:20.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:20.356 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:20.392 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:20.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:20.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:20.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:20.638 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:20.671 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:58:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:58:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:20.674 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:20.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.816 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:20.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:20.917 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:21.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:21.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:21.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:21.251 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:21 smithi003 ceph-mon[107278]: pgmap v1188: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 131/291216 objects degraded (0.045%); 31999/291216 objects misplaced (10.988%); 32 MiB/s, 510 objects/s recovering 2024-09-13T14:58:21.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:21 smithi003 ceph-mon[104007]: pgmap v1188: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 131/291216 objects degraded (0.045%); 31999/291216 objects misplaced (10.988%); 32 MiB/s, 510 objects/s recovering 2024-09-13T14:58:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:21.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:21.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:21.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:21 smithi019 ceph-mon[91296]: pgmap v1188: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 131/291216 objects degraded (0.045%); 31999/291216 objects misplaced (10.988%); 32 MiB/s, 510 objects/s recovering 2024-09-13T14:58:21.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:21.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:21.658 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:21.706 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:21.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:21.708 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:58:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:21.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:58:21.743 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:58:21.743 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:21.763 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:58:21.763 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:21.807 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:21.809 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:58:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:21.839 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:58:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.844 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:21.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:21.868 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:58:21.869 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:21.911 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:58:21.912 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:21.946 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:21.971 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:58:21.971 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:21.980 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:22.077 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:58:22.077 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:58:22.077 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T14:58:22.078 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.533365 2024-09-13T14:58:22.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 0.5333645897547002 2024-09-13T14:58:22.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:22.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:22.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.313 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:22.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:22.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1908570236' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:22.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1908570236' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:22.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1908570236' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:22.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:22.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:22.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:22.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:22.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:22.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:22.882 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:22.919 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:22.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:23.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:23.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:23.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:23.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:23.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:23.314 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:23 smithi003 ceph-mon[107278]: pgmap v1189: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 707 op/s; 325/289602 objects degraded (0.112%); 31738/289602 objects misplaced (10.959%); 27 MiB/s, 440 objects/s recovering 2024-09-13T14:58:23.314 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:23 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 325/289602 objects degraded (0.112%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:23.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:23 smithi003 ceph-mon[104007]: pgmap v1189: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 707 op/s; 325/289602 objects degraded (0.112%); 31738/289602 objects misplaced (10.959%); 27 MiB/s, 440 objects/s recovering 2024-09-13T14:58:23.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:23 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 325/289602 objects degraded (0.112%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:23.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:23.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:23 smithi019 ceph-mon[91296]: pgmap v1189: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 707 op/s; 325/289602 objects degraded (0.112%); 31738/289602 objects misplaced (10.959%); 27 MiB/s, 440 objects/s recovering 2024-09-13T14:58:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:23 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 325/289602 objects degraded (0.112%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:23.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:23.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:23.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:23.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:23.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:23.802 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:23.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:24.053 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:24.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.088 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:24.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:24.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:58:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:58:24] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:58:24.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:24.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:24.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:24.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:24.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:24.499 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:24.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:24.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:24.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:24.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:24.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:58:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.789 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:24.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:24.890 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.925 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:24.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:25.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:25.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:25.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:25.241 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:25.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.386 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:25.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:25.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:25 smithi003 ceph-mon[107278]: pgmap v1190: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 325/278535 objects degraded (0.117%); 27552/278535 objects misplaced (9.892%); 41 MiB/s, 649 objects/s recovering 2024-09-13T14:58:25.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:25 smithi003 ceph-mon[104007]: pgmap v1190: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 325/278535 objects degraded (0.117%); 27552/278535 objects misplaced (9.892%); 41 MiB/s, 649 objects/s recovering 2024-09-13T14:58:25.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:25.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:25.602 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:25.602 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:25.606 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:25 smithi019 ceph-mon[91296]: pgmap v1190: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 325/278535 objects degraded (0.117%); 27552/278535 objects misplaced (9.892%); 41 MiB/s, 649 objects/s recovering 2024-09-13T14:58:25.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:25.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:25.775 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.816 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:25.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:25.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:26.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:26.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:26.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:26.256 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:26.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:58:26.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.407 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:26.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:26.508 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.553 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:26.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:26.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:26.696 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:26.754 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:26.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:26.830 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:26.932 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:26.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:26.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:27.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:27.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:27.221 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:27.256 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:27.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:27.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:27.374 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:58:27.384 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.5 primary-affinity to 0.533365 (8349542) 2024-09-13T14:58:27.393 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:27.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:27.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:27.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:27.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:27.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:27 smithi019 ceph-mon[91296]: pgmap v1191: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 838 op/s; 519/277071 objects degraded (0.187%); 27294/277071 objects misplaced (9.851%); 41 MiB/s, 651 objects/s recovering 2024-09-13T14:58:27.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:27 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2380399431' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5333645897547002} : dispatch 2024-09-13T14:58:27.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:27 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5333645897547002} : dispatch 2024-09-13T14:58:27.666 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:27 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3443066260' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:27.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:27 smithi003 ceph-mon[104007]: pgmap v1191: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 838 op/s; 519/277071 objects degraded (0.187%); 27294/277071 objects misplaced (9.851%); 41 MiB/s, 651 objects/s recovering 2024-09-13T14:58:27.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:27 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2380399431' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5333645897547002} : dispatch 2024-09-13T14:58:27.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:27 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5333645897547002} : dispatch 2024-09-13T14:58:27.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:27 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3443066260' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:27 smithi003 ceph-mon[107278]: pgmap v1191: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 838 op/s; 519/277071 objects degraded (0.187%); 27294/277071 objects misplaced (9.851%); 41 MiB/s, 651 objects/s recovering 2024-09-13T14:58:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:27 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2380399431' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5333645897547002} : dispatch 2024-09-13T14:58:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:27 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5333645897547002} : dispatch 2024-09-13T14:58:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:27 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3443066260' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:27.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:27.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:27.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:27.917 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:28.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:28.044 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:28.146 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:28.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:28.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.401 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:28.426 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5333645897547002}]': finished 2024-09-13T14:58:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3443066260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[104007]: osdmap e338: 8 total, 8 up, 7 in 2024-09-13T14:58:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5333645897547002}]': finished 2024-09-13T14:58:28.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3443066260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:28.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[107278]: osdmap e338: 8 total, 8 up, 7 in 2024-09-13T14:58:28.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:28.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:28.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:28.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:28.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:28 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:28 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5333645897547002}]': finished 2024-09-13T14:58:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3443066260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:28 smithi019 ceph-mon[91296]: osdmap e338: 8 total, 8 up, 7 in 2024-09-13T14:58:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:28.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.811 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:28.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:28.912 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:28.938 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:29.064 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:29.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:29.191 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:29.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:29.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:29.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:29.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:29.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:29.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:29.662 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:29 smithi003 ceph-mon[104007]: pgmap v1193: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 519/275436 objects degraded (0.188%); 26498/275436 objects misplaced (9.620%); 40 MiB/s, 642 objects/s recovering 2024-09-13T14:58:29.662 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:29 smithi003 ceph-mon[104007]: osdmap e339: 8 total, 8 up, 7 in 2024-09-13T14:58:29.662 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:29 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 519/275436 objects degraded (0.188%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:29.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:29 smithi003 ceph-mon[107278]: pgmap v1193: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 519/275436 objects degraded (0.188%); 26498/275436 objects misplaced (9.620%); 40 MiB/s, 642 objects/s recovering 2024-09-13T14:58:29.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:29 smithi003 ceph-mon[107278]: osdmap e339: 8 total, 8 up, 7 in 2024-09-13T14:58:29.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:29 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 519/275436 objects degraded (0.188%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:29 smithi019 ceph-mon[91296]: pgmap v1193: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 113 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 519/275436 objects degraded (0.188%); 26498/275436 objects misplaced (9.620%); 40 MiB/s, 642 objects/s recovering 2024-09-13T14:58:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:29 smithi019 ceph-mon[91296]: osdmap e339: 8 total, 8 up, 7 in 2024-09-13T14:58:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:29 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 519/275436 objects degraded (0.188%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:29.695 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:29.725 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:29.860 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:29.962 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:29.991 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:30.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:30.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:30.236 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.268 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:30.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:30.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:30.379 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:58:30.379 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:30.391 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:30.402 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:58:30.402 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:30.442 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:58:30.442 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:30.482 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:58:30.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:30.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:30.515 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:58:30.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:30.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:30.546 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:58:30.546 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:30.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:30 smithi003 ceph-mon[107278]: 12.c continuing backfill to osd.7 from (314'1095,337'8890] 12:3aac1103:::benchmark_data_smithi003_259359_object46515:head to 337'8890 2024-09-13T14:58:30.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:30 smithi003 ceph-mon[107278]: osdmap e340: 8 total, 8 up, 7 in 2024-09-13T14:58:30.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:30 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3589624964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:30.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:30 smithi003 ceph-mon[104007]: 12.c continuing backfill to osd.7 from (314'1095,337'8890] 12:3aac1103:::benchmark_data_smithi003_259359_object46515:head to 337'8890 2024-09-13T14:58:30.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:30 smithi003 ceph-mon[104007]: osdmap e340: 8 total, 8 up, 7 in 2024-09-13T14:58:30.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:30 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3589624964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:30.578 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:58:30.578 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:30.610 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:58:30.610 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:30.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:30.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:30 smithi019 ceph-mon[91296]: 12.c continuing backfill to osd.7 from (314'1095,337'8890] 12:3aac1103:::benchmark_data_smithi003_259359_object46515:head to 337'8890 2024-09-13T14:58:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:30 smithi019 ceph-mon[91296]: osdmap e340: 8 total, 8 up, 7 in 2024-09-13T14:58:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:30 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3589624964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:58:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:58:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:30.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:30.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:30.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:30.958 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:31.060 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:31.082 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:31.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:31.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:31.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:58:31.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:31.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:31.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:31.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:31.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:31.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:31.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:31 smithi019 ceph-mon[91296]: pgmap v1196: 119 pgs: 1 remapped+peering, 4 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 109 active+clean; 5.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 519/267150 objects degraded (0.194%); 21381/267150 objects misplaced (8.003%); 33 MiB/s, 534 objects/s recovering 2024-09-13T14:58:31.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:31 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:58:31.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:31 smithi003 ceph-mon[107278]: pgmap v1196: 119 pgs: 1 remapped+peering, 4 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 109 active+clean; 5.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 519/267150 objects degraded (0.194%); 21381/267150 objects misplaced (8.003%); 33 MiB/s, 534 objects/s recovering 2024-09-13T14:58:31.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:31 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:58:31.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:31 smithi003 ceph-mon[104007]: pgmap v1196: 119 pgs: 1 remapped+peering, 4 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 109 active+clean; 5.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 519/267150 objects degraded (0.194%); 21381/267150 objects misplaced (8.003%); 33 MiB/s, 534 objects/s recovering 2024-09-13T14:58:31.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:31 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:58:31.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:31.808 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:31.910 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:31.923 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:31.937 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:32.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:32.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:32.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:32.193 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:32.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:32.295 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:32.318 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:32.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:32.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:32.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:32.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:32.674 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:32.697 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:32.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:32.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:32.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:32.958 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:33.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:33.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:32 smithi019 ceph-mon[91296]: osdmap e341: 8 total, 8 up, 7 in 2024-09-13T14:58:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:32 smithi019 ceph-mon[91296]: pgmap v1198: 119 pgs: 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 109 active+clean; 5.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 588/265992 objects degraded (0.221%); 22962/265992 objects misplaced (8.633%); 44 MiB/s, 708 objects/s recovering 2024-09-13T14:58:33.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:32 smithi003 ceph-mon[107278]: osdmap e341: 8 total, 8 up, 7 in 2024-09-13T14:58:33.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:32 smithi003 ceph-mon[107278]: pgmap v1198: 119 pgs: 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 109 active+clean; 5.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 588/265992 objects degraded (0.221%); 22962/265992 objects misplaced (8.633%); 44 MiB/s, 708 objects/s recovering 2024-09-13T14:58:33.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:32 smithi003 ceph-mon[104007]: osdmap e341: 8 total, 8 up, 7 in 2024-09-13T14:58:33.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:32 smithi003 ceph-mon[104007]: pgmap v1198: 119 pgs: 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 109 active+clean; 5.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 588/265992 objects degraded (0.221%); 22962/265992 objects misplaced (8.633%); 44 MiB/s, 708 objects/s recovering 2024-09-13T14:58:33.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:33.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:33.339 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:33.370 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:33.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:33.504 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:33.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:33.606 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:33.640 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:33.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:33.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:33.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:58:33.871 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:33 smithi003 ceph-mon[104007]: osdmap e342: 8 total, 8 up, 7 in 2024-09-13T14:58:33.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:33 smithi003 ceph-mon[107278]: osdmap e342: 8 total, 8 up, 7 in 2024-09-13T14:58:33.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:33.914 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:33 smithi019 ceph-mon[91296]: osdmap e342: 8 total, 8 up, 7 in 2024-09-13T14:58:34.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:34.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:34.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:34.185 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:58:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:58:34] "GET /metrics HTTP/1.1" 200 36555 "" "Prometheus/2.51.0" 2024-09-13T14:58:34.193 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:34.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:34.308 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:34.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:34.441 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:34.471 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:34.573 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:34.605 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:34.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:34.707 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:58:34.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:58:34.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:58:34.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:34.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:34.884 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:58:34.884 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:58:34.884 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:58:34.884 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:58:34.884 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:58:34.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:35.013 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:35.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:35.148 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:35.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:34 smithi003 ceph-mon[104007]: pgmap v1200: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+remapped+backfilling, 114 active+clean; 5.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 372 op/s; 588/260865 objects degraded (0.225%); 19727/260865 objects misplaced (7.562%); 34 MiB/s, 543 objects/s recovering 2024-09-13T14:58:35.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:34 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/24275209' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:35.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:34 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:35.169 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:34 smithi003 ceph-mon[107278]: pgmap v1200: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+remapped+backfilling, 114 active+clean; 5.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 372 op/s; 588/260865 objects degraded (0.225%); 19727/260865 objects misplaced (7.562%); 34 MiB/s, 543 objects/s recovering 2024-09-13T14:58:35.169 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:34 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/24275209' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:35.169 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:34 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:35.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:34 smithi019 ceph-mon[91296]: pgmap v1200: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+remapped+backfilling, 114 active+clean; 5.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 372 op/s; 588/260865 objects degraded (0.225%); 19727/260865 objects misplaced (7.562%); 34 MiB/s, 543 objects/s recovering 2024-09-13T14:58:35.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:34 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/24275209' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:35.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:34 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:35.249 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:35.279 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:35.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:35.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:35.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:35.556 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:58:35.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:35.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:35.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:35.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:35.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:35.822 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:35 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:35.822 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 588/260865 objects degraded (0.225%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:35.822 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:35 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:58:35.822 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:35 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:35.822 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:35 smithi019 ceph-mon[91296]: osdmap e343: 8 total, 8 up, 7 in 2024-09-13T14:58:35.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:35.936 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:35.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.985 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:35.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:36.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:36.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:36.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:36.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 588/260865 objects degraded (0.225%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:36.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:58:36.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:36.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[107278]: osdmap e343: 8 total, 8 up, 7 in 2024-09-13T14:58:36.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:36.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 588/260865 objects degraded (0.225%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:36.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:58:36.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:36.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:35 smithi003 ceph-mon[104007]: osdmap e343: 8 total, 8 up, 7 in 2024-09-13T14:58:36.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:36.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:36.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:36.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:36.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:36.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:36.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:36.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:36.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:36.911 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:36.939 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:37.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:37.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:37.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:36 smithi003 ceph-mon[104007]: pgmap v1202: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+remapped+backfilling, 114 active+clean; 5.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 456 op/s; 599/258936 objects degraded (0.231%); 19544/258936 objects misplaced (7.548%); 33 MiB/s, 527 objects/s recovering 2024-09-13T14:58:37.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:36 smithi003 ceph-mon[107278]: pgmap v1202: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+remapped+backfilling, 114 active+clean; 5.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 456 op/s; 599/258936 objects degraded (0.231%); 19544/258936 objects misplaced (7.548%); 33 MiB/s, 527 objects/s recovering 2024-09-13T14:58:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:36 smithi019 ceph-mon[91296]: pgmap v1202: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+remapped+backfilling, 114 active+clean; 5.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 456 op/s; 599/258936 objects degraded (0.231%); 19544/258936 objects misplaced (7.548%); 33 MiB/s, 527 objects/s recovering 2024-09-13T14:58:37.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:37.208 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:37.310 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:37.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:37.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:37.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:37.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:37.611 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:37.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:37.855 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:37.896 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:37.959 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:58:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T14:58:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T14:58:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:211567 10.2 10:4f67d55c:::benchmark_data_smithi003_197468_object94838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:50:59.069429+0000", 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 458.886030097, 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.208739855, 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:58:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211567 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069429+0000", 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069429+0000", 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069429+0000", 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069431+0000", 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.599e-06 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069439+0000", 2024-09-13T14:58:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5079999999999999e-06 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069443+0000", 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8600000000000003e-06 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069490+0000", 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7778999999999998e-05 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069592+0000", 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101412 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.069710+0000", 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118328 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073241+0000", 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0035303719999999999 2024-09-13T14:58:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.077160+0000", 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039191169999999997 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.278063+0000", 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.20090324400000001 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.278122+0000", 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9210999999999999e-05 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.278169+0000", 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6844000000000001e-05 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:211573 10.2 10:49dbb05f:::benchmark_data_smithi003_197468_object94844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-13T14:58:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:50:59.073455+0000", 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 458.88200446299999, 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21269655600000001, 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211573 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073455+0000", 2024-09-13T14:58:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073455+0000", 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073453+0000", 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073459+0000", 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8039999999999998e-06 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073469+0000", 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8859999999999999e-06 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073474+0000", 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1320000000000002e-06 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073519+0000", 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4366000000000002e-05 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073624+0000", 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000105553 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.073732+0000", 2024-09-13T14:58:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000107787 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.077346+0000", 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036141430000000002 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.079713+0000", 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0023672340000000002 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.286072+0000", 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.206358184 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.286116+0000", 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4048000000000002e-05 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:50:59.286151+0000", 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5629000000000001e-05 2024-09-13T14:58:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:203628 11.e 11:7a291014:::benchmark_data_smithi003_229782_object81248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:54:56.790582+0000", 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 221.16487738500001, 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.105247854, 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203628 2024-09-13T14:58:37.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.974 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:37.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:37.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.790582+0000", 2024-09-13T14:58:37.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:37.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.790582+0000", 2024-09-13T14:58:37.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.790580+0000", 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.790588+0000", 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1419999999999993e-06 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.790606+0000", 2024-09-13T14:58:37.976 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7844999999999999e-05 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.790613+0000", 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1629999999999999e-06 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.790658+0000", 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5467999999999999e-05 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:37.977 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.790779+0000", 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120506 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.790901+0000", 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012238 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.794853+0000", 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039511329999999999 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.794893+0000", 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0516999999999998e-05 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.895737+0000", 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.100844076 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.895787+0000", 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0256000000000001e-05 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:54:56.895830+0000", 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2265999999999998e-05 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211647 11.e 11:743fb1be:::benchmark_data_smithi003_229782_object89267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.898191+0000", 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 212.05726814100001, 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.45462776399999999, 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:37.980 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211647 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.898191+0000", 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:37.981 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.898191+0000", 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.898190+0000", 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.898196+0000", 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.37e-06 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.898208+0000", 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2143e-05 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.898215+0000", 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5910000000000001e-06 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.898263+0000", 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8207000000000001e-05 2024-09-13T14:58:37.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.898341+0000", 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8108999999999994e-05 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.898410+0000", 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8639000000000005e-05 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:37.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.901883+0000", 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0034727970000000001 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.904024+0000", 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0021418180000000002 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.352710+0000", 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.44868571099999999 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.352775+0000", 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5029000000000004e-05 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.352819+0000", 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3650999999999999e-05 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.986 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211660 11.e 11:7f5d2e6d:::benchmark_data_smithi003_229782_object89280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.911225+0000", 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 212.044234171, 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.44174664499999999, 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211660 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:37.987 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.911225+0000", 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.911225+0000", 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.911223+0000", 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:58:37.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.911234+0000", 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1129e-05 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.911252+0000", 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8074e-05 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:37.989 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.911259+0000", 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4509999999999997e-06 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.911307+0000", 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8004000000000003e-05 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.911400+0000", 2024-09-13T14:58:37.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3000999999999999e-05 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.911476+0000", 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6744999999999998e-05 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.915491+0000", 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0040142370000000004 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.915697+0000", 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00020618899999999999 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.352867+0000", 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.43717051699999998 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.352920+0000", 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2259000000000002e-05 2024-09-13T14:58:37.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.352972+0000", 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1943000000000001e-05 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211665 11.e 11:7d5c94dc:::benchmark_data_smithi003_229782_object89285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.918099+0000", 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 212.037359783, 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.43499264100000001, 2024-09-13T14:58:37.993 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211665 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.918099+0000", 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.918099+0000", 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.918098+0000", 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:37.995 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.918106+0000", 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2840000000000006e-06 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.918122+0000", 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5936000000000001e-05 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.918130+0000", 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7710000000000006e-06 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.996 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.918181+0000", 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1366999999999998e-05 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.918324+0000", 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014221200000000001 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.918472+0000", 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014788900000000001 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.922374+0000", 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039023959999999998 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.922390+0000", 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6220999999999999e-05 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353017+0000", 2024-09-13T14:58:37.998 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.43062655999999999 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353060+0000", 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2966000000000002e-05 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353092+0000", 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2345000000000003e-05 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:37.999 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:235426 11.e 11:7748e745:::benchmark_data_smithi003_229782_object113046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:33.141220+0000", 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 184.81423890100001, 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26063872799999999, 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 235426 2024-09-13T14:58:38.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.141220+0000", 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.141220+0000", 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.001 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.141218+0000", 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.141229+0000", 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0567000000000001e-05 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.141255+0000", 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6645e-05 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.141264+0000", 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3000000000000002e-06 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.141332+0000", 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8380999999999995e-05 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.141465+0000", 2024-09-13T14:58:38.003 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013320000000000001 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.141585+0000", 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012000100000000001 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.145580+0000", 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039949130000000001 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.004 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150139+0000", 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0045592239999999997 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.401787+0000", 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25164736500000001 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.401830+0000", 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3696000000000003e-05 2024-09-13T14:58:38.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.401859+0000", 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8568000000000002e-05 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:235431 11.e 11:7b5ea85a:::benchmark_data_smithi003_229782_object113051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:33.146077+0000", 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 184.80938171, 2024-09-13T14:58:38.006 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26874714900000002, 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 235431 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.146077+0000", 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.146077+0000", 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.146076+0000", 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.146084+0000", 2024-09-13T14:58:38.008 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5720000000000003e-06 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.146101+0000", 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6954999999999999e-05 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.146108+0000", 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4110000000000001e-06 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.146153+0000", 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4929999999999998e-05 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.146261+0000", 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000108373 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.146381+0000", 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120485 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150224+0000", 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0038421420000000002 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150672+0000", 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00044860700000000001 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.011 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.414729+0000", 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26405665499999997 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.414783+0000", 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4107000000000002e-05 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.414825+0000", 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.163e-05 2024-09-13T14:58:38.012 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:235452 11.e 11:791d1d89:::benchmark_data_smithi003_229782_object113072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:33.165347+0000", 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 184.79011263300001, 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24962479100000001, 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:58:38.013 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 235452 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.165347+0000", 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.165347+0000", 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.014 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.165345+0000", 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.165355+0000", 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0281e-05 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.165376+0000", 2024-09-13T14:58:38.015 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0370999999999999e-05 2024-09-13T14:58:38.016 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.016 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.016 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.016 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.165383+0000", 2024-09-13T14:58:38.016 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8779999999999998e-06 2024-09-13T14:58:38.016 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.016 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.016 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.016 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.165423+0000", 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9920999999999999e-05 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.165530+0000", 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106465 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.165652+0000", 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000122638 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.018 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.018 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.168837+0000", 2024-09-13T14:58:38.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0031846589999999998 2024-09-13T14:58:38.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.018 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.018 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.169504+0000", 2024-09-13T14:58:38.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00066724600000000003 2024-09-13T14:58:38.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:38.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.414870+0000", 2024-09-13T14:58:38.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.245365425 2024-09-13T14:58:38.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.414931+0000", 2024-09-13T14:58:38.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0921000000000002e-05 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.414971+0000", 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0652000000000003e-05 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:23363 12.0 12:0f203484:::benchmark_data_smithi003_259359_object23362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:56:31.107675+0000", 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 126.847783972, 2024-09-13T14:58:38.021 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.111718478, 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 23363 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.107675+0000", 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.022 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.107675+0000", 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.107674+0000", 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.107717+0000", 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3739999999999998e-05 2024-09-13T14:58:38.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.107730+0000", 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2796000000000001e-05 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.107737+0000", 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5910000000000001e-06 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.024 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.107780+0000", 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3528000000000001e-05 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.107904+0000", 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000123175 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.107999+0000", 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5229000000000006e-05 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.025 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110229+0000", 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022303380000000001 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.111521+0000", 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001292315 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.219310+0000", 2024-09-13T14:58:38.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.10778815899999999 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.219356+0000", 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6406999999999998e-05 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.219394+0000", 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7657000000000002e-05 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:23366 12.0 12:03e2ea83:::benchmark_data_smithi003_259359_object23365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:56:31.108433+0000", 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 126.847025746, 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.111143108, 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 23366 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.108433+0000", 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.108433+0000", 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.108432+0000", 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.108494+0000", 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1107999999999998e-05 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.108506+0000", 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2108e-05 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.108511+0000", 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7450000000000001e-06 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.108967+0000", 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00045608200000000002 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.109086+0000", 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011868 2024-09-13T14:58:38.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.109218+0000", 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013162300000000001 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.111266+0000", 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0020485669999999998 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.032 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.112277+0000", 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001011023 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.219493+0000", 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.10721536399999999 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.033 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.036 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.219541+0000", 2024-09-13T14:58:38.036 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8411000000000001e-05 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.219577+0000", 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5395999999999998e-05 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:23372 12.0 12:02cf93dd:::benchmark_data_smithi003_259359_object23371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:56:31.110168+0000", 2024-09-13T14:58:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 126.845291342, 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.10953360099999999, 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 23372 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.038 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110168+0000", 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110168+0000", 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110167+0000", 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.039 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110210+0000", 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3319000000000003e-05 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110222+0000", 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2065000000000001e-05 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110228+0000", 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5670000000000002e-06 2024-09-13T14:58:38.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110285+0000", 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7334999999999999e-05 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110390+0000", 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000105011 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:58:38.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.110518+0000", 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000128045 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.111418+0000", 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000900387 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.112462+0000", 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0010439679999999999 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.219624+0000", 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.10716210700000001 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.219669+0000", 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4557e-05 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:31.219701+0000", 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2396e-05 2024-09-13T14:58:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:30709 12.2 12:4fcdb13e:::benchmark_data_smithi003_259359_object30708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:56:35.157429+0000", 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 122.79802987399999, 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.116592429, 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 30709 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.157429+0000", 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.157429+0000", 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.157428+0000", 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.157452+0000", 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4334000000000001e-05 2024-09-13T14:58:38.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.157464+0000", 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2421e-05 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.157470+0000", 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1610000000000002e-06 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.157523+0000", 2024-09-13T14:58:38.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3081999999999998e-05 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.157624+0000", 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101716 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.157772+0000", 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014734599999999999 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.159104+0000", 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001332771 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.159284+0000", 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017992799999999999 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.273923+0000", 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.114638263 2024-09-13T14:58:38.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.273979+0000", 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6820000000000001e-05 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:56:35.274022+0000", 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2252000000000001e-05 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:75414 12.0 12:093bed76:::benchmark_data_smithi003_259359_object75413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:57:06.747703+0000", 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 91.207755801999994, 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.145262637, 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 75414 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.051 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.747703+0000", 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.747703+0000", 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.747702+0000", 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.747732+0000", 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0136000000000001e-05 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.747750+0000", 2024-09-13T14:58:38.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7781000000000001e-05 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.747756+0000", 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0079999999999999e-06 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.747904+0000", 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014837100000000001 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.748033+0000", 2024-09-13T14:58:38.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012930600000000001 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.748171+0000", 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013735099999999999 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.750866+0000", 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026957159999999999 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.055 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.750931+0000", 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4769000000000003e-05 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.892729+0000", 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.141798111 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.892877+0000", 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014800100000000001 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.892966+0000", 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.886e-05 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:75419 12.0 12:0e9f0cfc:::benchmark_data_smithi003_259359_object75418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:57:06.754745+0000", 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 91.200713672000006, 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.139073632, 2024-09-13T14:58:38.057 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 75419 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.754745+0000", 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.754745+0000", 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.754744+0000", 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.754814+0000", 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0325000000000002e-05 2024-09-13T14:58:38.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.754830+0000", 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5905000000000001e-05 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.754837+0000", 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2010000000000002e-06 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.754936+0000", 2024-09-13T14:58:38.060 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8869000000000005e-05 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.755041+0000", 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000105234 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.755166+0000", 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012498199999999999 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.757560+0000", 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002393484 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.757954+0000", 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00039442600000000002 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.893009+0000", 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.135054639 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.062 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.893778+0000", 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00076925799999999999 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:57:06.893819+0000", 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1060000000000003e-05 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.063 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:143952 12.0 12:0d792aec:::benchmark_data_smithi003_259359_object28326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.542283+0000", 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 6.4131765469999999, 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.066030456000000001, 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 143952 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.542283+0000", 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.542283+0000", 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.542282+0000", 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.065 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.542286+0000", 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9760000000000003e-06 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.542296+0000", 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2730000000000008e-06 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.542300+0000", 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.053e-06 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.542333+0000", 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2988000000000003e-05 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.542405+0000", 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2367999999999996e-05 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.542470+0000", 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5320999999999993e-05 2024-09-13T14:58:38.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.600749+0000", 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.058278754000000002 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.608183+0000", 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0074334800000000001 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.608231+0000", 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8365999999999997e-05 2024-09-13T14:58:38.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.608277+0000", 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6053000000000001e-05 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.608313+0000", 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5923000000000001e-05 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.069 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:147067 12.e 12:7c2c8741:::benchmark_data_smithi003_259359_object31441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:36.096546+0000", 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 1.85891286, 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.37244545699999998, 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 147067 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.070 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.096546+0000", 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.096546+0000", 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.096546+0000", 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:58:38.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.072 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.072 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.072 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.096550+0000", 2024-09-13T14:58:38.072 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5379999999999996e-06 2024-09-13T14:58:38.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.072 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.072 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.075 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.096558+0000", 2024-09-13T14:58:38.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3960000000000006e-06 2024-09-13T14:58:38.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.076 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.096563+0000", 2024-09-13T14:58:38.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6650000000000002e-06 2024-09-13T14:58:38.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.096607+0000", 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3577e-05 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.096720+0000", 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000113631 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,3,7", 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.096894+0000", 2024-09-13T14:58:38.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017359400000000001 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.100329+0000", 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0034348120000000002 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.100339+0000", 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0570999999999999e-05 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.078 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.100892+0000", 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00055250899999999996 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.468776+0000", 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.367883866 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.468842+0000", 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6605000000000004e-05 2024-09-13T14:58:38.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.468992+0000", 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000149488 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:147107 12.e 12:7fd9cc9c:::benchmark_data_smithi003_259359_object31481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:36.187487+0000", 2024-09-13T14:58:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 1.767971851, 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.281759922, 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 147107 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.187487+0000", 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.187487+0000", 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.187486+0000", 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.082 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.187495+0000", 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3689999999999996e-06 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.187515+0000", 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0398999999999999e-05 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.187523+0000", 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.311e-06 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.083 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.187579+0000", 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6376999999999999e-05 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.187727+0000", 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014749000000000001 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,3,7", 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.187895+0000", 2024-09-13T14:58:38.084 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000168431 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.193838+0000", 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0059432699999999996 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.193920+0000", 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1966999999999996e-05 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.193930+0000", 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0169999999999999e-05 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.469053+0000", 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27512281 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.469115+0000", 2024-09-13T14:58:38.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2156000000000006e-05 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.469247+0000", 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013181600000000001 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:147108 12.e 12:74aef62e:::benchmark_data_smithi003_259359_object31482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:36.188857+0000", 2024-09-13T14:58:38.087 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 1.766602207, 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28471417100000002, 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 147108 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.188857+0000", 2024-09-13T14:58:38.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.188857+0000", 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.188856+0000", 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.089 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.188861+0000", 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4240000000000001e-06 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.188880+0000", 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8168999999999998e-05 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.188892+0000", 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2259000000000001e-05 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.190676+0000", 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0017836620000000001 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.190826+0000", 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015094500000000001 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,3,7", 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.190991+0000", 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016492899999999999 2024-09-13T14:58:38.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.193985+0000", 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029937100000000001 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.194027+0000", 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2392999999999999e-05 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.194033+0000", 2024-09-13T14:58:38.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.13e-06 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.473392+0000", 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27935891600000001 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.473448+0000", 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6467e-05 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.093 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.473571+0000", 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012311900000000001 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:147109 12.e 12:721a98cb:::benchmark_data_smithi003_259359_object31483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:36.191220+0000", 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 1.7642393699999999, 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28257981399999998, 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:58:38.094 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 147109 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.191220+0000", 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.191220+0000", 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.191219+0000", 2024-09-13T14:58:38.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.191225+0000", 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9920000000000004e-06 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.191234+0000", 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5659999999999996e-06 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.191239+0000", 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.386e-06 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.194058+0000", 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028191549999999998 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.194194+0000", 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000136635 2024-09-13T14:58:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,3,7", 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.194349+0000", 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015452599999999999 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.196883+0000", 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0025344479999999999 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.197151+0000", 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00026722900000000001 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.197199+0000", 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8199999999999999e-05 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.473617+0000", 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.276418777 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.473674+0000", 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6076999999999999e-05 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:36.473800+0000", 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012604000000000001 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:38.101 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:58:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:38.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:38.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:38.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.425 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:38.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:38.526 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.558 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:38.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:38.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:38.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:38.796 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:38.824 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:38.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:38.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:39.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:39.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:39.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:58:39.212 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:39.247 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:39.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:39.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.390 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:39.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:39 smithi019 ceph-mon[91296]: pgmap v1203: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 456 op/s; 599/257475 objects degraded (0.233%); 18561/257475 objects misplaced (7.209%); 40 MiB/s, 635 objects/s recovering 2024-09-13T14:58:39.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:39 smithi003 ceph-mon[107278]: pgmap v1203: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 456 op/s; 599/257475 objects degraded (0.233%); 18561/257475 objects misplaced (7.209%); 40 MiB/s, 635 objects/s recovering 2024-09-13T14:58:39.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:39 smithi003 ceph-mon[104007]: pgmap v1203: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 456 op/s; 599/257475 objects degraded (0.233%); 18561/257475 objects misplaced (7.209%); 40 MiB/s, 635 objects/s recovering 2024-09-13T14:58:39.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:39.511 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:39.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:39.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:39.668 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:39.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:39.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:39.933 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:40.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:40.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:40.176 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.203 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:40.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:40.298 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:58:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:58:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:40.298 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:40 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 599/249147 objects degraded (0.240%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:40.304 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:40.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:40.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:40 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 599/249147 objects degraded (0.240%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:40.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:40 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 599/249147 objects degraded (0.240%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:40.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:40.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:40.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:40.600 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:40.636 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:40.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:40.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:40.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:40.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:40.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:41.034 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:41.065 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:41.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:41.203 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:41.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:41.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:41.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:41 smithi019 ceph-mon[91296]: pgmap v1204: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 5.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 701 op/s; 599/249147 objects degraded (0.240%); 18072/249147 objects misplaced (7.254%); 31 MiB/s, 493 objects/s recovering 2024-09-13T14:58:41.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:41 smithi003 ceph-mon[107278]: pgmap v1204: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 5.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 701 op/s; 599/249147 objects degraded (0.240%); 18072/249147 objects misplaced (7.254%); 31 MiB/s, 493 objects/s recovering 2024-09-13T14:58:41.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:41 smithi003 ceph-mon[104007]: pgmap v1204: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 5.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 701 op/s; 599/249147 objects degraded (0.240%); 18072/249147 objects misplaced (7.254%); 31 MiB/s, 493 objects/s recovering 2024-09-13T14:58:41.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:41.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:41.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:41.617 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:41.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:41.748 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:41.851 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.874 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:41.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:41.975 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:42.007 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:42.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:42.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:42.133 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:42.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.269 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:42.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:42.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:42.393 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:42.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:58:42.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.46703.0:152450 12.f e343/329 12:f3f6dd12:::benchmark_data_smithi003_259359_object36824:head v 343'9641, mlcod=343'9638)", 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:42.521930+0000", 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.0012475050000000001, 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0012768619999999999, 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "started", 2024-09-13T14:58:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:42.521930+0000", 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:42.521930+0000", 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:42.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:42.521929+0000", 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:42.521936+0000", 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8630000000000002e-06 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:42.521952+0000", 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5979999999999999e-05 2024-09-13T14:58:42.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:42.521958+0000", 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2360000000000001e-06 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:42.522049+0000", 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.0564999999999998e-05 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:42.522082+0000", 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2749000000000001e-05 2024-09-13T14:58:42.529 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:42.530 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:58:42.530 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:42.530 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:58:42.530 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-09-13T14:58:42.530 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 1 2024-09-13T14:58:42.530 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:58:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:42.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:42.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:42.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:42.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:42.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:42.846 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:42.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:43.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:43.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:43.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:43 smithi019 ceph-mon[91296]: pgmap v1205: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 5.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 671 op/s; 779/246753 objects degraded (0.316%); 17721/246753 objects misplaced (7.182%); 16 MiB/s, 249 objects/s recovering 2024-09-13T14:58:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:43.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:43.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:43.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:58:43.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:43 smithi003 ceph-mon[104007]: pgmap v1205: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 5.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 671 op/s; 779/246753 objects degraded (0.316%); 17721/246753 objects misplaced (7.182%); 16 MiB/s, 249 objects/s recovering 2024-09-13T14:58:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:43 smithi003 ceph-mon[107278]: pgmap v1205: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 5.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 671 op/s; 779/246753 objects degraded (0.316%); 17721/246753 objects misplaced (7.182%); 16 MiB/s, 249 objects/s recovering 2024-09-13T14:58:43.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:43.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:43.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:43.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:43.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.631 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:43.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:43.704 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:43.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:43.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:43.980 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:44.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:44.118 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:44.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:44.248 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:44.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:44.365 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:58:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:58:44] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:58:44.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:44.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:44.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:44.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:44.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.653 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:44.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:44.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:44.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:44.903 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:44.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:45.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:45.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:45.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:45.231 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:58:45.232 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:58:45.232 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:58:45.232 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:58:45.232 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:45.232 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:45.232 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:45.232 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:45.233 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:58:45.233 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:58:45.233 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:58:45.233 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:58:45.233 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:45.233 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:45.233 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:45.233 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:45.258 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.294 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:45.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:45.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:45 smithi019 ceph-mon[91296]: pgmap v1206: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 822 op/s; 779/236193 objects degraded (0.330%); 16510/236193 objects misplaced (6.990%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:58:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:45 smithi019 ceph-mon[91296]: osdmap e344: 8 total, 8 up, 7 in 2024-09-13T14:58:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:45 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3326230367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:45 smithi003 ceph-mon[104007]: pgmap v1206: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 822 op/s; 779/236193 objects degraded (0.330%); 16510/236193 objects misplaced (6.990%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:58:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:45 smithi003 ceph-mon[104007]: osdmap e344: 8 total, 8 up, 7 in 2024-09-13T14:58:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:45 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3326230367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:45 smithi003 ceph-mon[107278]: pgmap v1206: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 822 op/s; 779/236193 objects degraded (0.330%); 16510/236193 objects misplaced (6.990%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:58:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:45 smithi003 ceph-mon[107278]: osdmap e344: 8 total, 8 up, 7 in 2024-09-13T14:58:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:45 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3326230367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:45.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:45.539 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:45.569 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:45.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:45.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:45.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:45.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:45.970 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:46.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:46.101 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:46.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:46.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:46.334 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:46 smithi019 ceph-mon[91296]: osdmap e345: 8 total, 8 up, 7 in 2024-09-13T14:58:46.335 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 779/236193 objects degraded (0.330%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:46.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:46.362 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:58:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:46.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:46 smithi003 ceph-mon[107278]: osdmap e345: 8 total, 8 up, 7 in 2024-09-13T14:58:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 779/236193 objects degraded (0.330%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:46.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:46 smithi003 ceph-mon[104007]: osdmap e345: 8 total, 8 up, 7 in 2024-09-13T14:58:46.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 779/236193 objects degraded (0.330%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:46.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:46.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:46.613 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.666 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:46.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:46.743 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:46.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:46.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:46.916 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:46.949 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:47.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:47.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:47.165 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:47.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:47.219 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:47.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:47.343 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:47 smithi003 ceph-mon[107278]: pgmap v1209: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 972 op/s; 946/234138 objects degraded (0.404%); 16193/234138 objects misplaced (6.916%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:58:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:47 smithi003 ceph-mon[104007]: pgmap v1209: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 972 op/s; 946/234138 objects degraded (0.404%); 16193/234138 objects misplaced (6.916%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:58:47.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:47.480 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:47.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:47.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:47.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:47 smithi019 ceph-mon[91296]: pgmap v1209: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 972 op/s; 946/234138 objects degraded (0.404%); 16193/234138 objects misplaced (6.916%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:58:47.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:47.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:47.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:47.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:47.990 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:48.020 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:48.122 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:48.153 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:48.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:48.287 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:48.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.412 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:48.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:48.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:48.638 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:48.770 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:48.772 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:48.801 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:48.903 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:48.936 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:49.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:49.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:49.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:49.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.342 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:49.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:49.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:49 smithi003 ceph-mon[107278]: pgmap v1210: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 115 active+clean; 4.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 683 op/s; 946/232731 objects degraded (0.406%); 16029/232731 objects misplaced (6.887%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:58:49.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:49 smithi003 ceph-mon[104007]: pgmap v1210: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 115 active+clean; 4.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 683 op/s; 946/232731 objects degraded (0.406%); 16029/232731 objects misplaced (6.887%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:58:49.363 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:49 smithi019 ceph-mon[91296]: pgmap v1210: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 115 active+clean; 4.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 683 op/s; 946/232731 objects degraded (0.406%); 16029/232731 objects misplaced (6.887%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:58:49.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:49.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:49.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:49.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:49.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:49.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:49.862 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:49.897 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:50.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:50.040 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:50.040 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:50.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:50.144 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.184 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:50.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:50.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:50.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:50.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:50.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:50.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:50.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:58:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:58:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:50.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:50.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:50.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:50.772 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:58:50.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:58:50.844 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:50.873 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:50.975 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.003 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:51.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:51.129 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:51.138 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:51.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.267 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:51.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:51.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:51.453 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:51 smithi019 ceph-mon[91296]: pgmap v1211: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 946/225126 objects degraded (0.420%); 14660/225126 objects misplaced (6.512%); 14 MiB/s, 229 objects/s recovering 2024-09-13T14:58:51.453 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 946/225126 objects degraded (0.420%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:51.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.556 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:51.658 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:51 smithi003 ceph-mon[107278]: pgmap v1211: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 946/225126 objects degraded (0.420%); 14660/225126 objects misplaced (6.512%); 14 MiB/s, 229 objects/s recovering 2024-09-13T14:58:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 946/225126 objects degraded (0.420%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:51 smithi003 ceph-mon[104007]: pgmap v1211: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 946/225126 objects degraded (0.420%); 14660/225126 objects misplaced (6.512%); 14 MiB/s, 229 objects/s recovering 2024-09-13T14:58:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 946/225126 objects degraded (0.420%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:51.687 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:51.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:51.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:51.926 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.955 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:52.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:52.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:52.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:52.229 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:52.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:52.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:52.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:52.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:52.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:52.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:52.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:52.787 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:52.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:52.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:53.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:53.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:53.082 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:53.082 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:53.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:53.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:53.320 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:53 smithi019 ceph-mon[91296]: pgmap v1212: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 565 op/s; 1125/222618 objects degraded (0.505%); 14308/222618 objects misplaced (6.427%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:58:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:53.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:53.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:53.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:53.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:53 smithi003 ceph-mon[104007]: pgmap v1212: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 565 op/s; 1125/222618 objects degraded (0.505%); 14308/222618 objects misplaced (6.427%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:58:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:53 smithi003 ceph-mon[107278]: pgmap v1212: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 565 op/s; 1125/222618 objects degraded (0.505%); 14308/222618 objects misplaced (6.427%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:58:53.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:53.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:53.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:58:53.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:53.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:54.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.052 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:54.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:54.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:54.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:58:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:58:54] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:58:54.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:54.217 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:54.295 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:54.325 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:54.427 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:54.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:54.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:54.581 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:54.595 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:54.698 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:54.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:54.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:54.867 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:54.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:55.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:55.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.135 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:55.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.201 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:58:55.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:55.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:55.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:55.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:55 smithi003 ceph-mon[104007]: pgmap v1213: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 829 op/s; 1125/211641 objects degraded (0.532%); 13153/211641 objects misplaced (6.215%); 12 MiB/s, 197 objects/s recovering 2024-09-13T14:58:55.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1158696853' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:58:55.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:55 smithi003 ceph-mon[107278]: pgmap v1213: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 829 op/s; 1125/211641 objects degraded (0.532%); 13153/211641 objects misplaced (6.215%); 12 MiB/s, 197 objects/s recovering 2024-09-13T14:58:55.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1158696853' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:58:55.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:55.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:55.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:55.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:55.596 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:55 smithi019 ceph-mon[91296]: pgmap v1213: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 829 op/s; 1125/211641 objects degraded (0.532%); 13153/211641 objects misplaced (6.215%); 12 MiB/s, 197 objects/s recovering 2024-09-13T14:58:55.596 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1158696853' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:58:55.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:55.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:55.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.862 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:56.097 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:56.127 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:56.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:56.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:56.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:56.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1158696853' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:58:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[107278]: osdmap e346: 8 total, 8 up, 7 in 2024-09-13T14:58:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1125/211641 objects degraded (0.532%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2141438585' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:56.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1158696853' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:58:56.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[104007]: osdmap e346: 8 total, 8 up, 7 in 2024-09-13T14:58:56.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1125/211641 objects degraded (0.532%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:56.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2141438585' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:56.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:56 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:56.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:56.657 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1158696853' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:58:56.657 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:56 smithi019 ceph-mon[91296]: osdmap e346: 8 total, 8 up, 7 in 2024-09-13T14:58:56.658 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1125/211641 objects degraded (0.532%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:58:56.658 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2141438585' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:56.658 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:56 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:56.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:56.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:56.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:56.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:56.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:56.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.982 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:56.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:57.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:58:57.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:57.106 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:58:57.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:57.245 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:57.276 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:57.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:57.305 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:57.305 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:57.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:57.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:57.539 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:57.575 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:57.641 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:57 smithi019 ceph-mon[91296]: pgmap v1215: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 833 op/s; 1329/209100 objects degraded (0.636%); 12777/209100 objects misplaced (6.110%); 12 MiB/s, 193 objects/s recovering 2024-09-13T14:58:57.641 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:57 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:57.642 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:57 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:57.642 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:57 smithi019 ceph-mon[91296]: osdmap e347: 8 total, 8 up, 7 in 2024-09-13T14:58:57.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:57 smithi003 ceph-mon[107278]: pgmap v1215: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 833 op/s; 1329/209100 objects degraded (0.636%); 12777/209100 objects misplaced (6.110%); 12 MiB/s, 193 objects/s recovering 2024-09-13T14:58:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:57 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:57 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:57 smithi003 ceph-mon[107278]: osdmap e347: 8 total, 8 up, 7 in 2024-09-13T14:58:57.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:57 smithi003 ceph-mon[104007]: pgmap v1215: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 833 op/s; 1329/209100 objects degraded (0.636%); 12777/209100 objects misplaced (6.110%); 12 MiB/s, 193 objects/s recovering 2024-09-13T14:58:57.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:57 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:57.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:57 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:57.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:57 smithi003 ceph-mon[104007]: osdmap e347: 8 total, 8 up, 7 in 2024-09-13T14:58:57.707 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:57.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:57.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:57.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:57.985 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:58:58.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:58.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:58.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:58.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:58.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:58.386 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:58.403 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:58.435 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:58.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:58.566 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:58.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:58:58.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:58:58.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:58:58.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:58:58.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:58:58.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:58:58.645 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:58.664 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:58:58.664 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:58.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:58.699 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:58:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:58.723 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:58.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:58.833 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:58.866 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:58.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:59.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:59.109 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:59.127 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:59.128 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:59.128 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:58 smithi019 ceph-mon[91296]: pgmap v1217: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 1329/207456 objects degraded (0.641%); 12777/207456 objects misplaced (6.159%); 6.4 MiB/s, 103 objects/s recovering 2024-09-13T14:58:59.128 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:58:58 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3449216552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:59.142 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:58 smithi003 ceph-mon[107278]: pgmap v1217: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 1329/207456 objects degraded (0.641%); 12777/207456 objects misplaced (6.159%); 6.4 MiB/s, 103 objects/s recovering 2024-09-13T14:58:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:58:58 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3449216552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:58:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:58 smithi003 ceph-mon[104007]: pgmap v1217: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 1329/207456 objects degraded (0.641%); 12777/207456 objects misplaced (6.159%); 6.4 MiB/s, 103 objects/s recovering 2024-09-13T14:58:59.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:58:58 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3449216552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:59.244 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:59.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:59.379 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:59.418 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:59.520 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:59.551 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:59.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.680 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:59.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:59.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:59.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:58:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.811 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:58:59.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:59.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:58:59.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:00.069 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.105 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:00.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:00.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:00.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:00.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.382 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:00.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:00.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:59:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:59:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:00.471 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117266 10.f 10:f2aa6f81:::benchmark_data_smithi003_197468_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.720592+0000", 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 590.74760026800004, 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.327086822, 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.472 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117266 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720592+0000", 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720592+0000", 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.473 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720591+0000", 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720594+0000", 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8480000000000001e-06 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.474 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720600+0000", 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4200000000000004e-06 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.720604+0000", 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4860000000000002e-06 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.746485+0000", 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.025880826999999999 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.475 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.746592+0000", 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010717000000000001 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.746704+0000", 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011236800000000001 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.751485+0000", 2024-09-13T14:59:00.476 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0047811640000000001 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.759932+0000", 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0084469260000000004 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.047614+0000", 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28768163600000002 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.047651+0000", 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6480000000000003e-05 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.047679+0000", 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7974999999999999e-05 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117269 10.7 10:e83f946d:::benchmark_data_smithi003_197468_object540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:59:00.478 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.722459+0000", 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 590.74573312099994, 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.33327838999999998, 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117269 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722459+0000", 2024-09-13T14:59:00.479 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722459+0000", 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722458+0000", 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.480 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722463+0000", 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2809999999999999e-06 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722475+0000", 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2201000000000001e-05 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.722481+0000", 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5419999999999996e-06 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.481 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764160+0000", 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.041679714999999999 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764315+0000", 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015493000000000001 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764457+0000", 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014210500000000001 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.482 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772074+0000", 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0076163630000000001 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.788564+0000", 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016490134 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.483 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051129+0000", 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.262565304 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051204+0000", 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4845999999999995e-05 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055737+0000", 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0045331210000000002 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.484 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117274 10.f 10:fbc2e15f:::benchmark_data_smithi003_197468_object545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.735218+0000", 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 590.73297363100005, 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.316058477, 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117274 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.485 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.486 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:00.487 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.487 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735218+0000", 2024-09-13T14:59:00.487 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.487 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.487 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.487 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.487 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735218+0000", 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735216+0000", 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735230+0000", 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3811999999999999e-05 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.488 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735253+0000", 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2646e-05 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.735260+0000", 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5530000000000004e-06 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764197+0000", 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.028936565000000001 2024-09-13T14:59:00.489 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764308+0000", 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000110838 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.764437+0000", 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000128823 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.771774+0000", 2024-09-13T14:59:00.490 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.007336927 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772083+0000", 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00030953699999999997 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051206+0000", 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27912247099999998 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.491 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051246+0000", 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0608000000000002e-05 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.051277+0000", 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0546999999999997e-05 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.492 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117285 10.f 10:fd6a389f:::benchmark_data_smithi003_197468_object556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.752229+0000", 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 590.71596293699997, 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30415821399999998, 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117285 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.493 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752229+0000", 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752229+0000", 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752228+0000", 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.494 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752234+0000", 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8540000000000002e-06 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752247+0000", 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3555e-05 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.752253+0000", 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4539999999999998e-06 2024-09-13T14:59:00.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768099+0000", 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015846628000000001 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768230+0000", 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000130617 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:00.496 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768448+0000", 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000218381 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772088+0000", 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036394679999999999 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772225+0000", 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013752 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.497 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055783+0000", 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28355824600000001 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055834+0000", 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0442000000000003e-05 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.056387+0000", 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00055330699999999995 2024-09-13T14:59:00.498 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117292 10.7 10:e057952f:::benchmark_data_smithi003_197468_object563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.760602+0000", 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 590.70759017399996, 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29983130600000002, 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.499 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117292 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760602+0000", 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760602+0000", 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.500 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760600+0000", 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760607+0000", 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2910000000000002e-06 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760622+0000", 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4978e-05 2024-09-13T14:59:00.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.760628+0000", 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5559999999999998e-06 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768389+0000", 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0077610559999999997 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768548+0000", 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015928000000000001 2024-09-13T14:59:00.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.768699+0000", 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015094800000000001 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772239+0000", 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0035398029999999998 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.788617+0000", 2024-09-13T14:59:00.503 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016378344999999999 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055881+0000", 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26726359500000002 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.055933+0000", 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2432999999999998e-05 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.504 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060433+0000", 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0044996050000000003 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117298 10.7 10:e5727b6c:::benchmark_data_smithi003_197468_object569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.772611+0000", 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 590.69558074300005, 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.29277972899999999, 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.505 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117298 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772611+0000", 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.506 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772611+0000", 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772610+0000", 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772616+0000", 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3389999999999998e-06 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.507 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772628+0000", 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1877e-05 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772633+0000", 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0509999999999996e-06 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772692+0000", 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8433000000000001e-05 2024-09-13T14:59:00.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772798+0000", 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106079 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.772899+0000", 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010132999999999999 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.776210+0000", 2024-09-13T14:59:00.509 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0033107729999999999 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.788648+0000", 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012438289999999999 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060503+0000", 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27185490499999998 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.510 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060546+0000", 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3467000000000002e-05 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.065391+0000", 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0048445220000000004 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:117301 10.f 10:f4f3ddb9:::benchmark_data_smithi003_197468_object572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:09.779418+0000", 2024-09-13T14:59:00.511 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 590.68877412200004, 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28115621899999999, 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 117301 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779418+0000", 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.512 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779418+0000", 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779416+0000", 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779424+0000", 2024-09-13T14:59:00.513 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8809999999999996e-06 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779445+0000", 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1067999999999999e-05 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779453+0000", 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4630000000000004e-06 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.514 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779515+0000", 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1679000000000004e-05 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779658+0000", 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000142948 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.779790+0000", 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013255799999999999 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.515 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.516 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.783333+0000", 2024-09-13T14:59:00.516 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0035433719999999999 2024-09-13T14:59:00.516 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.516 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.516 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.516 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:09.783430+0000", 2024-09-13T14:59:00.516 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6834000000000002e-05 2024-09-13T14:59:00.516 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.516 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.517 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.517 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060490+0000", 2024-09-13T14:59:00.517 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.277059208 2024-09-13T14:59:00.518 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.518 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.518 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.518 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060540+0000", 2024-09-13T14:59:00.518 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.028e-05 2024-09-13T14:59:00.518 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.518 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.519 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.519 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:10.060574+0000", 2024-09-13T14:59:00.519 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4264000000000003e-05 2024-09-13T14:59:00.519 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.519 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.519 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.519 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.519 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.520 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:123351 10.f 10:f28c766d:::benchmark_data_smithi003_197468_object6622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-13T14:59:00.520 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:17.374873+0000", 2024-09-13T14:59:00.520 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 583.09331897599998, 2024-09-13T14:59:00.520 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28567656299999999, 2024-09-13T14:59:00.520 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.521 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.521 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.521 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.522 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.522 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123351 2024-09-13T14:59:00.522 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.523 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.523 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.523 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.523 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374873+0000", 2024-09-13T14:59:00.523 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.523 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.524 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.524 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.524 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374873+0000", 2024-09-13T14:59:00.524 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.524 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.524 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.524 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.525 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374872+0000", 2024-09-13T14:59:00.525 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:59:00.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.525 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.526 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.526 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374877+0000", 2024-09-13T14:59:00.526 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2849999999999996e-06 2024-09-13T14:59:00.526 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.527 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.527 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374893+0000", 2024-09-13T14:59:00.527 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6758000000000001e-05 2024-09-13T14:59:00.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.527 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.527 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.374898+0000", 2024-09-13T14:59:00.528 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0869999999999997e-06 2024-09-13T14:59:00.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.528 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.393467+0000", 2024-09-13T14:59:00.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.018568474000000001 2024-09-13T14:59:00.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.530 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.393577+0000", 2024-09-13T14:59:00.530 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000110605 2024-09-13T14:59:00.530 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.531 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.531 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:00.531 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.393687+0000", 2024-09-13T14:59:00.531 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000109844 2024-09-13T14:59:00.531 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.532 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.532 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.532 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.401477+0000", 2024-09-13T14:59:00.533 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0077901990000000003 2024-09-13T14:59:00.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.533 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.533 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.534 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.403044+0000", 2024-09-13T14:59:00.534 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001566556 2024-09-13T14:59:00.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.534 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.535 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.535 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.660438+0000", 2024-09-13T14:59:00.535 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25739424900000002 2024-09-13T14:59:00.535 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.535 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.535 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.536 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.660513+0000", 2024-09-13T14:59:00.536 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4787000000000001e-05 2024-09-13T14:59:00.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.536 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.536 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.536 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:17.660550+0000", 2024-09-13T14:59:00.537 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6458000000000002e-05 2024-09-13T14:59:00.537 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.537 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.538 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.538 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149388 10.7 10:ea963fcf:::benchmark_data_smithi003_197468_object32659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:59:00.538 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.303553+0000", 2024-09-13T14:59:00.539 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 553.16463851900005, 2024-09-13T14:59:00.539 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.34338312199999999, 2024-09-13T14:59:00.539 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.539 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.539 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149388 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303553+0000", 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303553+0000", 2024-09-13T14:59:00.540 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303552+0000", 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303558+0000", 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8259999999999998e-06 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.541 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303568+0000", 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7389999999999994e-06 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.303572+0000", 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2459999999999997e-06 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307829+0000", 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0042576009999999997 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.542 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307977+0000", 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014806699999999999 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.308129+0000", 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015176799999999999 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.312145+0000", 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0040153669999999997 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.543 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.544 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.340964+0000", 2024-09-13T14:59:00.544 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.028819079000000001 2024-09-13T14:59:00.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.545 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.545 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642410+0000", 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30144587499999997 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642464+0000", 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4509999999999998e-05 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.646937+0000", 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0044724949999999999 2024-09-13T14:59:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149395 10.f 10:ff0ca942:::benchmark_data_smithi003_197468_object32666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.307942+0000", 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 553.16025032300001, 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.334537682, 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.547 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149395 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307942+0000", 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307942+0000", 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.548 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307941+0000", 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307946+0000", 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2129999999999999e-06 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.549 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307956+0000", 2024-09-13T14:59:00.550 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7189999999999996e-06 2024-09-13T14:59:00.550 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.550 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.550 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.550 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.307961+0000", 2024-09-13T14:59:00.550 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3190000000000002e-06 2024-09-13T14:59:00.550 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.550 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.553 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.553 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.308054+0000", 2024-09-13T14:59:00.553 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3014000000000001e-05 2024-09-13T14:59:00.553 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.553 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.553 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.308152+0000", 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8295000000000007e-05 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.308259+0000", 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106173 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.312303+0000", 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0040445589999999997 2024-09-13T14:59:00.554 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314659+0000", 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0023557449999999998 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642416+0000", 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.32775716700000002 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642452+0000", 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6291999999999997e-05 2024-09-13T14:59:00.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.642479+0000", 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6988e-05 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149399 10.7 10:e7552c3b:::benchmark_data_smithi003_197468_object32670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.314862+0000", 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 553.15333029299995, 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.33632368200000001, 2024-09-13T14:59:00.556 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149399 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314862+0000", 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.557 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314862+0000", 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314861+0000", 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314866+0000", 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.977e-06 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314874+0000", 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3399999999999998e-06 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314879+0000", 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4989999999999996e-06 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.559 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.314929+0000", 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0234e-05 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.315034+0000", 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010527 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.315167+0000", 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013303800000000001 2024-09-13T14:59:00.560 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.319534+0000", 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0043670159999999996 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.341004+0000", 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.021470073999999999 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.561 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.647004+0000", 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30600010599999999 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.647052+0000", 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8063999999999999e-05 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.651185+0000", 2024-09-13T14:59:00.562 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0041329449999999998 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149422 10.f 10:f518014d:::benchmark_data_smithi003_197468_object32693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.359274+0000", 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 553.10891774900006, 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28782790600000002, 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.563 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149422 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359274+0000", 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.564 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359274+0000", 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359274+0000", 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359277+0000", 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8639999999999998e-06 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359285+0000", 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0220000000000004e-06 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359289+0000", 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3809999999999999e-06 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359319+0000", 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0329e-05 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359398+0000", 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8954000000000003e-05 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.359469+0000", 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1149999999999995e-05 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.362252+0000", 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027822929999999999 2024-09-13T14:59:00.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.362419+0000", 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016755700000000001 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.646982+0000", 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.284562708 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.647069+0000", 2024-09-13T14:59:00.568 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7306999999999996e-05 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.647102+0000", 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3006999999999998e-05 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36168.0:149426 10.7 10:e1c5b3e0:::benchmark_data_smithi003_197468_object32697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:49:47.376151+0000", 2024-09-13T14:59:00.569 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 553.09204139899998, 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.28245281700000002, 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36168", 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1106080813", 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 149426 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376151+0000", 2024-09-13T14:59:00.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376151+0000", 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376149+0000", 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376158+0000", 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4509999999999992e-06 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376175+0000", 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7207999999999999e-05 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376181+0000", 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3679999999999998e-06 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.572 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376240+0000", 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8353999999999997e-05 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376435+0000", 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000194916 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.376622+0000", 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00018775000000000001 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.379581+0000", 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029589740000000001 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.382484+0000", 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0029022779999999999 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.651267+0000", 2024-09-13T14:59:00.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26878351900000003 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.651315+0000", 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7701999999999998e-05 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:49:47.658603+0000", 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0072886119999999999 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211667 11.3 11:c090de96:::benchmark_data_smithi003_229782_object89287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.920496+0000", 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 234.54769594999999, 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.41771117699999999, 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211667 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.576 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920496+0000", 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920496+0000", 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920494+0000", 2024-09-13T14:59:00.577 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920502+0000", 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3189999999999995e-06 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920517+0000", 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4761e-05 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.578 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920524+0000", 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7159999999999997e-06 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920587+0000", 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3699000000000001e-05 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920724+0000", 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013690299999999999 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.921260+0000", 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00053544300000000003 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.924130+0000", 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028703309999999998 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.924568+0000", 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00043812400000000002 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.580 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.338055+0000", 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.41348705299999999 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.338147+0000", 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1294999999999999e-05 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.338207+0000", 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0582999999999999e-05 2024-09-13T14:59:00.581 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:153223 12.5 12:a07e3d64:::benchmark_data_smithi003_259359_object37597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:43.429828+0000", 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 17.038363729, 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.211590522, 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:59:00.582 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 153223 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.429828+0000", 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.429828+0000", 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.429827+0000", 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.429833+0000", 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9359999999999996e-06 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.429847+0000", 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4198e-05 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.429853+0000", 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.835e-06 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.429912+0000", 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9386999999999999e-05 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.585 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.430041+0000", 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012854300000000001 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.430595+0000", 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00055377599999999996 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.438207+0000", 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.007612756 2024-09-13T14:59:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.438605+0000", 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00039736799999999999 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.641325+0000", 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.20271983299999999 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.641379+0000", 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4252999999999997e-05 2024-09-13T14:59:00.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.641419+0000", 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9975000000000003e-05 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:153232 12.5 12:af4c6e80:::benchmark_data_smithi003_259359_object37606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:43.439281+0000", 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 17.028910867, 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.23204219800000001, 2024-09-13T14:59:00.588 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 153232 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439281+0000", 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439281+0000", 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439279+0000", 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439287+0000", 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5909999999999998e-06 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439305+0000", 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8099999999999999e-05 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439312+0000", 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5830000000000003e-06 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439435+0000", 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012348 2024-09-13T14:59:00.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439566+0000", 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000131073 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439695+0000", 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000128961 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.446857+0000", 2024-09-13T14:59:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0071618189999999998 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.451096+0000", 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0042388520000000004 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671214+0000", 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22011769 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.593 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671279+0000", 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5673000000000006e-05 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671323+0000", 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4024999999999999e-05 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:153238 12.5 12:a410bc8c:::benchmark_data_smithi003_259359_object37612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:43.444203+0000", 2024-09-13T14:59:00.594 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 17.02398938, 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22747585400000001, 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 153238 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.595 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.596 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.444203+0000", 2024-09-13T14:59:00.596 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.596 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.596 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.596 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.596 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.444203+0000", 2024-09-13T14:59:00.596 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.596 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.444201+0000", 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.444207+0000", 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9819999999999996e-06 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.597 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.444218+0000", 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0533999999999999e-05 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.444223+0000", 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.356e-06 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.446880+0000", 2024-09-13T14:59:00.598 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026565989999999999 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.446999+0000", 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011932 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.447108+0000", 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000109326 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.599 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.451391+0000", 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0042831409999999999 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.452217+0000", 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00082568500000000005 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.600 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671566+0000", 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21934851 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671634+0000", 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8190999999999998e-05 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671678+0000", 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4595000000000002e-05 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:153239 12.4 12:27b13740:::benchmark_data_smithi003_259359_object37613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:43.445321+0000", 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 17.022871078000001, 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19178216100000001, 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:59:00.602 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 153239 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.445321+0000", 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.445321+0000", 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.445320+0000", 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.445330+0000", 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0831e-05 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.604 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.445346+0000", 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5305e-05 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.445352+0000", 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8309999999999997e-06 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.445403+0000", 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1131999999999998e-05 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.445503+0000", 2024-09-13T14:59:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010021100000000001 2024-09-13T14:59:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:59:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.445602+0000", 2024-09-13T14:59:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8874000000000002e-05 2024-09-13T14:59:00.606 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.607 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.607 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.453974+0000", 2024-09-13T14:59:00.607 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0083723840000000001 2024-09-13T14:59:00.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.607 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.607 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.454020+0000", 2024-09-13T14:59:00.607 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5760000000000002e-05 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.636999+0000", 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.18297907899999999 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.637056+0000", 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6694999999999998e-05 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.637103+0000", 2024-09-13T14:59:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.727e-05 2024-09-13T14:59:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:153245 12.5 12:afb81c6b:::benchmark_data_smithi003_259359_object37619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:59:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:43.452537+0000", 2024-09-13T14:59:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 17.015655275, 2024-09-13T14:59:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21930446300000001, 2024-09-13T14:59:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:59:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:59:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 153245 2024-09-13T14:59:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.452537+0000", 2024-09-13T14:59:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.452537+0000", 2024-09-13T14:59:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.452535+0000", 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.452541+0000", 2024-09-13T14:59:00.612 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5560000000000004e-06 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.452588+0000", 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7160999999999998e-05 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.452595+0000", 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9129999999999997e-06 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.613 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.452654+0000", 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8690999999999998e-05 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.452773+0000", 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011892600000000001 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.453053+0000", 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00028020399999999999 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.614 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.456270+0000", 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003217391 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.456276+0000", 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8470000000000001e-06 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671761+0000", 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21548488499999999 2024-09-13T14:59:00.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671806+0000", 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5216999999999998e-05 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671841+0000", 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4712000000000002e-05 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.616 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:153249 12.5 12:ac3a1e58:::benchmark_data_smithi003_259359_object37623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:43.458875+0000", 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 17.009317184, 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.213107994, 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 153249 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:00.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.458875+0000", 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.458875+0000", 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.618 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.619 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:00.619 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.458873+0000", 2024-09-13T14:59:00.619 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:00.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.619 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:00.620 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.458880+0000", 2024-09-13T14:59:00.620 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6579999999999996e-06 2024-09-13T14:59:00.620 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.620 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.620 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:00.621 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.458904+0000", 2024-09-13T14:59:00.621 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3787000000000001e-05 2024-09-13T14:59:00.621 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.622 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.622 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:00.622 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.458910+0000", 2024-09-13T14:59:00.622 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8649999999999998e-06 2024-09-13T14:59:00.622 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.622 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.623 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:00.623 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.458948+0000", 2024-09-13T14:59:00.623 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7722999999999998e-05 2024-09-13T14:59:00.623 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.623 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.459034+0000", 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6093000000000004e-05 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.459106+0000", 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2589000000000006e-05 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.624 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.461793+0000", 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026868899999999999 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.461804+0000", 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0676000000000001e-05 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671893+0000", 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21008927399999999 2024-09-13T14:59:00.625 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671943+0000", 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9583000000000003e-05 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671983+0000", 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0125e-05 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:00.626 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:59:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:00.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:00.729 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:00.753 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:00.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:00.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:00.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:01.009 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:01.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:01.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:01.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:01 smithi003 ceph-mon[107278]: pgmap v1218: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 1329/198384 objects degraded (0.670%); 11632/198384 objects misplaced (5.863%); 12 MiB/s, 188 objects/s recovering 2024-09-13T14:59:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:01 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1329/198384 objects degraded (0.670%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:59:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:01 smithi003 ceph-mon[104007]: pgmap v1218: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 1329/198384 objects degraded (0.670%); 11632/198384 objects misplaced (5.863%); 12 MiB/s, 188 objects/s recovering 2024-09-13T14:59:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:01 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1329/198384 objects degraded (0.670%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:59:01.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:01.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:01.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:01.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:01.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:01 smithi019 ceph-mon[91296]: pgmap v1218: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 1329/198384 objects degraded (0.670%); 11632/198384 objects misplaced (5.863%); 12 MiB/s, 188 objects/s recovering 2024-09-13T14:59:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:01 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1329/198384 objects degraded (0.670%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:59:01.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:01.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:01.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:01.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:01.818 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:01.846 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:01.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:01.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:59:01.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:02.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:02.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:02.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:02.252 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:02.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:02.308 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:02.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.399 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:02.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:02.500 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:02.527 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:02.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:02.668 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:02.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:02.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:02.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:03.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:03.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:03.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:03 smithi003 ceph-mon[104007]: pgmap v1219: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 1528/195951 objects degraded (0.780%); 11287/195951 objects misplaced (5.760%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:59:03.189 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:03.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:03.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.354 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:03.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:03.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:03 smithi019 ceph-mon[91296]: pgmap v1219: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 1528/195951 objects degraded (0.780%); 11287/195951 objects misplaced (5.760%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:59:03.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:59:03.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:03 smithi003 ceph-mon[107278]: pgmap v1219: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 1528/195951 objects degraded (0.780%); 11287/195951 objects misplaced (5.760%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:59:03.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:59:03.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:59:03.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:03.490 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:03.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:03.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:03.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:03.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:03.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.916 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:03.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:04.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:04.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:04.170 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:04.203 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:04.263 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:59:04.263 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T14:59:04.263 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.263 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.46703.0:170441 12.2 e347/311 12:48aa4b13:::benchmark_data_smithi003_259359_object54815:head v 347'10655, mlcod=347'10653)", 2024-09-13T14:59:04.263 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:04.249928+0000", 2024-09-13T14:59:04.263 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.001859999, 2024-09-13T14:59:04.263 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001880077, 2024-09-13T14:59:04.263 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:04.263 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "started", 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.249928+0000", 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.249928+0000", 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:04.264 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.249928+0000", 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.249935+0000", 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1720000000000001e-06 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.249949+0000", 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4460999999999999e-05 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.265 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.249955+0000", 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7479999999999999e-06 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250026+0000", 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0596e-05 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250044+0000", 2024-09-13T14:59:04.266 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8516999999999999e-05 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.46703.0:170442 12.1 e347/311 12:80ab7c65:::benchmark_data_smithi003_259359_object54816:head v 347'10720, mlcod=347'10718)", 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:04.250252+0000", 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.001536356, 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001594467, 2024-09-13T14:59:04.267 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "started", 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250252+0000", 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250252+0000", 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.268 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250251+0000", 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250258+0000", 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6440000000000003e-06 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250267+0000", 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.4700000000000008e-06 2024-09-13T14:59:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250272+0000", 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.04e-06 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250345+0000", 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2695999999999997e-05 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250363+0000", 2024-09-13T14:59:04.270 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7509000000000001e-05 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:170443 12.d 12:b768ea47:::benchmark_data_smithi003_259359_object54817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:04.250427+0000", 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.0013608400000000001, 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0014540060000000001, 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T14:59:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 170443 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250427+0000", 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250427+0000", 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.272 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250426+0000", 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250431+0000", 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.775e-06 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250440+0000", 2024-09-13T14:59:04.273 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9770000000000006e-06 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250445+0000", 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8330000000000001e-06 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250870+0000", 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00042471000000000002 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.274 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.250979+0000", 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000109629 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:04.251082+0000", 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000102589 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 3 2024-09-13T14:59:04.275 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:59:04.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:04.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:04.412 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:59:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:59:04] "GET /metrics HTTP/1.1" 200 36559 "" "Prometheus/2.51.0" 2024-09-13T14:59:04.421 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:04.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:04.478 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:04.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:04.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:04.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:04.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:59:04.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:04.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:04.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:04.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:05.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:59:05.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:59:05.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:05.258 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:05.264 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:05 smithi019 ceph-mon[91296]: pgmap v1220: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 1528/186420 objects degraded (0.820%); 10085/186420 objects misplaced (5.410%); 13 MiB/s, 201 objects/s recovering 2024-09-13T14:59:05.264 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3370608844' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:59:05.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:05 smithi003 ceph-mon[107278]: pgmap v1220: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 1528/186420 objects degraded (0.820%); 10085/186420 objects misplaced (5.410%); 13 MiB/s, 201 objects/s recovering 2024-09-13T14:59:05.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3370608844' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:59:05.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:05 smithi003 ceph-mon[104007]: pgmap v1220: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 1528/186420 objects degraded (0.820%); 10085/186420 objects misplaced (5.410%); 13 MiB/s, 201 objects/s recovering 2024-09-13T14:59:05.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3370608844' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:59:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.320 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:05.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:05.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:05.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:05.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:05.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:05.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:05.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:05.838 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:05.891 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:05.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.895 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:05.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:05.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:06.037 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:06.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:59:06.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:06.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:06.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:06.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:06.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:06 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3370608844' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:59:06.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:06 smithi003 ceph-mon[107278]: osdmap e348: 8 total, 8 up, 7 in 2024-09-13T14:59:06.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1528/186420 objects degraded (0.820%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:59:06.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:06 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3370608844' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:59:06.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:06 smithi003 ceph-mon[104007]: osdmap e348: 8 total, 8 up, 7 in 2024-09-13T14:59:06.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1528/186420 objects degraded (0.820%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:59:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:06.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:06 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3370608844' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:59:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:06 smithi019 ceph-mon[91296]: osdmap e348: 8 total, 8 up, 7 in 2024-09-13T14:59:06.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1528/186420 objects degraded (0.820%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:59:06.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:06.500 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:06.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:06.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:06.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:06.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:06.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:06.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.906 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:06.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:07.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:07.041 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:07.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:07.232 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:07 smithi019 ceph-mon[91296]: pgmap v1222: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 1721/183993 objects degraded (0.935%); 9732/183993 objects misplaced (5.289%); 11 MiB/s, 181 objects/s recovering 2024-09-13T14:59:07.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:07.336 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:07.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:07.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:07 smithi003 ceph-mon[104007]: pgmap v1222: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 1721/183993 objects degraded (0.935%); 9732/183993 objects misplaced (5.289%); 11 MiB/s, 181 objects/s recovering 2024-09-13T14:59:07.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:07 smithi003 ceph-mon[107278]: pgmap v1222: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 1721/183993 objects degraded (0.935%); 9732/183993 objects misplaced (5.289%); 11 MiB/s, 181 objects/s recovering 2024-09-13T14:59:07.451 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:07.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:07.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:07.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:07.646 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:07.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:07.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:07.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:07.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:07.894 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:07.932 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:07.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:08.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:08.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:08.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:08.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:08.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:08.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:08.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:08.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:08.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:08.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:08.869 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T14:59:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T14:59:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T14:59:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T14:59:08.869 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T14:59:08.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:08.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:09.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:09.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:09.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:09.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:09.349 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:09.418 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:09 smithi003 ceph-mon[107278]: pgmap v1223: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 829 op/s; 1721/182568 objects degraded (0.943%); 9732/182568 objects misplaced (5.331%); 11 MiB/s, 177 objects/s recovering 2024-09-13T14:59:09.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:09 smithi003 ceph-mon[104007]: pgmap v1223: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 829 op/s; 1721/182568 objects degraded (0.943%); 9732/182568 objects misplaced (5.331%); 11 MiB/s, 177 objects/s recovering 2024-09-13T14:59:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:09 smithi019 ceph-mon[91296]: pgmap v1223: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 829 op/s; 1721/182568 objects degraded (0.943%); 9732/182568 objects misplaced (5.331%); 11 MiB/s, 177 objects/s recovering 2024-09-13T14:59:09.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:09.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:09.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:09.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:09.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:09.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:09.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:09.873 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.904 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:09.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:10.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:10.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:10.157 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:10.193 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:10.295 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:10.326 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:10.356 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:59:10.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:10.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:10.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:10.556 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:10.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:59:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:59:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:10.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:10.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:10.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:10.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:10.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:10.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:11.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:11.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:11.165 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T14:59:11.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:11.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:11.281 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:11.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:11.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:11.427 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:11 smithi019 ceph-mon[91296]: pgmap v1224: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 1721/173904 objects degraded (0.990%); 8733/173904 objects misplaced (5.022%); 11 MiB/s, 173 objects/s recovering 2024-09-13T14:59:11.427 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:11 smithi019 ceph-mon[91296]: osdmap e349: 8 total, 8 up, 7 in 2024-09-13T14:59:11.427 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1261120816' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:11.427 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:11 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:11.427 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1721/173904 objects degraded (0.990%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:59:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[107278]: pgmap v1224: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 1721/173904 objects degraded (0.990%); 8733/173904 objects misplaced (5.022%); 11 MiB/s, 173 objects/s recovering 2024-09-13T14:59:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[107278]: osdmap e349: 8 total, 8 up, 7 in 2024-09-13T14:59:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1261120816' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1721/173904 objects degraded (0.990%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:59:11.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[104007]: pgmap v1224: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 1721/173904 objects degraded (0.990%); 8733/173904 objects misplaced (5.022%); 11 MiB/s, 173 objects/s recovering 2024-09-13T14:59:11.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[104007]: osdmap e349: 8 total, 8 up, 7 in 2024-09-13T14:59:11.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1261120816' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:11.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:11.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1721/173904 objects degraded (0.990%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:59:11.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:11.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:11.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:11.747 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.789 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:11.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:11.890 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:11.946 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:11.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:12.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:12.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:12.188 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:12.224 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:12.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:12.365 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:12 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:12.365 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:12 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:12.365 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:12 smithi003 ceph-mon[107278]: osdmap e350: 8 total, 8 up, 7 in 2024-09-13T14:59:12.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:12 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:12.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:12 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:12.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:12 smithi003 ceph-mon[104007]: osdmap e350: 8 total, 8 up, 7 in 2024-09-13T14:59:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.366 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:12.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:12 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:12 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:12.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:12 smithi019 ceph-mon[91296]: osdmap e350: 8 total, 8 up, 7 in 2024-09-13T14:59:12.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:12.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:12.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:12.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:12.739 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:59:12.740 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:59:12.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:12.776 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:59:12.776 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:12.809 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:59:12.810 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:12.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:12.843 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:59:12.843 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:12.886 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:59:12.886 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:12.914 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:59:12.914 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:12.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:12.949 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:59:12.949 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:12.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:12.981 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:59:12.982 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:13.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:13.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:13.208 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.246 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:13.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:13.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:13.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:13.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:13 smithi019 ceph-mon[91296]: pgmap v1227: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 714 op/s; 1880/171588 objects degraded (1.096%); 8395/171588 objects misplaced (4.893%); 8.0 MiB/s, 127 objects/s recovering 2024-09-13T14:59:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:13 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1819693520' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:13.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:13 smithi003 ceph-mon[104007]: pgmap v1227: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 714 op/s; 1880/171588 objects degraded (1.096%); 8395/171588 objects misplaced (4.893%); 8.0 MiB/s, 127 objects/s recovering 2024-09-13T14:59:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:13 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1819693520' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:13 smithi003 ceph-mon[107278]: pgmap v1227: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 714 op/s; 1880/171588 objects degraded (1.096%); 8395/171588 objects misplaced (4.893%); 8.0 MiB/s, 127 objects/s recovering 2024-09-13T14:59:13.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:13 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1819693520' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:13.487 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:13.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:13.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:13.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:13.763 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:13.800 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:13.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:14.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:14.083 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:14.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:59:14.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:14.215 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:14.215 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:14.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:14.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:14.379 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:59:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:59:14] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:59:14.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:14.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:14.599 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:14.630 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:14.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:14.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:14.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:14.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:15.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:15.065 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:15.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:15.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:15.315 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:15.343 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:15 smithi019 ceph-mon[91296]: pgmap v1228: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 1880/162711 objects degraded (1.155%); 8025/162711 objects misplaced (4.932%); 7.5 MiB/s, 120 objects/s recovering 2024-09-13T14:59:15.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:15 smithi003 ceph-mon[107278]: pgmap v1228: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 1880/162711 objects degraded (1.155%); 8025/162711 objects misplaced (4.932%); 7.5 MiB/s, 120 objects/s recovering 2024-09-13T14:59:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:15 smithi003 ceph-mon[104007]: pgmap v1228: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 1880/162711 objects degraded (1.155%); 8025/162711 objects misplaced (4.932%); 7.5 MiB/s, 120 objects/s recovering 2024-09-13T14:59:15.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:15.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:15.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:15.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:15.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:15.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:15.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:15.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:16.009 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:16.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:16.133 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:16.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:59:16.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:16.257 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:16.352 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:59:16.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:16.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:16 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2701461204' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:59:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:16 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:59:16.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1880/162711 objects degraded (1.155%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:16 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2701461204' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:59:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:16 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:59:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1880/162711 objects degraded (1.155%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:16 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2701461204' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:59:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:16 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:59:16.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1880/162711 objects degraded (1.155%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:16.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:16.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:16.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:16.787 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:16.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:16.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:16.955 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:17.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:17.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:17.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:59:17.200 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.234 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:17.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:17.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:17.362 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:17.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:17 smithi019 ceph-mon[91296]: pgmap v1229: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 2023/160431 objects degraded (1.261%); 7684/160431 objects misplaced (4.790%); 8.3 MiB/s, 133 objects/s recovering 2024-09-13T14:59:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:17 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:59:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:17 smithi019 ceph-mon[91296]: osdmap e351: 8 total, 8 up, 7 in 2024-09-13T14:59:17.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:17.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:17.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:17.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:17 smithi003 ceph-mon[107278]: pgmap v1229: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 2023/160431 objects degraded (1.261%); 7684/160431 objects misplaced (4.790%); 8.3 MiB/s, 133 objects/s recovering 2024-09-13T14:59:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:17 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:59:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:17 smithi003 ceph-mon[107278]: osdmap e351: 8 total, 8 up, 7 in 2024-09-13T14:59:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:17 smithi003 ceph-mon[104007]: pgmap v1229: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 2023/160431 objects degraded (1.261%); 7684/160431 objects misplaced (4.790%); 8.3 MiB/s, 133 objects/s recovering 2024-09-13T14:59:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:17 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:59:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:17 smithi003 ceph-mon[104007]: osdmap e351: 8 total, 8 up, 7 in 2024-09-13T14:59:17.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:17.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:17.901 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:17.937 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:18.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:18.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:18.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:18.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:18.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:18.343 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:18.394 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:18.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:18.473 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:18.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:18.611 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:18.713 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:18.743 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:18.844 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:18.876 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:18.978 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:19.008 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:19.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:19.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:19.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:18 smithi019 ceph-mon[91296]: pgmap v1231: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 635 op/s; 2023/158871 objects degraded (1.273%); 7684/158871 objects misplaced (4.837%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:59:19.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:19.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:19.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:18 smithi003 ceph-mon[107278]: pgmap v1231: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 635 op/s; 2023/158871 objects degraded (1.273%); 7684/158871 objects misplaced (4.837%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:59:19.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:19.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:19.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:18 smithi003 ceph-mon[104007]: pgmap v1231: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 635 op/s; 2023/158871 objects degraded (1.273%); 7684/158871 objects misplaced (4.837%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T14:59:19.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:19.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:19.398 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:19.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:19.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:19.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:19.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:19.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:19.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:19.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:19.949 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:20.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:20.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:20.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:20.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:20.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:20.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:20.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:20.479 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:20.507 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:20.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:20.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:59:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:20.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:59:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:59:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:20.762 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:20.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:20.892 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:20.918 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:21.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:21.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:21.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:21.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:21.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:21.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:21 smithi019 ceph-mon[91296]: pgmap v1232: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 2023/149850 objects degraded (1.350%); 7402/149850 objects misplaced (4.940%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:59:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2023/149850 objects degraded (1.350%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:21.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:21 smithi003 ceph-mon[107278]: pgmap v1232: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 2023/149850 objects degraded (1.350%); 7402/149850 objects misplaced (4.940%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:59:21.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2023/149850 objects degraded (1.350%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:21 smithi003 ceph-mon[104007]: pgmap v1232: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 2023/149850 objects degraded (1.350%); 7402/149850 objects misplaced (4.940%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:59:21.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2023/149850 objects degraded (1.350%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:21.463 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:21.494 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:21.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:21.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:21.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:21.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:21.877 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:21.911 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:22.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:22.045 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:22.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:22.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:22.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:22.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:22.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:22.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:22.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:22.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:22.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:22.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:22.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:22.864 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:22.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:22.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:23.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:23.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:23.142 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:23.144 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:23.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:23 smithi003 ceph-mon[107278]: pgmap v1233: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 2204/147282 objects degraded (1.496%); 7033/147282 objects misplaced (4.775%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:59:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:23 smithi003 ceph-mon[104007]: pgmap v1233: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 2204/147282 objects degraded (1.496%); 7033/147282 objects misplaced (4.775%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:59:23.244 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:23.377 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:23.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.410 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:23.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:23 smithi019 ceph-mon[91296]: pgmap v1233: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 2204/147282 objects degraded (1.496%); 7033/147282 objects misplaced (4.775%); 2.2 MiB/s, 34 objects/s recovering 2024-09-13T14:59:23.510 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:23.543 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:23.644 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:23.757 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:23.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:23.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:23.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:23.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:23.964 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:23.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:24.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:24.093 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:24.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:24.227 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:24.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:24.343 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:59:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:59:24] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:59:24.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:24.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:24.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:24.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:24.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.823 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:24.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:24.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:24.952 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:25.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:25.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:25.200 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:25.231 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:25.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:25.333 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:25.361 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:25.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:25 smithi019 ceph-mon[91296]: pgmap v1234: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 2204/135474 objects degraded (1.627%); 6744/135474 objects misplaced (4.978%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:59:25.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:25 smithi003 ceph-mon[107278]: pgmap v1234: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 2204/135474 objects degraded (1.627%); 6744/135474 objects misplaced (4.978%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:59:25.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:25 smithi003 ceph-mon[104007]: pgmap v1234: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 2204/135474 objects degraded (1.627%); 6744/135474 objects misplaced (4.978%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:59:25.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:25.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:25.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:25.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:25.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:25.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:25.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.961 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:25.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:26.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:26.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:26.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:26.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:26.357 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:26.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2204/135474 objects degraded (1.627%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:26.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2204/135474 objects degraded (1.627%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:26.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2204/135474 objects degraded (1.627%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:26.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:26.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:26.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:26.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:26.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:26.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:26.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:26.848 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:26.848 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:26.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:26.928 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:26.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:27.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:27.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:27.121 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:27 smithi003 ceph-mon[104007]: pgmap v1235: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 2441/132774 objects degraded (1.838%); 6340/132774 objects misplaced (4.775%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:59:27.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:27.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:27.357 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:27.394 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:27.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:27 smithi019 ceph-mon[91296]: pgmap v1235: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 2441/132774 objects degraded (1.838%); 6340/132774 objects misplaced (4.775%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:59:27.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:27 smithi003 ceph-mon[107278]: pgmap v1235: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 2441/132774 objects degraded (1.838%); 6340/132774 objects misplaced (4.775%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:59:27.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:27.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:27.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:27.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:27.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:59:27.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:27.936 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:27.967 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:27.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:28.069 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:28.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:28.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:28.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:28.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:28.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1621045176' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:28 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:28.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1621045176' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:28 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1621045176' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:28 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:28.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:28.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:28.641 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:28.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:28.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:28.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.803 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:28.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:28.875 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T14:59:28.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:28.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:29.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:29.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:29.196 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:29.226 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:29.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:29.359 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:29.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[104007]: pgmap v1236: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 2441/131199 objects degraded (1.861%); 6340/131199 objects misplaced (4.832%); 890 KiB/s, 13 objects/s recovering 2024-09-13T14:59:29.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:29.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:29.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[104007]: osdmap e352: 8 total, 8 up, 7 in 2024-09-13T14:59:29.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:29.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:29.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:59:29.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:59:29.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:29.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[107278]: pgmap v1236: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 2441/131199 objects degraded (1.861%); 6340/131199 objects misplaced (4.832%); 890 KiB/s, 13 objects/s recovering 2024-09-13T14:59:29.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:29.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:29.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[107278]: osdmap e352: 8 total, 8 up, 7 in 2024-09-13T14:59:29.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:29.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:29.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:59:29.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:59:29.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:29 smithi019 ceph-mon[91296]: pgmap v1236: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 2441/131199 objects degraded (1.861%); 6340/131199 objects misplaced (4.832%); 890 KiB/s, 13 objects/s recovering 2024-09-13T14:59:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:29 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:29 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:29 smithi019 ceph-mon[91296]: osdmap e352: 8 total, 8 up, 7 in 2024-09-13T14:59:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:59:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:59:29.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T14:59:29.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:29.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:29.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.634 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:29.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:29.735 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.765 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:29.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:29.866 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:29.898 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:30.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:30.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:30.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:30.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:30.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:30.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:30.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:30.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:30.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:30.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:59:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:59:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:30.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:30.761 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:30.862 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:30.895 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:30.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:31.022 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:31.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:31.078 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:59:31.079 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:59:31.128 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:59:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:31.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:31.173 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:59:31.173 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:31.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:31.213 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:59:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:31.243 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:59:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:31.272 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:59:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:31.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:31.307 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:59:31.307 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:31.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:31.342 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:59:31.343 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:31 smithi019 ceph-mon[91296]: pgmap v1238: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 949 op/s; 2441/121359 objects degraded (2.011%); 6056/121359 objects misplaced (4.990%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:59:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2441/121359 objects degraded (2.011%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1178284485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:31.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:31 smithi003 ceph-mon[104007]: pgmap v1238: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 949 op/s; 2441/121359 objects degraded (2.011%); 6056/121359 objects misplaced (4.990%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:59:31.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2441/121359 objects degraded (2.011%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:31.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1178284485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:31.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:31 smithi003 ceph-mon[107278]: pgmap v1238: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 949 op/s; 2441/121359 objects degraded (2.011%); 6056/121359 objects misplaced (4.990%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:59:31.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2441/121359 objects degraded (2.011%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:31.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1178284485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:31.439 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:31.463 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:31.566 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:31.598 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:31.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:31.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:31.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:31.877 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:31.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:32.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:32.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:32.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:32.149 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:32.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:32.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:32.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:32.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:32.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:32.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:32.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:32.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:59:32.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:32.718 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:32.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:32.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:32.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:33.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:33.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:33.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:33.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:33.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:33 smithi019 ceph-mon[91296]: pgmap v1239: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 955 op/s; 2692/118602 objects degraded (2.270%); 5642/118602 objects misplaced (4.757%); 896 KiB/s, 13 objects/s recovering 2024-09-13T14:59:33.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:33 smithi003 ceph-mon[107278]: pgmap v1239: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 955 op/s; 2692/118602 objects degraded (2.270%); 5642/118602 objects misplaced (4.757%); 896 KiB/s, 13 objects/s recovering 2024-09-13T14:59:33.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:33 smithi003 ceph-mon[104007]: pgmap v1239: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 955 op/s; 2692/118602 objects degraded (2.270%); 5642/118602 objects misplaced (4.757%); 896 KiB/s, 13 objects/s recovering 2024-09-13T14:59:33.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:33.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:33.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:33.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:33.709 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:33.741 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:33.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:33.871 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:33.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:33.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:34.002 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:34.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:34.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:34.370 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:34.409 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:59:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:59:34] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:59:34.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:34.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:34.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:34.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:34.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:34.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:34.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:34.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:34.978 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:34.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:35.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:35.117 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:35.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:35.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:35.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:35.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:35.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:35 smithi019 ceph-mon[91296]: pgmap v1240: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 910 op/s; 2692/108159 objects degraded (2.489%); 5352/108159 objects misplaced (4.948%); 896 KiB/s, 13 objects/s recovering 2024-09-13T14:59:35.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:35 smithi003 ceph-mon[107278]: pgmap v1240: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 910 op/s; 2692/108159 objects degraded (2.489%); 5352/108159 objects misplaced (4.948%); 896 KiB/s, 13 objects/s recovering 2024-09-13T14:59:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:35 smithi003 ceph-mon[104007]: pgmap v1240: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 910 op/s; 2692/108159 objects degraded (2.489%); 5352/108159 objects misplaced (4.948%); 896 KiB/s, 13 objects/s recovering 2024-09-13T14:59:35.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:35.530 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:35.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:35.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:35.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:35.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:35.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:35.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:35.940 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:36.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:36.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:36.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:36.209 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:36.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:36.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:36.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:36 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2692/108159 objects degraded (2.489%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:36.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:36 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2692/108159 objects degraded (2.489%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:36.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:36.481 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:36.582 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:36.612 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:36.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:36 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2692/108159 objects degraded (2.489%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:36.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:36.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:36.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:36.901 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:37.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:37.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:37.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:37.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:37.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:37.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:37.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:37.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:37.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:37 smithi003 ceph-mon[107278]: pgmap v1241: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 906 op/s; 2881/105579 objects degraded (2.729%); 4978/105579 objects misplaced (4.715%); 953 KiB/s, 14 objects/s recovering 2024-09-13T14:59:37.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:37 smithi003 ceph-mon[104007]: pgmap v1241: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 906 op/s; 2881/105579 objects degraded (2.729%); 4978/105579 objects misplaced (4.715%); 953 KiB/s, 14 objects/s recovering 2024-09-13T14:59:37.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:37.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:37.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:37.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:37 smithi019 ceph-mon[91296]: pgmap v1241: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 906 op/s; 2881/105579 objects degraded (2.729%); 4978/105579 objects misplaced (4.715%); 953 KiB/s, 14 objects/s recovering 2024-09-13T14:59:37.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:37.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.894 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:37.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:37.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:38.023 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:38.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:38.198 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:59:38.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:59:38.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:38.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:38.424 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:38.456 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:38.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:38.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:38.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:38.639 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/497536588' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:59:38.639 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:38 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:59:38.640 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/497536588' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:59:38.640 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:38 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:59:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/497536588' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:59:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:38 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:59:38.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:38.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:38.816 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:38.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:38.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:39.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:39.058 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:39.160 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:39.192 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:39.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:39.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:39.325 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:39 smithi003 ceph-mon[107278]: pgmap v1242: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 2881/103797 objects degraded (2.776%); 4978/103797 objects misplaced (4.796%); 953 KiB/s, 14 objects/s recovering 2024-09-13T14:59:39.325 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:39 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:59:39.325 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:39 smithi003 ceph-mon[107278]: osdmap e353: 8 total, 8 up, 7 in 2024-09-13T14:59:39.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:39 smithi003 ceph-mon[104007]: pgmap v1242: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 2881/103797 objects degraded (2.776%); 4978/103797 objects misplaced (4.796%); 953 KiB/s, 14 objects/s recovering 2024-09-13T14:59:39.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:39 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:59:39.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:39 smithi003 ceph-mon[104007]: osdmap e353: 8 total, 8 up, 7 in 2024-09-13T14:59:39.327 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:39.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:39.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:39.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:39.490 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T14:59:39.490 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T14:59:39.490 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T14:59:39.490 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T14:59:39.490 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T14:59:39.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:39.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:59:39.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:39 smithi019 ceph-mon[91296]: pgmap v1242: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 2881/103797 objects degraded (2.776%); 4978/103797 objects misplaced (4.796%); 953 KiB/s, 14 objects/s recovering 2024-09-13T14:59:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:39 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:59:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:39 smithi019 ceph-mon[91296]: osdmap e353: 8 total, 8 up, 7 in 2024-09-13T14:59:39.727 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:39.759 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:39.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:39.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:40.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:40.048 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:40.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:40.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:40.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:40.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:59:40.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:40.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:40.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:40.536 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:40.582 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:59:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:59:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:40.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:40.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:40.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:40.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:40.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:41.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:41.109 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:41.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:41.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:41.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:41.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.383 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:41.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:41.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:41 smithi019 ceph-mon[91296]: pgmap v1244: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 912 op/s; 2881/93963 objects degraded (3.066%); 4668/93963 objects misplaced (4.968%); 953 KiB/s, 14 objects/s recovering 2024-09-13T14:59:41.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2881/93963 objects degraded (3.066%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:41.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:41.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:41.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:41.669 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:41.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:41 smithi003 ceph-mon[104007]: pgmap v1244: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 912 op/s; 2881/93963 objects degraded (3.066%); 4668/93963 objects misplaced (4.968%); 953 KiB/s, 14 objects/s recovering 2024-09-13T14:59:41.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2881/93963 objects degraded (3.066%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:41.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:41 smithi003 ceph-mon[107278]: pgmap v1244: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 912 op/s; 2881/93963 objects degraded (3.066%); 4668/93963 objects misplaced (4.968%); 953 KiB/s, 14 objects/s recovering 2024-09-13T14:59:41.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2881/93963 objects degraded (3.066%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:41.771 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:41.801 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:41.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:41.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:42.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:42.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:42.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:42.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:42.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:42.355 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:42.390 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:42.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:42.518 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:42.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:42.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:42.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:42.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:42.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:42.904 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:42.940 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:43.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:43.071 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:43.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:43.206 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:43.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:43.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:43.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:43.489 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:43.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:43.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:43.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:43 smithi003 ceph-mon[107278]: pgmap v1245: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 912 op/s; 3138/91206 objects degraded (3.441%); 4243/91206 objects misplaced (4.652%); 998 KiB/s, 15 objects/s recovering 2024-09-13T14:59:43.649 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:43.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:43 smithi003 ceph-mon[104007]: pgmap v1245: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 912 op/s; 3138/91206 objects degraded (3.441%); 4243/91206 objects misplaced (4.652%); 998 KiB/s, 15 objects/s recovering 2024-09-13T14:59:43.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:43 smithi019 ceph-mon[91296]: pgmap v1245: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 912 op/s; 3138/91206 objects degraded (3.441%); 4243/91206 objects misplaced (4.652%); 998 KiB/s, 15 objects/s recovering 2024-09-13T14:59:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:43.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:43.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:43.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:43.877 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:43.913 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:44.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:44.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:44.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:44.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:44.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:44.286 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:44.326 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:44.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:44.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:59:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:59:44] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:59:44.464 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:44.566 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:44.598 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:44.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:44.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:44.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:44.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:44.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:45.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:45.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:45.149 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:45.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:45.283 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:45.384 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:45.419 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:45.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:45.435 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:45.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:45.543 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:45.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:45 smithi019 ceph-mon[91296]: pgmap v1246: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 933 op/s; 3138/80133 objects degraded (3.916%); 3965/80133 objects misplaced (4.948%); 998 KiB/s, 15 objects/s recovering 2024-09-13T14:59:45.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:45 smithi003 ceph-mon[104007]: pgmap v1246: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 933 op/s; 3138/80133 objects degraded (3.916%); 3965/80133 objects misplaced (4.948%); 998 KiB/s, 15 objects/s recovering 2024-09-13T14:59:45.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:45 smithi003 ceph-mon[107278]: pgmap v1246: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 933 op/s; 3138/80133 objects degraded (3.916%); 3965/80133 objects misplaced (4.948%); 998 KiB/s, 15 objects/s recovering 2024-09-13T14:59:45.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:45.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:45.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:45.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:45.956 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:46.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:46.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:46.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:46.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:46.337 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:46.366 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:46.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3138/80133 objects degraded (3.916%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:46.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3138/80133 objects degraded (3.916%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:46.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:46.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:46.615 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:46.646 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3138/80133 objects degraded (3.916%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:46.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:46.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:46.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:46.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:47.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:47.072 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:47.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:47.204 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:47.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:47.262 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T14:59:47.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:47.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:47.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:47.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:47.467 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:47.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:47.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:47.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:47.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:47 smithi019 ceph-mon[91296]: pgmap v1247: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 937 op/s; 3368/77433 objects degraded (4.350%); 3549/77433 objects misplaced (4.583%); 1023 KiB/s, 15 objects/s recovering 2024-09-13T14:59:47.685 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/704531427' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:47.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:47 smithi003 ceph-mon[104007]: pgmap v1247: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 937 op/s; 3368/77433 objects degraded (4.350%); 3549/77433 objects misplaced (4.583%); 1023 KiB/s, 15 objects/s recovering 2024-09-13T14:59:47.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/704531427' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:47.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:47 smithi003 ceph-mon[107278]: pgmap v1247: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 937 op/s; 3368/77433 objects degraded (4.350%); 3549/77433 objects misplaced (4.583%); 1023 KiB/s, 15 objects/s recovering 2024-09-13T14:59:47.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/704531427' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:47.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:47.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:47.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:47.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:47.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:48.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:48.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:48.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:48.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:48.298 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:48.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:48.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:48.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:48.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:48 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/704531427' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:48 smithi019 ceph-mon[91296]: osdmap e354: 8 total, 8 up, 7 in 2024-09-13T14:59:48.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:48 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:48.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/704531427' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:48.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:48 smithi003 ceph-mon[104007]: osdmap e354: 8 total, 8 up, 7 in 2024-09-13T14:59:48.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:48 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:48.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/704531427' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:48.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:48 smithi003 ceph-mon[107278]: osdmap e354: 8 total, 8 up, 7 in 2024-09-13T14:59:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:48.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:48.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:48.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:49.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:49.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:49.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:49.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:49.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:49.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:49.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:49.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:49.547 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:49.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:49 smithi003 ceph-mon[107278]: pgmap v1249: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 3368/75654 objects degraded (4.452%); 3549/75654 objects misplaced (4.691%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:59:49.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:49 smithi003 ceph-mon[104007]: pgmap v1249: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 3368/75654 objects degraded (4.452%); 3549/75654 objects misplaced (4.691%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:59:49.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:49 smithi019 ceph-mon[91296]: pgmap v1249: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 3368/75654 objects degraded (4.452%); 3549/75654 objects misplaced (4.691%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:59:49.715 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:49.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:49.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:49.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:49.959 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:49.991 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:50.093 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.124 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:50.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:50.225 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:50.255 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:50.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:50.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:50.502 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:50.535 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:50.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:50.669 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 14:59:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T14:59:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:50.771 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:50.798 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:50.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:50.925 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:51.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:51.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:51.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:51.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:51.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T14:59:51.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T14:59:51.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T14:59:51.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T14:59:51.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T14:59:51.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T14:59:51.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T14:59:51.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T14:59:51.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:59:51.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:51.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:51.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:51.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:51.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:51.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:51.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:51.317 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:51.344 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:51.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:51.471 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:51.471 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:51.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:51 smithi003 ceph-mon[107278]: pgmap v1250: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 946 op/s; 3368/65562 objects degraded (5.137%); 3274/65562 objects misplaced (4.994%); 1023 KiB/s, 15 objects/s recovering 2024-09-13T14:59:51.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:51 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/818574519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:51.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3368/65562 objects degraded (5.137%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:51.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:51 smithi003 ceph-mon[104007]: pgmap v1250: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 946 op/s; 3368/65562 objects degraded (5.137%); 3274/65562 objects misplaced (4.994%); 1023 KiB/s, 15 objects/s recovering 2024-09-13T14:59:51.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:51 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/818574519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:51.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3368/65562 objects degraded (5.137%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:51.475 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:51.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:51.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:51 smithi019 ceph-mon[91296]: pgmap v1250: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 946 op/s; 3368/65562 objects degraded (5.137%); 3274/65562 objects misplaced (4.994%); 1023 KiB/s, 15 objects/s recovering 2024-09-13T14:59:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:51 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/818574519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3368/65562 objects degraded (5.137%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:51.721 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:51.753 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:51.855 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:51.887 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:51.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:52.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:52.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:59:52.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:52.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:52.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:52.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:52.494 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:52.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:52.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:52.628 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:52.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:52.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:52.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:52.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:53.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:53.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:53.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:59:53.177 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:53.209 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:53.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:53.338 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:53.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:53 smithi003 ceph-mon[104007]: pgmap v1251: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 945 op/s; 3586/62844 objects degraded (5.706%); 2890/62844 objects misplaced (4.599%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:59:53.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:53 smithi003 ceph-mon[107278]: pgmap v1251: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 945 op/s; 3586/62844 objects degraded (5.706%); 2890/62844 objects misplaced (4.599%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:59:53.440 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.471 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:53.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:53.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:53.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:53 smithi019 ceph-mon[91296]: pgmap v1251: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 945 op/s; 3586/62844 objects degraded (5.706%); 2890/62844 objects misplaced (4.599%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:59:53.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:53.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:53.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:53.823 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:53.824 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:53.879 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:53.906 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:54.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:54.033 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:54.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:54.164 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:54.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:59:54.265 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 14:59:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:14:59:54] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:59:54.266 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.295 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:54.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:54.300 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:59:54.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:54.427 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:54.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:54.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:54.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:54 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2466321864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:59:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:54 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:59:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:54 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2466321864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:59:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:54 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:59:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:54 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2466321864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:59:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:54 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:59:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.704 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:54.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:54.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:54.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:54.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:54.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:55.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:55.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:55.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:55.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:55.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:55.409 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:55.511 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.544 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:55.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:55.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:55 smithi019 ceph-mon[91296]: pgmap v1252: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 968 op/s; 3586/51057 objects degraded (7.024%); 2592/51057 objects misplaced (5.077%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:59:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:55 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:59:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:55 smithi019 ceph-mon[91296]: osdmap e355: 8 total, 8 up, 7 in 2024-09-13T14:59:55.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:55 smithi003 ceph-mon[104007]: pgmap v1252: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 968 op/s; 3586/51057 objects degraded (7.024%); 2592/51057 objects misplaced (5.077%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:59:55.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:55 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:59:55.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:55 smithi003 ceph-mon[104007]: osdmap e355: 8 total, 8 up, 7 in 2024-09-13T14:59:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:55 smithi003 ceph-mon[107278]: pgmap v1252: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 968 op/s; 3586/51057 objects degraded (7.024%); 2592/51057 objects misplaced (5.077%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:59:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:55 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:59:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:55 smithi003 ceph-mon[107278]: osdmap e355: 8 total, 8 up, 7 in 2024-09-13T14:59:55.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:55.787 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:55.815 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:55.917 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:55.947 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:56.049 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:56.079 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:56.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:56.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:56.331 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:56.361 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:56.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:56.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:56.607 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:56.639 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3586/51057 objects degraded (7.024%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:56.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3586/51057 objects degraded (7.024%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:56.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3586/51057 objects degraded (7.024%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:59:56.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:56.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:56.892 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:56.919 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:57.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:57.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:57.309 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:57.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:57.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:57.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:57.622 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:57 smithi019 ceph-mon[91296]: pgmap v1254: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1009 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 3798/48477 objects degraded (7.835%); 2202/48477 objects misplaced (4.542%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:59:57.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:57 smithi003 ceph-mon[104007]: pgmap v1254: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1009 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 3798/48477 objects degraded (7.835%); 2202/48477 objects misplaced (4.542%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:59:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:57 smithi003 ceph-mon[107278]: pgmap v1254: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 1009 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 3798/48477 objects degraded (7.835%); 2202/48477 objects misplaced (4.542%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:59:57.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:57.755 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:57.857 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:57.882 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:57.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:58.011 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:58.113 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:58.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:58.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:58.400 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:58.427 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:58.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:58.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:58.607 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:58.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:58.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:58.725 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:58.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:58.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:58.876 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:58.920 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:58.920 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:58.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:59.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:59.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:59.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:59.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:59.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:59.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:59.369 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T14:59:59.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.432 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:59.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:59.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:59.571 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:59.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:59.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:59.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:59 smithi019 ceph-mon[91296]: pgmap v1255: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 973 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 963 op/s; 3798/46743 objects degraded (8.125%); 2202/46743 objects misplaced (4.711%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:59:59.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2045777184' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 14:59:59 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:59 smithi003 ceph-mon[104007]: pgmap v1255: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 973 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 963 op/s; 3798/46743 objects degraded (8.125%); 2202/46743 objects misplaced (4.711%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:59:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2045777184' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 14:59:59 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:59 smithi003 ceph-mon[107278]: pgmap v1255: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 973 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 963 op/s; 3798/46743 objects degraded (8.125%); 2202/46743 objects misplaced (4.711%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:59:59.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2045777184' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 14:59:59 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:59.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:59.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T14:59:59.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:59.953 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:59.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T14:59:59.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:00.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:00.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:00.231 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:00.260 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:00.362 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:00.385 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:00.487 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.510 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:00.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:00.611 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:00.643 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:00:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:00:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:00 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:00:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:00 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:00 smithi019 ceph-mon[91296]: osdmap e356: 8 total, 8 up, 7 in 2024-09-13T15:00:00.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:00 smithi019 ceph-mon[91296]: Health detail: HEALTH_WARN Degraded data redundancy: 3798/46743 objects degraded (8.125%), 1 pg degraded, 1 pg undersized 2024-09-13T15:00:00.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:00 smithi019 ceph-mon[91296]: [WRN] PG_DEGRADED: Degraded data redundancy: 3798/46743 objects degraded (8.125%), 1 pg degraded, 1 pg undersized 2024-09-13T15:00:00.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:00 smithi019 ceph-mon[91296]: pg 12.a is stuck undersized for 106s, current state active+recovering+undersized+degraded+remapped, last acting [2,3] 2024-09-13T15:00:00.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:00:00.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:00.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[107278]: osdmap e356: 8 total, 8 up, 7 in 2024-09-13T15:00:00.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[107278]: Health detail: HEALTH_WARN Degraded data redundancy: 3798/46743 objects degraded (8.125%), 1 pg degraded, 1 pg undersized 2024-09-13T15:00:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[107278]: [WRN] PG_DEGRADED: Degraded data redundancy: 3798/46743 objects degraded (8.125%), 1 pg degraded, 1 pg undersized 2024-09-13T15:00:00.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[107278]: pg 12.a is stuck undersized for 106s, current state active+recovering+undersized+degraded+remapped, last acting [2,3] 2024-09-13T15:00:00.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:00:00.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:00.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[104007]: osdmap e356: 8 total, 8 up, 7 in 2024-09-13T15:00:00.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[104007]: Health detail: HEALTH_WARN Degraded data redundancy: 3798/46743 objects degraded (8.125%), 1 pg degraded, 1 pg undersized 2024-09-13T15:00:00.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[104007]: [WRN] PG_DEGRADED: Degraded data redundancy: 3798/46743 objects degraded (8.125%), 1 pg degraded, 1 pg undersized 2024-09-13T15:00:00.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:00 smithi003 ceph-mon[104007]: pg 12.a is stuck undersized for 106s, current state active+recovering+undersized+degraded+remapped, last acting [2,3] 2024-09-13T15:00:00.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:00.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:00.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:00.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:01.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:01.058 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:01.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:01.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:01.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:01.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:01.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:01 smithi003 ceph-mon[104007]: pgmap v1257: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 779 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 3798/37461 objects degraded (10.139%); 1905/37461 objects misplaced (5.085%); 744 KiB/s, 11 objects/s recovering 2024-09-13T15:00:01.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:01 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3798/37461 objects degraded (10.139%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:01.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:01 smithi003 ceph-mon[107278]: pgmap v1257: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 779 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 3798/37461 objects degraded (10.139%); 1905/37461 objects misplaced (5.085%); 744 KiB/s, 11 objects/s recovering 2024-09-13T15:00:01.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:01 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3798/37461 objects degraded (10.139%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:01.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:01.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:01.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:01 smithi019 ceph-mon[91296]: pgmap v1257: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 779 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 3798/37461 objects degraded (10.139%); 1905/37461 objects misplaced (5.085%); 744 KiB/s, 11 objects/s recovering 2024-09-13T15:00:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:01 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3798/37461 objects degraded (10.139%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:01.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:01.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:01.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:01.878 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:01.980 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:02.007 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:02.109 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:02.139 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:02.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:02.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:02.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:02.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:02.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:02.577 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:02.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:02.707 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:02.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:02.835 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:02.936 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:02.966 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:03.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:03.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:03.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:03.241 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:03.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:03.372 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:03.474 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.507 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:03.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:03.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:03.643 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:03.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:03 smithi019 ceph-mon[91296]: pgmap v1258: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 729 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 3979/35034 objects degraded (11.358%); 1535/35034 objects misplaced (4.381%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T15:00:03.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:03 smithi003 ceph-mon[104007]: pgmap v1258: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 729 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 3979/35034 objects degraded (11.358%); 1535/35034 objects misplaced (4.381%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T15:00:03.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:03 smithi003 ceph-mon[107278]: pgmap v1258: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 729 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 3979/35034 objects degraded (11.358%); 1535/35034 objects misplaced (4.381%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T15:00:03.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:03.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:03.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.917 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:03.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:04.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:04.056 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:04.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:04.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:04.303 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.337 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:04.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:04.369 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:00:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:00:04] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T15:00:04.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:00:04.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:04.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:04.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:04.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:04.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:04.794 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:04.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:04.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:05.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:05.610 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:05.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:05 smithi019 ceph-mon[91296]: pgmap v1259: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 504 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 916 op/s; 3979/24231 objects degraded (16.421%); 1263/24231 objects misplaced (5.212%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:05.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:05 smithi003 ceph-mon[107278]: pgmap v1259: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 504 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 916 op/s; 3979/24231 objects degraded (16.421%); 1263/24231 objects misplaced (5.212%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:05.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:05 smithi003 ceph-mon[104007]: pgmap v1259: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 504 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 916 op/s; 3979/24231 objects degraded (16.421%); 1263/24231 objects misplaced (5.212%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:05.711 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:05.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:05.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:05.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:05.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:05.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:06.032 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:06.134 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:06.167 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:06.244 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:06.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:06.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:06.424 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:06.453 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:06.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3979/24231 objects degraded (16.421%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:06.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3979/24231 objects degraded (16.421%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:06.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3979/24231 objects degraded (16.421%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:06.694 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:06.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:06.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:06.861 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:06.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:07.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:07.112 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.142 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:07.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:07.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:07.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:07.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:07.406 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:07.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:07.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:07.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:07 smithi019 ceph-mon[91296]: pgmap v1260: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 449 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 4161/21594 objects degraded (19.269%); 900/21594 objects misplaced (4.168%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:07.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:07 smithi003 ceph-mon[104007]: pgmap v1260: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 449 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 4161/21594 objects degraded (19.269%); 900/21594 objects misplaced (4.168%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:07.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:07 smithi003 ceph-mon[107278]: pgmap v1260: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 449 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 4161/21594 objects degraded (19.269%); 900/21594 objects misplaced (4.168%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:07.694 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:07.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:07.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:07.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:07.975 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:08.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:08.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:08.186 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T15:00:08.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:08.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:00:08.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:08.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:08.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:08 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2341019310' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:08.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:08.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:08.566 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:08.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:08 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2341019310' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:08 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:08 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:00:08.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:08 smithi019 ceph-mon[91296]: osdmap e357: 8 total, 8 up, 7 in 2024-09-13T15:00:08.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:08 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2341019310' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:08.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:08 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:08.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:08 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:00:08.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:08 smithi003 ceph-mon[107278]: osdmap e357: 8 total, 8 up, 7 in 2024-09-13T15:00:08.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:08 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:08.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:08 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:00:08.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:08 smithi003 ceph-mon[104007]: osdmap e357: 8 total, 8 up, 7 in 2024-09-13T15:00:08.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:08.720 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:00:08.720 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:08.761 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:00:08.761 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:08.806 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:00:08.806 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:08.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:08.849 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:00:08.850 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:08.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:08.878 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:00:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:08.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:00:08.908 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:08.945 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:00:08.945 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:08.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:08.977 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:00:08.978 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:09.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:09.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:09.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:09.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:09.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:09.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:09.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:09.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:09.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T15:00:09.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:09.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:09.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:09 smithi003 ceph-mon[107278]: pgmap v1262: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 412 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 675 op/s; 4161/19830 objects degraded (20.983%); 900/19830 objects misplaced (4.539%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:00:09.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/81599325' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:09.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:09 smithi003 ceph-mon[104007]: pgmap v1262: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 412 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 675 op/s; 4161/19830 objects degraded (20.983%); 900/19830 objects misplaced (4.539%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:00:09.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/81599325' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:09 smithi019 ceph-mon[91296]: pgmap v1262: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 412 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 675 op/s; 4161/19830 objects degraded (20.983%); 900/19830 objects misplaced (4.539%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:00:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/81599325' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:09.714 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:00:09.715 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:00:09.715 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:00:09.715 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:00:09.715 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:00:09.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:09.746 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:09.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:09.878 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:09.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:10.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:10.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:10.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:10.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:10.305 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:10.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:10.441 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:10.543 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:10.578 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:10.582 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:00:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:00:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:10.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:10.718 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:10.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:10.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:10.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:10.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:10.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:11.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:11.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:11.144 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:11.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:00:11.244 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:11.273 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:11.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:11.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 115623 objects 2024-09-13T15:00:11.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :136.933 2024-09-13T15:00:11.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:11.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:11.502 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:11.524 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:11.581 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:11 smithi019 ceph-mon[91296]: pgmap v1263: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 207 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 915 op/s; 4161/9999 objects degraded (41.614%); 581/9999 objects misplaced (5.811%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:11.582 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4161/9999 objects degraded (41.614%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:11.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:11.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:11.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:11.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:11 smithi003 ceph-mon[107278]: pgmap v1263: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 207 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 915 op/s; 4161/9999 objects degraded (41.614%); 581/9999 objects misplaced (5.811%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:11.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4161/9999 objects degraded (41.614%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:11.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:11 smithi003 ceph-mon[104007]: pgmap v1263: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 207 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 915 op/s; 4161/9999 objects degraded (41.614%); 581/9999 objects misplaced (5.811%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:11.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4161/9999 objects degraded (41.614%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:11.752 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:11.774 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:11.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:11.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:11.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:12.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:12.024 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:12.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:12.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:12.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:12.272 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:12.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:12.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:12.395 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:12.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:12.520 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:12.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:12.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:12.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:00:12.746 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:12.769 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:12.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:12.894 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:12.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:13.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:13.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:13.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:13.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:13.246 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:13.272 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:13.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:13.401 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:13.502 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:13.528 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:13.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:13.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:13 smithi019 ceph-mon[91296]: pgmap v1264: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 152 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 923 op/s; 4381/7335 objects degraded (59.727%); 177/7335 objects misplaced (2.413%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:13.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:13 smithi003 ceph-mon[104007]: pgmap v1264: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 152 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 923 op/s; 4381/7335 objects degraded (59.727%); 177/7335 objects misplaced (2.413%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:13.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:13.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:13 smithi003 ceph-mon[107278]: pgmap v1264: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 152 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 923 op/s; 4381/7335 objects degraded (59.727%); 177/7335 objects misplaced (2.413%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:13.755 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:13.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:13.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:13.911 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:14.039 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:14.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:14.168 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.217 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-09-13T15:00:14.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-09-13T15:00:14.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:14.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:14.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:14.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:00:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:00:14] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-13T15:00:14.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:14.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:14.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:14.678 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:14.806 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.908 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:14.935 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:15.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:15.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:15.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:15.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:15.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:15.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:15.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:15.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:15.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:15.536 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:15.665 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:15 smithi019 ceph-mon[91296]: pgmap v1265: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.4 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 800 op/s; 4381/213 objects degraded (2056.808%); 26/213 objects misplaced (12.207%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:15.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:15 smithi003 ceph-mon[104007]: pgmap v1265: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.4 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 800 op/s; 4381/213 objects degraded (2056.808%); 26/213 objects misplaced (12.207%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:15 smithi003 ceph-mon[107278]: pgmap v1265: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.4 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 800 op/s; 4381/213 objects degraded (2056.808%); 26/213 objects misplaced (12.207%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T15:00:15.691 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:15.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:15.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:15.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:15.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:15.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:16.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:16.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:16.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:16.194 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:16.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:16.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:16.421 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:16.448 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:16.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:16.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:16.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4381/213 objects degraded (2056.808%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:16.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4381/213 objects degraded (2056.808%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:16.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4381/213 objects degraded (2056.808%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:16.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:16.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:16.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:16.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:16.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:16.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:17.050 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:17.076 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:17.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:17.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:17.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:17.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:17.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:17.427 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:17.454 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:17.556 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:17.583 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:17 smithi019 ceph-mon[91296]: pgmap v1266: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.4 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 712 op/s; 4381/213 objects degraded (2056.808%); 26/213 objects misplaced (12.207%); 595 KiB/s, 9 objects/s recovering 2024-09-13T15:00:17.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:17 smithi003 ceph-mon[107278]: pgmap v1266: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.4 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 712 op/s; 4381/213 objects degraded (2056.808%); 26/213 objects misplaced (12.207%); 595 KiB/s, 9 objects/s recovering 2024-09-13T15:00:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:17 smithi003 ceph-mon[104007]: pgmap v1266: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 3.4 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 712 op/s; 4381/213 objects degraded (2056.808%); 26/213 objects misplaced (12.207%); 595 KiB/s, 9 objects/s recovering 2024-09-13T15:00:17.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:17.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:17.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:17.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:17.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:17.966 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:18.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:18.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:18.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:18.219 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:18.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:18.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:18.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:18.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:18.571 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:18.597 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:18.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:18.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:18.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:18.948 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:18.975 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:19.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:19.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:19.206 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:19.233 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:19.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:19.361 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:19.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:19.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:19.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:19.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:19 smithi019 ceph-mon[91296]: pgmap v1267: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 643 op/s; 4399/78 objects degraded (5639.744%); 607 KiB/s, 54 objects/s recovering 2024-09-13T15:00:19.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:19 smithi003 ceph-mon[107278]: pgmap v1267: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 643 op/s; 4399/78 objects degraded (5639.744%); 607 KiB/s, 54 objects/s recovering 2024-09-13T15:00:19.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:19 smithi003 ceph-mon[104007]: pgmap v1267: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 643 op/s; 4399/78 objects degraded (5639.744%); 607 KiB/s, 54 objects/s recovering 2024-09-13T15:00:19.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:19.741 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:19.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:19.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:19.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.997 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:19.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:20.098 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.125 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:20.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:20.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:20.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:20.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.379 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:20.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:20.480 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:20.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:20.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:20.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:00:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:00:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:20.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:20.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:20.857 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:20.884 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:20.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:21.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:21.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:21.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:21.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:21.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:21.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:21.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:21.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:21.483 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:21.510 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:21.566 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:00:21.612 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:21.638 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:21 smithi003 ceph-mon[107278]: pgmap v1268: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 4399/78 objects degraded (5639.744%); 517 KiB/s, 46 objects/s recovering 2024-09-13T15:00:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4399/78 objects degraded (5639.744%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2177991087' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:21 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:21 smithi003 ceph-mon[104007]: pgmap v1268: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 4399/78 objects degraded (5639.744%); 517 KiB/s, 46 objects/s recovering 2024-09-13T15:00:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4399/78 objects degraded (5639.744%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2177991087' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:21 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:21.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:21.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:21.855 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:21 smithi019 ceph-mon[91296]: pgmap v1268: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 4399/78 objects degraded (5639.744%); 517 KiB/s, 46 objects/s recovering 2024-09-13T15:00:21.855 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4399/78 objects degraded (5639.744%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:21.856 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2177991087' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:21.856 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:21 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:21.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:21.890 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:21.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:22.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:22.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:22.144 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:22.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:22.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:22.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:22.398 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:22.500 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:22.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:22.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:22.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:22.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:22.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:22.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:22.877 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:22.904 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:22.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:22 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:22.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:22 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:00:22.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:22 smithi019 ceph-mon[91296]: osdmap e358: 8 total, 8 up, 7 in 2024-09-13T15:00:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:22 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:22 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:00:22.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:22 smithi003 ceph-mon[107278]: osdmap e358: 8 total, 8 up, 7 in 2024-09-13T15:00:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:22 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:22 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:00:22.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:22 smithi003 ceph-mon[104007]: osdmap e358: 8 total, 8 up, 7 in 2024-09-13T15:00:23.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:23.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:23.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:23.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:23.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:23.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:23.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:23.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:23.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:23.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:23.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:23.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:23.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:23.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:23.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:23.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:23.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:23 smithi019 ceph-mon[91296]: pgmap v1270: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 241 op/s; 4195/78 objects degraded (5378.205%); 26 KiB/s, 99 objects/s recovering 2024-09-13T15:00:23.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:23 smithi003 ceph-mon[107278]: pgmap v1270: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 241 op/s; 4195/78 objects degraded (5378.205%); 26 KiB/s, 99 objects/s recovering 2024-09-13T15:00:23.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:23 smithi003 ceph-mon[104007]: pgmap v1270: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 241 op/s; 4195/78 objects degraded (5378.205%); 26 KiB/s, 99 objects/s recovering 2024-09-13T15:00:24.014 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:24.041 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:24.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:24.170 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:24.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:24.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:24.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:24.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:00:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:00:24] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-13T15:00:24.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:24.550 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:24.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:24.652 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:24.678 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:24.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:24.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:24.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:24.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:24.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:24 smithi019 ceph-mon[91296]: pgmap v1271: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 4195/78 objects degraded (5378.205%); 26 KiB/s, 99 objects/s recovering 2024-09-13T15:00:24.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:24 smithi003 ceph-mon[107278]: pgmap v1271: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 4195/78 objects degraded (5378.205%); 26 KiB/s, 99 objects/s recovering 2024-09-13T15:00:24.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:24 smithi003 ceph-mon[104007]: pgmap v1271: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 4195/78 objects degraded (5378.205%); 26 KiB/s, 99 objects/s recovering 2024-09-13T15:00:25.030 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:25.058 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:25.160 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:25.187 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:25.289 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:25.314 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:25.416 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:25.441 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:25.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:25.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:25.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:25.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:25.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:26.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:26.170 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:26.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:25 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 4195/78 objects degraded (5378.205%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:26.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:25 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 4195/78 objects degraded (5378.205%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:26.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:25 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 4195/78 objects degraded (5378.205%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:26.197 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:26.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:26.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:26.424 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:26.451 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:26.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:26.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:26.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:26.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:26.703 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:26.805 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:26.830 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:26.931 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:26.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:27.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:27.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:27.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:26 smithi019 ceph-mon[91296]: pgmap v1272: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 4195/78 objects degraded (5378.205%); 26 KiB/s, 99 objects/s recovering 2024-09-13T15:00:27.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:26 smithi003 ceph-mon[107278]: pgmap v1272: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 4195/78 objects degraded (5378.205%); 26 KiB/s, 99 objects/s recovering 2024-09-13T15:00:27.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:26 smithi003 ceph-mon[104007]: pgmap v1272: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 4195/78 objects degraded (5378.205%); 26 KiB/s, 99 objects/s recovering 2024-09-13T15:00:27.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:27.211 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:27.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:27.339 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:27.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:27.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:27.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:27.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:27.676 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:27.677 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:27.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:27.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:27.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:27.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:27.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:27.973 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:27.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:28.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:28.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:28.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:28.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:28.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:28.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:28.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:28.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:28.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:28.615 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:28.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:28.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T15:00:28.845 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:28.871 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:28.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:28.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:29.100 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:29.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:29.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:29.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[107278]: pgmap v1273: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 3395/78 objects degraded (4352.564%); 0 B/s, 133 objects/s recovering 2024-09-13T15:00:29.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/410604916' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:29.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:29.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:29.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:29.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[104007]: pgmap v1273: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 3395/78 objects degraded (4352.564%); 0 B/s, 133 objects/s recovering 2024-09-13T15:00:29.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/410604916' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:29.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:28 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:29.127 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:28 smithi019 ceph-mon[91296]: pgmap v1273: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 3395/78 objects degraded (4352.564%); 0 B/s, 133 objects/s recovering 2024-09-13T15:00:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/410604916' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:28 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:29.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:29.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:29.357 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:29.380 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:29.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:00:29.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:29.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:29.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:29.632 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:29.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:29.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:29.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:29.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:29.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:30.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:30.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:30.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:30.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:29 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:00:30.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:29 smithi003 ceph-mon[107278]: osdmap e359: 8 total, 8 up, 7 in 2024-09-13T15:00:30.148 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:00:30.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:29 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:00:30.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:29 smithi003 ceph-mon[104007]: osdmap e359: 8 total, 8 up, 7 in 2024-09-13T15:00:30.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:00:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:29 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:00:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:29 smithi019 ceph-mon[91296]: osdmap e359: 8 total, 8 up, 7 in 2024-09-13T15:00:30.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:00:30.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:30.365 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:30.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:30.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:30.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:30.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:30.634 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:00:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:00:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:30.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:30.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:30.878 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:30.900 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:30 smithi019 ceph-mon[91296]: pgmap v1275: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 3395/78 objects degraded (4352.564%); 0 B/s, 92 objects/s recovering 2024-09-13T15:00:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:30 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3395/78 objects degraded (4352.564%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:31.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:31.026 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:31.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:31.150 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:30 smithi003 ceph-mon[104007]: pgmap v1275: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 3395/78 objects degraded (4352.564%); 0 B/s, 92 objects/s recovering 2024-09-13T15:00:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:30 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3395/78 objects degraded (4352.564%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:30 smithi003 ceph-mon[107278]: pgmap v1275: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 3395/78 objects degraded (4352.564%); 0 B/s, 92 objects/s recovering 2024-09-13T15:00:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:30 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3395/78 objects degraded (4352.564%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:31.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:31.275 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:31.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:31.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:31.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:31.528 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:31.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:31.630 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:31.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:31.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:31.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:31.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:31.905 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:32.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:32.029 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:32.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:32.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:32.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:32.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:32.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:32.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:32.403 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:32.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:32.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:32.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:32.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:32.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:32.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:32.881 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:32.903 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:33.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:33.031 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:33.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:33.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:33.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:33.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:33.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:33.389 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:33 smithi019 ceph-mon[91296]: pgmap v1276: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 2504/78 objects degraded (3210.256%); 0 B/s, 169 objects/s recovering 2024-09-13T15:00:33.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:33.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:33 smithi003 ceph-mon[107278]: pgmap v1276: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 2504/78 objects degraded (3210.256%); 0 B/s, 169 objects/s recovering 2024-09-13T15:00:33.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:33 smithi003 ceph-mon[104007]: pgmap v1276: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 2504/78 objects degraded (3210.256%); 0 B/s, 169 objects/s recovering 2024-09-13T15:00:33.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:33.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:33.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:33.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:33.761 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.785 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:33.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:33.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:34.010 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.033 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:34.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:34.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:34.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:34.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:00:34] "GET /metrics HTTP/1.1" 200 36534 "" "Prometheus/2.51.0" 2024-09-13T15:00:34.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:34.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:34.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:34.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:34.409 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:34.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:34.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:34.613 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T15:00:34.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:00:34.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:34.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:34.762 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:34.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:34.908 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:35.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:35.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:35.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:35.138 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.138 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.139 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:34 smithi019 ceph-mon[91296]: pgmap v1277: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 2504/78 objects degraded (3210.256%); 0 B/s, 169 objects/s recovering 2024-09-13T15:00:35.139 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.139 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:35.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[107278]: pgmap v1277: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 2504/78 objects degraded (3210.256%); 0 B/s, 169 objects/s recovering 2024-09-13T15:00:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[104007]: pgmap v1277: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 2504/78 objects degraded (3210.256%); 0 B/s, 169 objects/s recovering 2024-09-13T15:00:35.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:35.260 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:35.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:35.386 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:35.408 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:35.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:35.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:35.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:35.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:35.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.784 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:35.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:35.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:35.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:36.011 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.052 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:36.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:36.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:36.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:36.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:35 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2504/78 objects degraded (3210.256%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:36.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:35 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2504/78 objects degraded (3210.256%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:36.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:35 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2504/78 objects degraded (3210.256%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:00:36.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.304 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:36.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:36.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:36.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:36.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:36.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:36.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:36.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:36.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:36.804 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:36.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:36.928 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:37.030 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:37.053 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:37.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:37.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:37.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:37.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:36 smithi019 ceph-mon[91296]: pgmap v1278: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 2504/78 objects degraded (3210.256%); 0 B/s, 169 objects/s recovering 2024-09-13T15:00:37.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:36 smithi003 ceph-mon[104007]: pgmap v1278: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 2504/78 objects degraded (3210.256%); 0 B/s, 169 objects/s recovering 2024-09-13T15:00:37.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:36 smithi003 ceph-mon[107278]: pgmap v1278: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 2504/78 objects degraded (3210.256%); 0 B/s, 169 objects/s recovering 2024-09-13T15:00:37.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:37.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:37.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:37.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.427 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:37.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:37.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:37.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:37.656 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:37.679 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:37.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:37.803 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:37.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:37.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:38.012 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:00:38.012 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T15:00:38.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T15:00:38.012 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:00:38.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.012 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94200 11.10 11:0fc22623:::benchmark_data_smithi003_229782_object94199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.012 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.711793+0000", 2024-09-13T15:00:38.012 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28419949599999, 2024-09-13T15:00:38.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.439892046, 2024-09-13T15:00:38.013 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.013 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94200 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711793+0000", 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711793+0000", 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711795+0000", 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.046e-06 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711838+0000", 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2548e-05 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711861+0000", 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3754e-05 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711871+0000", 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6840000000000008e-06 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712997+0000", 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011257089999999999 2024-09-13T15:00:38.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713006+0000", 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4299999999999995e-06 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.125730+0000", 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4127240780000001 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.125938+0000", 2024-09-13T15:00:38.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000208128 2024-09-13T15:00:38.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.126154+0000", 2024-09-13T15:00:38.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021603200000000001 2024-09-13T15:00:38.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134250+0000", 2024-09-13T15:00:38.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080953889999999997 2024-09-13T15:00:38.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134267+0000", 2024-09-13T15:00:38.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6864e-05 2024-09-13T15:00:38.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134403+0000", 2024-09-13T15:00:38.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000136105 2024-09-13T15:00:38.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.149934+0000", 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015531420000000001 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.149998+0000", 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4157000000000003e-05 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.151685+0000", 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001686702 2024-09-13T15:00:38.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94227 11.10 11:0a18fc76:::benchmark_data_smithi003_229782_object94226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.711885+0000", 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28410788399998, 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4469483990000001, 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.022 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94227 2024-09-13T15:00:38.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.023 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711885+0000", 2024-09-13T15:00:38.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711885+0000", 2024-09-13T15:00:38.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711885+0000", 2024-09-13T15:00:38.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.51e-07 2024-09-13T15:00:38.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711932+0000", 2024-09-13T15:00:38.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7104000000000002e-05 2024-09-13T15:00:38.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711941+0000", 2024-09-13T15:00:38.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1339999999999993e-06 2024-09-13T15:00:38.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711947+0000", 2024-09-13T15:00:38.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0619999999999997e-06 2024-09-13T15:00:38.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713030+0000", 2024-09-13T15:00:38.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001082548 2024-09-13T15:00:38.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713036+0000", 2024-09-13T15:00:38.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3099999999999997e-06 2024-09-13T15:00:38.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.129572+0000", 2024-09-13T15:00:38.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.416535189 2024-09-13T15:00:38.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.030 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.129723+0000", 2024-09-13T15:00:38.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015137399999999999 2024-09-13T15:00:38.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.129885+0000", 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000161743 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134480+0000", 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045955980000000002 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134497+0000", 2024-09-13T15:00:38.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6827e-05 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.151980+0000", 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017482971 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.156634+0000", 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004653857 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.156752+0000", 2024-09-13T15:00:38.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117652 2024-09-13T15:00:38.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.033 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.158833+0000", 2024-09-13T15:00:38.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020814789999999998 2024-09-13T15:00:38.033 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.034 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.034 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.034 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94238 11.10 11:0ada4fde:::benchmark_data_smithi003_229782_object94237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.034 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.711957+0000", 2024-09-13T15:00:38.034 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28403512199998, 2024-09-13T15:00:38.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4488024209999999, 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94238 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711957+0000", 2024-09-13T15:00:38.035 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711957+0000", 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.711958+0000", 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3399999999999999e-07 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712132+0000", 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017390999999999999 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712143+0000", 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1022000000000001e-05 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712149+0000", 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5719999999999999e-06 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713056+0000", 2024-09-13T15:00:38.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00090699699999999999 2024-09-13T15:00:38.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713062+0000", 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9259999999999997e-06 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.132064+0000", 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4190016240000001 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.132265+0000", 2024-09-13T15:00:38.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020077800000000001 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.132471+0000", 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000206741 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134508+0000", 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020368769999999999 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.138832+0000", 2024-09-13T15:00:38.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043240589999999999 2024-09-13T15:00:38.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152136+0000", 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013303756999999999 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.159014+0000", 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068779590000000003 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.159104+0000", 2024-09-13T15:00:38.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9450000000000006e-05 2024-09-13T15:00:38.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.160760+0000", 2024-09-13T15:00:38.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016562149999999999 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94322 11.10 11:0b36970b:::benchmark_data_smithi003_229782_object94321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712159+0000", 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28383343100001, 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4504645759999999, 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.044 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94322 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712159+0000", 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.045 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712159+0000", 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712160+0000", 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8100000000000003e-07 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.046 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712179+0000", 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9034000000000001e-05 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712185+0000", 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6810000000000001e-06 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712191+0000", 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2689999999999999e-06 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713082+0000", 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00089189900000000003 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713089+0000", 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0680000000000002e-06 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134543+0000", 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.421454663 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134719+0000", 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017592199999999999 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.134915+0000", 2024-09-13T15:00:38.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019601699999999999 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.138916+0000", 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040005939999999997 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.151928+0000", 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013012767999999999 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152184+0000", 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025554000000000001 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.160833+0000", 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0086491569999999993 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.160922+0000", 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8930000000000004e-05 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.162624+0000", 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001701553 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94324 11.10 11:0e3d6c56:::benchmark_data_smithi003_229782_object94323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.052 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712197+0000", 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28379569200001, 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4524904750000001, 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94324 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712197+0000", 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712197+0000", 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712197+0000", 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6600000000000002e-07 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712243+0000", 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5791000000000003e-05 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712250+0000", 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3429999999999998e-06 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712256+0000", 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8860000000000001e-06 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713109+0000", 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00085227100000000004 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713115+0000", 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1469999999999998e-06 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.136848+0000", 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.423733149 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.136985+0000", 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013682599999999999 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.137129+0000", 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000144647 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.141149+0000", 2024-09-13T15:00:38.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040198680000000002 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152001+0000", 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010852194000000001 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152265+0000", 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026327499999999999 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.162693+0000", 2024-09-13T15:00:38.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010428174 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.162890+0000", 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019689600000000001 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.164687+0000", 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017975420000000001 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.059 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94374 11.10 11:09485132:::benchmark_data_smithi003_229782_object94373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712266+0000", 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28372682999998, 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4743467940000001, 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94374 2024-09-13T15:00:38.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712266+0000", 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712266+0000", 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712266+0000", 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9100000000000004e-07 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712308+0000", 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1468999999999999e-05 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712317+0000", 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8519999999999993e-06 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712325+0000", 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4020000000000002e-06 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713135+0000", 2024-09-13T15:00:38.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00081022799999999997 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713141+0000", 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0499999999999997e-06 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.138952+0000", 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4258111630000001 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.139106+0000", 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015347599999999999 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.139274+0000", 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016841 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.143560+0000", 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004285634 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152009+0000", 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084487620000000003 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.164758+0000", 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012749442 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.181149+0000", 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016390873 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.183513+0000", 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023641220000000002 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.186612+0000", 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00309932 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.067 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94402 11.10 11:0ea6a2db:::benchmark_data_smithi003_229782_object94401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712334+0000", 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28365855200002, 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.499560526, 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94402 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.068 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712334+0000", 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712334+0000", 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712335+0000", 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6499999999999999e-07 2024-09-13T15:00:38.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712354+0000", 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9945999999999998e-05 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712362+0000", 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4270000000000002e-06 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712368+0000", 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1809999999999998e-06 2024-09-13T15:00:38.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713161+0000", 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00079314499999999998 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713167+0000", 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9920000000000004e-06 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.141180+0000", 2024-09-13T15:00:38.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.428012394 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.141341+0000", 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016166100000000001 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.141512+0000", 2024-09-13T15:00:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017027700000000001 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.143609+0000", 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020970279999999999 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152018+0000", 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084095290000000007 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174707+0000", 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022689227999999999 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.210055+0000", 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035347283 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.210328+0000", 2024-09-13T15:00:38.074 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027289600000000001 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.211894+0000", 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015669740000000001 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94404 11.10 11:0c74a415:::benchmark_data_smithi003_229782_object94403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712377+0000", 2024-09-13T15:00:38.075 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28361535099998, 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.50167396, 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94404 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712377+0000", 2024-09-13T15:00:38.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712377+0000", 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712378+0000", 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4099999999999999e-07 2024-09-13T15:00:38.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:38.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712399+0000", 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1491000000000001e-05 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712406+0000", 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1760000000000004e-06 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712412+0000", 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9050000000000002e-06 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713188+0000", 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00077545700000000001 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713194+0000", 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0519999999999998e-06 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.143644+0000", 2024-09-13T15:00:38.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4304505789999999 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.143816+0000", 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017148599999999999 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.144002+0000", 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018566200000000001 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.149883+0000", 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058818059999999998 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152024+0000", 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021409860000000001 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174825+0000", 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022800606000000001 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.211951+0000", 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.037126064 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.212009+0000", 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8159999999999999e-05 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.214051+0000", 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020419890000000001 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.087 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94466 11.10 11:0be2a9ea:::benchmark_data_smithi003_229782_object94465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712420+0000", 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.283572101, 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.503092879, 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.088 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94466 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712420+0000", 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712420+0000", 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712421+0000", 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1799999999999995e-07 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712445+0000", 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4474e-05 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712570+0000", 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012472000000000001 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712579+0000", 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3449999999999995e-06 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713214+0000", 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00063451600000000001 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713220+0000", 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1009999999999997e-06 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.145696+0000", 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4324761859999999 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.145830+0000", 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013381100000000001 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.145993+0000", 2024-09-13T15:00:38.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016244299999999999 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.151767+0000", 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057746070000000002 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152032+0000", 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000264966 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174928+0000", 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022896382 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.214109+0000", 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039180461999999999 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.214171+0000", 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1915999999999995e-05 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.215513+0000", 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001342432 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94472 11.10 11:0db08c66:::benchmark_data_smithi003_229782_object94471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.712593+0000", 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28339985500003, 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5046174130000001, 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.095 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94472 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712593+0000", 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712593+0000", 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712593+0000", 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8299999999999997e-07 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712616+0000", 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2486000000000001e-05 2024-09-13T15:00:38.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712624+0000", 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4940000000000003e-06 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.712630+0000", 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8560000000000003e-06 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713243+0000", 2024-09-13T15:00:38.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00061295200000000005 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713249+0000", 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.105e-06 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.148138+0000", 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.434889173 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.148257+0000", 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011830800000000001 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.148447+0000", 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019088699999999999 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152040+0000", 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035921659999999999 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152087+0000", 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7513999999999999e-05 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.186679+0000", 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.034591898000000003 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.215562+0000", 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028882642 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.215728+0000", 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000166253 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.217210+0000", 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001482096 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94521 11.10 11:0f0380e7:::benchmark_data_smithi003_229782_object94520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713089+0000", 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28290324, 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.510600586, 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94521 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713089+0000", 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713089+0000", 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713090+0000", 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2399999999999998e-07 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713113+0000", 2024-09-13T15:00:38.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2945000000000001e-05 2024-09-13T15:00:38.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713124+0000", 2024-09-13T15:00:38.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1613e-05 2024-09-13T15:00:38.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713132+0000", 2024-09-13T15:00:38.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4819999999999997e-06 2024-09-13T15:00:38.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713269+0000", 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000137546 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713276+0000", 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1489999999999999e-06 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152328+0000", 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.439052405 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152513+0000", 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018483599999999999 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.152708+0000", 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019519599999999999 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.158865+0000", 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061571389999999998 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.158937+0000", 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1451000000000004e-05 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.186776+0000", 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027839702000000001 2024-09-13T15:00:38.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.217262+0000", 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.030485763999999999 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.217324+0000", 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2100000000000005e-05 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.223690+0000", 2024-09-13T15:00:38.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063656340000000002 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94522 11.10 11:0fe3a0c0:::benchmark_data_smithi003_229782_object94521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713142+0000", 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28285021800002, 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5127048080000001, 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.111 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94522 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713142+0000", 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713142+0000", 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713143+0000", 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8599999999999998e-07 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713168+0000", 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5605999999999998e-05 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713177+0000", 2024-09-13T15:00:38.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2979999999999992e-06 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713183+0000", 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2570000000000004e-06 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713298+0000", 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114678 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713304+0000", 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1190000000000002e-06 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.154748+0000", 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.441443987 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.154941+0000", 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019285999999999999 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.155133+0000", 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019257899999999999 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.158961+0000", 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038275319999999998 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.164814+0000", 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058529740000000004 2024-09-13T15:00:38.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207491+0000", 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.042677155000000001 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.223750+0000", 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016259447 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.223900+0000", 2024-09-13T15:00:38.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014954599999999999 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.225847+0000", 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019472840000000001 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94575 11.10 11:0c0f3ecd:::benchmark_data_smithi003_229782_object94574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713191+0000", 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.282801081, 2024-09-13T15:00:38.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.51553099, 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94575 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713191+0000", 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713191+0000", 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713192+0000", 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7399999999999998e-07 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713216+0000", 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3824000000000001e-05 2024-09-13T15:00:38.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713224+0000", 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6000000000000007e-06 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713230+0000", 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9139999999999996e-06 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713324+0000", 2024-09-13T15:00:38.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3720000000000004e-05 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713330+0000", 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9460000000000003e-06 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.164861+0000", 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4515310960000001 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.165011+0000", 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015026300000000001 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.165194+0000", 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018294599999999999 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174721+0000", 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095271079999999994 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174785+0000", 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4045000000000001e-05 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207535+0000", 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032749160999999999 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.225911+0000", 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018376445000000002 2024-09-13T15:00:38.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.225973+0000", 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2070999999999996e-05 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.228722+0000", 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027493769999999999 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94605 11.10 11:0d24cfc4:::benchmark_data_smithi003_229782_object94604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713238+0000", 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28275468700002, 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.517634414, 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94605 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713238+0000", 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713238+0000", 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713238+0000", 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8299999999999997e-07 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713260+0000", 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1301000000000002e-05 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713267+0000", 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.272e-06 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713272+0000", 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6130000000000003e-06 2024-09-13T15:00:38.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713350+0000", 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7585999999999999e-05 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713358+0000", 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5100000000000001e-06 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.167319+0000", 2024-09-13T15:00:38.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4539618830000001 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.167492+0000", 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017285999999999999 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.167697+0000", 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000204448 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174728+0000", 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070309939999999996 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174877+0000", 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000149631 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207572+0000", 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032694873999999999 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.228787+0000", 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021214551000000002 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.228856+0000", 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9036999999999996e-05 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.230872+0000", 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020163709999999999 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.132 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94638 11.10 11:0b0578dd:::benchmark_data_smithi003_229782_object94637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713292+0000", 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28270009200003, 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5195766050000001, 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94638 2024-09-13T15:00:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713292+0000", 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713292+0000", 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713293+0000", 2024-09-13T15:00:38.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6300000000000005e-07 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713347+0000", 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4518999999999998e-05 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713357+0000", 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4709999999999996e-06 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713363+0000", 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1149999999999999e-06 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713525+0000", 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000162342 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713534+0000", 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6859999999999995e-06 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.169822+0000", 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4562879689999999 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.169982+0000", 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016018799999999999 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.170142+0000", 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016017900000000001 2024-09-13T15:00:38.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.174733+0000", 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045909169999999999 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.179836+0000", 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051030240000000003 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207632+0000", 2024-09-13T15:00:38.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027795306999999998 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.230931+0000", 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023299626 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.230992+0000", 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0327e-05 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.232869+0000", 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001877372 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:94657 11.10 11:0ea01dfd:::benchmark_data_smithi003_229782_object94656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:53:00.713432+0000", 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 457.28256075199999, 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5214419449999999, 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:38.140 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94657 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713432+0000", 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713432+0000", 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713432+0000", 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1900000000000002e-07 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713487+0000", 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4701999999999999e-05 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713498+0000", 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1035e-05 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713505+0000", 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5729999999999996e-06 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713559+0000", 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4731999999999997e-05 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:00.713566+0000", 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.956e-06 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.172320+0000", 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.458753387 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.172629+0000", 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000309019 2024-09-13T15:00:38.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.172812+0000", 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000183606 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.177428+0000", 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046158479999999997 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.179899+0000", 2024-09-13T15:00:38.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002470875 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.207665+0000", 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027765517 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.232925+0000", 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025259884 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:00:38.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.232985+0000", 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0173999999999997e-05 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:53:02.234874+0000", 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018890179999999999 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129065 12.6 12:6bd62a78:::benchmark_data_smithi003_259359_object13439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.462455+0000", 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 148.533537878, 2024-09-13T15:00:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2997067840000001, 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129065 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462455+0000", 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462455+0000", 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462456+0000", 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5859999999999999e-06 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462462+0000", 2024-09-13T15:00:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3380000000000004e-06 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462477+0000", 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5766000000000001e-05 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462484+0000", 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1770000000000001e-06 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.469103+0000", 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066184740000000001 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.469401+0000", 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029828199999999997 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.469826+0000", 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042470499999999999 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481700+0000", 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011874312 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.483337+0000", 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016369069999999999 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.522366+0000", 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039028411999999998 2024-09-13T15:00:38.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.522380+0000", 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4521999999999999e-05 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498225+0000", 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97584474499999996 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498239+0000", 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3729e-05 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762079+0000", 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.26384074 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762161+0000", 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2089000000000006e-05 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.154 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129080 12.6 12:6497ebd7:::benchmark_data_smithi003_259359_object13454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.476318+0000", 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 148.519674065, 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.286167098, 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129080 2024-09-13T15:00:38.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476318+0000", 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476318+0000", 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476320+0000", 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7290000000000001e-06 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476326+0000", 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6239999999999999e-06 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476388+0000", 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2374999999999998e-05 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476396+0000", 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0919999999999998e-06 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.483366+0000", 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069699829999999999 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.483476+0000", 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010993399999999999 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.483590+0000", 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113804 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.497960+0000", 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01436978 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.500120+0000", 2024-09-13T15:00:38.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002160655 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523174+0000", 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023053361000000001 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523184+0000", 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0451e-05 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498273+0000", 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97508908500000002 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498284+0000", 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0513000000000001e-05 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762447+0000", 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.26416269 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762486+0000", 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9022e-05 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129083 12.6 12:68209590:::benchmark_data_smithi003_259359_object13457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.481116+0000", 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 148.514876136, 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.28149063, 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.162 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129083 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481116+0000", 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481116+0000", 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481118+0000", 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6789999999999999e-06 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481124+0000", 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1569999999999997e-06 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481136+0000", 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2269000000000001e-05 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481144+0000", 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.79e-06 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.487171+0000", 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060264100000000003 2024-09-13T15:00:38.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.487298+0000", 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012718399999999999 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.487424+0000", 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012619500000000001 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.500581+0000", 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013157294999999999 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.501814+0000", 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012331410000000001 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523215+0000", 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021400395999999999 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523224+0000", 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1989999999999995e-06 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498314+0000", 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97509020000000002 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498325+0000", 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.024e-05 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762534+0000", 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2642099520000001 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762607+0000", 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2522999999999997e-05 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129089 12.6 12:667fb162:::benchmark_data_smithi003_259359_object13463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-13T15:00:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.484895+0000", 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 148.511097409, 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3123250710000001, 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129089 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:00:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484895+0000", 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484895+0000", 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484897+0000", 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5340000000000001e-06 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:00:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484901+0000", 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4630000000000003e-06 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484914+0000", 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2524000000000001e-05 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484921+0000", 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5079999999999999e-06 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.494354+0000", 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094326810000000001 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.494480+0000", 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000126456 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.494602+0000", 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121773 2024-09-13T15:00:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.501835+0000", 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072334019999999999 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523645+0000", 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021809151999999998 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523654+0000", 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1889999999999996e-06 2024-09-13T15:00:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498354+0000", 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97470043299999998 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498364+0000", 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0203e-05 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762734+0000", 2024-09-13T15:00:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2643698400000001 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for missing object", 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762922+0000", 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000187519 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.796756+0000", 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033834649000000001 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.797220+0000", 2024-09-13T15:00:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000463745 2024-09-13T15:00:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:00:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:00:38.177 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:00:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.201 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:38.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:38.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:38.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:38.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:38.453 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:38.555 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:38.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.705 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:38.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:38.806 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.830 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:38.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:38.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:38.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:38.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:39.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:39.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:39.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:39.181 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.209 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:39.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:39.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:39.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_7' removed 2024-09-13T15:00:39.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[107278]: pgmap v1279: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1389/78 objects degraded (1780.769%); 0 B/s, 200 objects/s recovering 2024-09-13T15:00:39.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:39.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:39.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/855315876' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:00:39.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2017365842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:39.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[104007]: pgmap v1279: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1389/78 objects degraded (1780.769%); 0 B/s, 200 objects/s recovering 2024-09-13T15:00:39.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:39.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:39.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/855315876' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:00:39.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2017365842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:39.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:39.460 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:00:39.460 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:39.461 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:39.462 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:00:39.462 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:00:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:39.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:39.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:39.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:39 smithi019 ceph-mon[91296]: pgmap v1279: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1389/78 objects degraded (1780.769%); 0 B/s, 200 objects/s recovering 2024-09-13T15:00:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:39 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:39.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:39 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:39.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/855315876' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:00:39.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2017365842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:39.688 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:39.715 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:39.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:39.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:39.986 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:40.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:40.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:40.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:40.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:40.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:40.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:40.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:40.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:40.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:40 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/855315876' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:00:40.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:40 smithi003 ceph-mon[107278]: osdmap e360: 8 total, 8 up, 7 in 2024-09-13T15:00:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:40 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/855315876' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:00:40.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:40 smithi003 ceph-mon[104007]: osdmap e360: 8 total, 8 up, 7 in 2024-09-13T15:00:40.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:40.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:40.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:00:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:00:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:40.584 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:40 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/855315876' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:00:40.584 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:40 smithi019 ceph-mon[91296]: osdmap e360: 8 total, 8 up, 7 in 2024-09-13T15:00:40.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:00:40.589 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:40.612 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:40.714 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:40.736 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:40.832 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:40.834 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:40.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:40.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:40.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:41.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:41.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:41.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:41.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:41.313 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:41.317 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T15:00:41.318 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T15:00:41.318 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_8 2024-09-13T15:00:41.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_8 16 2024-09-13T15:00:41.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:41.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:41.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:41.552 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:41 smithi003 ceph-mon[107278]: pgmap v1281: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:41.552 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:41 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1389/78 objects degraded (1780.769%), 1 pg degraded, 1 pg undersized) 2024-09-13T15:00:41.552 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:41 smithi003 ceph-mon[104007]: pgmap v1281: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:41.552 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:41 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1389/78 objects degraded (1780.769%), 1 pg degraded, 1 pg undersized) 2024-09-13T15:00:41.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:41.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:41 smithi019 ceph-mon[91296]: pgmap v1281: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:41 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1389/78 objects degraded (1780.769%), 1 pg degraded, 1 pg undersized) 2024-09-13T15:00:41.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.761 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:41.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:41.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:41.884 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:41.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:42.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:42.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:42.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:42.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:42.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.384 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:42.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:42.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:42.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:42.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:42.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:42.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:42.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:42.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:42.866 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:42.892 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:42.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:43.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:43.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:43.145 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:43.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:43.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:43.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:43.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:43.401 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:43.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:43.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:43 smithi003 ceph-mon[107278]: pgmap v1282: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:43.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:43.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:43.617 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:43 smithi003 ceph-mon[104007]: pgmap v1282: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:43.617 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:43.617 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:43.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:43.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:43 smithi019 ceph-mon[91296]: pgmap v1282: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:43.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:43.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:43.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:43.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:44.000 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.029 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:44.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:44.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:44.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:44.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:44.261 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:00:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:00:44] "GET /metrics HTTP/1.1" 200 34116 "" "Prometheus/2.51.0" 2024-09-13T15:00:44.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:44.382 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:44.409 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:44.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:44.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:44.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:44.663 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:44.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:44.765 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:44.792 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:44.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:44.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:45.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:45.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:45.178 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:45.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:45.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:45.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:45.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:45.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:00:45.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:45.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:45.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:45 smithi003 ceph-mon[104007]: pgmap v1283: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:45.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:45 smithi003 ceph-mon[107278]: pgmap v1283: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:45.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:45 smithi019 ceph-mon[91296]: pgmap v1283: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:45.707 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:45.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:45.835 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:45.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:45.964 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:46.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:46.092 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:46.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:46.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:46.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:46.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:46.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:46.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:46.603 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:46.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:46.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:46.834 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:46.860 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:46.963 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:47.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:47.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:47.215 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:47.241 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:47.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:47.364 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:00:47.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:47.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:47.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:47.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:47.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:47.624 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:47 smithi019 ceph-mon[91296]: pgmap v1284: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/5689047' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:47.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:47 smithi003 ceph-mon[107278]: pgmap v1284: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:47.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/5689047' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:47.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:47 smithi003 ceph-mon[104007]: pgmap v1284: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:47.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/5689047' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:47.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:47.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:47.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:47.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:47.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:48.003 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:48.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:48.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:48.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:48.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:48.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:48.353 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:48.381 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:48.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:48 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/5689047' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:48 smithi003 ceph-mon[107278]: osdmap e361: 8 total, 8 up, 7 in 2024-09-13T15:00:48.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:48 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:48.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/5689047' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:48.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:48 smithi003 ceph-mon[104007]: osdmap e361: 8 total, 8 up, 7 in 2024-09-13T15:00:48.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:48.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:48.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:48.637 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:48 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/5689047' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:48 smithi019 ceph-mon[91296]: osdmap e361: 8 total, 8 up, 7 in 2024-09-13T15:00:48.728 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:48.729 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:48.741 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:48.768 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:48.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:48.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:48.994 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.022 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:49.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:49.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:49.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:49.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:49.275 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:49.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:49.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:49.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:00:49.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:49.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:49.595 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:00:49.597 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:00:49.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:49.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:49 smithi019 ceph-mon[91296]: pgmap v1286: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:49 smithi019 ceph-mon[91296]: from='client.36870 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:00:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:49 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:49 smithi003 ceph-mon[107278]: pgmap v1286: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:49 smithi003 ceph-mon[107278]: from='client.36870 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:00:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:49 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:49 smithi003 ceph-mon[104007]: pgmap v1286: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:49 smithi003 ceph-mon[104007]: from='client.36870 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:00:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:49 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:49.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:49.872 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:49.882 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:49.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:49.914 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:50.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:50.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:50.142 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:50.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:50.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:50.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:50.298 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:50.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:00:50.351 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1286,"stamp":"2024-09-13T15:00:48.067858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":328147,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3159,"ondisk_log_size":3159,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":311,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8691888,"kb_used_data":88288,"kb_used_omap":23,"kb_used_meta":8603496,"kb_avail":647524176,"statfs":{"total":671965249536,"available":663064756224,"internally_reserved":0,"allocated":90406912,"data_stored":75482646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24109,"internal_metadata":8809980371},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.726957"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":106,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676185+0000","last_change":"2024-09-13T14:57:42.607827+0000","last_active":"2024-09-13T15:00:39.676185+0000","last_peered":"2024-09-13T15:00:39.676185+0000","last_clean":"2024-09-13T15:00:39.676185+0000","last_became_active":"2024-09-13T14:57:25.285562+0000","last_became_peered":"2024-09-13T14:57:25.285562+0000","last_unstale":"2024-09-13T15:00:39.676185+0000","last_undegraded":"2024-09-13T15:00:39.676185+0000","last_fullsized":"2024-09-13T15:00:39.676185+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.607753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.607753+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:21:26.227325+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":67,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.346615+0000","last_change":"2024-09-13T14:58:28.426659+0000","last_active":"2024-09-13T15:00:39.346615+0000","last_peered":"2024-09-13T15:00:39.346615+0000","last_clean":"2024-09-13T15:00:39.346615+0000","last_became_active":"2024-09-13T14:58:28.426336+0000","last_became_peered":"2024-09-13T14:58:28.426336+0000","last_unstale":"2024-09-13T15:00:39.346615+0000","last_undegraded":"2024-09-13T15:00:39.346615+0000","last_fullsized":"2024-09-13T15:00:39.346615+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:55.713969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":332,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354744+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T15:00:29.354744+0000","last_peered":"2024-09-13T15:00:29.354744+0000","last_clean":"2024-09-13T15:00:29.354744+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T15:00:29.354744+0000","last_undegraded":"2024-09-13T15:00:29.354744+0000","last_fullsized":"2024-09-13T15:00:29.354744+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":58,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.353731+0000","last_change":"2024-09-13T14:57:58.200463+0000","last_active":"2024-09-13T15:00:29.353731+0000","last_peered":"2024-09-13T15:00:29.353731+0000","last_clean":"2024-09-13T15:00:29.353731+0000","last_became_active":"2024-09-13T14:57:25.279086+0000","last_became_peered":"2024-09-13T14:57:25.279086+0000","last_unstale":"2024-09-13T15:00:29.353731+0000","last_undegraded":"2024-09-13T15:00:29.353731+0000","last_fullsized":"2024-09-13T15:00:29.353731+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:58.986716+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":83,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.675789+0000","last_change":"2024-09-13T14:57:35.356151+0000","last_active":"2024-09-13T15:00:39.675789+0000","last_peered":"2024-09-13T15:00:39.675789+0000","last_clean":"2024-09-13T15:00:39.675789+0000","last_became_active":"2024-09-13T14:57:35.355770+0000","last_became_peered":"2024-09-13T14:57:35.355770+0000","last_unstale":"2024-09-13T15:00:39.675789+0000","last_undegraded":"2024-09-13T15:00:39.675789+0000","last_fullsized":"2024-09-13T15:00:39.675789+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:37.887286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":568,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.677660+0000","last_change":"2024-09-13T14:58:10.513979+0000","last_active":"2024-09-13T15:00:39.677660+0000","last_peered":"2024-09-13T15:00:39.677660+0000","last_clean":"2024-09-13T15:00:39.677660+0000","last_became_active":"2024-09-13T14:58:10.513313+0000","last_became_peered":"2024-09-13T14:58:10.513313+0000","last_unstale":"2024-09-13T15:00:39.677660+0000","last_undegraded":"2024-09-13T15:00:39.677660+0000","last_fullsized":"2024-09-13T15:00:39.677660+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:25:15.376408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":63,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355480+0000","last_change":"2024-09-13T14:58:28.426763+0000","last_active":"2024-09-13T15:00:29.355480+0000","last_peered":"2024-09-13T15:00:29.355480+0000","last_clean":"2024-09-13T15:00:29.355480+0000","last_became_active":"2024-09-13T14:58:28.426270+0000","last_became_peered":"2024-09-13T14:58:28.426270+0000","last_unstale":"2024-09-13T15:00:29.355480+0000","last_undegraded":"2024-09-13T15:00:29.355480+0000","last_fullsized":"2024-09-13T15:00:29.355480+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.991576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":349,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355504+0000","last_change":"2024-09-13T14:58:10.515535+0000","last_active":"2024-09-13T15:00:29.355504+0000","last_peered":"2024-09-13T15:00:29.355504+0000","last_clean":"2024-09-13T15:00:29.355504+0000","last_became_active":"2024-09-13T14:58:10.515043+0000","last_became_peered":"2024-09-13T14:58:10.515043+0000","last_unstale":"2024-09-13T15:00:29.355504+0000","last_undegraded":"2024-09-13T15:00:29.355504+0000","last_fullsized":"2024-09-13T15:00:29.355504+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:53.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":108,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676744+0000","last_change":"2024-09-13T14:57:43.658933+0000","last_active":"2024-09-13T15:00:39.676744+0000","last_peered":"2024-09-13T15:00:39.676744+0000","last_clean":"2024-09-13T15:00:39.676744+0000","last_became_active":"2024-09-13T14:57:25.291285+0000","last_became_peered":"2024-09-13T14:57:25.291285+0000","last_unstale":"2024-09-13T15:00:39.676744+0000","last_undegraded":"2024-09-13T15:00:39.676744+0000","last_fullsized":"2024-09-13T15:00:39.676744+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:43.658833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:43.658833+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:22.120697+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":63,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355126+0000","last_change":"2024-09-13T14:58:28.429061+0000","last_active":"2024-09-13T15:00:29.355126+0000","last_peered":"2024-09-13T15:00:29.355126+0000","last_clean":"2024-09-13T15:00:29.355126+0000","last_became_active":"2024-09-13T14:58:28.428642+0000","last_became_peered":"2024-09-13T14:58:28.428642+0000","last_unstale":"2024-09-13T15:00:29.355126+0000","last_undegraded":"2024-09-13T15:00:29.355126+0000","last_fullsized":"2024-09-13T15:00:29.355126+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:03.197824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":369,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354584+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T15:00:29.354584+0000","last_peered":"2024-09-13T15:00:29.354584+0000","last_clean":"2024-09-13T15:00:29.354584+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T15:00:29.354584+0000","last_undegraded":"2024-09-13T15:00:29.354584+0000","last_fullsized":"2024-09-13T15:00:29.354584+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":115,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.675892+0000","last_change":"2024-09-13T14:58:10.521468+0000","last_active":"2024-09-13T15:00:39.675892+0000","last_peered":"2024-09-13T15:00:39.675892+0000","last_clean":"2024-09-13T15:00:39.675892+0000","last_became_active":"2024-09-13T14:58:10.520775+0000","last_became_peered":"2024-09-13T14:58:10.520775+0000","last_unstale":"2024-09-13T15:00:39.675892+0000","last_undegraded":"2024-09-13T15:00:39.675892+0000","last_fullsized":"2024-09-13T15:00:39.675892+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:27.363579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":66,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355205+0000","last_change":"2024-09-13T14:58:10.523188+0000","last_active":"2024-09-13T15:00:29.355205+0000","last_peered":"2024-09-13T15:00:29.355205+0000","last_clean":"2024-09-13T15:00:29.355205+0000","last_became_active":"2024-09-13T14:58:10.522815+0000","last_became_peered":"2024-09-13T14:58:10.522815+0000","last_unstale":"2024-09-13T15:00:29.355205+0000","last_undegraded":"2024-09-13T15:00:29.355205+0000","last_fullsized":"2024-09-13T15:00:29.355205+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:27.568999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":586,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.346691+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T15:00:39.346691+0000","last_peered":"2024-09-13T15:00:39.346691+0000","last_clean":"2024-09-13T15:00:39.346691+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T15:00:39.346691+0000","last_undegraded":"2024-09-13T15:00:39.346691+0000","last_fullsized":"2024-09-13T15:00:39.346691+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":55,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354445+0000","last_change":"2024-09-13T14:58:09.011981+0000","last_active":"2024-09-13T15:00:29.354445+0000","last_peered":"2024-09-13T15:00:29.354445+0000","last_clean":"2024-09-13T15:00:29.354445+0000","last_became_active":"2024-09-13T14:57:53.357674+0000","last_became_peered":"2024-09-13T14:57:53.357674+0000","last_unstale":"2024-09-13T15:00:29.354445+0000","last_undegraded":"2024-09-13T15:00:29.354445+0000","last_fullsized":"2024-09-13T15:00:29.354445+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.701912+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":341,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355594+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T15:00:29.355594+0000","last_peered":"2024-09-13T15:00:29.355594+0000","last_clean":"2024-09-13T15:00:29.355594+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T15:00:29.355594+0000","last_undegraded":"2024-09-13T15:00:29.355594+0000","last_fullsized":"2024-09-13T15:00:29.355594+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":528,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676233+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T15:00:39.676233+0000","last_peered":"2024-09-13T15:00:39.676233+0000","last_clean":"2024-09-13T15:00:39.676233+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T15:00:39.676233+0000","last_undegraded":"2024-09-13T15:00:39.676233+0000","last_fullsized":"2024-09-13T15:00:39.676233+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":61,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.356082+0000","last_change":"2024-09-13T14:58:10.513244+0000","last_active":"2024-09-13T15:00:29.356082+0000","last_peered":"2024-09-13T15:00:29.356082+0000","last_clean":"2024-09-13T15:00:29.356082+0000","last_became_active":"2024-09-13T14:58:10.511485+0000","last_became_peered":"2024-09-13T14:58:10.511485+0000","last_unstale":"2024-09-13T15:00:29.356082+0000","last_undegraded":"2024-09-13T15:00:29.356082+0000","last_fullsized":"2024-09-13T15:00:29.356082+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:23.283605+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":243,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.344008+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T15:00:39.344008+0000","last_peered":"2024-09-13T15:00:39.344008+0000","last_clean":"2024-09-13T15:00:39.344008+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T15:00:39.344008+0000","last_undegraded":"2024-09-13T15:00:39.344008+0000","last_fullsized":"2024-09-13T15:00:39.344008+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":430,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355070+0000","last_change":"2024-09-13T14:58:10.536485+0000","last_active":"2024-09-13T15:00:29.355070+0000","last_peered":"2024-09-13T15:00:29.355070+0000","last_clean":"2024-09-13T15:00:29.355070+0000","last_became_active":"2024-09-13T14:58:10.536134+0000","last_became_peered":"2024-09-13T14:58:10.536134+0000","last_unstale":"2024-09-13T15:00:29.355070+0000","last_undegraded":"2024-09-13T15:00:29.355070+0000","last_fullsized":"2024-09-13T15:00:29.355070+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:59.095681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":58,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354036+0000","last_change":"2024-09-13T14:58:05.212101+0000","last_active":"2024-09-13T15:00:29.354036+0000","last_peered":"2024-09-13T15:00:29.354036+0000","last_clean":"2024-09-13T15:00:29.354036+0000","last_became_active":"2024-09-13T14:57:25.275734+0000","last_became_peered":"2024-09-13T14:57:25.275734+0000","last_unstale":"2024-09-13T15:00:29.354036+0000","last_undegraded":"2024-09-13T15:00:29.354036+0000","last_fullsized":"2024-09-13T15:00:29.354036+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.212008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.212008+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:01.720211+0000","scrub_duration":0.0048969690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":567,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676328+0000","last_change":"2024-09-13T14:58:10.515043+0000","last_active":"2024-09-13T15:00:39.676328+0000","last_peered":"2024-09-13T15:00:39.676328+0000","last_clean":"2024-09-13T15:00:39.676328+0000","last_became_active":"2024-09-13T14:58:10.513487+0000","last_became_peered":"2024-09-13T14:58:10.513487+0000","last_unstale":"2024-09-13T15:00:39.676328+0000","last_undegraded":"2024-09-13T15:00:39.676328+0000","last_fullsized":"2024-09-13T15:00:39.676328+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:11.392005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":342,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.317410+0000","last_change":"2024-09-13T14:57:53.364039+0000","last_active":"2024-09-13T15:00:29.317410+0000","last_peered":"2024-09-13T15:00:29.317410+0000","last_clean":"2024-09-13T15:00:29.317410+0000","last_became_active":"2024-09-13T14:57:53.363550+0000","last_became_peered":"2024-09-13T14:57:53.363550+0000","last_unstale":"2024-09-13T15:00:29.317410+0000","last_undegraded":"2024-09-13T15:00:29.317410+0000","last_fullsized":"2024-09-13T15:00:29.317410+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:36.171089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":84,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355774+0000","last_change":"2024-09-13T14:58:04.168950+0000","last_active":"2024-09-13T15:00:29.355774+0000","last_peered":"2024-09-13T15:00:29.355774+0000","last_clean":"2024-09-13T15:00:29.355774+0000","last_became_active":"2024-09-13T14:57:53.351301+0000","last_became_peered":"2024-09-13T14:57:53.351301+0000","last_unstale":"2024-09-13T15:00:29.355774+0000","last_undegraded":"2024-09-13T15:00:29.355774+0000","last_fullsized":"2024-09-13T15:00:29.355774+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:42.270600+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":529,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.675912+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T15:00:39.675912+0000","last_peered":"2024-09-13T15:00:39.675912+0000","last_clean":"2024-09-13T15:00:39.675912+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T15:00:39.675912+0000","last_undegraded":"2024-09-13T15:00:39.675912+0000","last_fullsized":"2024-09-13T15:00:39.675912+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":583,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.356421+0000","last_change":"2024-09-13T14:58:10.514433+0000","last_active":"2024-09-13T15:00:29.356421+0000","last_peered":"2024-09-13T15:00:29.356421+0000","last_clean":"2024-09-13T15:00:29.356421+0000","last_became_active":"2024-09-13T14:58:10.513446+0000","last_became_peered":"2024-09-13T14:58:10.513446+0000","last_unstale":"2024-09-13T15:00:29.356421+0000","last_undegraded":"2024-09-13T15:00:29.356421+0000","last_fullsized":"2024-09-13T15:00:29.356421+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:24.535893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":534,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.677140+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T15:00:39.677140+0000","last_peered":"2024-09-13T15:00:39.677140+0000","last_clean":"2024-09-13T15:00:39.677140+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T15:00:39.677140+0000","last_undegraded":"2024-09-13T15:00:39.677140+0000","last_fullsized":"2024-09-13T15:00:39.677140+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":41,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355967+0000","last_change":"2024-09-13T14:57:53.350622+0000","last_active":"2024-09-13T15:00:29.355967+0000","last_peered":"2024-09-13T15:00:29.355967+0000","last_clean":"2024-09-13T15:00:29.355967+0000","last_became_active":"2024-09-13T14:57:53.350292+0000","last_became_peered":"2024-09-13T14:57:53.350292+0000","last_unstale":"2024-09-13T15:00:29.355967+0000","last_undegraded":"2024-09-13T15:00:29.355967+0000","last_fullsized":"2024-09-13T15:00:29.355967+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:16.088338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":286,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.346497+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T15:00:39.346497+0000","last_peered":"2024-09-13T15:00:39.346497+0000","last_clean":"2024-09-13T15:00:39.346497+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T15:00:39.346497+0000","last_undegraded":"2024-09-13T15:00:39.346497+0000","last_fullsized":"2024-09-13T15:00:39.346497+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.284651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":51,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.356526+0000","last_change":"2024-09-13T14:58:10.527352+0000","last_active":"2024-09-13T15:00:29.356526+0000","last_peered":"2024-09-13T15:00:29.356526+0000","last_clean":"2024-09-13T15:00:29.356526+0000","last_became_active":"2024-09-13T14:58:10.526614+0000","last_became_peered":"2024-09-13T14:58:10.526614+0000","last_unstale":"2024-09-13T15:00:29.356526+0000","last_undegraded":"2024-09-13T15:00:29.356526+0000","last_fullsized":"2024-09-13T15:00:29.356526+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:04.459255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":123,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355353+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T15:00:29.355353+0000","last_peered":"2024-09-13T15:00:29.355353+0000","last_clean":"2024-09-13T15:00:29.355353+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T15:00:29.355353+0000","last_undegraded":"2024-09-13T15:00:29.355353+0000","last_fullsized":"2024-09-13T15:00:29.355353+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":261,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.317051+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T15:00:29.317051+0000","last_peered":"2024-09-13T15:00:29.317051+0000","last_clean":"2024-09-13T15:00:29.317051+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T15:00:29.317051+0000","last_undegraded":"2024-09-13T15:00:29.317051+0000","last_fullsized":"2024-09-13T15:00:29.317051+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":113,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.342379+0000","last_change":"2024-09-13T14:58:10.511301+0000","last_active":"2024-09-13T15:00:39.342379+0000","last_peered":"2024-09-13T15:00:39.342379+0000","last_clean":"2024-09-13T15:00:39.342379+0000","last_became_active":"2024-09-13T14:58:10.510934+0000","last_became_peered":"2024-09-13T14:58:10.510934+0000","last_unstale":"2024-09-13T15:00:39.342379+0000","last_undegraded":"2024-09-13T15:00:39.342379+0000","last_fullsized":"2024-09-13T15:00:39.342379+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:12.152076+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":329,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355215+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T15:00:29.355215+0000","last_peered":"2024-09-13T15:00:29.355215+0000","last_clean":"2024-09-13T15:00:29.355215+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T15:00:29.355215+0000","last_undegraded":"2024-09-13T15:00:29.355215+0000","last_fullsized":"2024-09-13T15:00:29.355215+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":322,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355357+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T15:00:29.355357+0000","last_peered":"2024-09-13T15:00:29.355357+0000","last_clean":"2024-09-13T15:00:29.355357+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T15:00:29.355357+0000","last_undegraded":"2024-09-13T15:00:29.355357+0000","last_fullsized":"2024-09-13T15:00:29.355357+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":325,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354233+0000","last_change":"2024-09-13T14:57:53.351374+0000","last_active":"2024-09-13T15:00:29.354233+0000","last_peered":"2024-09-13T15:00:29.354233+0000","last_clean":"2024-09-13T15:00:29.354233+0000","last_became_active":"2024-09-13T14:57:53.350803+0000","last_became_peered":"2024-09-13T14:57:53.350803+0000","last_unstale":"2024-09-13T15:00:29.354233+0000","last_undegraded":"2024-09-13T15:00:29.354233+0000","last_fullsized":"2024-09-13T15:00:29.354233+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:30.535837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":355,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.317441+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T15:00:29.317441+0000","last_peered":"2024-09-13T15:00:29.317441+0000","last_clean":"2024-09-13T15:00:29.317441+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T15:00:29.317441+0000","last_undegraded":"2024-09-13T15:00:29.317441+0000","last_fullsized":"2024-09-13T15:00:29.317441+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":53,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355903+0000","last_change":"2024-09-13T14:58:10.539542+0000","last_active":"2024-09-13T15:00:29.355903+0000","last_peered":"2024-09-13T15:00:29.355903+0000","last_clean":"2024-09-13T15:00:29.355903+0000","last_became_active":"2024-09-13T14:58:10.536848+0000","last_became_peered":"2024-09-13T14:58:10.536848+0000","last_unstale":"2024-09-13T15:00:29.355903+0000","last_undegraded":"2024-09-13T15:00:29.355903+0000","last_fullsized":"2024-09-13T15:00:29.355903+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:52.133867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":308,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676221+0000","last_change":"2024-09-13T14:57:53.347772+0000","last_active":"2024-09-13T15:00:39.676221+0000","last_peered":"2024-09-13T15:00:39.676221+0000","last_clean":"2024-09-13T15:00:39.676221+0000","last_became_active":"2024-09-13T14:57:53.347373+0000","last_became_peered":"2024-09-13T14:57:53.347373+0000","last_unstale":"2024-09-13T15:00:39.676221+0000","last_undegraded":"2024-09-13T15:00:39.676221+0000","last_fullsized":"2024-09-13T15:00:39.676221+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:13.081908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":579,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676017+0000","last_change":"2024-09-13T14:58:10.517480+0000","last_active":"2024-09-13T15:00:39.676017+0000","last_peered":"2024-09-13T15:00:39.676017+0000","last_clean":"2024-09-13T15:00:39.676017+0000","last_became_active":"2024-09-13T14:58:10.517178+0000","last_became_peered":"2024-09-13T14:58:10.517178+0000","last_unstale":"2024-09-13T15:00:39.676017+0000","last_undegraded":"2024-09-13T15:00:39.676017+0000","last_fullsized":"2024-09-13T15:00:39.676017+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:27.472859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":196,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.317388+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T15:00:29.317388+0000","last_peered":"2024-09-13T15:00:29.317388+0000","last_clean":"2024-09-13T15:00:29.317388+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T15:00:29.317388+0000","last_undegraded":"2024-09-13T15:00:29.317388+0000","last_fullsized":"2024-09-13T15:00:29.317388+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":571,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676726+0000","last_change":"2024-09-13T14:58:10.532248+0000","last_active":"2024-09-13T15:00:39.676726+0000","last_peered":"2024-09-13T15:00:39.676726+0000","last_clean":"2024-09-13T15:00:39.676726+0000","last_became_active":"2024-09-13T14:58:10.522929+0000","last_became_peered":"2024-09-13T14:58:10.522929+0000","last_unstale":"2024-09-13T15:00:39.676726+0000","last_undegraded":"2024-09-13T15:00:39.676726+0000","last_fullsized":"2024-09-13T15:00:39.676726+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:38:40.572917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":163,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355438+0000","last_change":"2024-09-13T14:57:56.132062+0000","last_active":"2024-09-13T15:00:29.355438+0000","last_peered":"2024-09-13T15:00:29.355438+0000","last_clean":"2024-09-13T15:00:29.355438+0000","last_became_active":"2024-09-13T14:57:19.218181+0000","last_became_peered":"2024-09-13T14:57:19.218181+0000","last_unstale":"2024-09-13T15:00:29.355438+0000","last_undegraded":"2024-09-13T15:00:29.355438+0000","last_fullsized":"2024-09-13T15:00:29.355438+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:56.131976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:56.131976+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:31.359825+0000","scrub_duration":0.0061470450000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":568,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355873+0000","last_change":"2024-09-13T14:58:10.521645+0000","last_active":"2024-09-13T15:00:29.355873+0000","last_peered":"2024-09-13T15:00:29.355873+0000","last_clean":"2024-09-13T15:00:29.355873+0000","last_became_active":"2024-09-13T14:58:10.521223+0000","last_became_peered":"2024-09-13T14:58:10.521223+0000","last_unstale":"2024-09-13T15:00:29.355873+0000","last_undegraded":"2024-09-13T15:00:29.355873+0000","last_fullsized":"2024-09-13T15:00:29.355873+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:18.930422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":345,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355848+0000","last_change":"2024-09-13T14:58:10.524720+0000","last_active":"2024-09-13T15:00:29.355848+0000","last_peered":"2024-09-13T15:00:29.355848+0000","last_clean":"2024-09-13T15:00:29.355848+0000","last_became_active":"2024-09-13T14:58:10.522828+0000","last_became_peered":"2024-09-13T14:58:10.522828+0000","last_unstale":"2024-09-13T15:00:29.355848+0000","last_undegraded":"2024-09-13T15:00:29.355848+0000","last_fullsized":"2024-09-13T15:00:29.355848+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:57.737026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":559,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.346600+0000","last_change":"2024-09-13T14:57:53.351358+0000","last_active":"2024-09-13T15:00:39.346600+0000","last_peered":"2024-09-13T15:00:39.346600+0000","last_clean":"2024-09-13T15:00:39.346600+0000","last_became_active":"2024-09-13T14:57:53.351021+0000","last_became_peered":"2024-09-13T14:57:53.351021+0000","last_unstale":"2024-09-13T15:00:39.346600+0000","last_undegraded":"2024-09-13T15:00:39.346600+0000","last_fullsized":"2024-09-13T15:00:39.346600+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.067202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":326,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.317230+0000","last_change":"2024-09-13T14:58:10.515266+0000","last_active":"2024-09-13T15:00:29.317230+0000","last_peered":"2024-09-13T15:00:29.317230+0000","last_clean":"2024-09-13T15:00:29.317230+0000","last_became_active":"2024-09-13T14:58:10.514858+0000","last_became_peered":"2024-09-13T14:58:10.514858+0000","last_unstale":"2024-09-13T15:00:29.317230+0000","last_undegraded":"2024-09-13T15:00:29.317230+0000","last_fullsized":"2024-09-13T15:00:29.317230+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:55.856922+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":41,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.356226+0000","last_change":"2024-09-13T14:57:53.351087+0000","last_active":"2024-09-13T15:00:29.356226+0000","last_peered":"2024-09-13T15:00:29.356226+0000","last_clean":"2024-09-13T15:00:29.356226+0000","last_became_active":"2024-09-13T14:57:53.350744+0000","last_became_peered":"2024-09-13T14:57:53.350744+0000","last_unstale":"2024-09-13T15:00:29.356226+0000","last_undegraded":"2024-09-13T15:00:29.356226+0000","last_fullsized":"2024-09-13T15:00:29.356226+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:43.300957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":103,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354509+0000","last_change":"2024-09-13T14:58:10.521049+0000","last_active":"2024-09-13T15:00:29.354509+0000","last_peered":"2024-09-13T15:00:29.354509+0000","last_clean":"2024-09-13T15:00:29.354509+0000","last_became_active":"2024-09-13T14:58:10.520663+0000","last_became_peered":"2024-09-13T14:58:10.520663+0000","last_unstale":"2024-09-13T15:00:29.354509+0000","last_undegraded":"2024-09-13T15:00:29.354509+0000","last_fullsized":"2024-09-13T15:00:29.354509+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:59.410821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":41,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.356196+0000","last_change":"2024-09-13T14:57:53.347982+0000","last_active":"2024-09-13T15:00:29.356196+0000","last_peered":"2024-09-13T15:00:29.356196+0000","last_clean":"2024-09-13T15:00:29.356196+0000","last_became_active":"2024-09-13T14:57:53.347623+0000","last_became_peered":"2024-09-13T14:57:53.347623+0000","last_unstale":"2024-09-13T15:00:29.356196+0000","last_undegraded":"2024-09-13T15:00:29.356196+0000","last_fullsized":"2024-09-13T15:00:29.356196+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:15.734246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":336,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.317367+0000","last_change":"2024-09-13T14:58:05.197923+0000","last_active":"2024-09-13T15:00:29.317367+0000","last_peered":"2024-09-13T15:00:29.317367+0000","last_clean":"2024-09-13T15:00:29.317367+0000","last_became_active":"2024-09-13T14:57:19.208023+0000","last_became_peered":"2024-09-13T14:57:19.208023+0000","last_unstale":"2024-09-13T15:00:29.317367+0000","last_undegraded":"2024-09-13T15:00:29.317367+0000","last_fullsized":"2024-09-13T15:00:29.317367+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:06.994746+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":457,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.349565+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T15:00:39.349565+0000","last_peered":"2024-09-13T15:00:39.349565+0000","last_clean":"2024-09-13T15:00:39.349565+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T15:00:39.349565+0000","last_undegraded":"2024-09-13T15:00:39.349565+0000","last_fullsized":"2024-09-13T15:00:39.349565+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":333,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355505+0000","last_change":"2024-09-13T14:58:10.516308+0000","last_active":"2024-09-13T15:00:29.355505+0000","last_peered":"2024-09-13T15:00:29.355505+0000","last_clean":"2024-09-13T15:00:29.355505+0000","last_became_active":"2024-09-13T14:58:10.515894+0000","last_became_peered":"2024-09-13T14:58:10.515894+0000","last_unstale":"2024-09-13T15:00:29.355505+0000","last_undegraded":"2024-09-13T15:00:29.355505+0000","last_fullsized":"2024-09-13T15:00:29.355505+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:17.962091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":90,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.344593+0000","last_change":"2024-09-13T14:58:28.428183+0000","last_active":"2024-09-13T15:00:39.344593+0000","last_peered":"2024-09-13T15:00:39.344593+0000","last_clean":"2024-09-13T15:00:39.344593+0000","last_became_active":"2024-09-13T14:58:28.427553+0000","last_became_peered":"2024-09-13T14:58:28.427553+0000","last_unstale":"2024-09-13T15:00:39.344593+0000","last_undegraded":"2024-09-13T15:00:39.344593+0000","last_fullsized":"2024-09-13T15:00:39.344593+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:59.108147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":57,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.344124+0000","last_change":"2024-09-13T14:58:28.424704+0000","last_active":"2024-09-13T15:00:39.344124+0000","last_peered":"2024-09-13T15:00:39.344124+0000","last_clean":"2024-09-13T15:00:39.344124+0000","last_became_active":"2024-09-13T14:58:28.424375+0000","last_became_peered":"2024-09-13T14:58:28.424375+0000","last_unstale":"2024-09-13T15:00:39.344124+0000","last_undegraded":"2024-09-13T15:00:39.344124+0000","last_fullsized":"2024-09-13T15:00:39.344124+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:28.071652+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":305,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.344293+0000","last_change":"2024-09-13T14:58:10.519956+0000","last_active":"2024-09-13T15:00:39.344293+0000","last_peered":"2024-09-13T15:00:39.344293+0000","last_clean":"2024-09-13T15:00:39.344293+0000","last_became_active":"2024-09-13T14:58:10.519409+0000","last_became_peered":"2024-09-13T14:58:10.519409+0000","last_unstale":"2024-09-13T15:00:39.344293+0000","last_undegraded":"2024-09-13T15:00:39.344293+0000","last_fullsized":"2024-09-13T15:00:39.344293+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:32.082107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":397,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.342913+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T15:00:39.342913+0000","last_peered":"2024-09-13T15:00:39.342913+0000","last_clean":"2024-09-13T15:00:39.342913+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T15:00:39.342913+0000","last_undegraded":"2024-09-13T15:00:39.342913+0000","last_fullsized":"2024-09-13T15:00:39.342913+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":595,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.344440+0000","last_change":"2024-09-13T14:58:10.511063+0000","last_active":"2024-09-13T15:00:39.344440+0000","last_peered":"2024-09-13T15:00:39.344440+0000","last_clean":"2024-09-13T15:00:39.344440+0000","last_became_active":"2024-09-13T14:58:10.510571+0000","last_became_peered":"2024-09-13T14:58:10.510571+0000","last_unstale":"2024-09-13T15:00:39.344440+0000","last_undegraded":"2024-09-13T15:00:39.344440+0000","last_fullsized":"2024-09-13T15:00:39.344440+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:20.803368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":127,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354927+0000","last_change":"2024-09-13T14:58:10.523618+0000","last_active":"2024-09-13T15:00:29.354927+0000","last_peered":"2024-09-13T15:00:29.354927+0000","last_clean":"2024-09-13T15:00:29.354927+0000","last_became_active":"2024-09-13T14:58:10.522166+0000","last_became_peered":"2024-09-13T14:58:10.522166+0000","last_unstale":"2024-09-13T15:00:29.354927+0000","last_undegraded":"2024-09-13T15:00:29.354927+0000","last_fullsized":"2024-09-13T15:00:29.354927+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:14.151008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":161,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.345897+0000","last_change":"2024-09-13T14:58:10.525829+0000","last_active":"2024-09-13T15:00:39.345897+0000","last_peered":"2024-09-13T15:00:39.345897+0000","last_clean":"2024-09-13T15:00:39.345897+0000","last_became_active":"2024-09-13T14:58:10.525316+0000","last_became_peered":"2024-09-13T14:58:10.525316+0000","last_unstale":"2024-09-13T15:00:39.345897+0000","last_undegraded":"2024-09-13T15:00:39.345897+0000","last_fullsized":"2024-09-13T15:00:39.345897+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:41.689562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":417,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.316758+0000","last_change":"2024-09-13T14:58:10.517828+0000","last_active":"2024-09-13T15:00:29.316758+0000","last_peered":"2024-09-13T15:00:29.316758+0000","last_clean":"2024-09-13T15:00:29.316758+0000","last_became_active":"2024-09-13T14:58:10.517009+0000","last_became_peered":"2024-09-13T14:58:10.517009+0000","last_unstale":"2024-09-13T15:00:29.316758+0000","last_undegraded":"2024-09-13T15:00:29.316758+0000","last_fullsized":"2024-09-13T15:00:29.316758+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:29.728706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":382,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354363+0000","last_change":"2024-09-13T14:58:10.517445+0000","last_active":"2024-09-13T15:00:29.354363+0000","last_peered":"2024-09-13T15:00:29.354363+0000","last_clean":"2024-09-13T15:00:29.354363+0000","last_became_active":"2024-09-13T14:58:10.517101+0000","last_became_peered":"2024-09-13T14:58:10.517101+0000","last_unstale":"2024-09-13T15:00:29.354363+0000","last_undegraded":"2024-09-13T15:00:29.354363+0000","last_fullsized":"2024-09-13T15:00:29.354363+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:35.589919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":296,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.316973+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T15:00:29.316973+0000","last_peered":"2024-09-13T15:00:29.316973+0000","last_clean":"2024-09-13T15:00:29.316973+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T15:00:29.316973+0000","last_undegraded":"2024-09-13T15:00:29.316973+0000","last_fullsized":"2024-09-13T15:00:29.316973+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":305,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.317304+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T15:00:29.317304+0000","last_peered":"2024-09-13T15:00:29.317304+0000","last_clean":"2024-09-13T15:00:29.317304+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T15:00:29.317304+0000","last_undegraded":"2024-09-13T15:00:29.317304+0000","last_fullsized":"2024-09-13T15:00:29.317304+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":534,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676558+0000","last_change":"2024-09-13T14:57:53.351036+0000","last_active":"2024-09-13T15:00:39.676558+0000","last_peered":"2024-09-13T15:00:39.676558+0000","last_clean":"2024-09-13T15:00:39.676558+0000","last_became_active":"2024-09-13T14:57:53.350553+0000","last_became_peered":"2024-09-13T14:57:53.350553+0000","last_unstale":"2024-09-13T15:00:39.676558+0000","last_undegraded":"2024-09-13T15:00:39.676558+0000","last_fullsized":"2024-09-13T15:00:39.676558+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:17.354351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":45,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.356159+0000","last_change":"2024-09-13T14:57:53.360296+0000","last_active":"2024-09-13T15:00:29.356159+0000","last_peered":"2024-09-13T15:00:29.356159+0000","last_clean":"2024-09-13T15:00:29.356159+0000","last_became_active":"2024-09-13T14:57:53.347710+0000","last_became_peered":"2024-09-13T14:57:53.347710+0000","last_unstale":"2024-09-13T15:00:29.356159+0000","last_undegraded":"2024-09-13T15:00:29.356159+0000","last_fullsized":"2024-09-13T15:00:29.356159+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:43.099884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":127,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354026+0000","last_change":"2024-09-13T14:58:10.521307+0000","last_active":"2024-09-13T15:00:29.354026+0000","last_peered":"2024-09-13T15:00:29.354026+0000","last_clean":"2024-09-13T15:00:29.354026+0000","last_became_active":"2024-09-13T14:58:10.520905+0000","last_became_peered":"2024-09-13T14:58:10.520905+0000","last_unstale":"2024-09-13T15:00:29.354026+0000","last_undegraded":"2024-09-13T15:00:29.354026+0000","last_fullsized":"2024-09-13T15:00:29.354026+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:23.693312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":243,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.346469+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T15:00:39.346469+0000","last_peered":"2024-09-13T15:00:39.346469+0000","last_clean":"2024-09-13T15:00:39.346469+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T15:00:39.346469+0000","last_undegraded":"2024-09-13T15:00:39.346469+0000","last_fullsized":"2024-09-13T15:00:39.346469+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":530,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676897+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T15:00:39.676897+0000","last_peered":"2024-09-13T15:00:39.676897+0000","last_clean":"2024-09-13T15:00:39.676897+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T15:00:39.676897+0000","last_undegraded":"2024-09-13T15:00:39.676897+0000","last_fullsized":"2024-09-13T15:00:39.676897+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":83,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.344955+0000","last_change":"2024-09-13T14:58:28.428608+0000","last_active":"2024-09-13T15:00:39.344955+0000","last_peered":"2024-09-13T15:00:39.344955+0000","last_clean":"2024-09-13T15:00:39.344955+0000","last_became_active":"2024-09-13T14:58:28.428282+0000","last_became_peered":"2024-09-13T14:58:28.428282+0000","last_unstale":"2024-09-13T15:00:39.344955+0000","last_undegraded":"2024-09-13T15:00:39.344955+0000","last_fullsized":"2024-09-13T15:00:39.344955+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.603606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.603606+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:44.323633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":342,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355979+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T15:00:29.355979+0000","last_peered":"2024-09-13T15:00:29.355979+0000","last_clean":"2024-09-13T15:00:29.355979+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T15:00:29.355979+0000","last_undegraded":"2024-09-13T15:00:29.355979+0000","last_fullsized":"2024-09-13T15:00:29.355979+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":323,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354510+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T15:00:29.354510+0000","last_peered":"2024-09-13T15:00:29.354510+0000","last_clean":"2024-09-13T15:00:29.354510+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T15:00:29.354510+0000","last_undegraded":"2024-09-13T15:00:29.354510+0000","last_fullsized":"2024-09-13T15:00:29.354510+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":592,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355967+0000","last_change":"2024-09-13T14:58:10.512659+0000","last_active":"2024-09-13T15:00:29.355967+0000","last_peered":"2024-09-13T15:00:29.355967+0000","last_clean":"2024-09-13T15:00:29.355967+0000","last_became_active":"2024-09-13T14:58:10.511997+0000","last_became_peered":"2024-09-13T14:58:10.511997+0000","last_unstale":"2024-09-13T15:00:29.355967+0000","last_undegraded":"2024-09-13T15:00:29.355967+0000","last_fullsized":"2024-09-13T15:00:29.355967+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.415299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.415299+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:39.431834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":339,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355637+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T15:00:29.355637+0000","last_peered":"2024-09-13T15:00:29.355637+0000","last_clean":"2024-09-13T15:00:29.355637+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T15:00:29.355637+0000","last_undegraded":"2024-09-13T15:00:29.355637+0000","last_fullsized":"2024-09-13T15:00:29.355637+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":343,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.317108+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T15:00:29.317108+0000","last_peered":"2024-09-13T15:00:29.317108+0000","last_clean":"2024-09-13T15:00:29.317108+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T15:00:29.317108+0000","last_undegraded":"2024-09-13T15:00:29.317108+0000","last_fullsized":"2024-09-13T15:00:29.317108+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":138,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.342810+0000","last_change":"2024-09-13T14:58:28.424238+0000","last_active":"2024-09-13T15:00:39.342810+0000","last_peered":"2024-09-13T15:00:39.342810+0000","last_clean":"2024-09-13T15:00:39.342810+0000","last_became_active":"2024-09-13T14:58:28.423844+0000","last_became_peered":"2024-09-13T14:58:28.423844+0000","last_unstale":"2024-09-13T15:00:39.342810+0000","last_undegraded":"2024-09-13T15:00:39.342810+0000","last_fullsized":"2024-09-13T15:00:39.342810+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:55.429885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":322,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354553+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T15:00:29.354553+0000","last_peered":"2024-09-13T15:00:29.354553+0000","last_clean":"2024-09-13T15:00:29.354553+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T15:00:29.354553+0000","last_undegraded":"2024-09-13T15:00:29.354553+0000","last_fullsized":"2024-09-13T15:00:29.354553+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":571,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.677003+0000","last_change":"2024-09-13T14:58:10.522983+0000","last_active":"2024-09-13T15:00:39.677003+0000","last_peered":"2024-09-13T15:00:39.677003+0000","last_clean":"2024-09-13T15:00:39.677003+0000","last_became_active":"2024-09-13T14:58:10.513377+0000","last_became_peered":"2024-09-13T14:58:10.513377+0000","last_unstale":"2024-09-13T15:00:39.677003+0000","last_undegraded":"2024-09-13T15:00:39.677003+0000","last_fullsized":"2024-09-13T15:00:39.677003+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:12.579644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":163,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355270+0000","last_change":"2024-09-13T14:57:57.223728+0000","last_active":"2024-09-13T15:00:29.355270+0000","last_peered":"2024-09-13T15:00:29.355270+0000","last_clean":"2024-09-13T15:00:29.355270+0000","last_became_active":"2024-09-13T14:57:19.233883+0000","last_became_peered":"2024-09-13T14:57:19.233883+0000","last_unstale":"2024-09-13T15:00:29.355270+0000","last_undegraded":"2024-09-13T15:00:29.355270+0000","last_fullsized":"2024-09-13T15:00:29.355270+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:41.520060+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":91,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355053+0000","last_change":"2024-09-13T14:58:10.513713+0000","last_active":"2024-09-13T15:00:29.355053+0000","last_peered":"2024-09-13T15:00:29.355053+0000","last_clean":"2024-09-13T15:00:29.355053+0000","last_became_active":"2024-09-13T14:58:10.513334+0000","last_became_peered":"2024-09-13T14:58:10.513334+0000","last_unstale":"2024-09-13T15:00:29.355053+0000","last_undegraded":"2024-09-13T15:00:29.355053+0000","last_fullsized":"2024-09-13T15:00:29.355053+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:41.326218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":162,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676562+0000","last_change":"2024-09-13T14:58:28.428028+0000","last_active":"2024-09-13T15:00:39.676562+0000","last_peered":"2024-09-13T15:00:39.676562+0000","last_clean":"2024-09-13T15:00:39.676562+0000","last_became_active":"2024-09-13T14:58:28.427598+0000","last_became_peered":"2024-09-13T14:58:28.427598+0000","last_unstale":"2024-09-13T15:00:39.676562+0000","last_undegraded":"2024-09-13T15:00:39.676562+0000","last_fullsized":"2024-09-13T15:00:39.676562+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:32:50.930142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":163,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355764+0000","last_change":"2024-09-13T14:58:00.146439+0000","last_active":"2024-09-13T15:00:29.355764+0000","last_peered":"2024-09-13T15:00:29.355764+0000","last_clean":"2024-09-13T15:00:29.355764+0000","last_became_active":"2024-09-13T14:57:19.206097+0000","last_became_peered":"2024-09-13T14:57:19.206097+0000","last_unstale":"2024-09-13T15:00:29.355764+0000","last_undegraded":"2024-09-13T15:00:29.355764+0000","last_fullsized":"2024-09-13T15:00:29.355764+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:00.146384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:00.146384+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:09.697538+0000","scrub_duration":0.0014649660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":345,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.316977+0000","last_change":"2024-09-13T14:57:35.357982+0000","last_active":"2024-09-13T15:00:29.316977+0000","last_peered":"2024-09-13T15:00:29.316977+0000","last_clean":"2024-09-13T15:00:29.316977+0000","last_became_active":"2024-09-13T14:57:35.357618+0000","last_became_peered":"2024-09-13T14:57:35.357618+0000","last_unstale":"2024-09-13T15:00:29.316977+0000","last_undegraded":"2024-09-13T15:00:29.316977+0000","last_fullsized":"2024-09-13T15:00:29.316977+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:50.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":364,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355697+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:00:29.355697+0000","last_peered":"2024-09-13T15:00:29.355697+0000","last_clean":"2024-09-13T15:00:29.355697+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:00:29.355697+0000","last_undegraded":"2024-09-13T15:00:29.355697+0000","last_fullsized":"2024-09-13T15:00:29.355697+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":316,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.317400+0000","last_change":"2024-09-13T14:57:53.363216+0000","last_active":"2024-09-13T15:00:29.317400+0000","last_peered":"2024-09-13T15:00:29.317400+0000","last_clean":"2024-09-13T15:00:29.317400+0000","last_became_active":"2024-09-13T14:57:53.362679+0000","last_became_peered":"2024-09-13T14:57:53.362679+0000","last_unstale":"2024-09-13T15:00:29.317400+0000","last_undegraded":"2024-09-13T15:00:29.317400+0000","last_fullsized":"2024-09-13T15:00:29.317400+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.885757+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":349,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354374+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T15:00:29.354374+0000","last_peered":"2024-09-13T15:00:29.354374+0000","last_clean":"2024-09-13T15:00:29.354374+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T15:00:29.354374+0000","last_undegraded":"2024-09-13T15:00:29.354374+0000","last_fullsized":"2024-09-13T15:00:29.354374+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":326,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355163+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T15:00:29.355163+0000","last_peered":"2024-09-13T15:00:29.355163+0000","last_clean":"2024-09-13T15:00:29.355163+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T15:00:29.355163+0000","last_undegraded":"2024-09-13T15:00:29.355163+0000","last_fullsized":"2024-09-13T15:00:29.355163+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":329,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676332+0000","last_change":"2024-09-13T14:58:10.531563+0000","last_active":"2024-09-13T15:00:39.676332+0000","last_peered":"2024-09-13T15:00:39.676332+0000","last_clean":"2024-09-13T15:00:39.676332+0000","last_became_active":"2024-09-13T14:58:10.515595+0000","last_became_peered":"2024-09-13T14:58:10.515595+0000","last_unstale":"2024-09-13T15:00:39.676332+0000","last_undegraded":"2024-09-13T15:00:39.676332+0000","last_fullsized":"2024-09-13T15:00:39.676332+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:19.723855+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":41,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354368+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:00:29.354368+0000","last_peered":"2024-09-13T15:00:29.354368+0000","last_clean":"2024-09-13T15:00:29.354368+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:00:29.354368+0000","last_undegraded":"2024-09-13T15:00:29.354368+0000","last_fullsized":"2024-09-13T15:00:29.354368+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":135,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355133+0000","last_change":"2024-09-13T14:58:10.530803+0000","last_active":"2024-09-13T15:00:29.355133+0000","last_peered":"2024-09-13T15:00:29.355133+0000","last_clean":"2024-09-13T15:00:29.355133+0000","last_became_active":"2024-09-13T14:58:10.522078+0000","last_became_peered":"2024-09-13T14:58:10.522078+0000","last_unstale":"2024-09-13T15:00:29.355133+0000","last_undegraded":"2024-09-13T15:00:29.355133+0000","last_fullsized":"2024-09-13T15:00:29.355133+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:50.807687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":147,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355429+0000","last_change":"2024-09-13T14:57:55.208230+0000","last_active":"2024-09-13T15:00:29.355429+0000","last_peered":"2024-09-13T15:00:29.355429+0000","last_clean":"2024-09-13T15:00:29.355429+0000","last_became_active":"2024-09-13T14:57:19.211671+0000","last_became_peered":"2024-09-13T14:57:19.211671+0000","last_unstale":"2024-09-13T15:00:29.355429+0000","last_undegraded":"2024-09-13T15:00:29.355429+0000","last_fullsized":"2024-09-13T15:00:29.355429+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.066938+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":431,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.316899+0000","last_change":"2024-09-13T14:58:10.529854+0000","last_active":"2024-09-13T15:00:29.316899+0000","last_peered":"2024-09-13T15:00:29.316899+0000","last_clean":"2024-09-13T15:00:29.316899+0000","last_became_active":"2024-09-13T14:58:10.514813+0000","last_became_peered":"2024-09-13T14:58:10.514813+0000","last_unstale":"2024-09-13T15:00:29.316899+0000","last_undegraded":"2024-09-13T15:00:29.316899+0000","last_fullsized":"2024-09-13T15:00:29.316899+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:46.296441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":568,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676887+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T15:00:39.676887+0000","last_peered":"2024-09-13T15:00:39.676887+0000","last_clean":"2024-09-13T15:00:39.676887+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T15:00:39.676887+0000","last_undegraded":"2024-09-13T15:00:39.676887+0000","last_fullsized":"2024-09-13T15:00:39.676887+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":41,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.356152+0000","last_change":"2024-09-13T14:57:53.356010+0000","last_active":"2024-09-13T15:00:29.356152+0000","last_peered":"2024-09-13T15:00:29.356152+0000","last_clean":"2024-09-13T15:00:29.356152+0000","last_became_active":"2024-09-13T14:57:53.354979+0000","last_became_peered":"2024-09-13T14:57:53.354979+0000","last_unstale":"2024-09-13T15:00:29.356152+0000","last_undegraded":"2024-09-13T15:00:29.356152+0000","last_fullsized":"2024-09-13T15:00:29.356152+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:38.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":327,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355187+0000","last_change":"2024-09-13T14:57:35.358990+0000","last_active":"2024-09-13T15:00:29.355187+0000","last_peered":"2024-09-13T15:00:29.355187+0000","last_clean":"2024-09-13T15:00:29.355187+0000","last_became_active":"2024-09-13T14:57:35.358547+0000","last_became_peered":"2024-09-13T14:57:35.358547+0000","last_unstale":"2024-09-13T15:00:29.355187+0000","last_undegraded":"2024-09-13T15:00:29.355187+0000","last_fullsized":"2024-09-13T15:00:29.355187+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:45.386301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":448,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.677011+0000","last_change":"2024-09-13T14:58:10.536776+0000","last_active":"2024-09-13T15:00:39.677011+0000","last_peered":"2024-09-13T15:00:39.677011+0000","last_clean":"2024-09-13T15:00:39.677011+0000","last_became_active":"2024-09-13T14:58:10.536376+0000","last_became_peered":"2024-09-13T14:58:10.536376+0000","last_unstale":"2024-09-13T15:00:39.677011+0000","last_undegraded":"2024-09-13T15:00:39.677011+0000","last_fullsized":"2024-09-13T15:00:39.677011+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:08.237256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":286,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676768+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T15:00:39.676768+0000","last_peered":"2024-09-13T15:00:39.676768+0000","last_clean":"2024-09-13T15:00:39.676768+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T15:00:39.676768+0000","last_undegraded":"2024-09-13T15:00:39.676768+0000","last_fullsized":"2024-09-13T15:00:39.676768+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:04.244429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":428,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354092+0000","last_change":"2024-09-13T14:58:10.524047+0000","last_active":"2024-09-13T15:00:29.354092+0000","last_peered":"2024-09-13T15:00:29.354092+0000","last_clean":"2024-09-13T15:00:29.354092+0000","last_became_active":"2024-09-13T14:58:10.522007+0000","last_became_peered":"2024-09-13T14:58:10.522007+0000","last_unstale":"2024-09-13T15:00:29.354092+0000","last_undegraded":"2024-09-13T15:00:29.354092+0000","last_fullsized":"2024-09-13T15:00:29.354092+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:41.957759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":218,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355497+0000","last_change":"2024-09-13T14:58:28.426617+0000","last_active":"2024-09-13T15:00:29.355497+0000","last_peered":"2024-09-13T15:00:29.355497+0000","last_clean":"2024-09-13T15:00:29.355497+0000","last_became_active":"2024-09-13T14:58:28.425773+0000","last_became_peered":"2024-09-13T14:58:28.425773+0000","last_unstale":"2024-09-13T15:00:29.355497+0000","last_undegraded":"2024-09-13T15:00:29.355497+0000","last_fullsized":"2024-09-13T15:00:29.355497+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:59.659871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":49,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.354192+0000","last_change":"2024-09-13T14:58:28.428896+0000","last_active":"2024-09-13T15:00:29.354192+0000","last_peered":"2024-09-13T15:00:29.354192+0000","last_clean":"2024-09-13T15:00:29.354192+0000","last_became_active":"2024-09-13T14:58:28.428522+0000","last_became_peered":"2024-09-13T14:58:28.428522+0000","last_unstale":"2024-09-13T15:00:29.354192+0000","last_undegraded":"2024-09-13T15:00:29.354192+0000","last_fullsized":"2024-09-13T15:00:29.354192+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:05.418586+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":342,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.353676+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T15:00:29.353676+0000","last_peered":"2024-09-13T15:00:29.353676+0000","last_clean":"2024-09-13T15:00:29.353676+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T15:00:29.353676+0000","last_undegraded":"2024-09-13T15:00:29.353676+0000","last_fullsized":"2024-09-13T15:00:29.353676+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":63,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.676641+0000","last_change":"2024-09-13T14:58:10.517240+0000","last_active":"2024-09-13T15:00:39.676641+0000","last_peered":"2024-09-13T15:00:39.676641+0000","last_clean":"2024-09-13T15:00:39.676641+0000","last_became_active":"2024-09-13T14:58:10.516811+0000","last_became_peered":"2024-09-13T14:58:10.516811+0000","last_unstale":"2024-09-13T15:00:39.676641+0000","last_undegraded":"2024-09-13T15:00:39.676641+0000","last_fullsized":"2024-09-13T15:00:39.676641+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:52.711800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":558,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T15:00:29.355821+0000","last_change":"2024-09-13T14:58:10.512480+0000","last_active":"2024-09-13T15:00:29.355821+0000","last_peered":"2024-09-13T15:00:29.355821+0000","last_clean":"2024-09-13T15:00:29.355821+0000","last_became_active":"2024-09-13T14:58:10.511589+0000","last_became_peered":"2024-09-13T14:58:10.511589+0000","last_unstale":"2024-09-13T15:00:29.355821+0000","last_undegraded":"2024-09-13T15:00:29.355821+0000","last_fullsized":"2024-09-13T15:00:29.355821+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:52.302190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":448,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365292,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1521584,"kb_used_data":20580,"kb_used_omap":1,"kb_used_meta":1500990,"kb_avail":92223568,"statfs":{"total":95995035648,"available":94436933632,"internally_reserved":0,"allocated":21073920,"data_stored":18931209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1537014104},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561522,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501544,"kb_used_data":5148,"kb_used_omap":1,"kb_used_meta":1496382,"kb_avail":92243608,"statfs":{"total":95995035648,"available":94457454592,"internally_reserved":0,"allocated":5271552,"data_stored":3136882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1532295512},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790454,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491484,"kb_used_data":5072,"kb_used_omap":1,"kb_used_meta":1486398,"kb_avail":92253668,"statfs":{"total":95995035648,"available":94467756032,"internally_reserved":0,"allocated":5193728,"data_stored":3066894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1522071896},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986682,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1233572,"kb_used_data":42460,"kb_used_omap":1,"kb_used_meta":1191102,"kb_avail":92511580,"statfs":{"total":95995035648,"available":94731857920,"internally_reserved":0,"allocated":43479040,"data_stored":41344138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1219688792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519838,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890584,"kb_used_data":5132,"kb_used_omap":4,"kb_used_meta":885435,"kb_avail":92854568,"statfs":{"total":95995035648,"available":95083077632,"internally_reserved":0,"allocated":5255168,"data_stored":3131896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":906685888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304753,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912224,"kb_used_data":5076,"kb_used_omap":6,"kb_used_meta":907129,"kb_avail":92832928,"statfs":{"total":95995035648,"available":95060918272,"internally_reserved":0,"allocated":5197824,"data_stored":3067418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":928900551},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828056,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1140896,"kb_used_data":4820,"kb_used_omap":5,"kb_used_meta":1136058,"kb_avail":92604256,"statfs":{"total":95995035648,"available":94826758144,"internally_reserved":0,"allocated":4935680,"data_stored":2804209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1163323628},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":356482285921,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:00:50.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:00:50.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:50.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:50.585 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:00:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:00:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:50.687 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:50.711 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:50.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:50.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:50.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:50.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:50.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:51.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:51.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:51.187 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:51.214 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:51.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:51.343 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:51.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:51.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:51.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:51.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:51 smithi019 ceph-mon[91296]: pgmap v1287: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:51.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:51 smithi003 ceph-mon[104007]: pgmap v1287: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:51 smithi003 ceph-mon[107278]: pgmap v1287: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:51.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:51.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:51.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:51.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:51.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:51.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:52.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:52.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:52.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:52.230 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:52.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:52.359 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:52.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:52.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:52.487 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:52.539 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:52.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:52.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:52.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:52.734 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:52.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:52.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:52.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:52.867 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:52.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:52.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:52.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:53.098 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.126 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:53.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:53.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:53.253 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:53.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:53.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:53.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:53.507 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:53.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:00:53.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:00:53.571 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:00:53.571 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:00:53.571 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:00:53.571 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:53.571 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:53.571 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:53.571 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:53.571 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:53.581 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:00:53.581 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:53.597 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:00:53.597 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:53.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:53.613 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:00:53.613 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.635 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:53.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:53 smithi019 ceph-mon[91296]: pgmap v1288: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:53 smithi003 ceph-mon[107278]: pgmap v1288: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:53.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:53 smithi003 ceph-mon[104007]: pgmap v1288: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:53.736 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:53.762 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:53.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:53.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:53.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:54.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:54.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:54.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:54.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:54.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:54.361 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:54.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:54.394 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_8' created 2024-09-13T15:00:54.394 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:00:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:00:54] "GET /metrics HTTP/1.1" 200 34116 "" "Prometheus/2.51.0" 2024-09-13T15:00:54.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:54.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:54.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:54.641 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:54.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:54 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2989349297' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-13T15:00:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:54 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1633779832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:54.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:54 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2989349297' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-13T15:00:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:54 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1633779832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:54 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2989349297' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-13T15:00:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:54 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1633779832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:54.742 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:54.769 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:54.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:54.897 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:54.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:55.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:55.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:55.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:55.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_8 rados --yes-i-really-mean-it || true 2024-09-13T15:00:55.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:55.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:55.432 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:55.518 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:55.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:55.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:55.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:55 smithi003 ceph-mon[104007]: pgmap v1289: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:55.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2989349297' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-09-13T15:00:55.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:55 smithi003 ceph-mon[104007]: osdmap e362: 8 total, 8 up, 7 in 2024-09-13T15:00:55.619 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:55 smithi003 ceph-mon[107278]: pgmap v1289: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:55.619 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2989349297' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-09-13T15:00:55.619 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:55 smithi003 ceph-mon[107278]: osdmap e362: 8 total, 8 up, 7 in 2024-09-13T15:00:55.660 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:55 smithi019 ceph-mon[91296]: pgmap v1289: 103 pgs: 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2989349297' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-09-13T15:00:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:55 smithi019 ceph-mon[91296]: osdmap e362: 8 total, 8 up, 7 in 2024-09-13T15:00:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:55.788 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:55.816 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:55.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:55.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:56.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:56.072 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:56.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:56.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:56.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:56.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:56.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:56.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:56.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:56.581 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:56 smithi019 ceph-mon[91296]: osdmap e363: 8 total, 8 up, 7 in 2024-09-13T15:00:56.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:56 smithi003 ceph-mon[104007]: osdmap e363: 8 total, 8 up, 7 in 2024-09-13T15:00:56.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:56 smithi003 ceph-mon[107278]: osdmap e363: 8 total, 8 up, 7 in 2024-09-13T15:00:56.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:56.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:56.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:56.835 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:56.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:56.964 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:57.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:57.093 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:57.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:57.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:57.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:57.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:57.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:57.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:57.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:57.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:57.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:57.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:57.665 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:00:57.665 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:00:57.665 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:00:57.665 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:00:57.665 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:00:57.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:57 smithi003 ceph-mon[107278]: pgmap v1292: 119 pgs: 16 unknown, 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:57.666 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:57 smithi003 ceph-mon[107278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:00:57.666 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:57 smithi003 ceph-mon[107278]: osdmap e364: 8 total, 8 up, 7 in 2024-09-13T15:00:57.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:57 smithi003 ceph-mon[104007]: pgmap v1292: 119 pgs: 16 unknown, 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:57.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:57 smithi003 ceph-mon[104007]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:00:57.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:57 smithi003 ceph-mon[104007]: osdmap e364: 8 total, 8 up, 7 in 2024-09-13T15:00:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:57 smithi019 ceph-mon[91296]: pgmap v1292: 119 pgs: 16 unknown, 103 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:57 smithi019 ceph-mon[91296]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:00:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:57 smithi019 ceph-mon[91296]: osdmap e364: 8 total, 8 up, 7 in 2024-09-13T15:00:57.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:57.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:57.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:57.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:57.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:57.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:58.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:00:58.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:58.098 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:58.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:58.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:58.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:58.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:58.478 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:58.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:58.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:58.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:58.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:58.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:58.956 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:58.984 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:59.085 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:59.112 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:00:58 smithi019 ceph-mon[91296]: pgmap v1294: 119 pgs: 12 unknown, 107 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:59.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:00:58 smithi003 ceph-mon[107278]: pgmap v1294: 119 pgs: 12 unknown, 107 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:00:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:59.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:00:58 smithi003 ceph-mon[104007]: pgmap v1294: 119 pgs: 12 unknown, 107 active+clean; 582 KiB data, 8.3 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:00:59.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:59.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:59.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:00:59.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:59.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:59.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:59.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:59.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:59.647 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:00:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:59.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:59.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:59.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:59.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:00:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:59.998 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:00.025 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:00.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:00.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:00.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:00.283 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:00.385 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:00.411 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:00.459 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:00.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:00.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:00.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:00.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:01:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:01:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:00.763 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:00.789 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:00.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:00.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:01.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:01.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:01.131 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:01:01.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:01.174 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:01.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:01.382 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:01 smithi003 ceph-mon[107278]: pgmap v1295: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:01.382 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:01 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2860635262' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:01.382 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:01 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:01.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:01 smithi003 ceph-mon[104007]: pgmap v1295: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:01.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:01 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2860635262' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:01.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:01 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:01.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:01.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:01 smithi019 ceph-mon[91296]: pgmap v1295: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:01 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2860635262' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:01.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:01 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:01.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:01.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:01.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:01.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:01.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:01.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:01.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:01.949 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:01.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:02.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:02.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:02.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:02.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:02 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:02.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:02 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:02.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:02 smithi003 ceph-mon[107278]: osdmap e365: 8 total, 8 up, 7 in 2024-09-13T15:01:02.179 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:02 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:02.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:02 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:02.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:02 smithi003 ceph-mon[104007]: osdmap e365: 8 total, 8 up, 7 in 2024-09-13T15:01:02.180 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:02.206 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:02.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:02.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:02 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:02 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:02 smithi019 ceph-mon[91296]: osdmap e365: 8 total, 8 up, 7 in 2024-09-13T15:01:02.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:02.462 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:02.564 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:02.589 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:02.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.722 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:02.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:02.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:02.849 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:02.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:02.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:03.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:03.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:03.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:03.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:03.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:03.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:03.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:03 smithi003 ceph-mon[107278]: pgmap v1297: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:03.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:03.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:03 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:03.415 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:03 smithi003 ceph-mon[104007]: pgmap v1297: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:03.415 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:03.415 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:03 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:03 smithi019 ceph-mon[91296]: pgmap v1297: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:03 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:03.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:03.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:03.502 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:01:03.504 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:01:03.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:03.610 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:03.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:03.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:03.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:03.837 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:03.863 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:03.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:03.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:04.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:04.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:04.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:04.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:04.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:01:04.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:01:04.260 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:04.260 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:01:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:01:04] "GET /metrics HTTP/1.1" 200 36460 "" "Prometheus/2.51.0" 2024-09-13T15:01:04.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:01:04.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:01:04.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:04.340 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:04.367 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:01:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:01:04.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:04.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.499 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:04.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:04.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:04.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:04.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:04.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:04.753 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:04.769 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:04.854 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:04.882 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:04.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:04.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:05.011 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:05.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:05.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:05.265 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:05.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:05.308 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:05 smithi003 ceph-mon[107278]: from='client.36894 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:05.308 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:05 smithi003 ceph-mon[107278]: pgmap v1298: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:05.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:05 smithi003 ceph-mon[104007]: from='client.36894 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:05.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:05 smithi003 ceph-mon[104007]: pgmap v1298: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:05.337 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1297,"stamp":"2024-09-13T15:01:02.071008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":328147,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3159,"ondisk_log_size":3159,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":336,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8635420,"kb_used_data":44288,"kb_used_omap":23,"kb_used_meta":8591016,"kb_avail":647580644,"statfs":{"total":671965249536,"available":663122579456,"internally_reserved":0,"allocated":45350912,"data_stored":30335007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24109,"internal_metadata":8797200851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.675360"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":112,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400638+0000","last_change":"2024-09-13T14:57:42.607827+0000","last_active":"2024-09-13T15:00:55.400638+0000","last_peered":"2024-09-13T15:00:55.400638+0000","last_clean":"2024-09-13T15:00:55.400638+0000","last_became_active":"2024-09-13T14:57:25.285562+0000","last_became_peered":"2024-09-13T14:57:25.285562+0000","last_unstale":"2024-09-13T15:00:55.400638+0000","last_undegraded":"2024-09-13T15:00:55.400638+0000","last_fullsized":"2024-09-13T15:00:55.400638+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.607753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.607753+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:21:26.227325+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":75,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.279320+0000","last_change":"2024-09-13T14:58:28.426659+0000","last_active":"2024-09-13T15:00:58.279320+0000","last_peered":"2024-09-13T15:00:58.279320+0000","last_clean":"2024-09-13T15:00:58.279320+0000","last_became_active":"2024-09-13T14:58:28.426336+0000","last_became_peered":"2024-09-13T14:58:28.426336+0000","last_unstale":"2024-09-13T15:00:58.279320+0000","last_undegraded":"2024-09-13T15:00:58.279320+0000","last_fullsized":"2024-09-13T15:00:58.279320+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:55.713969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":336,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.406692+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T15:00:55.406692+0000","last_peered":"2024-09-13T15:00:55.406692+0000","last_clean":"2024-09-13T15:00:55.406692+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T15:00:55.406692+0000","last_undegraded":"2024-09-13T15:00:55.406692+0000","last_fullsized":"2024-09-13T15:00:55.406692+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":62,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401469+0000","last_change":"2024-09-13T14:57:58.200463+0000","last_active":"2024-09-13T15:00:55.401469+0000","last_peered":"2024-09-13T15:00:55.401469+0000","last_clean":"2024-09-13T15:00:55.401469+0000","last_became_active":"2024-09-13T14:57:25.279086+0000","last_became_peered":"2024-09-13T14:57:25.279086+0000","last_unstale":"2024-09-13T15:00:55.401469+0000","last_undegraded":"2024-09-13T15:00:55.401469+0000","last_fullsized":"2024-09-13T15:00:55.401469+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:58.986716+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":89,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400743+0000","last_change":"2024-09-13T14:57:35.356151+0000","last_active":"2024-09-13T15:00:55.400743+0000","last_peered":"2024-09-13T15:00:55.400743+0000","last_clean":"2024-09-13T15:00:55.400743+0000","last_became_active":"2024-09-13T14:57:35.355770+0000","last_became_peered":"2024-09-13T14:57:35.355770+0000","last_unstale":"2024-09-13T15:00:55.400743+0000","last_undegraded":"2024-09-13T15:00:55.400743+0000","last_fullsized":"2024-09-13T15:00:55.400743+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:37.887286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":574,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401315+0000","last_change":"2024-09-13T14:58:10.513979+0000","last_active":"2024-09-13T15:00:55.401315+0000","last_peered":"2024-09-13T15:00:55.401315+0000","last_clean":"2024-09-13T15:00:55.401315+0000","last_became_active":"2024-09-13T14:58:10.513313+0000","last_became_peered":"2024-09-13T14:58:10.513313+0000","last_unstale":"2024-09-13T15:00:55.401315+0000","last_undegraded":"2024-09-13T15:00:55.401315+0000","last_fullsized":"2024-09-13T15:00:55.401315+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:25:15.376408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":67,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.411268+0000","last_change":"2024-09-13T14:58:28.426763+0000","last_active":"2024-09-13T15:00:55.411268+0000","last_peered":"2024-09-13T15:00:55.411268+0000","last_clean":"2024-09-13T15:00:55.411268+0000","last_became_active":"2024-09-13T14:58:28.426270+0000","last_became_peered":"2024-09-13T14:58:28.426270+0000","last_unstale":"2024-09-13T15:00:55.411268+0000","last_undegraded":"2024-09-13T15:00:55.411268+0000","last_fullsized":"2024-09-13T15:00:55.411268+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.991576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":353,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.409670+0000","last_change":"2024-09-13T14:58:10.515535+0000","last_active":"2024-09-13T15:00:55.409670+0000","last_peered":"2024-09-13T15:00:55.409670+0000","last_clean":"2024-09-13T15:00:55.409670+0000","last_became_active":"2024-09-13T14:58:10.515043+0000","last_became_peered":"2024-09-13T14:58:10.515043+0000","last_unstale":"2024-09-13T15:00:55.409670+0000","last_undegraded":"2024-09-13T15:00:55.409670+0000","last_fullsized":"2024-09-13T15:00:55.409670+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:53.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":114,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401016+0000","last_change":"2024-09-13T14:57:43.658933+0000","last_active":"2024-09-13T15:00:55.401016+0000","last_peered":"2024-09-13T15:00:55.401016+0000","last_clean":"2024-09-13T15:00:55.401016+0000","last_became_active":"2024-09-13T14:57:25.291285+0000","last_became_peered":"2024-09-13T14:57:25.291285+0000","last_unstale":"2024-09-13T15:00:55.401016+0000","last_undegraded":"2024-09-13T15:00:55.401016+0000","last_fullsized":"2024-09-13T15:00:55.401016+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:43.658833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:43.658833+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:22.120697+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":67,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410654+0000","last_change":"2024-09-13T14:58:28.429061+0000","last_active":"2024-09-13T15:00:55.410654+0000","last_peered":"2024-09-13T15:00:55.410654+0000","last_clean":"2024-09-13T15:00:55.410654+0000","last_became_active":"2024-09-13T14:58:28.428642+0000","last_became_peered":"2024-09-13T14:58:28.428642+0000","last_unstale":"2024-09-13T15:00:55.410654+0000","last_undegraded":"2024-09-13T15:00:55.410654+0000","last_fullsized":"2024-09-13T15:00:55.410654+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:03.197824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":373,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410636+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T15:00:55.410636+0000","last_peered":"2024-09-13T15:00:55.410636+0000","last_clean":"2024-09-13T15:00:55.410636+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T15:00:55.410636+0000","last_undegraded":"2024-09-13T15:00:55.410636+0000","last_fullsized":"2024-09-13T15:00:55.410636+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":121,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400192+0000","last_change":"2024-09-13T14:58:10.521468+0000","last_active":"2024-09-13T15:00:55.400192+0000","last_peered":"2024-09-13T15:00:55.400192+0000","last_clean":"2024-09-13T15:00:55.400192+0000","last_became_active":"2024-09-13T14:58:10.520775+0000","last_became_peered":"2024-09-13T14:58:10.520775+0000","last_unstale":"2024-09-13T15:00:55.400192+0000","last_undegraded":"2024-09-13T15:00:55.400192+0000","last_fullsized":"2024-09-13T15:00:55.400192+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:27.363579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":70,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410480+0000","last_change":"2024-09-13T14:58:10.523188+0000","last_active":"2024-09-13T15:00:55.410480+0000","last_peered":"2024-09-13T15:00:55.410480+0000","last_clean":"2024-09-13T15:00:55.410480+0000","last_became_active":"2024-09-13T14:58:10.522815+0000","last_became_peered":"2024-09-13T14:58:10.522815+0000","last_unstale":"2024-09-13T15:00:55.410480+0000","last_undegraded":"2024-09-13T15:00:55.410480+0000","last_fullsized":"2024-09-13T15:00:55.410480+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:27.568999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":594,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.279288+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T15:00:58.279288+0000","last_peered":"2024-09-13T15:00:58.279288+0000","last_clean":"2024-09-13T15:00:58.279288+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T15:00:58.279288+0000","last_undegraded":"2024-09-13T15:00:58.279288+0000","last_fullsized":"2024-09-13T15:00:58.279288+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":59,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.402371+0000","last_change":"2024-09-13T14:58:09.011981+0000","last_active":"2024-09-13T15:00:55.402371+0000","last_peered":"2024-09-13T15:00:55.402371+0000","last_clean":"2024-09-13T15:00:55.402371+0000","last_became_active":"2024-09-13T14:57:53.357674+0000","last_became_peered":"2024-09-13T14:57:53.357674+0000","last_unstale":"2024-09-13T15:00:55.402371+0000","last_undegraded":"2024-09-13T15:00:55.402371+0000","last_fullsized":"2024-09-13T15:00:55.402371+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.701912+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":345,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.409826+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T15:00:55.409826+0000","last_peered":"2024-09-13T15:00:55.409826+0000","last_clean":"2024-09-13T15:00:55.409826+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T15:00:55.409826+0000","last_undegraded":"2024-09-13T15:00:55.409826+0000","last_fullsized":"2024-09-13T15:00:55.409826+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":534,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400667+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T15:00:55.400667+0000","last_peered":"2024-09-13T15:00:55.400667+0000","last_clean":"2024-09-13T15:00:55.400667+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T15:00:55.400667+0000","last_undegraded":"2024-09-13T15:00:55.400667+0000","last_fullsized":"2024-09-13T15:00:55.400667+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":65,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401310+0000","last_change":"2024-09-13T14:58:10.513244+0000","last_active":"2024-09-13T15:00:55.401310+0000","last_peered":"2024-09-13T15:00:55.401310+0000","last_clean":"2024-09-13T15:00:55.401310+0000","last_became_active":"2024-09-13T14:58:10.511485+0000","last_became_peered":"2024-09-13T14:58:10.511485+0000","last_unstale":"2024-09-13T15:00:55.401310+0000","last_undegraded":"2024-09-13T15:00:55.401310+0000","last_fullsized":"2024-09-13T15:00:55.401310+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:23.283605+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":251,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.278702+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T15:00:58.278702+0000","last_peered":"2024-09-13T15:00:58.278702+0000","last_clean":"2024-09-13T15:00:58.278702+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T15:00:58.278702+0000","last_undegraded":"2024-09-13T15:00:58.278702+0000","last_fullsized":"2024-09-13T15:00:58.278702+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":434,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.403938+0000","last_change":"2024-09-13T14:58:10.536485+0000","last_active":"2024-09-13T15:00:55.403938+0000","last_peered":"2024-09-13T15:00:55.403938+0000","last_clean":"2024-09-13T15:00:55.403938+0000","last_became_active":"2024-09-13T14:58:10.536134+0000","last_became_peered":"2024-09-13T14:58:10.536134+0000","last_unstale":"2024-09-13T15:00:55.403938+0000","last_undegraded":"2024-09-13T15:00:55.403938+0000","last_fullsized":"2024-09-13T15:00:55.403938+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:59.095681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":62,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.409441+0000","last_change":"2024-09-13T14:58:05.212101+0000","last_active":"2024-09-13T15:00:55.409441+0000","last_peered":"2024-09-13T15:00:55.409441+0000","last_clean":"2024-09-13T15:00:55.409441+0000","last_became_active":"2024-09-13T14:57:25.275734+0000","last_became_peered":"2024-09-13T14:57:25.275734+0000","last_unstale":"2024-09-13T15:00:55.409441+0000","last_undegraded":"2024-09-13T15:00:55.409441+0000","last_fullsized":"2024-09-13T15:00:55.409441+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.212008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.212008+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:01.720211+0000","scrub_duration":0.0048969690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":573,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400236+0000","last_change":"2024-09-13T14:58:10.515043+0000","last_active":"2024-09-13T15:00:55.400236+0000","last_peered":"2024-09-13T15:00:55.400236+0000","last_clean":"2024-09-13T15:00:55.400236+0000","last_became_active":"2024-09-13T14:58:10.513487+0000","last_became_peered":"2024-09-13T14:58:10.513487+0000","last_unstale":"2024-09-13T15:00:55.400236+0000","last_undegraded":"2024-09-13T15:00:55.400236+0000","last_fullsized":"2024-09-13T15:00:55.400236+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:11.392005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":346,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.398425+0000","last_change":"2024-09-13T14:57:53.364039+0000","last_active":"2024-09-13T15:00:55.398425+0000","last_peered":"2024-09-13T15:00:55.398425+0000","last_clean":"2024-09-13T15:00:55.398425+0000","last_became_active":"2024-09-13T14:57:53.363550+0000","last_became_peered":"2024-09-13T14:57:53.363550+0000","last_unstale":"2024-09-13T15:00:55.398425+0000","last_undegraded":"2024-09-13T15:00:55.398425+0000","last_fullsized":"2024-09-13T15:00:55.398425+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:36.171089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":88,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401911+0000","last_change":"2024-09-13T14:58:04.168950+0000","last_active":"2024-09-13T15:00:55.401911+0000","last_peered":"2024-09-13T15:00:55.401911+0000","last_clean":"2024-09-13T15:00:55.401911+0000","last_became_active":"2024-09-13T14:57:53.351301+0000","last_became_peered":"2024-09-13T14:57:53.351301+0000","last_unstale":"2024-09-13T15:00:55.401911+0000","last_undegraded":"2024-09-13T15:00:55.401911+0000","last_fullsized":"2024-09-13T15:00:55.401911+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:42.270600+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":535,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400127+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T15:00:55.400127+0000","last_peered":"2024-09-13T15:00:55.400127+0000","last_clean":"2024-09-13T15:00:55.400127+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T15:00:55.400127+0000","last_undegraded":"2024-09-13T15:00:55.400127+0000","last_fullsized":"2024-09-13T15:00:55.400127+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":587,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401806+0000","last_change":"2024-09-13T14:58:10.514433+0000","last_active":"2024-09-13T15:00:55.401806+0000","last_peered":"2024-09-13T15:00:55.401806+0000","last_clean":"2024-09-13T15:00:55.401806+0000","last_became_active":"2024-09-13T14:58:10.513446+0000","last_became_peered":"2024-09-13T14:58:10.513446+0000","last_unstale":"2024-09-13T15:00:55.401806+0000","last_undegraded":"2024-09-13T15:00:55.401806+0000","last_fullsized":"2024-09-13T15:00:55.401806+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:24.535893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":540,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400411+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T15:00:55.400411+0000","last_peered":"2024-09-13T15:00:55.400411+0000","last_clean":"2024-09-13T15:00:55.400411+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T15:00:55.400411+0000","last_undegraded":"2024-09-13T15:00:55.400411+0000","last_fullsized":"2024-09-13T15:00:55.400411+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":45,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401451+0000","last_change":"2024-09-13T14:57:53.350622+0000","last_active":"2024-09-13T15:00:55.401451+0000","last_peered":"2024-09-13T15:00:55.401451+0000","last_clean":"2024-09-13T15:00:55.401451+0000","last_became_active":"2024-09-13T14:57:53.350292+0000","last_became_peered":"2024-09-13T14:57:53.350292+0000","last_unstale":"2024-09-13T15:00:55.401451+0000","last_undegraded":"2024-09-13T15:00:55.401451+0000","last_fullsized":"2024-09-13T15:00:55.401451+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:16.088338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":294,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.279196+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T15:00:58.279196+0000","last_peered":"2024-09-13T15:00:58.279196+0000","last_clean":"2024-09-13T15:00:58.279196+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T15:00:58.279196+0000","last_undegraded":"2024-09-13T15:00:58.279196+0000","last_fullsized":"2024-09-13T15:00:58.279196+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.284651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":55,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401995+0000","last_change":"2024-09-13T14:58:10.527352+0000","last_active":"2024-09-13T15:00:55.401995+0000","last_peered":"2024-09-13T15:00:55.401995+0000","last_clean":"2024-09-13T15:00:55.401995+0000","last_became_active":"2024-09-13T14:58:10.526614+0000","last_became_peered":"2024-09-13T14:58:10.526614+0000","last_unstale":"2024-09-13T15:00:55.401995+0000","last_undegraded":"2024-09-13T15:00:55.401995+0000","last_fullsized":"2024-09-13T15:00:55.401995+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:04.459255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":127,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401444+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T15:00:55.401444+0000","last_peered":"2024-09-13T15:00:55.401444+0000","last_clean":"2024-09-13T15:00:55.401444+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T15:00:55.401444+0000","last_undegraded":"2024-09-13T15:00:55.401444+0000","last_fullsized":"2024-09-13T15:00:55.401444+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":265,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.397402+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T15:00:55.397402+0000","last_peered":"2024-09-13T15:00:55.397402+0000","last_clean":"2024-09-13T15:00:55.397402+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T15:00:55.397402+0000","last_undegraded":"2024-09-13T15:00:55.397402+0000","last_fullsized":"2024-09-13T15:00:55.397402+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":121,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:57.913238+0000","last_change":"2024-09-13T14:58:10.511301+0000","last_active":"2024-09-13T15:00:57.913238+0000","last_peered":"2024-09-13T15:00:57.913238+0000","last_clean":"2024-09-13T15:00:57.913238+0000","last_became_active":"2024-09-13T14:58:10.510934+0000","last_became_peered":"2024-09-13T14:58:10.510934+0000","last_unstale":"2024-09-13T15:00:57.913238+0000","last_undegraded":"2024-09-13T15:00:57.913238+0000","last_fullsized":"2024-09-13T15:00:57.913238+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:12.152076+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":333,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410744+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T15:00:55.410744+0000","last_peered":"2024-09-13T15:00:55.410744+0000","last_clean":"2024-09-13T15:00:55.410744+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T15:00:55.410744+0000","last_undegraded":"2024-09-13T15:00:55.410744+0000","last_fullsized":"2024-09-13T15:00:55.410744+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":326,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410724+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T15:00:55.410724+0000","last_peered":"2024-09-13T15:00:55.410724+0000","last_clean":"2024-09-13T15:00:55.410724+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T15:00:55.410724+0000","last_undegraded":"2024-09-13T15:00:55.410724+0000","last_fullsized":"2024-09-13T15:00:55.410724+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.278595+0000","last_change":"2024-09-13T15:00:55.405989+0000","last_active":"2024-09-13T15:00:58.278595+0000","last_peered":"2024-09-13T15:00:58.278595+0000","last_clean":"2024-09-13T15:00:58.278595+0000","last_became_active":"2024-09-13T15:00:55.405741+0000","last_became_peered":"2024-09-13T15:00:55.405741+0000","last_unstale":"2024-09-13T15:00:58.278595+0000","last_undegraded":"2024-09-13T15:00:58.278595+0000","last_fullsized":"2024-09-13T15:00:58.278595+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:43.197615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":329,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.406208+0000","last_change":"2024-09-13T14:57:53.351374+0000","last_active":"2024-09-13T15:00:55.406208+0000","last_peered":"2024-09-13T15:00:55.406208+0000","last_clean":"2024-09-13T15:00:55.406208+0000","last_became_active":"2024-09-13T14:57:53.350803+0000","last_became_peered":"2024-09-13T14:57:53.350803+0000","last_unstale":"2024-09-13T15:00:55.406208+0000","last_undegraded":"2024-09-13T15:00:55.406208+0000","last_fullsized":"2024-09-13T15:00:55.406208+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:30.535837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":359,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.398510+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T15:00:55.398510+0000","last_peered":"2024-09-13T15:00:55.398510+0000","last_clean":"2024-09-13T15:00:55.398510+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T15:00:55.398510+0000","last_undegraded":"2024-09-13T15:00:55.398510+0000","last_fullsized":"2024-09-13T15:00:55.398510+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":57,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.411300+0000","last_change":"2024-09-13T14:58:10.539542+0000","last_active":"2024-09-13T15:00:55.411300+0000","last_peered":"2024-09-13T15:00:55.411300+0000","last_clean":"2024-09-13T15:00:55.411300+0000","last_became_active":"2024-09-13T14:58:10.536848+0000","last_became_peered":"2024-09-13T14:58:10.536848+0000","last_unstale":"2024-09-13T15:00:55.411300+0000","last_undegraded":"2024-09-13T15:00:55.411300+0000","last_fullsized":"2024-09-13T15:00:55.411300+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:52.133867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":314,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400241+0000","last_change":"2024-09-13T14:57:53.347772+0000","last_active":"2024-09-13T15:00:55.400241+0000","last_peered":"2024-09-13T15:00:55.400241+0000","last_clean":"2024-09-13T15:00:55.400241+0000","last_became_active":"2024-09-13T14:57:53.347373+0000","last_became_peered":"2024-09-13T14:57:53.347373+0000","last_unstale":"2024-09-13T15:00:55.400241+0000","last_undegraded":"2024-09-13T15:00:55.400241+0000","last_fullsized":"2024-09-13T15:00:55.400241+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:13.081908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410083+0000","last_change":"2024-09-13T15:00:55.409140+0000","last_active":"2024-09-13T15:00:55.410083+0000","last_peered":"2024-09-13T15:00:55.410083+0000","last_clean":"2024-09-13T15:00:55.410083+0000","last_became_active":"2024-09-13T15:00:55.408820+0000","last_became_peered":"2024-09-13T15:00:55.408820+0000","last_unstale":"2024-09-13T15:00:55.410083+0000","last_undegraded":"2024-09-13T15:00:55.410083+0000","last_fullsized":"2024-09-13T15:00:55.410083+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:55.216821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":585,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400440+0000","last_change":"2024-09-13T14:58:10.517480+0000","last_active":"2024-09-13T15:00:55.400440+0000","last_peered":"2024-09-13T15:00:55.400440+0000","last_clean":"2024-09-13T15:00:55.400440+0000","last_became_active":"2024-09-13T14:58:10.517178+0000","last_became_peered":"2024-09-13T14:58:10.517178+0000","last_unstale":"2024-09-13T15:00:55.400440+0000","last_undegraded":"2024-09-13T15:00:55.400440+0000","last_fullsized":"2024-09-13T15:00:55.400440+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:27.472859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":200,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.398121+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T15:00:55.398121+0000","last_peered":"2024-09-13T15:00:55.398121+0000","last_clean":"2024-09-13T15:00:55.398121+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T15:00:55.398121+0000","last_undegraded":"2024-09-13T15:00:55.398121+0000","last_fullsized":"2024-09-13T15:00:55.398121+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":578,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401474+0000","last_change":"2024-09-13T14:58:10.532248+0000","last_active":"2024-09-13T15:00:55.401474+0000","last_peered":"2024-09-13T15:00:55.401474+0000","last_clean":"2024-09-13T15:00:55.401474+0000","last_became_active":"2024-09-13T14:58:10.522929+0000","last_became_peered":"2024-09-13T14:58:10.522929+0000","last_unstale":"2024-09-13T15:00:55.401474+0000","last_undegraded":"2024-09-13T15:00:55.401474+0000","last_fullsized":"2024-09-13T15:00:55.401474+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:38:40.572917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":167,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.411156+0000","last_change":"2024-09-13T14:57:56.132062+0000","last_active":"2024-09-13T15:00:55.411156+0000","last_peered":"2024-09-13T15:00:55.411156+0000","last_clean":"2024-09-13T15:00:55.411156+0000","last_became_active":"2024-09-13T14:57:19.218181+0000","last_became_peered":"2024-09-13T14:57:19.218181+0000","last_unstale":"2024-09-13T15:00:55.411156+0000","last_undegraded":"2024-09-13T15:00:55.411156+0000","last_fullsized":"2024-09-13T15:00:55.411156+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:56.131976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:56.131976+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:31.359825+0000","scrub_duration":0.0061470450000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.278780+0000","last_change":"2024-09-13T15:00:55.873094+0000","last_active":"2024-09-13T15:00:58.278780+0000","last_peered":"2024-09-13T15:00:58.278780+0000","last_clean":"2024-09-13T15:00:58.278780+0000","last_became_active":"2024-09-13T15:00:55.872743+0000","last_became_peered":"2024-09-13T15:00:55.872743+0000","last_unstale":"2024-09-13T15:00:58.278780+0000","last_undegraded":"2024-09-13T15:00:58.278780+0000","last_fullsized":"2024-09-13T15:00:58.278780+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:10.073117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":572,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401018+0000","last_change":"2024-09-13T14:58:10.521645+0000","last_active":"2024-09-13T15:00:55.401018+0000","last_peered":"2024-09-13T15:00:55.401018+0000","last_clean":"2024-09-13T15:00:55.401018+0000","last_became_active":"2024-09-13T14:58:10.521223+0000","last_became_peered":"2024-09-13T14:58:10.521223+0000","last_unstale":"2024-09-13T15:00:55.401018+0000","last_undegraded":"2024-09-13T15:00:55.401018+0000","last_fullsized":"2024-09-13T15:00:55.401018+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:18.930422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":349,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401998+0000","last_change":"2024-09-13T14:58:10.524720+0000","last_active":"2024-09-13T15:00:55.401998+0000","last_peered":"2024-09-13T15:00:55.401998+0000","last_clean":"2024-09-13T15:00:55.401998+0000","last_became_active":"2024-09-13T14:58:10.522828+0000","last_became_peered":"2024-09-13T14:58:10.522828+0000","last_unstale":"2024-09-13T15:00:55.401998+0000","last_undegraded":"2024-09-13T15:00:55.401998+0000","last_fullsized":"2024-09-13T15:00:55.401998+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:57.737026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":567,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.279161+0000","last_change":"2024-09-13T14:57:53.351358+0000","last_active":"2024-09-13T15:00:58.279161+0000","last_peered":"2024-09-13T15:00:58.279161+0000","last_clean":"2024-09-13T15:00:58.279161+0000","last_became_active":"2024-09-13T14:57:53.351021+0000","last_became_peered":"2024-09-13T14:57:53.351021+0000","last_unstale":"2024-09-13T15:00:58.279161+0000","last_undegraded":"2024-09-13T15:00:58.279161+0000","last_fullsized":"2024-09-13T15:00:58.279161+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.067202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":330,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.397815+0000","last_change":"2024-09-13T14:58:10.515266+0000","last_active":"2024-09-13T15:00:55.397815+0000","last_peered":"2024-09-13T15:00:55.397815+0000","last_clean":"2024-09-13T15:00:55.397815+0000","last_became_active":"2024-09-13T14:58:10.514858+0000","last_became_peered":"2024-09-13T14:58:10.514858+0000","last_unstale":"2024-09-13T15:00:55.397815+0000","last_undegraded":"2024-09-13T15:00:55.397815+0000","last_fullsized":"2024-09-13T15:00:55.397815+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:55.856922+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.405837+0000","last_change":"2024-09-13T15:00:55.405198+0000","last_active":"2024-09-13T15:00:55.405837+0000","last_peered":"2024-09-13T15:00:55.405837+0000","last_clean":"2024-09-13T15:00:55.405837+0000","last_became_active":"2024-09-13T15:00:55.404898+0000","last_became_peered":"2024-09-13T15:00:55.404898+0000","last_unstale":"2024-09-13T15:00:55.405837+0000","last_undegraded":"2024-09-13T15:00:55.405837+0000","last_fullsized":"2024-09-13T15:00:55.405837+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:01.901615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":45,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401253+0000","last_change":"2024-09-13T14:57:53.351087+0000","last_active":"2024-09-13T15:00:55.401253+0000","last_peered":"2024-09-13T15:00:55.401253+0000","last_clean":"2024-09-13T15:00:55.401253+0000","last_became_active":"2024-09-13T14:57:53.350744+0000","last_became_peered":"2024-09-13T14:57:53.350744+0000","last_unstale":"2024-09-13T15:00:55.401253+0000","last_undegraded":"2024-09-13T15:00:55.401253+0000","last_fullsized":"2024-09-13T15:00:55.401253+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:43.300957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":107,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401611+0000","last_change":"2024-09-13T14:58:10.521049+0000","last_active":"2024-09-13T15:00:55.401611+0000","last_peered":"2024-09-13T15:00:55.401611+0000","last_clean":"2024-09-13T15:00:55.401611+0000","last_became_active":"2024-09-13T14:58:10.520663+0000","last_became_peered":"2024-09-13T14:58:10.520663+0000","last_unstale":"2024-09-13T15:00:55.401611+0000","last_undegraded":"2024-09-13T15:00:55.401611+0000","last_fullsized":"2024-09-13T15:00:55.401611+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:59.410821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":45,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401512+0000","last_change":"2024-09-13T14:57:53.347982+0000","last_active":"2024-09-13T15:00:55.401512+0000","last_peered":"2024-09-13T15:00:55.401512+0000","last_clean":"2024-09-13T15:00:55.401512+0000","last_became_active":"2024-09-13T14:57:53.347623+0000","last_became_peered":"2024-09-13T14:57:53.347623+0000","last_unstale":"2024-09-13T15:00:55.401512+0000","last_undegraded":"2024-09-13T15:00:55.401512+0000","last_fullsized":"2024-09-13T15:00:55.401512+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:15.734246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":340,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.398310+0000","last_change":"2024-09-13T14:58:05.197923+0000","last_active":"2024-09-13T15:00:55.398310+0000","last_peered":"2024-09-13T15:00:55.398310+0000","last_clean":"2024-09-13T15:00:55.398310+0000","last_became_active":"2024-09-13T14:57:19.208023+0000","last_became_peered":"2024-09-13T14:57:19.208023+0000","last_unstale":"2024-09-13T15:00:55.398310+0000","last_undegraded":"2024-09-13T15:00:55.398310+0000","last_fullsized":"2024-09-13T15:00:55.398310+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:06.994746+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.874496+0000","last_change":"2024-09-13T15:00:55.874092+0000","last_active":"2024-09-13T15:00:55.874496+0000","last_peered":"2024-09-13T15:00:55.874496+0000","last_clean":"2024-09-13T15:00:55.874496+0000","last_became_active":"2024-09-13T15:00:55.873709+0000","last_became_peered":"2024-09-13T15:00:55.873709+0000","last_unstale":"2024-09-13T15:00:55.874496+0000","last_undegraded":"2024-09-13T15:00:55.874496+0000","last_fullsized":"2024-09-13T15:00:55.874496+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.813547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":465,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.279301+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T15:00:58.279301+0000","last_peered":"2024-09-13T15:00:58.279301+0000","last_clean":"2024-09-13T15:00:58.279301+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T15:00:58.279301+0000","last_undegraded":"2024-09-13T15:00:58.279301+0000","last_fullsized":"2024-09-13T15:00:58.279301+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":337,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410879+0000","last_change":"2024-09-13T14:58:10.516308+0000","last_active":"2024-09-13T15:00:55.410879+0000","last_peered":"2024-09-13T15:00:55.410879+0000","last_clean":"2024-09-13T15:00:55.410879+0000","last_became_active":"2024-09-13T14:58:10.515894+0000","last_became_peered":"2024-09-13T14:58:10.515894+0000","last_unstale":"2024-09-13T15:00:55.410879+0000","last_undegraded":"2024-09-13T15:00:55.410879+0000","last_fullsized":"2024-09-13T15:00:55.410879+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:17.962091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":98,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:57.913180+0000","last_change":"2024-09-13T14:58:28.428183+0000","last_active":"2024-09-13T15:00:57.913180+0000","last_peered":"2024-09-13T15:00:57.913180+0000","last_clean":"2024-09-13T15:00:57.913180+0000","last_became_active":"2024-09-13T14:58:28.427553+0000","last_became_peered":"2024-09-13T14:58:28.427553+0000","last_unstale":"2024-09-13T15:00:57.913180+0000","last_undegraded":"2024-09-13T15:00:57.913180+0000","last_fullsized":"2024-09-13T15:00:57.913180+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:59.108147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":65,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.278679+0000","last_change":"2024-09-13T14:58:28.424704+0000","last_active":"2024-09-13T15:00:58.278679+0000","last_peered":"2024-09-13T15:00:58.278679+0000","last_clean":"2024-09-13T15:00:58.278679+0000","last_became_active":"2024-09-13T14:58:28.424375+0000","last_became_peered":"2024-09-13T14:58:28.424375+0000","last_unstale":"2024-09-13T15:00:58.278679+0000","last_undegraded":"2024-09-13T15:00:58.278679+0000","last_fullsized":"2024-09-13T15:00:58.278679+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:28.071652+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.409251+0000","last_change":"2024-09-13T15:00:55.408720+0000","last_active":"2024-09-13T15:00:55.409251+0000","last_peered":"2024-09-13T15:00:55.409251+0000","last_clean":"2024-09-13T15:00:55.409251+0000","last_became_active":"2024-09-13T15:00:55.408379+0000","last_became_peered":"2024-09-13T15:00:55.408379+0000","last_unstale":"2024-09-13T15:00:55.409251+0000","last_undegraded":"2024-09-13T15:00:55.409251+0000","last_fullsized":"2024-09-13T15:00:55.409251+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:24.296460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":313,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:57.913522+0000","last_change":"2024-09-13T14:58:10.519956+0000","last_active":"2024-09-13T15:00:57.913522+0000","last_peered":"2024-09-13T15:00:57.913522+0000","last_clean":"2024-09-13T15:00:57.913522+0000","last_became_active":"2024-09-13T14:58:10.519409+0000","last_became_peered":"2024-09-13T14:58:10.519409+0000","last_unstale":"2024-09-13T15:00:57.913522+0000","last_undegraded":"2024-09-13T15:00:57.913522+0000","last_fullsized":"2024-09-13T15:00:57.913522+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:32.082107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":405,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.278778+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T15:00:58.278778+0000","last_peered":"2024-09-13T15:00:58.278778+0000","last_clean":"2024-09-13T15:00:58.278778+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T15:00:58.278778+0000","last_undegraded":"2024-09-13T15:00:58.278778+0000","last_fullsized":"2024-09-13T15:00:58.278778+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":603,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.278888+0000","last_change":"2024-09-13T14:58:10.511063+0000","last_active":"2024-09-13T15:00:58.278888+0000","last_peered":"2024-09-13T15:00:58.278888+0000","last_clean":"2024-09-13T15:00:58.278888+0000","last_became_active":"2024-09-13T14:58:10.510571+0000","last_became_peered":"2024-09-13T14:58:10.510571+0000","last_unstale":"2024-09-13T15:00:58.278888+0000","last_undegraded":"2024-09-13T15:00:58.278888+0000","last_fullsized":"2024-09-13T15:00:58.278888+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:20.803368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":131,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400823+0000","last_change":"2024-09-13T14:58:10.523618+0000","last_active":"2024-09-13T15:00:55.400823+0000","last_peered":"2024-09-13T15:00:55.400823+0000","last_clean":"2024-09-13T15:00:55.400823+0000","last_became_active":"2024-09-13T14:58:10.522166+0000","last_became_peered":"2024-09-13T14:58:10.522166+0000","last_unstale":"2024-09-13T15:00:55.400823+0000","last_undegraded":"2024-09-13T15:00:55.400823+0000","last_fullsized":"2024-09-13T15:00:55.400823+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:14.151008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.408997+0000","last_change":"2024-09-13T15:00:55.408205+0000","last_active":"2024-09-13T15:00:55.408997+0000","last_peered":"2024-09-13T15:00:55.408997+0000","last_clean":"2024-09-13T15:00:55.408997+0000","last_became_active":"2024-09-13T15:00:55.407801+0000","last_became_peered":"2024-09-13T15:00:55.407801+0000","last_unstale":"2024-09-13T15:00:55.408997+0000","last_undegraded":"2024-09-13T15:00:55.408997+0000","last_fullsized":"2024-09-13T15:00:55.408997+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:36.295309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":169,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.279225+0000","last_change":"2024-09-13T14:58:10.525829+0000","last_active":"2024-09-13T15:00:58.279225+0000","last_peered":"2024-09-13T15:00:58.279225+0000","last_clean":"2024-09-13T15:00:58.279225+0000","last_became_active":"2024-09-13T14:58:10.525316+0000","last_became_peered":"2024-09-13T14:58:10.525316+0000","last_unstale":"2024-09-13T15:00:58.279225+0000","last_undegraded":"2024-09-13T15:00:58.279225+0000","last_fullsized":"2024-09-13T15:00:58.279225+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:41.689562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":421,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.397247+0000","last_change":"2024-09-13T14:58:10.517828+0000","last_active":"2024-09-13T15:00:55.397247+0000","last_peered":"2024-09-13T15:00:55.397247+0000","last_clean":"2024-09-13T15:00:55.397247+0000","last_became_active":"2024-09-13T14:58:10.517009+0000","last_became_peered":"2024-09-13T14:58:10.517009+0000","last_unstale":"2024-09-13T15:00:55.397247+0000","last_undegraded":"2024-09-13T15:00:55.397247+0000","last_fullsized":"2024-09-13T15:00:55.397247+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:29.728706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":386,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401302+0000","last_change":"2024-09-13T14:58:10.517445+0000","last_active":"2024-09-13T15:00:55.401302+0000","last_peered":"2024-09-13T15:00:55.401302+0000","last_clean":"2024-09-13T15:00:55.401302+0000","last_became_active":"2024-09-13T14:58:10.517101+0000","last_became_peered":"2024-09-13T14:58:10.517101+0000","last_unstale":"2024-09-13T15:00:55.401302+0000","last_undegraded":"2024-09-13T15:00:55.401302+0000","last_fullsized":"2024-09-13T15:00:55.401302+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:35.589919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":300,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.397387+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T15:00:55.397387+0000","last_peered":"2024-09-13T15:00:55.397387+0000","last_clean":"2024-09-13T15:00:55.397387+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T15:00:55.397387+0000","last_undegraded":"2024-09-13T15:00:55.397387+0000","last_fullsized":"2024-09-13T15:00:55.397387+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.873981+0000","last_change":"2024-09-13T15:00:55.873540+0000","last_active":"2024-09-13T15:00:55.873981+0000","last_peered":"2024-09-13T15:00:55.873981+0000","last_clean":"2024-09-13T15:00:55.873981+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:00:55.873981+0000","last_undegraded":"2024-09-13T15:00:55.873981+0000","last_fullsized":"2024-09-13T15:00:55.873981+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:49.820345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":309,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.397910+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T15:00:55.397910+0000","last_peered":"2024-09-13T15:00:55.397910+0000","last_clean":"2024-09-13T15:00:55.397910+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T15:00:55.397910+0000","last_undegraded":"2024-09-13T15:00:55.397910+0000","last_fullsized":"2024-09-13T15:00:55.397910+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":540,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400331+0000","last_change":"2024-09-13T14:57:53.351036+0000","last_active":"2024-09-13T15:00:55.400331+0000","last_peered":"2024-09-13T15:00:55.400331+0000","last_clean":"2024-09-13T15:00:55.400331+0000","last_became_active":"2024-09-13T14:57:53.350553+0000","last_became_peered":"2024-09-13T14:57:53.350553+0000","last_unstale":"2024-09-13T15:00:55.400331+0000","last_undegraded":"2024-09-13T15:00:55.400331+0000","last_fullsized":"2024-09-13T15:00:55.400331+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:17.354351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":49,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401212+0000","last_change":"2024-09-13T14:57:53.360296+0000","last_active":"2024-09-13T15:00:55.401212+0000","last_peered":"2024-09-13T15:00:55.401212+0000","last_clean":"2024-09-13T15:00:55.401212+0000","last_became_active":"2024-09-13T14:57:53.347710+0000","last_became_peered":"2024-09-13T14:57:53.347710+0000","last_unstale":"2024-09-13T15:00:55.401212+0000","last_undegraded":"2024-09-13T15:00:55.401212+0000","last_fullsized":"2024-09-13T15:00:55.401212+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:43.099884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":131,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401440+0000","last_change":"2024-09-13T14:58:10.521307+0000","last_active":"2024-09-13T15:00:55.401440+0000","last_peered":"2024-09-13T15:00:55.401440+0000","last_clean":"2024-09-13T15:00:55.401440+0000","last_became_active":"2024-09-13T14:58:10.520905+0000","last_became_peered":"2024-09-13T14:58:10.520905+0000","last_unstale":"2024-09-13T15:00:55.401440+0000","last_undegraded":"2024-09-13T15:00:55.401440+0000","last_fullsized":"2024-09-13T15:00:55.401440+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:23.693312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.411494+0000","last_change":"2024-09-13T15:00:55.409383+0000","last_active":"2024-09-13T15:00:55.411494+0000","last_peered":"2024-09-13T15:00:55.411494+0000","last_clean":"2024-09-13T15:00:55.411494+0000","last_became_active":"2024-09-13T15:00:55.407034+0000","last_became_peered":"2024-09-13T15:00:55.407034+0000","last_unstale":"2024-09-13T15:00:55.411494+0000","last_undegraded":"2024-09-13T15:00:55.411494+0000","last_fullsized":"2024-09-13T15:00:55.411494+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:26.464719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":251,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.279212+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T15:00:58.279212+0000","last_peered":"2024-09-13T15:00:58.279212+0000","last_clean":"2024-09-13T15:00:58.279212+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T15:00:58.279212+0000","last_undegraded":"2024-09-13T15:00:58.279212+0000","last_fullsized":"2024-09-13T15:00:58.279212+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":537,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400400+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T15:00:55.400400+0000","last_peered":"2024-09-13T15:00:55.400400+0000","last_clean":"2024-09-13T15:00:55.400400+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T15:00:55.400400+0000","last_undegraded":"2024-09-13T15:00:55.400400+0000","last_fullsized":"2024-09-13T15:00:55.400400+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":91,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.278824+0000","last_change":"2024-09-13T14:58:28.428608+0000","last_active":"2024-09-13T15:00:58.278824+0000","last_peered":"2024-09-13T15:00:58.278824+0000","last_clean":"2024-09-13T15:00:58.278824+0000","last_became_active":"2024-09-13T14:58:28.428282+0000","last_became_peered":"2024-09-13T14:58:28.428282+0000","last_unstale":"2024-09-13T15:00:58.278824+0000","last_undegraded":"2024-09-13T15:00:58.278824+0000","last_fullsized":"2024-09-13T15:00:58.278824+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.603606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.603606+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:44.323633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.411595+0000","last_change":"2024-09-13T15:00:55.411034+0000","last_active":"2024-09-13T15:00:55.411595+0000","last_peered":"2024-09-13T15:00:55.411595+0000","last_clean":"2024-09-13T15:00:55.411595+0000","last_became_active":"2024-09-13T15:00:55.410636+0000","last_became_peered":"2024-09-13T15:00:55.410636+0000","last_unstale":"2024-09-13T15:00:55.411595+0000","last_undegraded":"2024-09-13T15:00:55.411595+0000","last_fullsized":"2024-09-13T15:00:55.411595+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:42.688532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":346,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.411462+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T15:00:55.411462+0000","last_peered":"2024-09-13T15:00:55.411462+0000","last_clean":"2024-09-13T15:00:55.411462+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T15:00:55.411462+0000","last_undegraded":"2024-09-13T15:00:55.411462+0000","last_fullsized":"2024-09-13T15:00:55.411462+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":327,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.406573+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T15:00:55.406573+0000","last_peered":"2024-09-13T15:00:55.406573+0000","last_clean":"2024-09-13T15:00:55.406573+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T15:00:55.406573+0000","last_undegraded":"2024-09-13T15:00:55.406573+0000","last_fullsized":"2024-09-13T15:00:55.406573+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":596,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401120+0000","last_change":"2024-09-13T14:58:10.512659+0000","last_active":"2024-09-13T15:00:55.401120+0000","last_peered":"2024-09-13T15:00:55.401120+0000","last_clean":"2024-09-13T15:00:55.401120+0000","last_became_active":"2024-09-13T14:58:10.511997+0000","last_became_peered":"2024-09-13T14:58:10.511997+0000","last_unstale":"2024-09-13T15:00:55.401120+0000","last_undegraded":"2024-09-13T15:00:55.401120+0000","last_fullsized":"2024-09-13T15:00:55.401120+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.415299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.415299+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:39.431834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.873552+0000","last_change":"2024-09-13T15:00:55.873173+0000","last_active":"2024-09-13T15:00:55.873552+0000","last_peered":"2024-09-13T15:00:55.873552+0000","last_clean":"2024-09-13T15:00:55.873552+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:00:55.873552+0000","last_undegraded":"2024-09-13T15:00:55.873552+0000","last_fullsized":"2024-09-13T15:00:55.873552+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:27.254765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":343,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.411447+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T15:00:55.411447+0000","last_peered":"2024-09-13T15:00:55.411447+0000","last_clean":"2024-09-13T15:00:55.411447+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T15:00:55.411447+0000","last_undegraded":"2024-09-13T15:00:55.411447+0000","last_fullsized":"2024-09-13T15:00:55.411447+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":347,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.397520+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T15:00:55.397520+0000","last_peered":"2024-09-13T15:00:55.397520+0000","last_clean":"2024-09-13T15:00:55.397520+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T15:00:55.397520+0000","last_undegraded":"2024-09-13T15:00:55.397520+0000","last_fullsized":"2024-09-13T15:00:55.397520+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":146,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-13T15:00:58.278733+0000","last_change":"2024-09-13T14:58:28.424238+0000","last_active":"2024-09-13T15:00:58.278733+0000","last_peered":"2024-09-13T15:00:58.278733+0000","last_clean":"2024-09-13T15:00:58.278733+0000","last_became_active":"2024-09-13T14:58:28.423844+0000","last_became_peered":"2024-09-13T14:58:28.423844+0000","last_unstale":"2024-09-13T15:00:58.278733+0000","last_undegraded":"2024-09-13T15:00:58.278733+0000","last_fullsized":"2024-09-13T15:00:58.278733+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:55.429885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410148+0000","last_change":"2024-09-13T15:00:55.407846+0000","last_active":"2024-09-13T15:00:55.410148+0000","last_peered":"2024-09-13T15:00:55.410148+0000","last_clean":"2024-09-13T15:00:55.410148+0000","last_became_active":"2024-09-13T15:00:55.407115+0000","last_became_peered":"2024-09-13T15:00:55.407115+0000","last_unstale":"2024-09-13T15:00:55.410148+0000","last_undegraded":"2024-09-13T15:00:55.410148+0000","last_fullsized":"2024-09-13T15:00:55.410148+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:29.545511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":326,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410561+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T15:00:55.410561+0000","last_peered":"2024-09-13T15:00:55.410561+0000","last_clean":"2024-09-13T15:00:55.410561+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T15:00:55.410561+0000","last_undegraded":"2024-09-13T15:00:55.410561+0000","last_fullsized":"2024-09-13T15:00:55.410561+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":577,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400442+0000","last_change":"2024-09-13T14:58:10.522983+0000","last_active":"2024-09-13T15:00:55.400442+0000","last_peered":"2024-09-13T15:00:55.400442+0000","last_clean":"2024-09-13T15:00:55.400442+0000","last_became_active":"2024-09-13T14:58:10.513377+0000","last_became_peered":"2024-09-13T14:58:10.513377+0000","last_unstale":"2024-09-13T15:00:55.400442+0000","last_undegraded":"2024-09-13T15:00:55.400442+0000","last_fullsized":"2024-09-13T15:00:55.400442+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:12.579644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":167,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401148+0000","last_change":"2024-09-13T14:57:57.223728+0000","last_active":"2024-09-13T15:00:55.401148+0000","last_peered":"2024-09-13T15:00:55.401148+0000","last_clean":"2024-09-13T15:00:55.401148+0000","last_became_active":"2024-09-13T14:57:19.233883+0000","last_became_peered":"2024-09-13T14:57:19.233883+0000","last_unstale":"2024-09-13T15:00:55.401148+0000","last_undegraded":"2024-09-13T15:00:55.401148+0000","last_fullsized":"2024-09-13T15:00:55.401148+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:41.520060+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410814+0000","last_change":"2024-09-13T15:00:55.410530+0000","last_active":"2024-09-13T15:00:55.410814+0000","last_peered":"2024-09-13T15:00:55.410814+0000","last_clean":"2024-09-13T15:00:55.410814+0000","last_became_active":"2024-09-13T15:00:55.410331+0000","last_became_peered":"2024-09-13T15:00:55.410331+0000","last_unstale":"2024-09-13T15:00:55.410814+0000","last_undegraded":"2024-09-13T15:00:55.410814+0000","last_fullsized":"2024-09-13T15:00:55.410814+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:57:20.295682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":95,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410266+0000","last_change":"2024-09-13T14:58:10.513713+0000","last_active":"2024-09-13T15:00:55.410266+0000","last_peered":"2024-09-13T15:00:55.410266+0000","last_clean":"2024-09-13T15:00:55.410266+0000","last_became_active":"2024-09-13T14:58:10.513334+0000","last_became_peered":"2024-09-13T14:58:10.513334+0000","last_unstale":"2024-09-13T15:00:55.410266+0000","last_undegraded":"2024-09-13T15:00:55.410266+0000","last_fullsized":"2024-09-13T15:00:55.410266+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:41.326218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":168,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400599+0000","last_change":"2024-09-13T14:58:28.428028+0000","last_active":"2024-09-13T15:00:55.400599+0000","last_peered":"2024-09-13T15:00:55.400599+0000","last_clean":"2024-09-13T15:00:55.400599+0000","last_became_active":"2024-09-13T14:58:28.427598+0000","last_became_peered":"2024-09-13T14:58:28.427598+0000","last_unstale":"2024-09-13T15:00:55.400599+0000","last_undegraded":"2024-09-13T15:00:55.400599+0000","last_fullsized":"2024-09-13T15:00:55.400599+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:32:50.930142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":167,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.411102+0000","last_change":"2024-09-13T14:58:00.146439+0000","last_active":"2024-09-13T15:00:55.411102+0000","last_peered":"2024-09-13T15:00:55.411102+0000","last_clean":"2024-09-13T15:00:55.411102+0000","last_became_active":"2024-09-13T14:57:19.206097+0000","last_became_peered":"2024-09-13T14:57:19.206097+0000","last_unstale":"2024-09-13T15:00:55.411102+0000","last_undegraded":"2024-09-13T15:00:55.411102+0000","last_fullsized":"2024-09-13T15:00:55.411102+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:00.146384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:00.146384+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:09.697538+0000","scrub_duration":0.0014649660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.410146+0000","last_change":"2024-09-13T15:00:55.407630+0000","last_active":"2024-09-13T15:00:55.410146+0000","last_peered":"2024-09-13T15:00:55.410146+0000","last_clean":"2024-09-13T15:00:55.410146+0000","last_became_active":"2024-09-13T15:00:55.407307+0000","last_became_peered":"2024-09-13T15:00:55.407307+0000","last_unstale":"2024-09-13T15:00:55.410146+0000","last_undegraded":"2024-09-13T15:00:55.410146+0000","last_fullsized":"2024-09-13T15:00:55.410146+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:54.284626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":349,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.397385+0000","last_change":"2024-09-13T14:57:35.357982+0000","last_active":"2024-09-13T15:00:55.397385+0000","last_peered":"2024-09-13T15:00:55.397385+0000","last_clean":"2024-09-13T15:00:55.397385+0000","last_became_active":"2024-09-13T14:57:35.357618+0000","last_became_peered":"2024-09-13T14:57:35.357618+0000","last_unstale":"2024-09-13T15:00:55.397385+0000","last_undegraded":"2024-09-13T15:00:55.397385+0000","last_fullsized":"2024-09-13T15:00:55.397385+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:50.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":368,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401807+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:00:55.401807+0000","last_peered":"2024-09-13T15:00:55.401807+0000","last_clean":"2024-09-13T15:00:55.401807+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:00:55.401807+0000","last_undegraded":"2024-09-13T15:00:55.401807+0000","last_fullsized":"2024-09-13T15:00:55.401807+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":320,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.398366+0000","last_change":"2024-09-13T14:57:53.363216+0000","last_active":"2024-09-13T15:00:55.398366+0000","last_peered":"2024-09-13T15:00:55.398366+0000","last_clean":"2024-09-13T15:00:55.398366+0000","last_became_active":"2024-09-13T14:57:53.362679+0000","last_became_peered":"2024-09-13T14:57:53.362679+0000","last_unstale":"2024-09-13T15:00:55.398366+0000","last_undegraded":"2024-09-13T15:00:55.398366+0000","last_fullsized":"2024-09-13T15:00:55.398366+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.885757+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.873629+0000","last_change":"2024-09-13T15:00:55.873177+0000","last_active":"2024-09-13T15:00:55.873629+0000","last_peered":"2024-09-13T15:00:55.873629+0000","last_clean":"2024-09-13T15:00:55.873629+0000","last_became_active":"2024-09-13T15:00:55.872769+0000","last_became_peered":"2024-09-13T15:00:55.872769+0000","last_unstale":"2024-09-13T15:00:55.873629+0000","last_undegraded":"2024-09-13T15:00:55.873629+0000","last_fullsized":"2024-09-13T15:00:55.873629+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:22.869519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":353,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.406372+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T15:00:55.406372+0000","last_peered":"2024-09-13T15:00:55.406372+0000","last_clean":"2024-09-13T15:00:55.406372+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T15:00:55.406372+0000","last_undegraded":"2024-09-13T15:00:55.406372+0000","last_fullsized":"2024-09-13T15:00:55.406372+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":330,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.404037+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T15:00:55.404037+0000","last_peered":"2024-09-13T15:00:55.404037+0000","last_clean":"2024-09-13T15:00:55.404037+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T15:00:55.404037+0000","last_undegraded":"2024-09-13T15:00:55.404037+0000","last_fullsized":"2024-09-13T15:00:55.404037+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":335,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400348+0000","last_change":"2024-09-13T14:58:10.531563+0000","last_active":"2024-09-13T15:00:55.400348+0000","last_peered":"2024-09-13T15:00:55.400348+0000","last_clean":"2024-09-13T15:00:55.400348+0000","last_became_active":"2024-09-13T14:58:10.515595+0000","last_became_peered":"2024-09-13T14:58:10.515595+0000","last_unstale":"2024-09-13T15:00:55.400348+0000","last_undegraded":"2024-09-13T15:00:55.400348+0000","last_fullsized":"2024-09-13T15:00:55.400348+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:19.723855+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.407713+0000","last_change":"2024-09-13T15:00:55.405838+0000","last_active":"2024-09-13T15:00:55.407713+0000","last_peered":"2024-09-13T15:00:55.407713+0000","last_clean":"2024-09-13T15:00:55.407713+0000","last_became_active":"2024-09-13T15:00:55.405095+0000","last_became_peered":"2024-09-13T15:00:55.405095+0000","last_unstale":"2024-09-13T15:00:55.407713+0000","last_undegraded":"2024-09-13T15:00:55.407713+0000","last_fullsized":"2024-09-13T15:00:55.407713+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:55.315749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":45,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.402270+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:00:55.402270+0000","last_peered":"2024-09-13T15:00:55.402270+0000","last_clean":"2024-09-13T15:00:55.402270+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:00:55.402270+0000","last_undegraded":"2024-09-13T15:00:55.402270+0000","last_fullsized":"2024-09-13T15:00:55.402270+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":139,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.406797+0000","last_change":"2024-09-13T14:58:10.530803+0000","last_active":"2024-09-13T15:00:55.406797+0000","last_peered":"2024-09-13T15:00:55.406797+0000","last_clean":"2024-09-13T15:00:55.406797+0000","last_became_active":"2024-09-13T14:58:10.522078+0000","last_became_peered":"2024-09-13T14:58:10.522078+0000","last_unstale":"2024-09-13T15:00:55.406797+0000","last_undegraded":"2024-09-13T15:00:55.406797+0000","last_fullsized":"2024-09-13T15:00:55.406797+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:50.807687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":151,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401543+0000","last_change":"2024-09-13T14:57:55.208230+0000","last_active":"2024-09-13T15:00:55.401543+0000","last_peered":"2024-09-13T15:00:55.401543+0000","last_clean":"2024-09-13T15:00:55.401543+0000","last_became_active":"2024-09-13T14:57:19.211671+0000","last_became_peered":"2024-09-13T14:57:19.211671+0000","last_unstale":"2024-09-13T15:00:55.401543+0000","last_undegraded":"2024-09-13T15:00:55.401543+0000","last_fullsized":"2024-09-13T15:00:55.401543+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.066938+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":435,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.397300+0000","last_change":"2024-09-13T14:58:10.529854+0000","last_active":"2024-09-13T15:00:55.397300+0000","last_peered":"2024-09-13T15:00:55.397300+0000","last_clean":"2024-09-13T15:00:55.397300+0000","last_became_active":"2024-09-13T14:58:10.514813+0000","last_became_peered":"2024-09-13T14:58:10.514813+0000","last_unstale":"2024-09-13T15:00:55.397300+0000","last_undegraded":"2024-09-13T15:00:55.397300+0000","last_fullsized":"2024-09-13T15:00:55.397300+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:46.296441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":574,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401268+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T15:00:55.401268+0000","last_peered":"2024-09-13T15:00:55.401268+0000","last_clean":"2024-09-13T15:00:55.401268+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T15:00:55.401268+0000","last_undegraded":"2024-09-13T15:00:55.401268+0000","last_fullsized":"2024-09-13T15:00:55.401268+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":45,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401162+0000","last_change":"2024-09-13T14:57:53.356010+0000","last_active":"2024-09-13T15:00:55.401162+0000","last_peered":"2024-09-13T15:00:55.401162+0000","last_clean":"2024-09-13T15:00:55.401162+0000","last_became_active":"2024-09-13T14:57:53.354979+0000","last_became_peered":"2024-09-13T14:57:53.354979+0000","last_unstale":"2024-09-13T15:00:55.401162+0000","last_undegraded":"2024-09-13T15:00:55.401162+0000","last_fullsized":"2024-09-13T15:00:55.401162+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:38.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":331,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401312+0000","last_change":"2024-09-13T14:57:35.358990+0000","last_active":"2024-09-13T15:00:55.401312+0000","last_peered":"2024-09-13T15:00:55.401312+0000","last_clean":"2024-09-13T15:00:55.401312+0000","last_became_active":"2024-09-13T14:57:35.358547+0000","last_became_peered":"2024-09-13T14:57:35.358547+0000","last_unstale":"2024-09-13T15:00:55.401312+0000","last_undegraded":"2024-09-13T15:00:55.401312+0000","last_fullsized":"2024-09-13T15:00:55.401312+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:45.386301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":454,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401114+0000","last_change":"2024-09-13T14:58:10.536776+0000","last_active":"2024-09-13T15:00:55.401114+0000","last_peered":"2024-09-13T15:00:55.401114+0000","last_clean":"2024-09-13T15:00:55.401114+0000","last_became_active":"2024-09-13T14:58:10.536376+0000","last_became_peered":"2024-09-13T14:58:10.536376+0000","last_unstale":"2024-09-13T15:00:55.401114+0000","last_undegraded":"2024-09-13T15:00:55.401114+0000","last_fullsized":"2024-09-13T15:00:55.401114+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:08.237256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":292,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400356+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T15:00:55.400356+0000","last_peered":"2024-09-13T15:00:55.400356+0000","last_clean":"2024-09-13T15:00:55.400356+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T15:00:55.400356+0000","last_undegraded":"2024-09-13T15:00:55.400356+0000","last_fullsized":"2024-09-13T15:00:55.400356+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:04.244429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":432,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400594+0000","last_change":"2024-09-13T14:58:10.524047+0000","last_active":"2024-09-13T15:00:55.400594+0000","last_peered":"2024-09-13T15:00:55.400594+0000","last_clean":"2024-09-13T15:00:55.400594+0000","last_became_active":"2024-09-13T14:58:10.522007+0000","last_became_peered":"2024-09-13T14:58:10.522007+0000","last_unstale":"2024-09-13T15:00:55.400594+0000","last_undegraded":"2024-09-13T15:00:55.400594+0000","last_fullsized":"2024-09-13T15:00:55.400594+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:41.957759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":222,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401635+0000","last_change":"2024-09-13T14:58:28.426617+0000","last_active":"2024-09-13T15:00:55.401635+0000","last_peered":"2024-09-13T15:00:55.401635+0000","last_clean":"2024-09-13T15:00:55.401635+0000","last_became_active":"2024-09-13T14:58:28.425773+0000","last_became_peered":"2024-09-13T14:58:28.425773+0000","last_unstale":"2024-09-13T15:00:55.401635+0000","last_undegraded":"2024-09-13T15:00:55.401635+0000","last_fullsized":"2024-09-13T15:00:55.401635+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:59.659871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":53,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401066+0000","last_change":"2024-09-13T14:58:28.428896+0000","last_active":"2024-09-13T15:00:55.401066+0000","last_peered":"2024-09-13T15:00:55.401066+0000","last_clean":"2024-09-13T15:00:55.401066+0000","last_became_active":"2024-09-13T14:58:28.428522+0000","last_became_peered":"2024-09-13T14:58:28.428522+0000","last_unstale":"2024-09-13T15:00:55.401066+0000","last_undegraded":"2024-09-13T15:00:55.401066+0000","last_fullsized":"2024-09-13T15:00:55.401066+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:05.418586+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":346,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.401352+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T15:00:55.401352+0000","last_peered":"2024-09-13T15:00:55.401352+0000","last_clean":"2024-09-13T15:00:55.401352+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T15:00:55.401352+0000","last_undegraded":"2024-09-13T15:00:55.401352+0000","last_fullsized":"2024-09-13T15:00:55.401352+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":69,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400207+0000","last_change":"2024-09-13T14:58:10.517240+0000","last_active":"2024-09-13T15:00:55.400207+0000","last_peered":"2024-09-13T15:00:55.400207+0000","last_clean":"2024-09-13T15:00:55.400207+0000","last_became_active":"2024-09-13T14:58:10.516811+0000","last_became_peered":"2024-09-13T14:58:10.516811+0000","last_unstale":"2024-09-13T15:00:55.400207+0000","last_undegraded":"2024-09-13T15:00:55.400207+0000","last_fullsized":"2024-09-13T15:00:55.400207+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:52.711800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":562,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.400758+0000","last_change":"2024-09-13T14:58:10.512480+0000","last_active":"2024-09-13T15:00:55.400758+0000","last_peered":"2024-09-13T15:00:55.400758+0000","last_clean":"2024-09-13T15:00:55.400758+0000","last_became_active":"2024-09-13T14:58:10.511589+0000","last_became_peered":"2024-09-13T14:58:10.511589+0000","last_unstale":"2024-09-13T15:00:55.400758+0000","last_undegraded":"2024-09-13T15:00:55.400758+0000","last_fullsized":"2024-09-13T15:00:55.400758+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:52.302190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":448,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365295,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506132,"kb_used_data":5128,"kb_used_omap":1,"kb_used_meta":1500990,"kb_avail":92239020,"statfs":{"total":95995035648,"available":94452756480,"internally_reserved":0,"allocated":5251072,"data_stored":3096792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1537014104},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561525,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501592,"kb_used_data":5196,"kb_used_omap":1,"kb_used_meta":1496382,"kb_avail":92243560,"statfs":{"total":95995035648,"available":94457405440,"internally_reserved":0,"allocated":5320704,"data_stored":3170913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1532295512},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790457,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491524,"kb_used_data":5108,"kb_used_omap":1,"kb_used_meta":1486398,"kb_avail":92253628,"statfs":{"total":95995035648,"available":94467715072,"internally_reserved":0,"allocated":5230592,"data_stored":3092189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1522071896},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986685,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208988,"kb_used_data":13708,"kb_used_omap":1,"kb_used_meta":1195262,"kb_avail":92536164,"statfs":{"total":95995035648,"available":94757031936,"internally_reserved":0,"allocated":14036992,"data_stored":11886969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1223948632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519840,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890620,"kb_used_data":5168,"kb_used_omap":4,"kb_used_meta":885435,"kb_avail":92854532,"statfs":{"total":95995035648,"available":95083040768,"internally_reserved":0,"allocated":5292032,"data_stored":3157191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":906685888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304756,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912272,"kb_used_data":5124,"kb_used_omap":6,"kb_used_meta":907129,"kb_avail":92832880,"statfs":{"total":95995035648,"available":95060869120,"internally_reserved":0,"allocated":5246976,"data_stored":3101449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":928900551},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828059,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124292,"kb_used_data":4856,"kb_used_omap":5,"kb_used_meta":1119418,"kb_avail":92620860,"statfs":{"total":95995035648,"available":94843760640,"internally_reserved":0,"allocated":4972544,"data_stored":2829504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1146284268},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":356482285924,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:01:05.342 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T15:01:05.342 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:01:05.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:05.392 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:05.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:05 smithi019 ceph-mon[91296]: from='client.36894 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:05.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:05 smithi019 ceph-mon[91296]: pgmap v1298: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:05.495 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:05.522 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:05.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:05.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:05.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:05.778 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:05.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:05.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:01:05.881 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:05.908 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:06.010 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.038 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:06.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:06.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:06.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:06.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:06.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:06.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:06.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:06.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:06.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:06.544 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:06.646 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.675 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:06.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:06.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:06.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:06.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:06.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:07.025 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:07.052 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:07.144 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_8' 2024-09-13T15:01:07.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:07.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:07.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:07.306 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:07.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:07 smithi019 ceph-mon[91296]: pgmap v1299: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:07 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1813922067' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:01:07.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:07 smithi003 ceph-mon[104007]: pgmap v1299: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:07.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:07 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1813922067' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:01:07.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:07 smithi003 ceph-mon[107278]: pgmap v1299: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:07.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:07 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1813922067' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:01:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:07.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:07.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:07.564 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:07.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:07.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:07.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:07.818 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:07.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:07.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:08.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:08.073 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:08.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:08.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:08.300 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.329 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:08.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:08.430 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:08 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1813922067' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:01:08.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:08 smithi019 ceph-mon[91296]: osdmap e366: 8 total, 8 up, 7 in 2024-09-13T15:01:08.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:08 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1813922067' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:01:08.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:08 smithi003 ceph-mon[107278]: osdmap e366: 8 total, 8 up, 7 in 2024-09-13T15:01:08.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:08 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1813922067' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:01:08.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:08 smithi003 ceph-mon[104007]: osdmap e366: 8 total, 8 up, 7 in 2024-09-13T15:01:08.457 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:08.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:08.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:08.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:08.712 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:08.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:08.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:08.901 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_8 bench 90 write' 2024-09-13T15:01:08.902 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T15:01:08.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:01:08.904 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T15:01:08.905 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T15:01:08.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:08.955 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T15:01:08.955 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T15:01:08.955 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_293363 2024-09-13T15:01:08.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:01:08.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T15:01:08.974 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:09.078 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:09.118 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:09.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:09.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:09.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:09 smithi003 ceph-mon[104007]: pgmap v1301: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:09.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:09 smithi003 ceph-mon[104007]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:01:09.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:09.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:09 smithi003 ceph-mon[107278]: pgmap v1301: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:09.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:09 smithi003 ceph-mon[107278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:01:09.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:09.327 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:01:09.328 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:01:09.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:09.386 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:01:09.386 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:01:09.386 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:01:09.386 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:01:09.386 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:01:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:09.387 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:09.387 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:09.387 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:09.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:09.414 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:01:09.414 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:09 smithi019 ceph-mon[91296]: pgmap v1301: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:09 smithi019 ceph-mon[91296]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:01:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:09.447 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:01:09.448 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:09.481 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:01:09.481 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:09.504 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:09.541 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:09.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:09.680 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:09.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:09.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:09.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:09.955 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3683 3667 229.174 229.188 0.00296366 0.00435103 2024-09-13T15:01:09.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:10.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:10.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:10.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:10.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:10.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:10.367 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/551232398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:10.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/551232398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:10.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:10.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:10.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:01:10.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:01:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:01:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:10.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/551232398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:10.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:10.512 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:10.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.657 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:10.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:10.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:10.795 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:10.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:10.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:10.955 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 7005 6989 218.389 207.625 0.0032433 0.00456853 2024-09-13T15:01:11.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:11.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:11.178 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.215 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:11.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:11.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:11.357 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:11 smithi019 ceph-mon[91296]: pgmap v1302: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:11.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:11 smithi003 ceph-mon[107278]: pgmap v1302: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:11 smithi003 ceph-mon[104007]: pgmap v1302: 119 pgs: 119 active+clean; 582 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail 2024-09-13T15:01:11.459 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:11.497 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:11.599 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:11.638 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:11.658 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:11.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.780 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:11.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:11.881 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:11.916 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:11.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 9852 9836 204.885 177.938 0.00404603 0.00487476 2024-09-13T15:01:12.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:12.059 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:12.161 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.198 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:12.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:12.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:12.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:12.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:12.475 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:12.532 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:12.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:12.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:12.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:12.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:12.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:12.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:12.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 15 12716 12701 198.425 179.062 0.00480734 0.00503159 2024-09-13T15:01:12.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:13.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:13.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:13.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:13.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:13.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:13 smithi003 ceph-mon[107278]: pgmap v1303: 119 pgs: 119 active+clean; 58 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 5.7 MiB/s wr, 91 op/s 2024-09-13T15:01:13.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:13.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:13.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:13 smithi003 ceph-mon[104007]: pgmap v1303: 119 pgs: 119 active+clean; 58 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 5.7 MiB/s wr, 91 op/s 2024-09-13T15:01:13.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:13.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:13.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:13.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:13.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:13.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:13.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:13 smithi019 ceph-mon[91296]: pgmap v1303: 119 pgs: 119 active+clean; 58 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 5.7 MiB/s wr, 91 op/s 2024-09-13T15:01:13.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:13.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:13.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:13.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:13.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:13.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:01:13.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:13.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:13.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:13.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:13.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:13.956 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:13.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 15646 15630 195.349 183.062 0.00366261 0.00511181 2024-09-13T15:01:13.995 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:14.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:14.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:14.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:14.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:14.370 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:14.381 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:01:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:01:14] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T15:01:14.415 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:14.517 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:14.551 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:14.653 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:14.693 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:14.795 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:14.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:14.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:14.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:14.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 18394 18378 191.414 171.75 0.00521095 0.00521718 2024-09-13T15:01:14.975 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:15.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:15.115 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:15.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:15.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:15.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:15.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:15.391 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:15.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:15 smithi019 ceph-mon[91296]: pgmap v1304: 119 pgs: 119 active+clean; 801 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T15:01:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:15 smithi003 ceph-mon[104007]: pgmap v1304: 119 pgs: 119 active+clean; 801 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T15:01:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:15 smithi003 ceph-mon[107278]: pgmap v1304: 119 pgs: 119 active+clean; 801 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T15:01:15.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:15.531 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:15.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:15.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:15.775 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:15.814 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:15.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:15.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:15.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:15.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 20951 20935 186.898 159.812 0.00418826 0.00534478 2024-09-13T15:01:16.052 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:16.090 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:16.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:16.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:16.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:16.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:16.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:16.506 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:16.571 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:01:16.572 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:01:16.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:16.647 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:16.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:16.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:16.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:16.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:16.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:16.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 23372 23356 182.448 151.312 0.00434628 0.00547579 2024-09-13T15:01:17.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:17.052 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:17.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:17.187 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:01:17.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:17.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:17.329 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:17.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:17.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:17.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:17.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:17.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:17 smithi019 ceph-mon[91296]: pgmap v1305: 119 pgs: 119 active+clean; 801 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T15:01:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:17 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3170035321' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:17 smithi003 ceph-mon[107278]: pgmap v1305: 119 pgs: 119 active+clean; 801 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T15:01:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:17 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3170035321' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:17 smithi003 ceph-mon[104007]: pgmap v1305: 119 pgs: 119 active+clean; 801 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T15:01:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:17 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3170035321' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:17.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:17.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:17.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:17.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:17.953 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:17.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 25274 25258 175.383 118.875 0.00680467 0.00569587 2024-09-13T15:01:17.993 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:18.095 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:18.136 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:18.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:18.280 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:18.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:01:18.382 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:18.423 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:18.525 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:18.566 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:18.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:18 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:18 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3170035321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:18 smithi019 ceph-mon[91296]: osdmap e367: 8 total, 8 up, 7 in 2024-09-13T15:01:18.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:18 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:01:18.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:18 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:01:18.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3170035321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:18.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:18 smithi003 ceph-mon[104007]: osdmap e367: 8 total, 8 up, 7 in 2024-09-13T15:01:18.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:18 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:01:18.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:18 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:01:18.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3170035321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:18.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:18 smithi003 ceph-mon[107278]: osdmap e367: 8 total, 8 up, 7 in 2024-09-13T15:01:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.705 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:18.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:18.806 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:18.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:18.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 27352 27336 170.831 129.875 0.0111285 0.00584818 2024-09-13T15:01:18.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:19.039 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:19.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:19.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:19.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:19.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:19.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:19 smithi003 ceph-mon[104007]: pgmap v1307: 119 pgs: 119 active+clean; 1014 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T15:01:19.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:19 smithi003 ceph-mon[107278]: pgmap v1307: 119 pgs: 119 active+clean; 1014 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T15:01:19.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:19.390 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:19.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:19.530 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:19.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:19 smithi019 ceph-mon[91296]: pgmap v1307: 119 pgs: 119 active+clean; 1014 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T15:01:19.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:19.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:19.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:19.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:19.956 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 29458 29442 167.266 131.625 0.00866972 0.00597312 2024-09-13T15:01:20.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:20.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:20.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:20.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:20.247 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:01:20.249 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:01:20.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:20.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:20.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:20.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:20.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:20.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:01:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:01:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:20.729 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:20.767 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:20.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:20.869 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:20.910 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:20.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 31453 31437 163.717 124.688 0.0146836 0.00610279 2024-09-13T15:01:21.011 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:21.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:01:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.053 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:21.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:21.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:21.195 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:21.207 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:21 smithi003 ceph-mon[107278]: pgmap v1308: 119 pgs: 119 active+clean; 1.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T15:01:21.208 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:21 smithi003 ceph-mon[104007]: pgmap v1308: 119 pgs: 119 active+clean; 1.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T15:01:21.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:21.334 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:21.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:21.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:21.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:21 smithi003 ceph-mon[107278]: from='client.46988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:21.483 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:21 smithi003 ceph-mon[104007]: from='client.46988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:21.568 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:21.609 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:21 smithi019 ceph-mon[91296]: pgmap v1308: 119 pgs: 119 active+clean; 1.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T15:01:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:21 smithi019 ceph-mon[91296]: from='client.46988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:21.711 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:21.750 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:21.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:21.890 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:21.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 33503 33487 160.978 128.125 0.00619515 0.00620714 2024-09-13T15:01:21.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:22.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:22.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:22.274 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:22.376 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.416 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:22.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:22.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:22.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:22.651 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:22.695 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:22.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:22.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:22.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:22.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:22.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 35486 35470 158.332 123.938 0.00881182 0.00630951 2024-09-13T15:01:22.979 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:23.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.123 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:23.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:23.224 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.264 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:23.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:23.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:23.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:23 smithi003 ceph-mon[107278]: pgmap v1309: 119 pgs: 119 active+clean; 1.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-09-13T15:01:23.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:23 smithi003 ceph-mon[104007]: pgmap v1309: 119 pgs: 119 active+clean; 1.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-09-13T15:01:23.502 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:23.540 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:23.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:23.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:23.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:23.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:23.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:23 smithi019 ceph-mon[91296]: pgmap v1309: 119 pgs: 119 active+clean; 1.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-09-13T15:01:23.777 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:23.815 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:23.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:23.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:23.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 37359 37343 155.58 117.062 0.0063611 0.0064219 2024-09-13T15:01:24.052 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:24.093 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:24.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:24.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:24.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:24.359 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:01:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:01:24] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T15:01:24.460 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:24.501 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:24.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:24.603 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:24.647 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:24.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:24.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:24.789 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:24.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:24.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:24.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 39148 39132 152.844 111.812 0.00699611 0.00653728 2024-09-13T15:01:24.973 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1308,"stamp":"2024-09-13T15:01:20.075760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1663506381,"num_objects":25400,"num_object_clones":0,"num_object_copies":76200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25438,"num_write_kb":1623968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":328147,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28533,"ondisk_log_size":28533,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13192928,"kb_used_data":4555988,"kb_used_omap":23,"kb_used_meta":8636840,"kb_avail":643023136,"statfs":{"total":671965249536,"available":658455691264,"internally_reserved":0,"allocated":4665331712,"data_stored":4650230434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24109,"internal_metadata":8844124627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1662910464,"num_objects":25374,"num_object_clones":0,"num_object_copies":76122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25374,"num_write_kb":1623936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002667"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":120,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199631+0000","last_change":"2024-09-13T14:57:42.607827+0000","last_active":"2024-09-13T15:01:17.199631+0000","last_peered":"2024-09-13T15:01:17.199631+0000","last_clean":"2024-09-13T15:01:17.199631+0000","last_became_active":"2024-09-13T14:57:25.285562+0000","last_became_peered":"2024-09-13T14:57:25.285562+0000","last_unstale":"2024-09-13T15:01:17.199631+0000","last_undegraded":"2024-09-13T15:01:17.199631+0000","last_fullsized":"2024-09-13T15:01:17.199631+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.607753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.607753+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:21:26.227325+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":79,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519733+0000","last_change":"2024-09-13T14:58:28.426659+0000","last_active":"2024-09-13T15:01:07.519733+0000","last_peered":"2024-09-13T15:01:07.519733+0000","last_clean":"2024-09-13T15:01:07.519733+0000","last_became_active":"2024-09-13T14:58:28.426336+0000","last_became_peered":"2024-09-13T14:58:28.426336+0000","last_unstale":"2024-09-13T15:01:07.519733+0000","last_undegraded":"2024-09-13T15:01:07.519733+0000","last_fullsized":"2024-09-13T15:01:07.519733+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:55.713969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":339,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.508765+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T15:01:08.508765+0000","last_peered":"2024-09-13T15:01:08.508765+0000","last_clean":"2024-09-13T15:01:08.508765+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T15:01:08.508765+0000","last_undegraded":"2024-09-13T15:01:08.508765+0000","last_fullsized":"2024-09-13T15:01:08.508765+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":65,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146095+0000","last_change":"2024-09-13T14:57:58.200463+0000","last_active":"2024-09-13T15:01:07.146095+0000","last_peered":"2024-09-13T15:01:07.146095+0000","last_clean":"2024-09-13T15:01:07.146095+0000","last_became_active":"2024-09-13T14:57:25.279086+0000","last_became_peered":"2024-09-13T14:57:25.279086+0000","last_unstale":"2024-09-13T15:01:07.146095+0000","last_undegraded":"2024-09-13T15:01:07.146095+0000","last_fullsized":"2024-09-13T15:01:07.146095+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:58.986716+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":97,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199422+0000","last_change":"2024-09-13T14:57:35.356151+0000","last_active":"2024-09-13T15:01:17.199422+0000","last_peered":"2024-09-13T15:01:17.199422+0000","last_clean":"2024-09-13T15:01:17.199422+0000","last_became_active":"2024-09-13T14:57:35.355770+0000","last_became_peered":"2024-09-13T14:57:35.355770+0000","last_unstale":"2024-09-13T15:01:17.199422+0000","last_undegraded":"2024-09-13T15:01:17.199422+0000","last_fullsized":"2024-09-13T15:01:17.199422+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:37.887286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":582,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.201217+0000","last_change":"2024-09-13T14:58:10.513979+0000","last_active":"2024-09-13T15:01:17.201217+0000","last_peered":"2024-09-13T15:01:17.201217+0000","last_clean":"2024-09-13T15:01:17.201217+0000","last_became_active":"2024-09-13T14:58:10.513313+0000","last_became_peered":"2024-09-13T14:58:10.513313+0000","last_unstale":"2024-09-13T15:01:17.201217+0000","last_undegraded":"2024-09-13T15:01:17.201217+0000","last_fullsized":"2024-09-13T15:01:17.201217+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:25:15.376408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":70,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.508722+0000","last_change":"2024-09-13T14:58:28.426763+0000","last_active":"2024-09-13T15:01:08.508722+0000","last_peered":"2024-09-13T15:01:08.508722+0000","last_clean":"2024-09-13T15:01:08.508722+0000","last_became_active":"2024-09-13T14:58:28.426270+0000","last_became_peered":"2024-09-13T14:58:28.426270+0000","last_unstale":"2024-09-13T15:01:08.508722+0000","last_undegraded":"2024-09-13T15:01:08.508722+0000","last_fullsized":"2024-09-13T15:01:08.508722+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.991576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":356,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509270+0000","last_change":"2024-09-13T14:58:10.515535+0000","last_active":"2024-09-13T15:01:08.509270+0000","last_peered":"2024-09-13T15:01:08.509270+0000","last_clean":"2024-09-13T15:01:08.509270+0000","last_became_active":"2024-09-13T14:58:10.515043+0000","last_became_peered":"2024-09-13T14:58:10.515043+0000","last_unstale":"2024-09-13T15:01:08.509270+0000","last_undegraded":"2024-09-13T15:01:08.509270+0000","last_fullsized":"2024-09-13T15:01:08.509270+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:53.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":122,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.203998+0000","last_change":"2024-09-13T14:57:43.658933+0000","last_active":"2024-09-13T15:01:17.203998+0000","last_peered":"2024-09-13T15:01:17.203998+0000","last_clean":"2024-09-13T15:01:17.203998+0000","last_became_active":"2024-09-13T14:57:25.291285+0000","last_became_peered":"2024-09-13T14:57:25.291285+0000","last_unstale":"2024-09-13T15:01:17.203998+0000","last_undegraded":"2024-09-13T15:01:17.203998+0000","last_fullsized":"2024-09-13T15:01:17.203998+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:43.658833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:43.658833+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:22.120697+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":70,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.508672+0000","last_change":"2024-09-13T14:58:28.429061+0000","last_active":"2024-09-13T15:01:08.508672+0000","last_peered":"2024-09-13T15:01:08.508672+0000","last_clean":"2024-09-13T15:01:08.508672+0000","last_became_active":"2024-09-13T14:58:28.428642+0000","last_became_peered":"2024-09-13T14:58:28.428642+0000","last_unstale":"2024-09-13T15:01:08.508672+0000","last_undegraded":"2024-09-13T15:01:08.508672+0000","last_fullsized":"2024-09-13T15:01:08.508672+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:03.197824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":376,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509573+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T15:01:08.509573+0000","last_peered":"2024-09-13T15:01:08.509573+0000","last_clean":"2024-09-13T15:01:08.509573+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T15:01:08.509573+0000","last_undegraded":"2024-09-13T15:01:08.509573+0000","last_fullsized":"2024-09-13T15:01:08.509573+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":129,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199691+0000","last_change":"2024-09-13T14:58:10.521468+0000","last_active":"2024-09-13T15:01:17.199691+0000","last_peered":"2024-09-13T15:01:17.199691+0000","last_clean":"2024-09-13T15:01:17.199691+0000","last_became_active":"2024-09-13T14:58:10.520775+0000","last_became_peered":"2024-09-13T14:58:10.520775+0000","last_unstale":"2024-09-13T15:01:17.199691+0000","last_undegraded":"2024-09-13T15:01:17.199691+0000","last_fullsized":"2024-09-13T15:01:17.199691+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:27.363579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":73,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509414+0000","last_change":"2024-09-13T14:58:10.523188+0000","last_active":"2024-09-13T15:01:08.509414+0000","last_peered":"2024-09-13T15:01:08.509414+0000","last_clean":"2024-09-13T15:01:08.509414+0000","last_became_active":"2024-09-13T14:58:10.522815+0000","last_became_peered":"2024-09-13T14:58:10.522815+0000","last_unstale":"2024-09-13T15:01:08.509414+0000","last_undegraded":"2024-09-13T15:01:08.509414+0000","last_fullsized":"2024-09-13T15:01:08.509414+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:27.568999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":598,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519714+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T15:01:07.519714+0000","last_peered":"2024-09-13T15:01:07.519714+0000","last_clean":"2024-09-13T15:01:07.519714+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T15:01:07.519714+0000","last_undegraded":"2024-09-13T15:01:07.519714+0000","last_fullsized":"2024-09-13T15:01:07.519714+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":62,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.961809+0000","last_change":"2024-09-13T14:58:09.011981+0000","last_active":"2024-09-13T15:01:08.961809+0000","last_peered":"2024-09-13T15:01:08.961809+0000","last_clean":"2024-09-13T15:01:08.961809+0000","last_became_active":"2024-09-13T14:57:53.357674+0000","last_became_peered":"2024-09-13T14:57:53.357674+0000","last_unstale":"2024-09-13T15:01:08.961809+0000","last_undegraded":"2024-09-13T15:01:08.961809+0000","last_fullsized":"2024-09-13T15:01:08.961809+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.701912+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":348,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509351+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T15:01:08.509351+0000","last_peered":"2024-09-13T15:01:08.509351+0000","last_clean":"2024-09-13T15:01:08.509351+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T15:01:08.509351+0000","last_undegraded":"2024-09-13T15:01:08.509351+0000","last_fullsized":"2024-09-13T15:01:08.509351+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":542,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199746+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T15:01:17.199746+0000","last_peered":"2024-09-13T15:01:17.199746+0000","last_clean":"2024-09-13T15:01:17.199746+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T15:01:17.199746+0000","last_undegraded":"2024-09-13T15:01:17.199746+0000","last_fullsized":"2024-09-13T15:01:17.199746+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":68,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.963040+0000","last_change":"2024-09-13T14:58:10.513244+0000","last_active":"2024-09-13T15:01:08.963040+0000","last_peered":"2024-09-13T15:01:08.963040+0000","last_clean":"2024-09-13T15:01:08.963040+0000","last_became_active":"2024-09-13T14:58:10.511485+0000","last_became_peered":"2024-09-13T14:58:10.511485+0000","last_unstale":"2024-09-13T15:01:08.963040+0000","last_undegraded":"2024-09-13T15:01:08.963040+0000","last_fullsized":"2024-09-13T15:01:08.963040+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:23.283605+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":255,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519087+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T15:01:07.519087+0000","last_peered":"2024-09-13T15:01:07.519087+0000","last_clean":"2024-09-13T15:01:07.519087+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T15:01:07.519087+0000","last_undegraded":"2024-09-13T15:01:07.519087+0000","last_fullsized":"2024-09-13T15:01:07.519087+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":437,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146553+0000","last_change":"2024-09-13T14:58:10.536485+0000","last_active":"2024-09-13T15:01:07.146553+0000","last_peered":"2024-09-13T15:01:07.146553+0000","last_clean":"2024-09-13T15:01:07.146553+0000","last_became_active":"2024-09-13T14:58:10.536134+0000","last_became_peered":"2024-09-13T14:58:10.536134+0000","last_unstale":"2024-09-13T15:01:07.146553+0000","last_undegraded":"2024-09-13T15:01:07.146553+0000","last_fullsized":"2024-09-13T15:01:07.146553+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:59.095681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":65,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509159+0000","last_change":"2024-09-13T14:58:05.212101+0000","last_active":"2024-09-13T15:01:08.509159+0000","last_peered":"2024-09-13T15:01:08.509159+0000","last_clean":"2024-09-13T15:01:08.509159+0000","last_became_active":"2024-09-13T14:57:25.275734+0000","last_became_peered":"2024-09-13T14:57:25.275734+0000","last_unstale":"2024-09-13T15:01:08.509159+0000","last_undegraded":"2024-09-13T15:01:08.509159+0000","last_fullsized":"2024-09-13T15:01:08.509159+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.212008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.212008+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:01.720211+0000","scrub_duration":0.0048969690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":582,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.200953+0000","last_change":"2024-09-13T14:58:10.515043+0000","last_active":"2024-09-13T15:01:17.200953+0000","last_peered":"2024-09-13T15:01:17.200953+0000","last_clean":"2024-09-13T15:01:17.200953+0000","last_became_active":"2024-09-13T14:58:10.513487+0000","last_became_peered":"2024-09-13T14:58:10.513487+0000","last_unstale":"2024-09-13T15:01:17.200953+0000","last_undegraded":"2024-09-13T15:01:17.200953+0000","last_fullsized":"2024-09-13T15:01:17.200953+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:11.392005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":349,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.419083+0000","last_change":"2024-09-13T14:57:53.364039+0000","last_active":"2024-09-13T15:01:07.419083+0000","last_peered":"2024-09-13T15:01:07.419083+0000","last_clean":"2024-09-13T15:01:07.419083+0000","last_became_active":"2024-09-13T14:57:53.363550+0000","last_became_peered":"2024-09-13T14:57:53.363550+0000","last_unstale":"2024-09-13T15:01:07.419083+0000","last_undegraded":"2024-09-13T15:01:07.419083+0000","last_fullsized":"2024-09-13T15:01:07.419083+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:36.171089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":91,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146700+0000","last_change":"2024-09-13T14:58:04.168950+0000","last_active":"2024-09-13T15:01:07.146700+0000","last_peered":"2024-09-13T15:01:07.146700+0000","last_clean":"2024-09-13T15:01:07.146700+0000","last_became_active":"2024-09-13T14:57:53.351301+0000","last_became_peered":"2024-09-13T14:57:53.351301+0000","last_unstale":"2024-09-13T15:01:07.146700+0000","last_undegraded":"2024-09-13T15:01:07.146700+0000","last_fullsized":"2024-09-13T15:01:07.146700+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:42.270600+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":543,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199482+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T15:01:17.199482+0000","last_peered":"2024-09-13T15:01:17.199482+0000","last_clean":"2024-09-13T15:01:17.199482+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T15:01:17.199482+0000","last_undegraded":"2024-09-13T15:01:17.199482+0000","last_fullsized":"2024-09-13T15:01:17.199482+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":590,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.963365+0000","last_change":"2024-09-13T14:58:10.514433+0000","last_active":"2024-09-13T15:01:08.963365+0000","last_peered":"2024-09-13T15:01:08.963365+0000","last_clean":"2024-09-13T15:01:08.963365+0000","last_became_active":"2024-09-13T14:58:10.513446+0000","last_became_peered":"2024-09-13T14:58:10.513446+0000","last_unstale":"2024-09-13T15:01:08.963365+0000","last_undegraded":"2024-09-13T15:01:08.963365+0000","last_fullsized":"2024-09-13T15:01:08.963365+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:24.535893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":548,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.200438+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T15:01:17.200438+0000","last_peered":"2024-09-13T15:01:17.200438+0000","last_clean":"2024-09-13T15:01:17.200438+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T15:01:17.200438+0000","last_undegraded":"2024-09-13T15:01:17.200438+0000","last_fullsized":"2024-09-13T15:01:17.200438+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":48,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.961873+0000","last_change":"2024-09-13T14:57:53.350622+0000","last_active":"2024-09-13T15:01:08.961873+0000","last_peered":"2024-09-13T15:01:08.961873+0000","last_clean":"2024-09-13T15:01:08.961873+0000","last_became_active":"2024-09-13T14:57:53.350292+0000","last_became_peered":"2024-09-13T14:57:53.350292+0000","last_unstale":"2024-09-13T15:01:08.961873+0000","last_undegraded":"2024-09-13T15:01:08.961873+0000","last_fullsized":"2024-09-13T15:01:08.961873+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:16.088338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":299,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.518971+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T15:01:07.518971+0000","last_peered":"2024-09-13T15:01:07.518971+0000","last_clean":"2024-09-13T15:01:07.518971+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T15:01:07.518971+0000","last_undegraded":"2024-09-13T15:01:07.518971+0000","last_fullsized":"2024-09-13T15:01:07.518971+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.284651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":58,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.963357+0000","last_change":"2024-09-13T14:58:10.527352+0000","last_active":"2024-09-13T15:01:08.963357+0000","last_peered":"2024-09-13T15:01:08.963357+0000","last_clean":"2024-09-13T15:01:08.963357+0000","last_became_active":"2024-09-13T14:58:10.526614+0000","last_became_peered":"2024-09-13T14:58:10.526614+0000","last_unstale":"2024-09-13T15:01:08.963357+0000","last_undegraded":"2024-09-13T15:01:08.963357+0000","last_fullsized":"2024-09-13T15:01:08.963357+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:04.459255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":130,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146073+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T15:01:07.146073+0000","last_peered":"2024-09-13T15:01:07.146073+0000","last_clean":"2024-09-13T15:01:07.146073+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T15:01:07.146073+0000","last_undegraded":"2024-09-13T15:01:07.146073+0000","last_fullsized":"2024-09-13T15:01:07.146073+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":268,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.418407+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T15:01:07.418407+0000","last_peered":"2024-09-13T15:01:07.418407+0000","last_clean":"2024-09-13T15:01:07.418407+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T15:01:07.418407+0000","last_undegraded":"2024-09-13T15:01:07.418407+0000","last_fullsized":"2024-09-13T15:01:07.418407+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":127,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.188090+0000","last_change":"2024-09-13T14:58:10.511301+0000","last_active":"2024-09-13T15:01:17.188090+0000","last_peered":"2024-09-13T15:01:17.188090+0000","last_clean":"2024-09-13T15:01:17.188090+0000","last_became_active":"2024-09-13T14:58:10.510934+0000","last_became_peered":"2024-09-13T14:58:10.510934+0000","last_unstale":"2024-09-13T15:01:17.188090+0000","last_undegraded":"2024-09-13T15:01:17.188090+0000","last_fullsized":"2024-09-13T15:01:17.188090+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:12.152076+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":336,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509572+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T15:01:08.509572+0000","last_peered":"2024-09-13T15:01:08.509572+0000","last_clean":"2024-09-13T15:01:08.509572+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T15:01:08.509572+0000","last_undegraded":"2024-09-13T15:01:08.509572+0000","last_fullsized":"2024-09-13T15:01:08.509572+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":329,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509562+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T15:01:08.509562+0000","last_peered":"2024-09-13T15:01:08.509562+0000","last_clean":"2024-09-13T15:01:08.509562+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T15:01:08.509562+0000","last_undegraded":"2024-09-13T15:01:08.509562+0000","last_fullsized":"2024-09-13T15:01:08.509562+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"366'1325","reported_seq":1344,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:16.091288+0000","last_change":"2024-09-13T15:00:55.405989+0000","last_active":"2024-09-13T15:01:16.091288+0000","last_peered":"2024-09-13T15:01:16.091288+0000","last_clean":"2024-09-13T15:01:16.091288+0000","last_became_active":"2024-09-13T15:00:55.405741+0000","last_became_peered":"2024-09-13T15:00:55.405741+0000","last_unstale":"2024-09-13T15:01:16.091288+0000","last_undegraded":"2024-09-13T15:01:16.091288+0000","last_fullsized":"2024-09-13T15:01:16.091288+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:32.041720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86835200,"num_objects":1325,"num_object_clones":0,"num_object_copies":3975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":84800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":332,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.508992+0000","last_change":"2024-09-13T14:57:53.351374+0000","last_active":"2024-09-13T15:01:08.508992+0000","last_peered":"2024-09-13T15:01:08.508992+0000","last_clean":"2024-09-13T15:01:08.508992+0000","last_became_active":"2024-09-13T14:57:53.350803+0000","last_became_peered":"2024-09-13T14:57:53.350803+0000","last_unstale":"2024-09-13T15:01:08.508992+0000","last_undegraded":"2024-09-13T15:01:08.508992+0000","last_fullsized":"2024-09-13T15:01:08.508992+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:30.535837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":362,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.419138+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T15:01:07.419138+0000","last_peered":"2024-09-13T15:01:07.419138+0000","last_clean":"2024-09-13T15:01:07.419138+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T15:01:07.419138+0000","last_undegraded":"2024-09-13T15:01:07.419138+0000","last_fullsized":"2024-09-13T15:01:07.419138+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":60,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509956+0000","last_change":"2024-09-13T14:58:10.539542+0000","last_active":"2024-09-13T15:01:08.509956+0000","last_peered":"2024-09-13T15:01:08.509956+0000","last_clean":"2024-09-13T15:01:08.509956+0000","last_became_active":"2024-09-13T14:58:10.536848+0000","last_became_peered":"2024-09-13T14:58:10.536848+0000","last_unstale":"2024-09-13T15:01:08.509956+0000","last_undegraded":"2024-09-13T15:01:08.509956+0000","last_fullsized":"2024-09-13T15:01:08.509956+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:52.133867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":322,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199828+0000","last_change":"2024-09-13T14:57:53.347772+0000","last_active":"2024-09-13T15:01:17.199828+0000","last_peered":"2024-09-13T15:01:17.199828+0000","last_clean":"2024-09-13T15:01:17.199828+0000","last_became_active":"2024-09-13T14:57:53.347373+0000","last_became_peered":"2024-09-13T14:57:53.347373+0000","last_unstale":"2024-09-13T15:01:17.199828+0000","last_undegraded":"2024-09-13T15:01:17.199828+0000","last_fullsized":"2024-09-13T15:01:17.199828+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:13.081908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"367'1666","reported_seq":1679,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.301487+0000","last_change":"2024-09-13T15:00:55.409140+0000","last_active":"2024-09-13T15:01:18.301487+0000","last_peered":"2024-09-13T15:01:18.301487+0000","last_clean":"2024-09-13T15:01:18.301487+0000","last_became_active":"2024-09-13T15:00:55.408820+0000","last_became_peered":"2024-09-13T15:00:55.408820+0000","last_unstale":"2024-09-13T15:01:18.301487+0000","last_undegraded":"2024-09-13T15:01:18.301487+0000","last_fullsized":"2024-09-13T15:01:18.301487+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1666,"log_dups_size":0,"ondisk_log_size":1666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:55.216821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109182976,"num_objects":1666,"num_object_clones":0,"num_object_copies":4998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1666,"num_write_kb":106624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":594,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199593+0000","last_change":"2024-09-13T14:58:10.517480+0000","last_active":"2024-09-13T15:01:17.199593+0000","last_peered":"2024-09-13T15:01:17.199593+0000","last_clean":"2024-09-13T15:01:17.199593+0000","last_became_active":"2024-09-13T14:58:10.517178+0000","last_became_peered":"2024-09-13T14:58:10.517178+0000","last_unstale":"2024-09-13T15:01:17.199593+0000","last_undegraded":"2024-09-13T15:01:17.199593+0000","last_fullsized":"2024-09-13T15:01:17.199593+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:27.472859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":203,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.418756+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T15:01:07.418756+0000","last_peered":"2024-09-13T15:01:07.418756+0000","last_clean":"2024-09-13T15:01:07.418756+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T15:01:07.418756+0000","last_undegraded":"2024-09-13T15:01:07.418756+0000","last_fullsized":"2024-09-13T15:01:07.418756+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":586,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.201773+0000","last_change":"2024-09-13T14:58:10.532248+0000","last_active":"2024-09-13T15:01:17.201773+0000","last_peered":"2024-09-13T15:01:17.201773+0000","last_clean":"2024-09-13T15:01:17.201773+0000","last_became_active":"2024-09-13T14:58:10.522929+0000","last_became_peered":"2024-09-13T14:58:10.522929+0000","last_unstale":"2024-09-13T15:01:17.201773+0000","last_undegraded":"2024-09-13T15:01:17.201773+0000","last_fullsized":"2024-09-13T15:01:17.201773+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:38:40.572917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":170,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509866+0000","last_change":"2024-09-13T14:57:56.132062+0000","last_active":"2024-09-13T15:01:08.509866+0000","last_peered":"2024-09-13T15:01:08.509866+0000","last_clean":"2024-09-13T15:01:08.509866+0000","last_became_active":"2024-09-13T14:57:19.218181+0000","last_became_peered":"2024-09-13T14:57:19.218181+0000","last_unstale":"2024-09-13T15:01:08.509866+0000","last_undegraded":"2024-09-13T15:01:08.509866+0000","last_fullsized":"2024-09-13T15:01:08.509866+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:56.131976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:56.131976+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:31.359825+0000","scrub_duration":0.0061470450000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"366'1306","reported_seq":1325,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:16.088538+0000","last_change":"2024-09-13T15:00:55.873094+0000","last_active":"2024-09-13T15:01:16.088538+0000","last_peered":"2024-09-13T15:01:16.088538+0000","last_clean":"2024-09-13T15:01:16.088538+0000","last_became_active":"2024-09-13T15:00:55.872743+0000","last_became_peered":"2024-09-13T15:00:55.872743+0000","last_unstale":"2024-09-13T15:01:16.088538+0000","last_undegraded":"2024-09-13T15:01:16.088538+0000","last_fullsized":"2024-09-13T15:01:16.088538+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1306,"log_dups_size":0,"ondisk_log_size":1306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:46.267062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85590016,"num_objects":1306,"num_object_clones":0,"num_object_copies":3918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1306,"num_write_kb":83584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":575,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.962780+0000","last_change":"2024-09-13T14:58:10.521645+0000","last_active":"2024-09-13T15:01:08.962780+0000","last_peered":"2024-09-13T15:01:08.962780+0000","last_clean":"2024-09-13T15:01:08.962780+0000","last_became_active":"2024-09-13T14:58:10.521223+0000","last_became_peered":"2024-09-13T14:58:10.521223+0000","last_unstale":"2024-09-13T15:01:08.962780+0000","last_undegraded":"2024-09-13T15:01:08.962780+0000","last_fullsized":"2024-09-13T15:01:08.962780+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:18.930422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":352,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146790+0000","last_change":"2024-09-13T14:58:10.524720+0000","last_active":"2024-09-13T15:01:07.146790+0000","last_peered":"2024-09-13T15:01:07.146790+0000","last_clean":"2024-09-13T15:01:07.146790+0000","last_became_active":"2024-09-13T14:58:10.522828+0000","last_became_peered":"2024-09-13T14:58:10.522828+0000","last_unstale":"2024-09-13T15:01:07.146790+0000","last_undegraded":"2024-09-13T15:01:07.146790+0000","last_fullsized":"2024-09-13T15:01:07.146790+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:57.737026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":571,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519620+0000","last_change":"2024-09-13T14:57:53.351358+0000","last_active":"2024-09-13T15:01:07.519620+0000","last_peered":"2024-09-13T15:01:07.519620+0000","last_clean":"2024-09-13T15:01:07.519620+0000","last_became_active":"2024-09-13T14:57:53.351021+0000","last_became_peered":"2024-09-13T14:57:53.351021+0000","last_unstale":"2024-09-13T15:01:07.519620+0000","last_undegraded":"2024-09-13T15:01:07.519620+0000","last_fullsized":"2024-09-13T15:01:07.519620+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.067202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":333,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.418658+0000","last_change":"2024-09-13T14:58:10.515266+0000","last_active":"2024-09-13T15:01:07.418658+0000","last_peered":"2024-09-13T15:01:07.418658+0000","last_clean":"2024-09-13T15:01:07.418658+0000","last_became_active":"2024-09-13T14:58:10.514858+0000","last_became_peered":"2024-09-13T14:58:10.514858+0000","last_unstale":"2024-09-13T15:01:07.418658+0000","last_undegraded":"2024-09-13T15:01:07.418658+0000","last_fullsized":"2024-09-13T15:01:07.418658+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:55.856922+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"367'1636","reported_seq":1657,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.717585+0000","last_change":"2024-09-13T15:00:55.405198+0000","last_active":"2024-09-13T15:01:17.717585+0000","last_peered":"2024-09-13T15:01:17.717585+0000","last_clean":"2024-09-13T15:01:17.717585+0000","last_became_active":"2024-09-13T15:00:55.404898+0000","last_became_peered":"2024-09-13T15:00:55.404898+0000","last_unstale":"2024-09-13T15:01:17.717585+0000","last_undegraded":"2024-09-13T15:01:17.717585+0000","last_fullsized":"2024-09-13T15:01:17.717585+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1636,"log_dups_size":0,"ondisk_log_size":1636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:42.847939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107216896,"num_objects":1636,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1636,"num_write_kb":104704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":48,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.964645+0000","last_change":"2024-09-13T14:57:53.351087+0000","last_active":"2024-09-13T15:01:08.964645+0000","last_peered":"2024-09-13T15:01:08.964645+0000","last_clean":"2024-09-13T15:01:08.964645+0000","last_became_active":"2024-09-13T14:57:53.350744+0000","last_became_peered":"2024-09-13T14:57:53.350744+0000","last_unstale":"2024-09-13T15:01:08.964645+0000","last_undegraded":"2024-09-13T15:01:08.964645+0000","last_fullsized":"2024-09-13T15:01:08.964645+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:43.300957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":110,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146248+0000","last_change":"2024-09-13T14:58:10.521049+0000","last_active":"2024-09-13T15:01:07.146248+0000","last_peered":"2024-09-13T15:01:07.146248+0000","last_clean":"2024-09-13T15:01:07.146248+0000","last_became_active":"2024-09-13T14:58:10.520663+0000","last_became_peered":"2024-09-13T14:58:10.520663+0000","last_unstale":"2024-09-13T15:01:07.146248+0000","last_undegraded":"2024-09-13T15:01:07.146248+0000","last_fullsized":"2024-09-13T15:01:07.146248+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:59.410821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":48,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.962373+0000","last_change":"2024-09-13T14:57:53.347982+0000","last_active":"2024-09-13T15:01:08.962373+0000","last_peered":"2024-09-13T15:01:08.962373+0000","last_clean":"2024-09-13T15:01:08.962373+0000","last_became_active":"2024-09-13T14:57:53.347623+0000","last_became_peered":"2024-09-13T14:57:53.347623+0000","last_unstale":"2024-09-13T15:01:08.962373+0000","last_undegraded":"2024-09-13T15:01:08.962373+0000","last_fullsized":"2024-09-13T15:01:08.962373+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:15.734246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":343,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.418978+0000","last_change":"2024-09-13T14:58:05.197923+0000","last_active":"2024-09-13T15:01:07.418978+0000","last_peered":"2024-09-13T15:01:07.418978+0000","last_clean":"2024-09-13T15:01:07.418978+0000","last_became_active":"2024-09-13T14:57:19.208023+0000","last_became_peered":"2024-09-13T14:57:19.208023+0000","last_unstale":"2024-09-13T15:01:07.418978+0000","last_undegraded":"2024-09-13T15:01:07.418978+0000","last_fullsized":"2024-09-13T15:01:07.418978+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:06.994746+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"367'1667","reported_seq":1680,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.327730+0000","last_change":"2024-09-13T15:00:55.874092+0000","last_active":"2024-09-13T15:01:18.327730+0000","last_peered":"2024-09-13T15:01:18.327730+0000","last_clean":"2024-09-13T15:01:18.327730+0000","last_became_active":"2024-09-13T15:00:55.873709+0000","last_became_peered":"2024-09-13T15:00:55.873709+0000","last_unstale":"2024-09-13T15:01:18.327730+0000","last_undegraded":"2024-09-13T15:01:18.327730+0000","last_fullsized":"2024-09-13T15:01:18.327730+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1667,"log_dups_size":0,"ondisk_log_size":1667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.813547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109248512,"num_objects":1667,"num_object_clones":0,"num_object_copies":5001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1667,"num_write_kb":106688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":469,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519802+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T15:01:07.519802+0000","last_peered":"2024-09-13T15:01:07.519802+0000","last_clean":"2024-09-13T15:01:07.519802+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T15:01:07.519802+0000","last_undegraded":"2024-09-13T15:01:07.519802+0000","last_fullsized":"2024-09-13T15:01:07.519802+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":340,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509703+0000","last_change":"2024-09-13T14:58:10.516308+0000","last_active":"2024-09-13T15:01:08.509703+0000","last_peered":"2024-09-13T15:01:08.509703+0000","last_clean":"2024-09-13T15:01:08.509703+0000","last_became_active":"2024-09-13T14:58:10.515894+0000","last_became_peered":"2024-09-13T14:58:10.515894+0000","last_unstale":"2024-09-13T15:01:08.509703+0000","last_undegraded":"2024-09-13T15:01:08.509703+0000","last_fullsized":"2024-09-13T15:01:08.509703+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:17.962091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":104,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.188782+0000","last_change":"2024-09-13T14:58:28.428183+0000","last_active":"2024-09-13T15:01:17.188782+0000","last_peered":"2024-09-13T15:01:17.188782+0000","last_clean":"2024-09-13T15:01:17.188782+0000","last_became_active":"2024-09-13T14:58:28.427553+0000","last_became_peered":"2024-09-13T14:58:28.427553+0000","last_unstale":"2024-09-13T15:01:17.188782+0000","last_undegraded":"2024-09-13T15:01:17.188782+0000","last_fullsized":"2024-09-13T15:01:17.188782+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:59.108147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":69,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519115+0000","last_change":"2024-09-13T14:58:28.424704+0000","last_active":"2024-09-13T15:01:07.519115+0000","last_peered":"2024-09-13T15:01:07.519115+0000","last_clean":"2024-09-13T15:01:07.519115+0000","last_became_active":"2024-09-13T14:58:28.424375+0000","last_became_peered":"2024-09-13T14:58:28.424375+0000","last_unstale":"2024-09-13T15:01:07.519115+0000","last_undegraded":"2024-09-13T15:01:07.519115+0000","last_fullsized":"2024-09-13T15:01:07.519115+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:28.071652+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.e","version":"367'1628","reported_seq":1641,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.371223+0000","last_change":"2024-09-13T15:00:55.408720+0000","last_active":"2024-09-13T15:01:18.371223+0000","last_peered":"2024-09-13T15:01:18.371223+0000","last_clean":"2024-09-13T15:01:18.371223+0000","last_became_active":"2024-09-13T15:00:55.408379+0000","last_became_peered":"2024-09-13T15:00:55.408379+0000","last_unstale":"2024-09-13T15:01:18.371223+0000","last_undegraded":"2024-09-13T15:01:18.371223+0000","last_fullsized":"2024-09-13T15:01:18.371223+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1628,"log_dups_size":0,"ondisk_log_size":1628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:24.296460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1628,"num_write_kb":104192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":319,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.188685+0000","last_change":"2024-09-13T14:58:10.519956+0000","last_active":"2024-09-13T15:01:17.188685+0000","last_peered":"2024-09-13T15:01:17.188685+0000","last_clean":"2024-09-13T15:01:17.188685+0000","last_became_active":"2024-09-13T14:58:10.519409+0000","last_became_peered":"2024-09-13T14:58:10.519409+0000","last_unstale":"2024-09-13T15:01:17.188685+0000","last_undegraded":"2024-09-13T15:01:17.188685+0000","last_fullsized":"2024-09-13T15:01:17.188685+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:32.082107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":409,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519222+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T15:01:07.519222+0000","last_peered":"2024-09-13T15:01:07.519222+0000","last_clean":"2024-09-13T15:01:07.519222+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T15:01:07.519222+0000","last_undegraded":"2024-09-13T15:01:07.519222+0000","last_fullsized":"2024-09-13T15:01:07.519222+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":607,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519324+0000","last_change":"2024-09-13T14:58:10.511063+0000","last_active":"2024-09-13T15:01:07.519324+0000","last_peered":"2024-09-13T15:01:07.519324+0000","last_clean":"2024-09-13T15:01:07.519324+0000","last_became_active":"2024-09-13T14:58:10.510571+0000","last_became_peered":"2024-09-13T14:58:10.510571+0000","last_unstale":"2024-09-13T15:01:07.519324+0000","last_undegraded":"2024-09-13T15:01:07.519324+0000","last_fullsized":"2024-09-13T15:01:07.519324+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:20.803368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":134,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146070+0000","last_change":"2024-09-13T14:58:10.523618+0000","last_active":"2024-09-13T15:01:07.146070+0000","last_peered":"2024-09-13T15:01:07.146070+0000","last_clean":"2024-09-13T15:01:07.146070+0000","last_became_active":"2024-09-13T14:58:10.522166+0000","last_became_peered":"2024-09-13T14:58:10.522166+0000","last_unstale":"2024-09-13T15:01:07.146070+0000","last_undegraded":"2024-09-13T15:01:07.146070+0000","last_fullsized":"2024-09-13T15:01:07.146070+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:14.151008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.f","version":"367'1636","reported_seq":1649,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.345912+0000","last_change":"2024-09-13T15:00:55.408205+0000","last_active":"2024-09-13T15:01:18.345912+0000","last_peered":"2024-09-13T15:01:18.345912+0000","last_clean":"2024-09-13T15:01:18.345912+0000","last_became_active":"2024-09-13T15:00:55.407801+0000","last_became_peered":"2024-09-13T15:00:55.407801+0000","last_unstale":"2024-09-13T15:01:18.345912+0000","last_undegraded":"2024-09-13T15:01:18.345912+0000","last_fullsized":"2024-09-13T15:01:18.345912+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1636,"log_dups_size":0,"ondisk_log_size":1636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:36.295309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107216896,"num_objects":1636,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1636,"num_write_kb":104704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":173,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519428+0000","last_change":"2024-09-13T14:58:10.525829+0000","last_active":"2024-09-13T15:01:07.519428+0000","last_peered":"2024-09-13T15:01:07.519428+0000","last_clean":"2024-09-13T15:01:07.519428+0000","last_became_active":"2024-09-13T14:58:10.525316+0000","last_became_peered":"2024-09-13T14:58:10.525316+0000","last_unstale":"2024-09-13T15:01:07.519428+0000","last_undegraded":"2024-09-13T15:01:07.519428+0000","last_fullsized":"2024-09-13T15:01:07.519428+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:41.689562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":424,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.418070+0000","last_change":"2024-09-13T14:58:10.517828+0000","last_active":"2024-09-13T15:01:07.418070+0000","last_peered":"2024-09-13T15:01:07.418070+0000","last_clean":"2024-09-13T15:01:07.418070+0000","last_became_active":"2024-09-13T14:58:10.517009+0000","last_became_peered":"2024-09-13T14:58:10.517009+0000","last_unstale":"2024-09-13T15:01:07.418070+0000","last_undegraded":"2024-09-13T15:01:07.418070+0000","last_fullsized":"2024-09-13T15:01:07.418070+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:29.728706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":389,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146058+0000","last_change":"2024-09-13T14:58:10.517445+0000","last_active":"2024-09-13T15:01:07.146058+0000","last_peered":"2024-09-13T15:01:07.146058+0000","last_clean":"2024-09-13T15:01:07.146058+0000","last_became_active":"2024-09-13T14:58:10.517101+0000","last_became_peered":"2024-09-13T14:58:10.517101+0000","last_unstale":"2024-09-13T15:01:07.146058+0000","last_undegraded":"2024-09-13T15:01:07.146058+0000","last_fullsized":"2024-09-13T15:01:07.146058+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:35.589919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":303,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.417704+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T15:01:07.417704+0000","last_peered":"2024-09-13T15:01:07.417704+0000","last_clean":"2024-09-13T15:01:07.417704+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T15:01:07.417704+0000","last_undegraded":"2024-09-13T15:01:07.417704+0000","last_fullsized":"2024-09-13T15:01:07.417704+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.c","version":"367'1593","reported_seq":1606,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.299402+0000","last_change":"2024-09-13T15:00:55.873540+0000","last_active":"2024-09-13T15:01:18.299402+0000","last_peered":"2024-09-13T15:01:18.299402+0000","last_clean":"2024-09-13T15:01:18.299402+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:01:18.299402+0000","last_undegraded":"2024-09-13T15:01:18.299402+0000","last_fullsized":"2024-09-13T15:01:18.299402+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1593,"log_dups_size":0,"ondisk_log_size":1593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:49.820345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104398848,"num_objects":1593,"num_object_clones":0,"num_object_copies":4779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1593,"num_write_kb":101952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":312,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.418602+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T15:01:07.418602+0000","last_peered":"2024-09-13T15:01:07.418602+0000","last_clean":"2024-09-13T15:01:07.418602+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T15:01:07.418602+0000","last_undegraded":"2024-09-13T15:01:07.418602+0000","last_fullsized":"2024-09-13T15:01:07.418602+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":548,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199800+0000","last_change":"2024-09-13T14:57:53.351036+0000","last_active":"2024-09-13T15:01:17.199800+0000","last_peered":"2024-09-13T15:01:17.199800+0000","last_clean":"2024-09-13T15:01:17.199800+0000","last_became_active":"2024-09-13T14:57:53.350553+0000","last_became_peered":"2024-09-13T14:57:53.350553+0000","last_unstale":"2024-09-13T15:01:17.199800+0000","last_undegraded":"2024-09-13T15:01:17.199800+0000","last_fullsized":"2024-09-13T15:01:17.199800+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:17.354351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":52,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.962947+0000","last_change":"2024-09-13T14:57:53.360296+0000","last_active":"2024-09-13T15:01:08.962947+0000","last_peered":"2024-09-13T15:01:08.962947+0000","last_clean":"2024-09-13T15:01:08.962947+0000","last_became_active":"2024-09-13T14:57:53.347710+0000","last_became_peered":"2024-09-13T14:57:53.347710+0000","last_unstale":"2024-09-13T15:01:08.962947+0000","last_undegraded":"2024-09-13T15:01:08.962947+0000","last_fullsized":"2024-09-13T15:01:08.962947+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:43.099884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":134,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.145623+0000","last_change":"2024-09-13T14:58:10.521307+0000","last_active":"2024-09-13T15:01:07.145623+0000","last_peered":"2024-09-13T15:01:07.145623+0000","last_clean":"2024-09-13T15:01:07.145623+0000","last_became_active":"2024-09-13T14:58:10.520905+0000","last_became_peered":"2024-09-13T14:58:10.520905+0000","last_unstale":"2024-09-13T15:01:07.145623+0000","last_undegraded":"2024-09-13T15:01:07.145623+0000","last_fullsized":"2024-09-13T15:01:07.145623+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:23.693312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"367'1568","reported_seq":1581,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.315921+0000","last_change":"2024-09-13T15:00:55.409383+0000","last_active":"2024-09-13T15:01:18.315921+0000","last_peered":"2024-09-13T15:01:18.315921+0000","last_clean":"2024-09-13T15:01:18.315921+0000","last_became_active":"2024-09-13T15:00:55.407034+0000","last_became_peered":"2024-09-13T15:00:55.407034+0000","last_unstale":"2024-09-13T15:01:18.315921+0000","last_undegraded":"2024-09-13T15:01:18.315921+0000","last_fullsized":"2024-09-13T15:01:18.315921+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:26.464719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102760448,"num_objects":1568,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1568,"num_write_kb":100352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":255,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519514+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T15:01:07.519514+0000","last_peered":"2024-09-13T15:01:07.519514+0000","last_clean":"2024-09-13T15:01:07.519514+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T15:01:07.519514+0000","last_undegraded":"2024-09-13T15:01:07.519514+0000","last_fullsized":"2024-09-13T15:01:07.519514+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":545,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.201010+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T15:01:17.201010+0000","last_peered":"2024-09-13T15:01:17.201010+0000","last_clean":"2024-09-13T15:01:17.201010+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T15:01:17.201010+0000","last_undegraded":"2024-09-13T15:01:17.201010+0000","last_fullsized":"2024-09-13T15:01:17.201010+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":95,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519389+0000","last_change":"2024-09-13T14:58:28.428608+0000","last_active":"2024-09-13T15:01:07.519389+0000","last_peered":"2024-09-13T15:01:07.519389+0000","last_clean":"2024-09-13T15:01:07.519389+0000","last_became_active":"2024-09-13T14:58:28.428282+0000","last_became_peered":"2024-09-13T14:58:28.428282+0000","last_unstale":"2024-09-13T15:01:07.519389+0000","last_undegraded":"2024-09-13T15:01:07.519389+0000","last_fullsized":"2024-09-13T15:01:07.519389+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.603606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.603606+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:44.323633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.7","version":"367'1624","reported_seq":1637,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.372073+0000","last_change":"2024-09-13T15:00:55.411034+0000","last_active":"2024-09-13T15:01:18.372073+0000","last_peered":"2024-09-13T15:01:18.372073+0000","last_clean":"2024-09-13T15:01:18.372073+0000","last_became_active":"2024-09-13T15:00:55.410636+0000","last_became_peered":"2024-09-13T15:00:55.410636+0000","last_unstale":"2024-09-13T15:01:18.372073+0000","last_undegraded":"2024-09-13T15:01:18.372073+0000","last_fullsized":"2024-09-13T15:01:18.372073+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:42.688532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106430464,"num_objects":1624,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1624,"num_write_kb":103936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":349,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.508711+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T15:01:08.508711+0000","last_peered":"2024-09-13T15:01:08.508711+0000","last_clean":"2024-09-13T15:01:08.508711+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T15:01:08.508711+0000","last_undegraded":"2024-09-13T15:01:08.508711+0000","last_fullsized":"2024-09-13T15:01:08.508711+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":330,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509243+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T15:01:08.509243+0000","last_peered":"2024-09-13T15:01:08.509243+0000","last_clean":"2024-09-13T15:01:08.509243+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T15:01:08.509243+0000","last_undegraded":"2024-09-13T15:01:08.509243+0000","last_fullsized":"2024-09-13T15:01:08.509243+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":599,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.962805+0000","last_change":"2024-09-13T14:58:10.512659+0000","last_active":"2024-09-13T15:01:08.962805+0000","last_peered":"2024-09-13T15:01:08.962805+0000","last_clean":"2024-09-13T15:01:08.962805+0000","last_became_active":"2024-09-13T14:58:10.511997+0000","last_became_peered":"2024-09-13T14:58:10.511997+0000","last_unstale":"2024-09-13T15:01:08.962805+0000","last_undegraded":"2024-09-13T15:01:08.962805+0000","last_fullsized":"2024-09-13T15:01:08.962805+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.415299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.415299+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:39.431834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"367'1705","reported_seq":1718,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.347900+0000","last_change":"2024-09-13T15:00:55.873173+0000","last_active":"2024-09-13T15:01:18.347900+0000","last_peered":"2024-09-13T15:01:18.347900+0000","last_clean":"2024-09-13T15:01:18.347900+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:01:18.347900+0000","last_undegraded":"2024-09-13T15:01:18.347900+0000","last_fullsized":"2024-09-13T15:01:18.347900+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1705,"log_dups_size":0,"ondisk_log_size":1705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:27.254765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111738880,"num_objects":1705,"num_object_clones":0,"num_object_copies":5115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1705,"num_write_kb":109120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":346,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.510007+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T15:01:08.510007+0000","last_peered":"2024-09-13T15:01:08.510007+0000","last_clean":"2024-09-13T15:01:08.510007+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T15:01:08.510007+0000","last_undegraded":"2024-09-13T15:01:08.510007+0000","last_fullsized":"2024-09-13T15:01:08.510007+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":350,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.418513+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T15:01:07.418513+0000","last_peered":"2024-09-13T15:01:07.418513+0000","last_clean":"2024-09-13T15:01:07.418513+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T15:01:07.418513+0000","last_undegraded":"2024-09-13T15:01:07.418513+0000","last_fullsized":"2024-09-13T15:01:07.418513+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":150,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.519165+0000","last_change":"2024-09-13T14:58:28.424238+0000","last_active":"2024-09-13T15:01:07.519165+0000","last_peered":"2024-09-13T15:01:07.519165+0000","last_clean":"2024-09-13T15:01:07.519165+0000","last_became_active":"2024-09-13T14:58:28.423844+0000","last_became_peered":"2024-09-13T14:58:28.423844+0000","last_unstale":"2024-09-13T15:01:07.519165+0000","last_undegraded":"2024-09-13T15:01:07.519165+0000","last_fullsized":"2024-09-13T15:01:07.519165+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:55.429885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"367'1659","reported_seq":1672,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.316226+0000","last_change":"2024-09-13T15:00:55.407846+0000","last_active":"2024-09-13T15:01:18.316226+0000","last_peered":"2024-09-13T15:01:18.316226+0000","last_clean":"2024-09-13T15:01:18.316226+0000","last_became_active":"2024-09-13T15:00:55.407115+0000","last_became_peered":"2024-09-13T15:00:55.407115+0000","last_unstale":"2024-09-13T15:01:18.316226+0000","last_undegraded":"2024-09-13T15:01:18.316226+0000","last_fullsized":"2024-09-13T15:01:18.316226+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1659,"log_dups_size":0,"ondisk_log_size":1659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:29.545511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108724224,"num_objects":1659,"num_object_clones":0,"num_object_copies":4977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1659,"num_write_kb":106176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":329,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509502+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T15:01:08.509502+0000","last_peered":"2024-09-13T15:01:08.509502+0000","last_clean":"2024-09-13T15:01:08.509502+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T15:01:08.509502+0000","last_undegraded":"2024-09-13T15:01:08.509502+0000","last_fullsized":"2024-09-13T15:01:08.509502+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":585,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199821+0000","last_change":"2024-09-13T14:58:10.522983+0000","last_active":"2024-09-13T15:01:17.199821+0000","last_peered":"2024-09-13T15:01:17.199821+0000","last_clean":"2024-09-13T15:01:17.199821+0000","last_became_active":"2024-09-13T14:58:10.513377+0000","last_became_peered":"2024-09-13T14:58:10.513377+0000","last_unstale":"2024-09-13T15:01:17.199821+0000","last_undegraded":"2024-09-13T15:01:17.199821+0000","last_fullsized":"2024-09-13T15:01:17.199821+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:12.579644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":170,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.145725+0000","last_change":"2024-09-13T14:57:57.223728+0000","last_active":"2024-09-13T15:01:07.145725+0000","last_peered":"2024-09-13T15:01:07.145725+0000","last_clean":"2024-09-13T15:01:07.145725+0000","last_became_active":"2024-09-13T14:57:19.233883+0000","last_became_peered":"2024-09-13T14:57:19.233883+0000","last_unstale":"2024-09-13T15:01:07.145725+0000","last_undegraded":"2024-09-13T15:01:07.145725+0000","last_fullsized":"2024-09-13T15:01:07.145725+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:41.520060+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2","version":"367'1543","reported_seq":1564,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.728700+0000","last_change":"2024-09-13T15:00:55.410530+0000","last_active":"2024-09-13T15:01:17.728700+0000","last_peered":"2024-09-13T15:01:17.728700+0000","last_clean":"2024-09-13T15:01:17.728700+0000","last_became_active":"2024-09-13T15:00:55.410331+0000","last_became_peered":"2024-09-13T15:00:55.410331+0000","last_unstale":"2024-09-13T15:01:17.728700+0000","last_undegraded":"2024-09-13T15:01:17.728700+0000","last_fullsized":"2024-09-13T15:01:17.728700+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1543,"log_dups_size":0,"ondisk_log_size":1543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:24.475954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":4629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1543,"num_write_kb":98752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":98,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509352+0000","last_change":"2024-09-13T14:58:10.513713+0000","last_active":"2024-09-13T15:01:08.509352+0000","last_peered":"2024-09-13T15:01:08.509352+0000","last_clean":"2024-09-13T15:01:08.509352+0000","last_became_active":"2024-09-13T14:58:10.513334+0000","last_became_peered":"2024-09-13T14:58:10.513334+0000","last_unstale":"2024-09-13T15:01:08.509352+0000","last_undegraded":"2024-09-13T15:01:08.509352+0000","last_fullsized":"2024-09-13T15:01:08.509352+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:41.326218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":176,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.200152+0000","last_change":"2024-09-13T14:58:28.428028+0000","last_active":"2024-09-13T15:01:17.200152+0000","last_peered":"2024-09-13T15:01:17.200152+0000","last_clean":"2024-09-13T15:01:17.200152+0000","last_became_active":"2024-09-13T14:58:28.427598+0000","last_became_peered":"2024-09-13T14:58:28.427598+0000","last_unstale":"2024-09-13T15:01:17.200152+0000","last_undegraded":"2024-09-13T15:01:17.200152+0000","last_fullsized":"2024-09-13T15:01:17.200152+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:32:50.930142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":170,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509835+0000","last_change":"2024-09-13T14:58:00.146439+0000","last_active":"2024-09-13T15:01:08.509835+0000","last_peered":"2024-09-13T15:01:08.509835+0000","last_clean":"2024-09-13T15:01:08.509835+0000","last_became_active":"2024-09-13T14:57:19.206097+0000","last_became_peered":"2024-09-13T14:57:19.206097+0000","last_unstale":"2024-09-13T15:01:08.509835+0000","last_undegraded":"2024-09-13T15:01:08.509835+0000","last_fullsized":"2024-09-13T15:01:08.509835+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:00.146384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:00.146384+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:09.697538+0000","scrub_duration":0.0014649660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"367'1572","reported_seq":1585,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.315229+0000","last_change":"2024-09-13T15:00:55.407630+0000","last_active":"2024-09-13T15:01:18.315229+0000","last_peered":"2024-09-13T15:01:18.315229+0000","last_clean":"2024-09-13T15:01:18.315229+0000","last_became_active":"2024-09-13T15:00:55.407307+0000","last_became_peered":"2024-09-13T15:00:55.407307+0000","last_unstale":"2024-09-13T15:01:18.315229+0000","last_undegraded":"2024-09-13T15:01:18.315229+0000","last_fullsized":"2024-09-13T15:01:18.315229+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1572,"log_dups_size":0,"ondisk_log_size":1572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:54.284626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103022592,"num_objects":1572,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1572,"num_write_kb":100608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":352,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.418258+0000","last_change":"2024-09-13T14:57:35.357982+0000","last_active":"2024-09-13T15:01:07.418258+0000","last_peered":"2024-09-13T15:01:07.418258+0000","last_clean":"2024-09-13T15:01:07.418258+0000","last_became_active":"2024-09-13T14:57:35.357618+0000","last_became_peered":"2024-09-13T14:57:35.357618+0000","last_unstale":"2024-09-13T15:01:07.418258+0000","last_undegraded":"2024-09-13T15:01:07.418258+0000","last_fullsized":"2024-09-13T15:01:07.418258+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:50.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":371,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146610+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:01:07.146610+0000","last_peered":"2024-09-13T15:01:07.146610+0000","last_clean":"2024-09-13T15:01:07.146610+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:01:07.146610+0000","last_undegraded":"2024-09-13T15:01:07.146610+0000","last_fullsized":"2024-09-13T15:01:07.146610+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":323,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.419079+0000","last_change":"2024-09-13T14:57:53.363216+0000","last_active":"2024-09-13T15:01:07.419079+0000","last_peered":"2024-09-13T15:01:07.419079+0000","last_clean":"2024-09-13T15:01:07.419079+0000","last_became_active":"2024-09-13T14:57:53.362679+0000","last_became_peered":"2024-09-13T14:57:53.362679+0000","last_unstale":"2024-09-13T15:01:07.419079+0000","last_undegraded":"2024-09-13T15:01:07.419079+0000","last_fullsized":"2024-09-13T15:01:07.419079+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.885757+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"367'1641","reported_seq":1654,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.365414+0000","last_change":"2024-09-13T15:00:55.873177+0000","last_active":"2024-09-13T15:01:18.365414+0000","last_peered":"2024-09-13T15:01:18.365414+0000","last_clean":"2024-09-13T15:01:18.365414+0000","last_became_active":"2024-09-13T15:00:55.872769+0000","last_became_peered":"2024-09-13T15:00:55.872769+0000","last_unstale":"2024-09-13T15:01:18.365414+0000","last_undegraded":"2024-09-13T15:01:18.365414+0000","last_fullsized":"2024-09-13T15:01:18.365414+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1641,"log_dups_size":0,"ondisk_log_size":1641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:22.869519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107544576,"num_objects":1641,"num_object_clones":0,"num_object_copies":4923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1641,"num_write_kb":105024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":356,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.509102+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T15:01:08.509102+0000","last_peered":"2024-09-13T15:01:08.509102+0000","last_clean":"2024-09-13T15:01:08.509102+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T15:01:08.509102+0000","last_undegraded":"2024-09-13T15:01:08.509102+0000","last_fullsized":"2024-09-13T15:01:08.509102+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":333,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146570+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T15:01:07.146570+0000","last_peered":"2024-09-13T15:01:07.146570+0000","last_clean":"2024-09-13T15:01:07.146570+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T15:01:07.146570+0000","last_undegraded":"2024-09-13T15:01:07.146570+0000","last_fullsized":"2024-09-13T15:01:07.146570+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":343,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199943+0000","last_change":"2024-09-13T14:58:10.531563+0000","last_active":"2024-09-13T15:01:17.199943+0000","last_peered":"2024-09-13T15:01:17.199943+0000","last_clean":"2024-09-13T15:01:17.199943+0000","last_became_active":"2024-09-13T14:58:10.515595+0000","last_became_peered":"2024-09-13T14:58:10.515595+0000","last_unstale":"2024-09-13T15:01:17.199943+0000","last_undegraded":"2024-09-13T15:01:17.199943+0000","last_fullsized":"2024-09-13T15:01:17.199943+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:19.723855+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"367'1605","reported_seq":1617,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:18.337037+0000","last_change":"2024-09-13T15:00:55.405838+0000","last_active":"2024-09-13T15:01:18.337037+0000","last_peered":"2024-09-13T15:01:18.337037+0000","last_clean":"2024-09-13T15:01:18.337037+0000","last_became_active":"2024-09-13T15:00:55.405095+0000","last_became_peered":"2024-09-13T15:00:55.405095+0000","last_unstale":"2024-09-13T15:01:18.337037+0000","last_undegraded":"2024-09-13T15:01:18.337037+0000","last_fullsized":"2024-09-13T15:01:18.337037+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":1605,"log_dups_size":0,"ondisk_log_size":1605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:55.315749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105185280,"num_objects":1605,"num_object_clones":0,"num_object_copies":4815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1605,"num_write_kb":102720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":48,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.961906+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:01:08.961906+0000","last_peered":"2024-09-13T15:01:08.961906+0000","last_clean":"2024-09-13T15:01:08.961906+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:01:08.961906+0000","last_undegraded":"2024-09-13T15:01:08.961906+0000","last_fullsized":"2024-09-13T15:01:08.961906+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":142,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146671+0000","last_change":"2024-09-13T14:58:10.530803+0000","last_active":"2024-09-13T15:01:07.146671+0000","last_peered":"2024-09-13T15:01:07.146671+0000","last_clean":"2024-09-13T15:01:07.146671+0000","last_became_active":"2024-09-13T14:58:10.522078+0000","last_became_peered":"2024-09-13T14:58:10.522078+0000","last_unstale":"2024-09-13T15:01:07.146671+0000","last_undegraded":"2024-09-13T15:01:07.146671+0000","last_fullsized":"2024-09-13T15:01:07.146671+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:50.807687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":154,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146800+0000","last_change":"2024-09-13T14:57:55.208230+0000","last_active":"2024-09-13T15:01:07.146800+0000","last_peered":"2024-09-13T15:01:07.146800+0000","last_clean":"2024-09-13T15:01:07.146800+0000","last_became_active":"2024-09-13T14:57:19.211671+0000","last_became_peered":"2024-09-13T14:57:19.211671+0000","last_unstale":"2024-09-13T15:01:07.146800+0000","last_undegraded":"2024-09-13T15:01:07.146800+0000","last_fullsized":"2024-09-13T15:01:07.146800+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.066938+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":438,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.418069+0000","last_change":"2024-09-13T14:58:10.529854+0000","last_active":"2024-09-13T15:01:07.418069+0000","last_peered":"2024-09-13T15:01:07.418069+0000","last_clean":"2024-09-13T15:01:07.418069+0000","last_became_active":"2024-09-13T14:58:10.514813+0000","last_became_peered":"2024-09-13T14:58:10.514813+0000","last_unstale":"2024-09-13T15:01:07.418069+0000","last_undegraded":"2024-09-13T15:01:07.418069+0000","last_fullsized":"2024-09-13T15:01:07.418069+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:46.296441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":583,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.201680+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T15:01:17.201680+0000","last_peered":"2024-09-13T15:01:17.201680+0000","last_clean":"2024-09-13T15:01:17.201680+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T15:01:17.201680+0000","last_undegraded":"2024-09-13T15:01:17.201680+0000","last_fullsized":"2024-09-13T15:01:17.201680+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":48,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.964558+0000","last_change":"2024-09-13T14:57:53.356010+0000","last_active":"2024-09-13T15:01:08.964558+0000","last_peered":"2024-09-13T15:01:08.964558+0000","last_clean":"2024-09-13T15:01:08.964558+0000","last_became_active":"2024-09-13T14:57:53.354979+0000","last_became_peered":"2024-09-13T14:57:53.354979+0000","last_unstale":"2024-09-13T15:01:08.964558+0000","last_undegraded":"2024-09-13T15:01:08.964558+0000","last_fullsized":"2024-09-13T15:01:08.964558+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:38.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":334,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.145922+0000","last_change":"2024-09-13T14:57:35.358990+0000","last_active":"2024-09-13T15:01:07.145922+0000","last_peered":"2024-09-13T15:01:07.145922+0000","last_clean":"2024-09-13T15:01:07.145922+0000","last_became_active":"2024-09-13T14:57:35.358547+0000","last_became_peered":"2024-09-13T14:57:35.358547+0000","last_unstale":"2024-09-13T15:01:07.145922+0000","last_undegraded":"2024-09-13T15:01:07.145922+0000","last_fullsized":"2024-09-13T15:01:07.145922+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:45.386301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":462,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.204023+0000","last_change":"2024-09-13T14:58:10.536776+0000","last_active":"2024-09-13T15:01:17.204023+0000","last_peered":"2024-09-13T15:01:17.204023+0000","last_clean":"2024-09-13T15:01:17.204023+0000","last_became_active":"2024-09-13T14:58:10.536376+0000","last_became_peered":"2024-09-13T14:58:10.536376+0000","last_unstale":"2024-09-13T15:01:17.204023+0000","last_undegraded":"2024-09-13T15:01:17.204023+0000","last_fullsized":"2024-09-13T15:01:17.204023+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:08.237256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":301,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199644+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T15:01:17.199644+0000","last_peered":"2024-09-13T15:01:17.199644+0000","last_clean":"2024-09-13T15:01:17.199644+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T15:01:17.199644+0000","last_undegraded":"2024-09-13T15:01:17.199644+0000","last_fullsized":"2024-09-13T15:01:17.199644+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:04.244429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":435,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.145669+0000","last_change":"2024-09-13T14:58:10.524047+0000","last_active":"2024-09-13T15:01:07.145669+0000","last_peered":"2024-09-13T15:01:07.145669+0000","last_clean":"2024-09-13T15:01:07.145669+0000","last_became_active":"2024-09-13T14:58:10.522007+0000","last_became_peered":"2024-09-13T14:58:10.522007+0000","last_unstale":"2024-09-13T15:01:07.145669+0000","last_undegraded":"2024-09-13T15:01:07.145669+0000","last_fullsized":"2024-09-13T15:01:07.145669+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:41.957759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":225,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146386+0000","last_change":"2024-09-13T14:58:28.426617+0000","last_active":"2024-09-13T15:01:07.146386+0000","last_peered":"2024-09-13T15:01:07.146386+0000","last_clean":"2024-09-13T15:01:07.146386+0000","last_became_active":"2024-09-13T14:58:28.425773+0000","last_became_peered":"2024-09-13T14:58:28.425773+0000","last_unstale":"2024-09-13T15:01:07.146386+0000","last_undegraded":"2024-09-13T15:01:07.146386+0000","last_fullsized":"2024-09-13T15:01:07.146386+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:59.659871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":56,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.145750+0000","last_change":"2024-09-13T14:58:28.428896+0000","last_active":"2024-09-13T15:01:07.145750+0000","last_peered":"2024-09-13T15:01:07.145750+0000","last_clean":"2024-09-13T15:01:07.145750+0000","last_became_active":"2024-09-13T14:58:28.428522+0000","last_became_peered":"2024-09-13T14:58:28.428522+0000","last_unstale":"2024-09-13T15:01:07.145750+0000","last_undegraded":"2024-09-13T15:01:07.145750+0000","last_fullsized":"2024-09-13T15:01:07.145750+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:05.418586+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":349,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:07.146000+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T15:01:07.146000+0000","last_peered":"2024-09-13T15:01:07.146000+0000","last_clean":"2024-09-13T15:01:07.146000+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T15:01:07.146000+0000","last_undegraded":"2024-09-13T15:01:07.146000+0000","last_fullsized":"2024-09-13T15:01:07.146000+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":77,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T15:01:17.199737+0000","last_change":"2024-09-13T14:58:10.517240+0000","last_active":"2024-09-13T15:01:17.199737+0000","last_peered":"2024-09-13T15:01:17.199737+0000","last_clean":"2024-09-13T15:01:17.199737+0000","last_became_active":"2024-09-13T14:58:10.516811+0000","last_became_peered":"2024-09-13T14:58:10.516811+0000","last_unstale":"2024-09-13T15:01:17.199737+0000","last_undegraded":"2024-09-13T15:01:17.199737+0000","last_fullsized":"2024-09-13T15:01:17.199737+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:52.711800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":565,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-13T15:01:08.965011+0000","last_change":"2024-09-13T14:58:10.512480+0000","last_active":"2024-09-13T15:01:08.965011+0000","last_peered":"2024-09-13T15:01:08.965011+0000","last_clean":"2024-09-13T15:01:08.965011+0000","last_became_active":"2024-09-13T14:58:10.511589+0000","last_became_peered":"2024-09-13T14:58:10.511589+0000","last_unstale":"2024-09-13T15:01:08.965011+0000","last_undegraded":"2024-09-13T15:01:08.965011+0000","last_fullsized":"2024-09-13T15:01:08.965011+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:52.302190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":1662910464,"num_objects":25374,"num_object_clones":0,"num_object_copies":76122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25374,"num_write_kb":1623936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4944691200,"data_stored":4944691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25374,"ondisk_log_size":25374,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":448,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365299,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2001156,"kb_used_data":491832,"kb_used_omap":1,"kb_used_meta":1509310,"kb_avail":91743996,"statfs":{"total":95995035648,"available":93945851904,"internally_reserved":0,"allocated":503635968,"data_stored":501466865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1545533784},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561529,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2083836,"kb_used_data":591600,"kb_used_omap":1,"kb_used_meta":1492222,"kb_avail":91661316,"statfs":{"total":95995035648,"available":93861187584,"internally_reserved":0,"allocated":605798400,"data_stored":603637338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1528035672},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790461,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2372016,"kb_used_data":881444,"kb_used_omap":1,"kb_used_meta":1490558,"kb_avail":91373136,"statfs":{"total":95995035648,"available":93566091264,"internally_reserved":0,"allocated":902598656,"data_stored":900445430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1526331736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986689,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2067132,"kb_used_data":884336,"kb_used_omap":1,"kb_used_meta":1182782,"kb_avail":91678020,"statfs":{"total":95995035648,"available":93878292480,"internally_reserved":0,"allocated":905560064,"data_stored":903398770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1211169112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519844,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1408924,"kb_used_data":477652,"kb_used_omap":4,"kb_used_meta":931259,"kb_avail":92336228,"statfs":{"total":95995035648,"available":94552297472,"internally_reserved":0,"allocated":489115648,"data_stored":486969850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":953609664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304759,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1629540,"kb_used_data":730716,"kb_used_omap":6,"kb_used_meta":898809,"kb_avail":92115612,"statfs":{"total":95995035648,"available":94326386688,"internally_reserved":0,"allocated":748253184,"data_stored":746100252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":920380871},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828063,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1630324,"kb_used_data":498408,"kb_used_omap":5,"kb_used_meta":1131898,"kb_avail":92114828,"statfs":{"total":95995035648,"available":94325583872,"internally_reserved":0,"allocated":510369792,"data_stored":508211929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1159063788},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":356482285927,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":543883264,"data_stored":543883264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":794755072,"data_stored":794755072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":507641856,"data_stored":507641856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":953876480,"data_stored":953876480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":960233472,"data_stored":960233472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":646119424,"data_stored":646119424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":538181632,"data_stored":538181632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:01:24.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-09-13T15:01:25.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:25.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:25.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:25.268 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:25.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:25.294 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:01:25.294 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:25.329 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:01:25.330 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:25.362 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:01:25.362 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:25.384 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:25.388 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:01:25.389 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:25.419 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:01:25.419 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:25.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:25 smithi003 ceph-mon[107278]: pgmap v1310: 119 pgs: 119 active+clean; 2.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-13T15:01:25.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:25 smithi003 ceph-mon[104007]: pgmap v1310: 119 pgs: 119 active+clean; 2.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-13T15:01:25.453 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:01:25.453 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:25.483 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:01:25.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:25.523 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:01:25.524 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:25.524 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.566 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:25.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:25.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:25 smithi019 ceph-mon[91296]: pgmap v1310: 119 pgs: 119 active+clean; 2.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-13T15:01:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:25.802 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:25.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:25.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 40944 40928 150.456 112.25 0.00936753 0.00664026 2024-09-13T15:01:25.985 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:26.086 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.127 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:26.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:26.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.269 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:26.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:26.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:26.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:26.504 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:26.541 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:26.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3774966115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:26.574 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3774966115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:26.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3774966115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:26.685 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:26.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:26.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:26.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:26.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:26.958 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 42754 42738 148.381 113.125 0.00761568 0.00673463 2024-09-13T15:01:27.058 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:27.095 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:27.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:27.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:27.333 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:27.374 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:27.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:27.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:27 smithi003 ceph-mon[107278]: pgmap v1311: 119 pgs: 119 active+clean; 2.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-13T15:01:27.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:27 smithi003 ceph-mon[104007]: pgmap v1311: 119 pgs: 119 active+clean; 2.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-13T15:01:27.618 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:27 smithi019 ceph-mon[91296]: pgmap v1311: 119 pgs: 119 active+clean; 2.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-13T15:01:27.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:27.789 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:27.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:27.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:27.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 44524 44508 146.393 110.625 0.00947676 0.0068254 2024-09-13T15:01:28.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:28.064 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:28.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:28.207 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:28.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:28.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:28.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:28.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:28.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:28.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.759 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:28.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:28.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:28.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:28 smithi003 ceph-mon[104007]: pgmap v1312: 119 pgs: 119 active+clean; 2.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-09-13T15:01:28.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]} : dispatch 2024-09-13T15:01:28.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:28.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:28.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:28 smithi003 ceph-mon[107278]: pgmap v1312: 119 pgs: 119 active+clean; 2.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-09-13T15:01:28.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]} : dispatch 2024-09-13T15:01:28.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T15:01:28.957011+0000 min lat: 0.00155173 max lat: 0.140489 avg lat: 0.00691816 2024-09-13T15:01:28.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:01:28.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 46243 46227 144.445 107.438 0.00711157 0.00691816 2024-09-13T15:01:28.997 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:29.037 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:29.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:28 smithi019 ceph-mon[91296]: pgmap v1312: 119 pgs: 119 active+clean; 2.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-09-13T15:01:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]} : dispatch 2024-09-13T15:01:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:29.287 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:29.323 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:29.331 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:01:29.331 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:01:29.331 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:01:29.331 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:01:29.331 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:01:29.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:29.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:29.574 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:29.611 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:29.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:29.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:29.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:29.852 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:01:29.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:01:29.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:29.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:29.929 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-09-13T15:01:29.929 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:29 smithi003 ceph-mon[107278]: osdmap e368: 8 total, 8 up, 7 in 2024-09-13T15:01:29.929 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:29 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4191439780' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:01:29.929 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:29 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:01:29.929 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-09-13T15:01:29.930 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:29 smithi003 ceph-mon[104007]: osdmap e368: 8 total, 8 up, 7 in 2024-09-13T15:01:29.930 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:29 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4191439780' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:01:29.930 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:29 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:01:29.957 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 47893 47877 142.477 103.125 0.00558062 0.00701306 2024-09-13T15:01:29.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:30.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:30.138 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:30.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-09-13T15:01:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:29 smithi019 ceph-mon[91296]: osdmap e368: 8 total, 8 up, 7 in 2024-09-13T15:01:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:29 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4191439780' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:01:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:29 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:01:30.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:30.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:30.421 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:30.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:30.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:30.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:01:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:01:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:30.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:30.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:30.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:30.959 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 49498 49482 140.549 100.312 0.00581076 0.00710933 2024-09-13T15:01:30.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:30.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:01:31.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:31.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:31.157 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:30 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:30 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:01:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:30 smithi019 ceph-mon[91296]: osdmap e369: 8 total, 8 up, 7 in 2024-09-13T15:01:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:30 smithi019 ceph-mon[91296]: pgmap v1315: 119 pgs: 119 active+clean; 2.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-13T15:01:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:30 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:30 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:01:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:30 smithi003 ceph-mon[107278]: osdmap e369: 8 total, 8 up, 7 in 2024-09-13T15:01:31.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:30 smithi003 ceph-mon[107278]: pgmap v1315: 119 pgs: 119 active+clean; 2.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-13T15:01:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:30 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:30 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:01:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:30 smithi003 ceph-mon[104007]: osdmap e369: 8 total, 8 up, 7 in 2024-09-13T15:01:31.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:30 smithi003 ceph-mon[104007]: pgmap v1315: 119 pgs: 119 active+clean; 2.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-13T15:01:31.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.298 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:31.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:31.399 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:31.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:31.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.581 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:31.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:31.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:31.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:31.760 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:31.815 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:31.818 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:31.859 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:31.959 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 51100 51084 138.792 100.125 0.0054225 0.00719999 2024-09-13T15:01:31.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:31.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:31.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:32.092 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.134 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:32.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:32.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:32.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:32.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:32.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:32.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:32.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:32.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:32.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:32.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:32.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:32.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:32.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:32.959 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 52688 52672 137.144 99.25 0.0170604 0.00728718 2024-09-13T15:01:33.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:33.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:33.188 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:33.229 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:33.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:33.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:33 smithi019 ceph-mon[91296]: pgmap v1316: 119 pgs: 119 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T15:01:33.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:33 smithi003 ceph-mon[104007]: pgmap v1316: 119 pgs: 119 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T15:01:33.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:33 smithi003 ceph-mon[107278]: pgmap v1316: 119 pgs: 119 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T15:01:33.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:33.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:33.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:33.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:33.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:01:33.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.773 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:33.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:33.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:33.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:33.960 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 54260 54244 135.586 98.25 0.0045221 0.00736991 2024-09-13T15:01:34.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:34.056 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:34.158 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:34.197 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:34.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:34.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:01:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:01:34] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-13T15:01:34.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:34.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:34.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:34.618 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:34.720 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:34.758 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:34.860 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:34.897 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:34.960 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 55761 55745 133.979 93.8125 0.00782535 0.00746011 2024-09-13T15:01:34.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:35.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:35.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:35.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:35.268 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:35.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:35.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:35 smithi003 ceph-mon[107278]: pgmap v1317: 119 pgs: 119 active+clean; 3.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-09-13T15:01:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:35 smithi003 ceph-mon[104007]: pgmap v1317: 119 pgs: 119 active+clean; 3.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-09-13T15:01:35.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:35 smithi019 ceph-mon[91296]: pgmap v1317: 119 pgs: 119 active+clean; 3.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-09-13T15:01:35.449 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:35.551 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:35.592 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:35.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:35.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:35.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:35.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:35.960 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 57263 57247 132.494 93.875 0.00854845 0.00754286 2024-09-13T15:01:35.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:36.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:01:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:01:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:01:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:36.002 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:36.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:36.104 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:36.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:36.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:36.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:36.427 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:36.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:36 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/797527222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:36.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:36 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/797527222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:36.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:36 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/797527222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:36.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:36.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:36.663 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.706 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:36.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:36.807 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:36.845 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:36.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:36.960 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 58742 58726 131.063 92.4375 0.0142546 0.00762442 2024-09-13T15:01:36.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:37.078 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:37.114 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:37.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:37.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:37.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:37.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:37.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:37 smithi019 ceph-mon[91296]: pgmap v1318: 119 pgs: 119 active+clean; 3.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 117 MiB/s wr, 1.86k op/s 2024-09-13T15:01:37.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:37 smithi003 ceph-mon[107278]: pgmap v1318: 119 pgs: 119 active+clean; 3.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 117 MiB/s wr, 1.86k op/s 2024-09-13T15:01:37.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:37 smithi003 ceph-mon[104007]: pgmap v1318: 119 pgs: 119 active+clean; 3.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 117 MiB/s wr, 1.86k op/s 2024-09-13T15:01:37.476 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:37.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:37.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:37.661 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:37.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:37.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:37.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:37.960 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 60127 60111 129.528 86.5625 0.00559333 0.00771511 2024-09-13T15:01:38.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:38.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:38.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:38.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:38.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:38.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:38.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:38.594 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:38.632 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:38.640 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:38.733 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:38.776 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:38.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:38.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:38.960 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 61551 61535 128.177 89 0.00630133 0.00779675 2024-09-13T15:01:39.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:39.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:39.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:39.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:39.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:39 smithi003 ceph-mon[104007]: pgmap v1319: 119 pgs: 119 active+clean; 3.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-13T15:01:39.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:39.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:39.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:39.425 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:39 smithi019 ceph-mon[91296]: pgmap v1319: 119 pgs: 119 active+clean; 3.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-13T15:01:39.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:39 smithi003 ceph-mon[107278]: pgmap v1319: 119 pgs: 119 active+clean; 3.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-13T15:01:39.461 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:39.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:39.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:39.696 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:39.735 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:39.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:39.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:39.960 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 62989 62973 126.942 89.875 0.0195019 0.00787302 2024-09-13T15:01:39.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:39.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:01:40.026 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:40.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:40.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:40.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:40.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:40.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:40.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.447 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:40.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:40.549 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:40.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.582 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:40.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:40.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:01:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:01:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:40.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:40.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:40.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:40.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:40.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 64259 64243 125.454 79.375 0.00712616 0.00796645 2024-09-13T15:01:40.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:41.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:41.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:41.266 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:41 smithi003 ceph-mon[107278]: pgmap v1320: 119 pgs: 119 active+clean; 3.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T15:01:41.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:41 smithi003 ceph-mon[104007]: pgmap v1320: 119 pgs: 119 active+clean; 3.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T15:01:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.270 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:41.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:41.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:41.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:41.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:41 smithi019 ceph-mon[91296]: pgmap v1320: 119 pgs: 119 active+clean; 3.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T15:01:41.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:41.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:41.659 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:41.702 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:41.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:41.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:41.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:41.961 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 65643 65627 124.272 86.5 0.00737249 0.00804215 2024-09-13T15:01:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:42.001 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:01:42.001 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:01:42.082 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:42.123 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:42.165 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pgp_num to 31 2024-09-13T15:01:42.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:42.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:42.290 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:42 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3198937118' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-13T15:01:42.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:42 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3198937118' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-13T15:01:42.357 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:42.403 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:42.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:42 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3198937118' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-13T15:01:42.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:42.545 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:42.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:42.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:42.729 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:01:42.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:42.823 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:42.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:42.961 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 66939 66923 122.999 81 0.0279509 0.00812494 2024-09-13T15:01:42.964 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:43.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.125 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:43.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:43.179 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:01:43.226 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:43.234 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:01:43.288 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:43.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:43 smithi019 ceph-mon[91296]: pgmap v1321: 119 pgs: 119 active+clean; 3.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-13T15:01:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3198937118' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-13T15:01:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:43 smithi019 ceph-mon[91296]: osdmap e370: 8 total, 8 up, 7 in 2024-09-13T15:01:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3331719075' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:43.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:43.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[104007]: pgmap v1321: 119 pgs: 119 active+clean; 3.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-13T15:01:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3198937118' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-13T15:01:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[104007]: osdmap e370: 8 total, 8 up, 7 in 2024-09-13T15:01:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3331719075' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:43.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:43.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[107278]: pgmap v1321: 119 pgs: 119 active+clean; 3.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-13T15:01:43.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3198937118' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-13T15:01:43.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[107278]: osdmap e370: 8 total, 8 up, 7 in 2024-09-13T15:01:43.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3331719075' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:43.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:01:43.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:43.435 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:43.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:43.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:43.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:43.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:43.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:43.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:43.926 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:43.961 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 68145 68129 121.638 75.375 0.00530286 0.00821561 2024-09-13T15:01:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:43.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:44.068 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:44.111 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:44.170 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:01:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:01:44] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-13T15:01:44.204 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:44.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.255 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:44.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:44.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:44.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:44 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3331719075' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:44.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:44 smithi019 ceph-mon[91296]: osdmap e371: 8 total, 8 up, 7 in 2024-09-13T15:01:44.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:44 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:44.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3331719075' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:44.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:44 smithi003 ceph-mon[104007]: osdmap e371: 8 total, 8 up, 7 in 2024-09-13T15:01:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:44 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3331719075' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:44 smithi003 ceph-mon[107278]: osdmap e371: 8 total, 8 up, 7 in 2024-09-13T15:01:44.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:44.533 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:44.635 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:44.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:44.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:44.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:44.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:44.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:44.961 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 69437 69421 120.502 80.75 0.00513013 0.00829336 2024-09-13T15:01:45.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:45.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:45.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:45.240 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:45 smithi019 ceph-mon[91296]: pgmap v1324: 119 pgs: 119 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T15:01:45.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:45.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:45.408 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:01:45.408 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:01:45.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:45.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:45.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:45.610 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:45.672 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:45 smithi003 ceph-mon[104007]: pgmap v1324: 119 pgs: 119 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T15:01:45.672 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:45 smithi003 ceph-mon[107278]: pgmap v1324: 119 pgs: 119 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T15:01:45.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:45.745 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:45.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:45.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:45.961 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 70619 70603 119.242 73.875 0.00450857 0.00838152 2024-09-13T15:01:45.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:46.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:46.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:46.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:46.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:46.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:01:46.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:46.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:46.503 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:46.526 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:46.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:46.668 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:46.770 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:46.813 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:46.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:46.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:46.961 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 71751 71735 117.966 70.75 0.007504 0.00847187 2024-09-13T15:01:47.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:47.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:47.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:47.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:47.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:47.359 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:47.428 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:47 smithi003 ceph-mon[107278]: pgmap v1325: 119 pgs: 119 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-09-13T15:01:47.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:47 smithi003 ceph-mon[104007]: pgmap v1325: 119 pgs: 119 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-09-13T15:01:47.460 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:47.479 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:01:47.480 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:47.492 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:47.498 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:47.505 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:01:47.506 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:47.537 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:01:47.540 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:47.565 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:01:47.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:47.599 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:47.617 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:01:47.617 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:47.627 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:47.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:47.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:01:47.645 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:47.673 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:01:47.674 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:47 smithi019 ceph-mon[91296]: pgmap v1325: 119 pgs: 119 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-09-13T15:01:47.705 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:01:47.705 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:47.729 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:47.768 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:47.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:47.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:47.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 72892 72876 116.769 71.3125 0.00865267 0.00855781 2024-09-13T15:01:48.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:48.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:48.131 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:48.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:48.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:48.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:48.400 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3528929460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3528929460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:48.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.448 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:48.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:48.527 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:01:48.527 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:01:48.527 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T15:01:48.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:01:48.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:48.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:48.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3528929460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:48.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:48.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:48.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:48.889 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:48.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T15:01:48.961456+0000 min lat: 0.00155173 max lat: 0.140489 avg lat: 0.00863732 2024-09-13T15:01:48.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:01:48.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 74089 74073 115.721 74.8125 0.0113566 0.00863732 2024-09-13T15:01:48.990 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:49.033 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:49.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:49.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:49.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:49.266 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:49.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:49.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:49.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:49 smithi019 ceph-mon[91296]: pgmap v1326: 119 pgs: 119 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T15:01:49.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:49 smithi003 ceph-mon[104007]: pgmap v1326: 119 pgs: 119 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T15:01:49.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:49 smithi003 ceph-mon[107278]: pgmap v1326: 119 pgs: 119 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T15:01:49.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:49.500 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:49.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:49.591 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:49.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:49.733 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:49.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:49.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:49.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:49.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 75212 75196 114.61 70.1875 0.0175247 0.00872042 2024-09-13T15:01:49.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:50.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:50.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:50.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:50.231 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.276 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:50.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:50.377 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:50.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:50.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:50.642 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:50.677 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:01:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:01:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:50.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:50.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:50.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:50.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:50.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 76300 76284 113.5 68 0.00996158 0.00880588 2024-09-13T15:01:51.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:01:51.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:51.073 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:51.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:51.205 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:01:51.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:51.315 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:51.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:51 smithi003 ceph-mon[104007]: pgmap v1327: 119 pgs: 119 active+clean; 4.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-13T15:01:51.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:51 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3824122525' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:51.331 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:51 smithi003 ceph-mon[107278]: pgmap v1327: 119 pgs: 119 active+clean; 4.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-13T15:01:51.331 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:51 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3824122525' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:51.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:51.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:51.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:51.583 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.623 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:51.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:51 smithi019 ceph-mon[91296]: pgmap v1327: 119 pgs: 119 active+clean; 4.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-13T15:01:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:51 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3824122525' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:51.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:51.763 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:51.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:51.901 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:51.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 77157 77141 112.106 53.5625 0.00353501 0.00890597 2024-09-13T15:01:52.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:52.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:52.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:52.172 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:52.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:52.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:52.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:52.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:52.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:52.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:52.570 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:52.571 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:52 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:52.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:52 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3824122525' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:52.572 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:52 smithi003 ceph-mon[107278]: osdmap e372: 8 total, 8 up, 7 in 2024-09-13T15:01:52.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:52 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:52.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:52 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3824122525' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:52.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:52 smithi003 ceph-mon[104007]: osdmap e372: 8 total, 8 up, 7 in 2024-09-13T15:01:52.608 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:52 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:52 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3824122525' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:52.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:52 smithi019 ceph-mon[91296]: osdmap e372: 8 total, 8 up, 7 in 2024-09-13T15:01:52.710 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:52.745 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:52.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:52.837 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:52.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:52.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:52.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 78042 78026 110.815 55.3125 0.00896445 0.00901934 2024-09-13T15:01:52.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:53.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:53.030 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:01:53.128 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:53.190 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:01:53.194 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:01:53.236 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:53.338 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:53.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.402 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:53.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:53.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:53 smithi003 ceph-mon[107278]: pgmap v1329: 119 pgs: 119 active+clean; 4.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T15:01:53.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:53 smithi003 ceph-mon[104007]: pgmap v1329: 119 pgs: 119 active+clean; 4.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T15:01:53.433 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:53 smithi019 ceph-mon[91296]: pgmap v1329: 119 pgs: 119 active+clean; 4.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T15:01:53.504 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:53.527 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:01:53.558 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:53.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:53.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:53.795 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:53.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:53.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:53.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 79111 79095 109.837 66.8125 0.00362356 0.00909712 2024-09-13T15:01:53.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:53.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:54.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:54.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:54.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:54.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:54.231 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:01:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:01:54] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-13T15:01:54.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:01:54.325 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1329,"stamp":"2024-09-13T15:01:52.084335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4827125709,"num_objects":73673,"num_object_clones":0,"num_object_copies":221019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73711,"num_write_kb":4713440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":328147,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76806,"ondisk_log_size":76806,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":22818596,"kb_used_data":14088684,"kb_used_omap":23,"kb_used_meta":8729832,"kb_avail":633397468,"statfs":{"total":671965249536,"available":648599007232,"internally_reserved":0,"allocated":14426812416,"data_stored":14411484131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24109,"internal_metadata":8939348435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,4,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":767885312,"num_objects":11717,"num_object_clones":0,"num_object_copies":35151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11717,"num_write_kb":749888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.902360"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":129,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.192610+0000","last_change":"2024-09-13T14:57:42.607827+0000","last_active":"2024-09-13T15:01:43.192610+0000","last_peered":"2024-09-13T15:01:43.192610+0000","last_clean":"2024-09-13T15:01:43.192610+0000","last_became_active":"2024-09-13T14:57:25.285562+0000","last_became_peered":"2024-09-13T14:57:25.285562+0000","last_unstale":"2024-09-13T15:01:43.192610+0000","last_undegraded":"2024-09-13T15:01:43.192610+0000","last_fullsized":"2024-09-13T15:01:43.192610+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.607753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.607753+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:13.317042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":89,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.190981+0000","last_change":"2024-09-13T14:58:28.426659+0000","last_active":"2024-09-13T15:01:43.190981+0000","last_peered":"2024-09-13T15:01:43.190981+0000","last_clean":"2024-09-13T15:01:43.190981+0000","last_became_active":"2024-09-13T14:58:28.426336+0000","last_became_peered":"2024-09-13T14:58:28.426336+0000","last_unstale":"2024-09-13T15:01:43.190981+0000","last_undegraded":"2024-09-13T15:01:43.190981+0000","last_fullsized":"2024-09-13T15:01:43.190981+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:55.713969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":343,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.175444+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T15:01:42.175444+0000","last_peered":"2024-09-13T15:01:42.175444+0000","last_clean":"2024-09-13T15:01:42.175444+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T15:01:42.175444+0000","last_undegraded":"2024-09-13T15:01:42.175444+0000","last_fullsized":"2024-09-13T15:01:42.175444+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":69,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176069+0000","last_change":"2024-09-13T14:57:58.200463+0000","last_active":"2024-09-13T15:01:42.176069+0000","last_peered":"2024-09-13T15:01:42.176069+0000","last_clean":"2024-09-13T15:01:42.176069+0000","last_became_active":"2024-09-13T14:57:25.279086+0000","last_became_peered":"2024-09-13T14:57:25.279086+0000","last_unstale":"2024-09-13T15:01:42.176069+0000","last_undegraded":"2024-09-13T15:01:42.176069+0000","last_fullsized":"2024-09-13T15:01:42.176069+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:58.986716+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":106,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.193515+0000","last_change":"2024-09-13T14:57:35.356151+0000","last_active":"2024-09-13T15:01:43.193515+0000","last_peered":"2024-09-13T15:01:43.193515+0000","last_clean":"2024-09-13T15:01:43.193515+0000","last_became_active":"2024-09-13T14:57:35.355770+0000","last_became_peered":"2024-09-13T14:57:35.355770+0000","last_unstale":"2024-09-13T15:01:43.193515+0000","last_undegraded":"2024-09-13T15:01:43.193515+0000","last_fullsized":"2024-09-13T15:01:43.193515+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:37.887286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":591,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.192393+0000","last_change":"2024-09-13T14:58:10.513979+0000","last_active":"2024-09-13T15:01:43.192393+0000","last_peered":"2024-09-13T15:01:43.192393+0000","last_clean":"2024-09-13T15:01:43.192393+0000","last_became_active":"2024-09-13T14:58:10.513313+0000","last_became_peered":"2024-09-13T14:58:10.513313+0000","last_unstale":"2024-09-13T15:01:43.192393+0000","last_undegraded":"2024-09-13T15:01:43.192393+0000","last_fullsized":"2024-09-13T15:01:43.192393+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:25:15.376408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":74,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.180016+0000","last_change":"2024-09-13T14:58:28.426763+0000","last_active":"2024-09-13T15:01:42.180016+0000","last_peered":"2024-09-13T15:01:42.180016+0000","last_clean":"2024-09-13T15:01:42.180016+0000","last_became_active":"2024-09-13T14:58:28.426270+0000","last_became_peered":"2024-09-13T14:58:28.426270+0000","last_unstale":"2024-09-13T15:01:42.180016+0000","last_undegraded":"2024-09-13T15:01:42.180016+0000","last_fullsized":"2024-09-13T15:01:42.180016+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.991576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":360,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176099+0000","last_change":"2024-09-13T14:58:10.515535+0000","last_active":"2024-09-13T15:01:42.176099+0000","last_peered":"2024-09-13T15:01:42.176099+0000","last_clean":"2024-09-13T15:01:42.176099+0000","last_became_active":"2024-09-13T14:58:10.515043+0000","last_became_peered":"2024-09-13T14:58:10.515043+0000","last_unstale":"2024-09-13T15:01:42.176099+0000","last_undegraded":"2024-09-13T15:01:42.176099+0000","last_fullsized":"2024-09-13T15:01:42.176099+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:53.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":132,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.193806+0000","last_change":"2024-09-13T14:57:43.658933+0000","last_active":"2024-09-13T15:01:43.193806+0000","last_peered":"2024-09-13T15:01:43.193806+0000","last_clean":"2024-09-13T15:01:43.193806+0000","last_became_active":"2024-09-13T14:57:25.291285+0000","last_became_peered":"2024-09-13T14:57:25.291285+0000","last_unstale":"2024-09-13T15:01:43.193806+0000","last_undegraded":"2024-09-13T15:01:43.193806+0000","last_fullsized":"2024-09-13T15:01:43.193806+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:43.658833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:43.658833+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.888776+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":74,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176163+0000","last_change":"2024-09-13T14:58:28.429061+0000","last_active":"2024-09-13T15:01:42.176163+0000","last_peered":"2024-09-13T15:01:42.176163+0000","last_clean":"2024-09-13T15:01:42.176163+0000","last_became_active":"2024-09-13T14:58:28.428642+0000","last_became_peered":"2024-09-13T14:58:28.428642+0000","last_unstale":"2024-09-13T15:01:42.176163+0000","last_undegraded":"2024-09-13T15:01:42.176163+0000","last_fullsized":"2024-09-13T15:01:42.176163+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:03.197824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":380,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176542+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T15:01:42.176542+0000","last_peered":"2024-09-13T15:01:42.176542+0000","last_clean":"2024-09-13T15:01:42.176542+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T15:01:42.176542+0000","last_undegraded":"2024-09-13T15:01:42.176542+0000","last_fullsized":"2024-09-13T15:01:42.176542+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":138,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.191846+0000","last_change":"2024-09-13T14:58:10.521468+0000","last_active":"2024-09-13T15:01:43.191846+0000","last_peered":"2024-09-13T15:01:43.191846+0000","last_clean":"2024-09-13T15:01:43.191846+0000","last_became_active":"2024-09-13T14:58:10.520775+0000","last_became_peered":"2024-09-13T14:58:10.520775+0000","last_unstale":"2024-09-13T15:01:43.191846+0000","last_undegraded":"2024-09-13T15:01:43.191846+0000","last_fullsized":"2024-09-13T15:01:43.191846+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:11.701399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":77,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176408+0000","last_change":"2024-09-13T14:58:10.523188+0000","last_active":"2024-09-13T15:01:42.176408+0000","last_peered":"2024-09-13T15:01:42.176408+0000","last_clean":"2024-09-13T15:01:42.176408+0000","last_became_active":"2024-09-13T14:58:10.522815+0000","last_became_peered":"2024-09-13T14:58:10.522815+0000","last_unstale":"2024-09-13T15:01:42.176408+0000","last_undegraded":"2024-09-13T15:01:42.176408+0000","last_fullsized":"2024-09-13T15:01:42.176408+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:27.568999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":608,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.195737+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T15:01:43.195737+0000","last_peered":"2024-09-13T15:01:43.195737+0000","last_clean":"2024-09-13T15:01:43.195737+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T15:01:43.195737+0000","last_undegraded":"2024-09-13T15:01:43.195737+0000","last_fullsized":"2024-09-13T15:01:43.195737+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":66,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.183091+0000","last_change":"2024-09-13T14:58:09.011981+0000","last_active":"2024-09-13T15:01:42.183091+0000","last_peered":"2024-09-13T15:01:42.183091+0000","last_clean":"2024-09-13T15:01:42.183091+0000","last_became_active":"2024-09-13T14:57:53.357674+0000","last_became_peered":"2024-09-13T14:57:53.357674+0000","last_unstale":"2024-09-13T15:01:42.183091+0000","last_undegraded":"2024-09-13T15:01:42.183091+0000","last_fullsized":"2024-09-13T15:01:42.183091+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.701912+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":352,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176265+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T15:01:42.176265+0000","last_peered":"2024-09-13T15:01:42.176265+0000","last_clean":"2024-09-13T15:01:42.176265+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T15:01:42.176265+0000","last_undegraded":"2024-09-13T15:01:42.176265+0000","last_fullsized":"2024-09-13T15:01:42.176265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":550,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.192984+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T15:01:43.192984+0000","last_peered":"2024-09-13T15:01:43.192984+0000","last_clean":"2024-09-13T15:01:43.192984+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T15:01:43.192984+0000","last_undegraded":"2024-09-13T15:01:43.192984+0000","last_fullsized":"2024-09-13T15:01:43.192984+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":72,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.184603+0000","last_change":"2024-09-13T14:58:10.513244+0000","last_active":"2024-09-13T15:01:42.184603+0000","last_peered":"2024-09-13T15:01:42.184603+0000","last_clean":"2024-09-13T15:01:42.184603+0000","last_became_active":"2024-09-13T14:58:10.511485+0000","last_became_peered":"2024-09-13T14:58:10.511485+0000","last_unstale":"2024-09-13T15:01:42.184603+0000","last_undegraded":"2024-09-13T15:01:42.184603+0000","last_fullsized":"2024-09-13T15:01:42.184603+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:23.283605+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":265,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.191125+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T15:01:43.191125+0000","last_peered":"2024-09-13T15:01:43.191125+0000","last_clean":"2024-09-13T15:01:43.191125+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T15:01:43.191125+0000","last_undegraded":"2024-09-13T15:01:43.191125+0000","last_fullsized":"2024-09-13T15:01:43.191125+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":441,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.178675+0000","last_change":"2024-09-13T14:58:10.536485+0000","last_active":"2024-09-13T15:01:42.178675+0000","last_peered":"2024-09-13T15:01:42.178675+0000","last_clean":"2024-09-13T15:01:42.178675+0000","last_became_active":"2024-09-13T14:58:10.536134+0000","last_became_peered":"2024-09-13T14:58:10.536134+0000","last_unstale":"2024-09-13T15:01:42.178675+0000","last_undegraded":"2024-09-13T15:01:42.178675+0000","last_fullsized":"2024-09-13T15:01:42.178675+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:59.095681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":69,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.175875+0000","last_change":"2024-09-13T14:58:05.212101+0000","last_active":"2024-09-13T15:01:42.175875+0000","last_peered":"2024-09-13T15:01:42.175875+0000","last_clean":"2024-09-13T15:01:42.175875+0000","last_became_active":"2024-09-13T14:57:25.275734+0000","last_became_peered":"2024-09-13T14:57:25.275734+0000","last_unstale":"2024-09-13T15:01:42.175875+0000","last_undegraded":"2024-09-13T15:01:42.175875+0000","last_fullsized":"2024-09-13T15:01:42.175875+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.212008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.212008+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:01.720211+0000","scrub_duration":0.0048969690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":590,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.191968+0000","last_change":"2024-09-13T14:58:10.515043+0000","last_active":"2024-09-13T15:01:43.191968+0000","last_peered":"2024-09-13T15:01:43.191968+0000","last_clean":"2024-09-13T15:01:43.191968+0000","last_became_active":"2024-09-13T14:58:10.513487+0000","last_became_peered":"2024-09-13T14:58:10.513487+0000","last_unstale":"2024-09-13T15:01:43.191968+0000","last_undegraded":"2024-09-13T15:01:43.191968+0000","last_fullsized":"2024-09-13T15:01:43.191968+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:11.392005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":353,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.181320+0000","last_change":"2024-09-13T14:57:53.364039+0000","last_active":"2024-09-13T15:01:42.181320+0000","last_peered":"2024-09-13T15:01:42.181320+0000","last_clean":"2024-09-13T15:01:42.181320+0000","last_became_active":"2024-09-13T14:57:53.363550+0000","last_became_peered":"2024-09-13T14:57:53.363550+0000","last_unstale":"2024-09-13T15:01:42.181320+0000","last_undegraded":"2024-09-13T15:01:42.181320+0000","last_fullsized":"2024-09-13T15:01:42.181320+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:36.171089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":95,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173913+0000","last_change":"2024-09-13T14:58:04.168950+0000","last_active":"2024-09-13T15:01:42.173913+0000","last_peered":"2024-09-13T15:01:42.173913+0000","last_clean":"2024-09-13T15:01:42.173913+0000","last_became_active":"2024-09-13T14:57:53.351301+0000","last_became_peered":"2024-09-13T14:57:53.351301+0000","last_unstale":"2024-09-13T15:01:42.173913+0000","last_undegraded":"2024-09-13T15:01:42.173913+0000","last_fullsized":"2024-09-13T15:01:42.173913+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:42.270600+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":551,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.191846+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T15:01:43.191846+0000","last_peered":"2024-09-13T15:01:43.191846+0000","last_clean":"2024-09-13T15:01:43.191846+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T15:01:43.191846+0000","last_undegraded":"2024-09-13T15:01:43.191846+0000","last_fullsized":"2024-09-13T15:01:43.191846+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":594,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.183222+0000","last_change":"2024-09-13T14:58:10.514433+0000","last_active":"2024-09-13T15:01:42.183222+0000","last_peered":"2024-09-13T15:01:42.183222+0000","last_clean":"2024-09-13T15:01:42.183222+0000","last_became_active":"2024-09-13T14:58:10.513446+0000","last_became_peered":"2024-09-13T14:58:10.513446+0000","last_unstale":"2024-09-13T15:01:42.183222+0000","last_undegraded":"2024-09-13T15:01:42.183222+0000","last_fullsized":"2024-09-13T15:01:42.183222+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:24.535893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":556,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.192087+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T15:01:43.192087+0000","last_peered":"2024-09-13T15:01:43.192087+0000","last_clean":"2024-09-13T15:01:43.192087+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T15:01:43.192087+0000","last_undegraded":"2024-09-13T15:01:43.192087+0000","last_fullsized":"2024-09-13T15:01:43.192087+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":52,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.183541+0000","last_change":"2024-09-13T14:57:53.350622+0000","last_active":"2024-09-13T15:01:42.183541+0000","last_peered":"2024-09-13T15:01:42.183541+0000","last_clean":"2024-09-13T15:01:42.183541+0000","last_became_active":"2024-09-13T14:57:53.350292+0000","last_became_peered":"2024-09-13T14:57:53.350292+0000","last_unstale":"2024-09-13T15:01:42.183541+0000","last_undegraded":"2024-09-13T15:01:42.183541+0000","last_fullsized":"2024-09-13T15:01:42.183541+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:16.088338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":310,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.195548+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T15:01:43.195548+0000","last_peered":"2024-09-13T15:01:43.195548+0000","last_clean":"2024-09-13T15:01:43.195548+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T15:01:43.195548+0000","last_undegraded":"2024-09-13T15:01:43.195548+0000","last_fullsized":"2024-09-13T15:01:43.195548+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:43:13.165444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":62,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.183331+0000","last_change":"2024-09-13T14:58:10.527352+0000","last_active":"2024-09-13T15:01:42.183331+0000","last_peered":"2024-09-13T15:01:42.183331+0000","last_clean":"2024-09-13T15:01:42.183331+0000","last_became_active":"2024-09-13T14:58:10.526614+0000","last_became_peered":"2024-09-13T14:58:10.526614+0000","last_unstale":"2024-09-13T15:01:42.183331+0000","last_undegraded":"2024-09-13T15:01:42.183331+0000","last_fullsized":"2024-09-13T15:01:42.183331+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:04.459255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":134,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173307+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T15:01:42.173307+0000","last_peered":"2024-09-13T15:01:42.173307+0000","last_clean":"2024-09-13T15:01:42.173307+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T15:01:42.173307+0000","last_undegraded":"2024-09-13T15:01:42.173307+0000","last_fullsized":"2024-09-13T15:01:42.173307+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":272,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.179473+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T15:01:42.179473+0000","last_peered":"2024-09-13T15:01:42.179473+0000","last_clean":"2024-09-13T15:01:42.179473+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T15:01:42.179473+0000","last_undegraded":"2024-09-13T15:01:42.179473+0000","last_fullsized":"2024-09-13T15:01:42.179473+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":136,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.188373+0000","last_change":"2024-09-13T14:58:10.511301+0000","last_active":"2024-09-13T15:01:43.188373+0000","last_peered":"2024-09-13T15:01:43.188373+0000","last_clean":"2024-09-13T15:01:43.188373+0000","last_became_active":"2024-09-13T14:58:10.510934+0000","last_became_peered":"2024-09-13T14:58:10.510934+0000","last_unstale":"2024-09-13T15:01:43.188373+0000","last_undegraded":"2024-09-13T15:01:43.188373+0000","last_fullsized":"2024-09-13T15:01:43.188373+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:00.007161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":340,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176255+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T15:01:42.176255+0000","last_peered":"2024-09-13T15:01:42.176255+0000","last_clean":"2024-09-13T15:01:42.176255+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T15:01:42.176255+0000","last_undegraded":"2024-09-13T15:01:42.176255+0000","last_fullsized":"2024-09-13T15:01:42.176255+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":333,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176646+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T15:01:42.176646+0000","last_peered":"2024-09-13T15:01:42.176646+0000","last_clean":"2024-09-13T15:01:42.176646+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T15:01:42.176646+0000","last_undegraded":"2024-09-13T15:01:42.176646+0000","last_fullsized":"2024-09-13T15:01:42.176646+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"371'4724","reported_seq":4753,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:51.028435+0000","last_change":"2024-09-13T15:00:55.405989+0000","last_active":"2024-09-13T15:01:51.028435+0000","last_peered":"2024-09-13T15:01:51.028435+0000","last_clean":"2024-09-13T15:01:51.028435+0000","last_became_active":"2024-09-13T15:00:55.405741+0000","last_became_peered":"2024-09-13T15:00:55.405741+0000","last_unstale":"2024-09-13T15:01:51.028435+0000","last_undegraded":"2024-09-13T15:01:51.028435+0000","last_fullsized":"2024-09-13T15:01:51.028435+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:32.041720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309592064,"num_objects":4724,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4724,"num_write_kb":302336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":336,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.177384+0000","last_change":"2024-09-13T14:57:53.351374+0000","last_active":"2024-09-13T15:01:42.177384+0000","last_peered":"2024-09-13T15:01:42.177384+0000","last_clean":"2024-09-13T15:01:42.177384+0000","last_became_active":"2024-09-13T14:57:53.350803+0000","last_became_peered":"2024-09-13T14:57:53.350803+0000","last_unstale":"2024-09-13T15:01:42.177384+0000","last_undegraded":"2024-09-13T15:01:42.177384+0000","last_fullsized":"2024-09-13T15:01:42.177384+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:30.535837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":366,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.181429+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T15:01:42.181429+0000","last_peered":"2024-09-13T15:01:42.181429+0000","last_clean":"2024-09-13T15:01:42.181429+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T15:01:42.181429+0000","last_undegraded":"2024-09-13T15:01:42.181429+0000","last_fullsized":"2024-09-13T15:01:42.181429+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":64,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.177514+0000","last_change":"2024-09-13T14:58:10.539542+0000","last_active":"2024-09-13T15:01:42.177514+0000","last_peered":"2024-09-13T15:01:42.177514+0000","last_clean":"2024-09-13T15:01:42.177514+0000","last_became_active":"2024-09-13T14:58:10.536848+0000","last_became_peered":"2024-09-13T14:58:10.536848+0000","last_unstale":"2024-09-13T15:01:42.177514+0000","last_undegraded":"2024-09-13T15:01:42.177514+0000","last_fullsized":"2024-09-13T15:01:42.177514+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:52.133867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":332,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.192528+0000","last_change":"2024-09-13T14:57:53.347772+0000","last_active":"2024-09-13T15:01:43.192528+0000","last_peered":"2024-09-13T15:01:43.192528+0000","last_clean":"2024-09-13T15:01:43.192528+0000","last_became_active":"2024-09-13T14:57:53.347373+0000","last_became_peered":"2024-09-13T14:57:53.347373+0000","last_unstale":"2024-09-13T15:01:43.192528+0000","last_undegraded":"2024-09-13T15:01:43.192528+0000","last_fullsized":"2024-09-13T15:01:43.192528+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:51.119909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"371'4624","reported_seq":4641,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.314727+0000","last_change":"2024-09-13T15:00:55.409140+0000","last_active":"2024-09-13T15:01:48.314727+0000","last_peered":"2024-09-13T15:01:48.314727+0000","last_clean":"2024-09-13T15:01:48.314727+0000","last_became_active":"2024-09-13T15:00:55.408820+0000","last_became_peered":"2024-09-13T15:00:55.408820+0000","last_unstale":"2024-09-13T15:01:48.314727+0000","last_undegraded":"2024-09-13T15:01:48.314727+0000","last_fullsized":"2024-09-13T15:01:48.314727+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:55.216821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303038464,"num_objects":4624,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4624,"num_write_kb":295936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":602,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.191878+0000","last_change":"2024-09-13T14:58:10.517480+0000","last_active":"2024-09-13T15:01:43.191878+0000","last_peered":"2024-09-13T15:01:43.191878+0000","last_clean":"2024-09-13T15:01:43.191878+0000","last_became_active":"2024-09-13T14:58:10.517178+0000","last_became_peered":"2024-09-13T14:58:10.517178+0000","last_unstale":"2024-09-13T15:01:43.191878+0000","last_undegraded":"2024-09-13T15:01:43.191878+0000","last_fullsized":"2024-09-13T15:01:43.191878+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:27.472859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":207,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.180475+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T15:01:42.180475+0000","last_peered":"2024-09-13T15:01:42.180475+0000","last_clean":"2024-09-13T15:01:42.180475+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T15:01:42.180475+0000","last_undegraded":"2024-09-13T15:01:42.180475+0000","last_fullsized":"2024-09-13T15:01:42.180475+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":594,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.193338+0000","last_change":"2024-09-13T14:58:10.532248+0000","last_active":"2024-09-13T15:01:43.193338+0000","last_peered":"2024-09-13T15:01:43.193338+0000","last_clean":"2024-09-13T15:01:43.193338+0000","last_became_active":"2024-09-13T14:58:10.522929+0000","last_became_peered":"2024-09-13T14:58:10.522929+0000","last_unstale":"2024-09-13T15:01:43.193338+0000","last_undegraded":"2024-09-13T15:01:43.193338+0000","last_fullsized":"2024-09-13T15:01:43.193338+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:38:40.572917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":174,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.179924+0000","last_change":"2024-09-13T14:57:56.132062+0000","last_active":"2024-09-13T15:01:42.179924+0000","last_peered":"2024-09-13T15:01:42.179924+0000","last_clean":"2024-09-13T15:01:42.179924+0000","last_became_active":"2024-09-13T14:57:19.218181+0000","last_became_peered":"2024-09-13T14:57:19.218181+0000","last_unstale":"2024-09-13T15:01:42.179924+0000","last_undegraded":"2024-09-13T15:01:42.179924+0000","last_fullsized":"2024-09-13T15:01:42.179924+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:56.131976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:56.131976+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:31.359825+0000","scrub_duration":0.0061470450000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"371'4728","reported_seq":4757,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:51.021521+0000","last_change":"2024-09-13T15:00:55.873094+0000","last_active":"2024-09-13T15:01:51.021521+0000","last_peered":"2024-09-13T15:01:51.021521+0000","last_clean":"2024-09-13T15:01:51.021521+0000","last_became_active":"2024-09-13T15:00:55.872743+0000","last_became_peered":"2024-09-13T15:00:55.872743+0000","last_unstale":"2024-09-13T15:01:51.021521+0000","last_undegraded":"2024-09-13T15:01:51.021521+0000","last_fullsized":"2024-09-13T15:01:51.021521+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4728,"log_dups_size":0,"ondisk_log_size":4728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:46.267062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309854208,"num_objects":4728,"num_object_clones":0,"num_object_copies":14184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4728,"num_write_kb":302592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":579,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.184459+0000","last_change":"2024-09-13T14:58:10.521645+0000","last_active":"2024-09-13T15:01:42.184459+0000","last_peered":"2024-09-13T15:01:42.184459+0000","last_clean":"2024-09-13T15:01:42.184459+0000","last_became_active":"2024-09-13T14:58:10.521223+0000","last_became_peered":"2024-09-13T14:58:10.521223+0000","last_unstale":"2024-09-13T15:01:42.184459+0000","last_undegraded":"2024-09-13T15:01:42.184459+0000","last_fullsized":"2024-09-13T15:01:42.184459+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:18.930422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":356,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.174038+0000","last_change":"2024-09-13T14:58:10.524720+0000","last_active":"2024-09-13T15:01:42.174038+0000","last_peered":"2024-09-13T15:01:42.174038+0000","last_clean":"2024-09-13T15:01:42.174038+0000","last_became_active":"2024-09-13T14:58:10.522828+0000","last_became_peered":"2024-09-13T14:58:10.522828+0000","last_unstale":"2024-09-13T15:01:42.174038+0000","last_undegraded":"2024-09-13T15:01:42.174038+0000","last_fullsized":"2024-09-13T15:01:42.174038+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:57.737026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":581,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.195649+0000","last_change":"2024-09-13T14:57:53.351358+0000","last_active":"2024-09-13T15:01:43.195649+0000","last_peered":"2024-09-13T15:01:43.195649+0000","last_clean":"2024-09-13T15:01:43.195649+0000","last_became_active":"2024-09-13T14:57:53.351021+0000","last_became_peered":"2024-09-13T14:57:53.351021+0000","last_unstale":"2024-09-13T15:01:43.195649+0000","last_undegraded":"2024-09-13T15:01:43.195649+0000","last_fullsized":"2024-09-13T15:01:43.195649+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.067202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":337,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.180283+0000","last_change":"2024-09-13T14:58:10.515266+0000","last_active":"2024-09-13T15:01:42.180283+0000","last_peered":"2024-09-13T15:01:42.180283+0000","last_clean":"2024-09-13T15:01:42.180283+0000","last_became_active":"2024-09-13T14:58:10.514858+0000","last_became_peered":"2024-09-13T14:58:10.514858+0000","last_unstale":"2024-09-13T15:01:42.180283+0000","last_undegraded":"2024-09-13T15:01:42.180283+0000","last_fullsized":"2024-09-13T15:01:42.180283+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:55.856922+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"371'4610","reported_seq":4639,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:47.697581+0000","last_change":"2024-09-13T15:00:55.405198+0000","last_active":"2024-09-13T15:01:47.697581+0000","last_peered":"2024-09-13T15:01:47.697581+0000","last_clean":"2024-09-13T15:01:47.697581+0000","last_became_active":"2024-09-13T15:00:55.404898+0000","last_became_peered":"2024-09-13T15:00:55.404898+0000","last_unstale":"2024-09-13T15:01:47.697581+0000","last_undegraded":"2024-09-13T15:01:47.697581+0000","last_fullsized":"2024-09-13T15:01:47.697581+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:42.847939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302120960,"num_objects":4610,"num_object_clones":0,"num_object_copies":13830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4610,"num_write_kb":295040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":52,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.183756+0000","last_change":"2024-09-13T14:57:53.351087+0000","last_active":"2024-09-13T15:01:42.183756+0000","last_peered":"2024-09-13T15:01:42.183756+0000","last_clean":"2024-09-13T15:01:42.183756+0000","last_became_active":"2024-09-13T14:57:53.350744+0000","last_became_peered":"2024-09-13T14:57:53.350744+0000","last_unstale":"2024-09-13T15:01:42.183756+0000","last_undegraded":"2024-09-13T15:01:42.183756+0000","last_fullsized":"2024-09-13T15:01:42.183756+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:43.300957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":114,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176264+0000","last_change":"2024-09-13T14:58:10.521049+0000","last_active":"2024-09-13T15:01:42.176264+0000","last_peered":"2024-09-13T15:01:42.176264+0000","last_clean":"2024-09-13T15:01:42.176264+0000","last_became_active":"2024-09-13T14:58:10.520663+0000","last_became_peered":"2024-09-13T14:58:10.520663+0000","last_unstale":"2024-09-13T15:01:42.176264+0000","last_undegraded":"2024-09-13T15:01:42.176264+0000","last_fullsized":"2024-09-13T15:01:42.176264+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:59.410821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":52,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.183851+0000","last_change":"2024-09-13T14:57:53.347982+0000","last_active":"2024-09-13T15:01:42.183851+0000","last_peered":"2024-09-13T15:01:42.183851+0000","last_clean":"2024-09-13T15:01:42.183851+0000","last_became_active":"2024-09-13T14:57:53.347623+0000","last_became_peered":"2024-09-13T14:57:53.347623+0000","last_unstale":"2024-09-13T15:01:42.183851+0000","last_undegraded":"2024-09-13T15:01:42.183851+0000","last_fullsized":"2024-09-13T15:01:42.183851+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:15.734246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":347,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.181006+0000","last_change":"2024-09-13T14:58:05.197923+0000","last_active":"2024-09-13T15:01:42.181006+0000","last_peered":"2024-09-13T15:01:42.181006+0000","last_clean":"2024-09-13T15:01:42.181006+0000","last_became_active":"2024-09-13T14:57:19.208023+0000","last_became_peered":"2024-09-13T14:57:19.208023+0000","last_unstale":"2024-09-13T15:01:42.181006+0000","last_undegraded":"2024-09-13T15:01:42.181006+0000","last_fullsized":"2024-09-13T15:01:42.181006+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:06.994746+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"371'4668","reported_seq":4685,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.303733+0000","last_change":"2024-09-13T15:00:55.874092+0000","last_active":"2024-09-13T15:01:48.303733+0000","last_peered":"2024-09-13T15:01:48.303733+0000","last_clean":"2024-09-13T15:01:48.303733+0000","last_became_active":"2024-09-13T15:00:55.873709+0000","last_became_peered":"2024-09-13T15:00:55.873709+0000","last_unstale":"2024-09-13T15:01:48.303733+0000","last_undegraded":"2024-09-13T15:01:48.303733+0000","last_fullsized":"2024-09-13T15:01:48.303733+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.813547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305922048,"num_objects":4668,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4668,"num_write_kb":298752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":479,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.195807+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T15:01:43.195807+0000","last_peered":"2024-09-13T15:01:43.195807+0000","last_clean":"2024-09-13T15:01:43.195807+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T15:01:43.195807+0000","last_undegraded":"2024-09-13T15:01:43.195807+0000","last_fullsized":"2024-09-13T15:01:43.195807+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":344,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176781+0000","last_change":"2024-09-13T14:58:10.516308+0000","last_active":"2024-09-13T15:01:42.176781+0000","last_peered":"2024-09-13T15:01:42.176781+0000","last_clean":"2024-09-13T15:01:42.176781+0000","last_became_active":"2024-09-13T14:58:10.515894+0000","last_became_peered":"2024-09-13T14:58:10.515894+0000","last_unstale":"2024-09-13T15:01:42.176781+0000","last_undegraded":"2024-09-13T15:01:42.176781+0000","last_fullsized":"2024-09-13T15:01:42.176781+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:17.962091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":113,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.188499+0000","last_change":"2024-09-13T14:58:28.428183+0000","last_active":"2024-09-13T15:01:43.188499+0000","last_peered":"2024-09-13T15:01:43.188499+0000","last_clean":"2024-09-13T15:01:43.188499+0000","last_became_active":"2024-09-13T14:58:28.427553+0000","last_became_peered":"2024-09-13T14:58:28.427553+0000","last_unstale":"2024-09-13T15:01:43.188499+0000","last_undegraded":"2024-09-13T15:01:43.188499+0000","last_fullsized":"2024-09-13T15:01:43.188499+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:59.108147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":80,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.191228+0000","last_change":"2024-09-13T14:58:28.424704+0000","last_active":"2024-09-13T15:01:43.191228+0000","last_peered":"2024-09-13T15:01:43.191228+0000","last_clean":"2024-09-13T15:01:43.191228+0000","last_became_active":"2024-09-13T14:58:28.424375+0000","last_became_peered":"2024-09-13T14:58:28.424375+0000","last_unstale":"2024-09-13T15:01:43.191228+0000","last_undegraded":"2024-09-13T15:01:43.191228+0000","last_fullsized":"2024-09-13T15:01:43.191228+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:03:16.604948+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.e","version":"371'4591","reported_seq":4607,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.388290+0000","last_change":"2024-09-13T15:00:55.408720+0000","last_active":"2024-09-13T15:01:48.388290+0000","last_peered":"2024-09-13T15:01:48.388290+0000","last_clean":"2024-09-13T15:01:48.388290+0000","last_became_active":"2024-09-13T15:00:55.408379+0000","last_became_peered":"2024-09-13T15:00:55.408379+0000","last_unstale":"2024-09-13T15:01:48.388290+0000","last_undegraded":"2024-09-13T15:01:48.388290+0000","last_fullsized":"2024-09-13T15:01:48.388290+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:24.296460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300875776,"num_objects":4591,"num_object_clones":0,"num_object_copies":13773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":293824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":327,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.188911+0000","last_change":"2024-09-13T14:58:10.519956+0000","last_active":"2024-09-13T15:01:43.188911+0000","last_peered":"2024-09-13T15:01:43.188911+0000","last_clean":"2024-09-13T15:01:43.188911+0000","last_became_active":"2024-09-13T14:58:10.519409+0000","last_became_peered":"2024-09-13T14:58:10.519409+0000","last_unstale":"2024-09-13T15:01:43.188911+0000","last_undegraded":"2024-09-13T15:01:43.188911+0000","last_fullsized":"2024-09-13T15:01:43.188911+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:32.082107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":420,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.191067+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T15:01:43.191067+0000","last_peered":"2024-09-13T15:01:43.191067+0000","last_clean":"2024-09-13T15:01:43.191067+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T15:01:43.191067+0000","last_undegraded":"2024-09-13T15:01:43.191067+0000","last_fullsized":"2024-09-13T15:01:43.191067+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":617,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.191740+0000","last_change":"2024-09-13T14:58:10.511063+0000","last_active":"2024-09-13T15:01:43.191740+0000","last_peered":"2024-09-13T15:01:43.191740+0000","last_clean":"2024-09-13T15:01:43.191740+0000","last_became_active":"2024-09-13T14:58:10.510571+0000","last_became_peered":"2024-09-13T14:58:10.510571+0000","last_unstale":"2024-09-13T15:01:43.191740+0000","last_undegraded":"2024-09-13T15:01:43.191740+0000","last_fullsized":"2024-09-13T15:01:43.191740+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:20.803368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":138,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173724+0000","last_change":"2024-09-13T14:58:10.523618+0000","last_active":"2024-09-13T15:01:42.173724+0000","last_peered":"2024-09-13T15:01:42.173724+0000","last_clean":"2024-09-13T15:01:42.173724+0000","last_became_active":"2024-09-13T14:58:10.522166+0000","last_became_peered":"2024-09-13T14:58:10.522166+0000","last_unstale":"2024-09-13T15:01:42.173724+0000","last_undegraded":"2024-09-13T15:01:42.173724+0000","last_fullsized":"2024-09-13T15:01:42.173724+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:14.151008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.f","version":"371'4661","reported_seq":4675,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.338376+0000","last_change":"2024-09-13T15:00:55.408205+0000","last_active":"2024-09-13T15:01:48.338376+0000","last_peered":"2024-09-13T15:01:48.338376+0000","last_clean":"2024-09-13T15:01:48.338376+0000","last_became_active":"2024-09-13T15:00:55.407801+0000","last_became_peered":"2024-09-13T15:00:55.407801+0000","last_unstale":"2024-09-13T15:01:48.338376+0000","last_undegraded":"2024-09-13T15:01:48.338376+0000","last_fullsized":"2024-09-13T15:01:48.338376+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:36.295309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305463296,"num_objects":4661,"num_object_clones":0,"num_object_copies":13983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4661,"num_write_kb":298304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":183,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.190468+0000","last_change":"2024-09-13T14:58:10.525829+0000","last_active":"2024-09-13T15:01:43.190468+0000","last_peered":"2024-09-13T15:01:43.190468+0000","last_clean":"2024-09-13T15:01:43.190468+0000","last_became_active":"2024-09-13T14:58:10.525316+0000","last_became_peered":"2024-09-13T14:58:10.525316+0000","last_unstale":"2024-09-13T15:01:43.190468+0000","last_undegraded":"2024-09-13T15:01:43.190468+0000","last_fullsized":"2024-09-13T15:01:43.190468+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:41.689562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":428,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.181537+0000","last_change":"2024-09-13T14:58:10.517828+0000","last_active":"2024-09-13T15:01:42.181537+0000","last_peered":"2024-09-13T15:01:42.181537+0000","last_clean":"2024-09-13T15:01:42.181537+0000","last_became_active":"2024-09-13T14:58:10.517009+0000","last_became_peered":"2024-09-13T14:58:10.517009+0000","last_unstale":"2024-09-13T15:01:42.181537+0000","last_undegraded":"2024-09-13T15:01:42.181537+0000","last_fullsized":"2024-09-13T15:01:42.181537+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:29.728706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":393,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.180434+0000","last_change":"2024-09-13T14:58:10.517445+0000","last_active":"2024-09-13T15:01:42.180434+0000","last_peered":"2024-09-13T15:01:42.180434+0000","last_clean":"2024-09-13T15:01:42.180434+0000","last_became_active":"2024-09-13T14:58:10.517101+0000","last_became_peered":"2024-09-13T14:58:10.517101+0000","last_unstale":"2024-09-13T15:01:42.180434+0000","last_undegraded":"2024-09-13T15:01:42.180434+0000","last_fullsized":"2024-09-13T15:01:42.180434+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:35.589919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":307,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.179712+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T15:01:42.179712+0000","last_peered":"2024-09-13T15:01:42.179712+0000","last_clean":"2024-09-13T15:01:42.179712+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T15:01:42.179712+0000","last_undegraded":"2024-09-13T15:01:42.179712+0000","last_fullsized":"2024-09-13T15:01:42.179712+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.c","version":"371'4625","reported_seq":4642,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.343941+0000","last_change":"2024-09-13T15:00:55.873540+0000","last_active":"2024-09-13T15:01:48.343941+0000","last_peered":"2024-09-13T15:01:48.343941+0000","last_clean":"2024-09-13T15:01:48.343941+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:01:48.343941+0000","last_undegraded":"2024-09-13T15:01:48.343941+0000","last_fullsized":"2024-09-13T15:01:48.343941+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:49.820345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303104000,"num_objects":4625,"num_object_clones":0,"num_object_copies":13875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":296000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":316,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.180525+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T15:01:42.180525+0000","last_peered":"2024-09-13T15:01:42.180525+0000","last_clean":"2024-09-13T15:01:42.180525+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T15:01:42.180525+0000","last_undegraded":"2024-09-13T15:01:42.180525+0000","last_fullsized":"2024-09-13T15:01:42.180525+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":556,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.192120+0000","last_change":"2024-09-13T14:57:53.351036+0000","last_active":"2024-09-13T15:01:43.192120+0000","last_peered":"2024-09-13T15:01:43.192120+0000","last_clean":"2024-09-13T15:01:43.192120+0000","last_became_active":"2024-09-13T14:57:53.350553+0000","last_became_peered":"2024-09-13T14:57:53.350553+0000","last_unstale":"2024-09-13T15:01:43.192120+0000","last_undegraded":"2024-09-13T15:01:43.192120+0000","last_fullsized":"2024-09-13T15:01:43.192120+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:17.354351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":56,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.184553+0000","last_change":"2024-09-13T14:57:53.360296+0000","last_active":"2024-09-13T15:01:42.184553+0000","last_peered":"2024-09-13T15:01:42.184553+0000","last_clean":"2024-09-13T15:01:42.184553+0000","last_became_active":"2024-09-13T14:57:53.347710+0000","last_became_peered":"2024-09-13T14:57:53.347710+0000","last_unstale":"2024-09-13T15:01:42.184553+0000","last_undegraded":"2024-09-13T15:01:42.184553+0000","last_fullsized":"2024-09-13T15:01:42.184553+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:43.099884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":138,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.180099+0000","last_change":"2024-09-13T14:58:10.521307+0000","last_active":"2024-09-13T15:01:42.180099+0000","last_peered":"2024-09-13T15:01:42.180099+0000","last_clean":"2024-09-13T15:01:42.180099+0000","last_became_active":"2024-09-13T14:58:10.520905+0000","last_became_peered":"2024-09-13T14:58:10.520905+0000","last_unstale":"2024-09-13T15:01:42.180099+0000","last_undegraded":"2024-09-13T15:01:42.180099+0000","last_fullsized":"2024-09-13T15:01:42.180099+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:23.693312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"371'4466","reported_seq":4483,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.327145+0000","last_change":"2024-09-13T15:00:55.409383+0000","last_active":"2024-09-13T15:01:48.327145+0000","last_peered":"2024-09-13T15:01:48.327145+0000","last_clean":"2024-09-13T15:01:48.327145+0000","last_became_active":"2024-09-13T15:00:55.407034+0000","last_became_peered":"2024-09-13T15:00:55.407034+0000","last_unstale":"2024-09-13T15:01:48.327145+0000","last_undegraded":"2024-09-13T15:01:48.327145+0000","last_fullsized":"2024-09-13T15:01:48.327145+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:26.464719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292683776,"num_objects":4466,"num_object_clones":0,"num_object_copies":13398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4466,"num_write_kb":285824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":265,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.190063+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T15:01:43.190063+0000","last_peered":"2024-09-13T15:01:43.190063+0000","last_clean":"2024-09-13T15:01:43.190063+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T15:01:43.190063+0000","last_undegraded":"2024-09-13T15:01:43.190063+0000","last_fullsized":"2024-09-13T15:01:43.190063+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":553,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.191991+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T15:01:43.191991+0000","last_peered":"2024-09-13T15:01:43.191991+0000","last_clean":"2024-09-13T15:01:43.191991+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T15:01:43.191991+0000","last_undegraded":"2024-09-13T15:01:43.191991+0000","last_fullsized":"2024-09-13T15:01:43.191991+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":106,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.190454+0000","last_change":"2024-09-13T14:58:28.428608+0000","last_active":"2024-09-13T15:01:43.190454+0000","last_peered":"2024-09-13T15:01:43.190454+0000","last_clean":"2024-09-13T15:01:43.190454+0000","last_became_active":"2024-09-13T14:58:28.428282+0000","last_became_peered":"2024-09-13T14:58:28.428282+0000","last_unstale":"2024-09-13T15:01:43.190454+0000","last_undegraded":"2024-09-13T15:01:43.190454+0000","last_fullsized":"2024-09-13T15:01:43.190454+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.603606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.603606+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:10.695417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.7","version":"371'4618","reported_seq":4635,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.378668+0000","last_change":"2024-09-13T15:00:55.411034+0000","last_active":"2024-09-13T15:01:48.378668+0000","last_peered":"2024-09-13T15:01:48.378668+0000","last_clean":"2024-09-13T15:01:48.378668+0000","last_became_active":"2024-09-13T15:00:55.410636+0000","last_became_peered":"2024-09-13T15:00:55.410636+0000","last_unstale":"2024-09-13T15:01:48.378668+0000","last_undegraded":"2024-09-13T15:01:48.378668+0000","last_fullsized":"2024-09-13T15:01:48.378668+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:42.688532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302645248,"num_objects":4618,"num_object_clones":0,"num_object_copies":13854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":295552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":353,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.177681+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T15:01:42.177681+0000","last_peered":"2024-09-13T15:01:42.177681+0000","last_clean":"2024-09-13T15:01:42.177681+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T15:01:42.177681+0000","last_undegraded":"2024-09-13T15:01:42.177681+0000","last_fullsized":"2024-09-13T15:01:42.177681+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":334,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.177770+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T15:01:42.177770+0000","last_peered":"2024-09-13T15:01:42.177770+0000","last_clean":"2024-09-13T15:01:42.177770+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T15:01:42.177770+0000","last_undegraded":"2024-09-13T15:01:42.177770+0000","last_fullsized":"2024-09-13T15:01:42.177770+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":603,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.184487+0000","last_change":"2024-09-13T14:58:10.512659+0000","last_active":"2024-09-13T15:01:42.184487+0000","last_peered":"2024-09-13T15:01:42.184487+0000","last_clean":"2024-09-13T15:01:42.184487+0000","last_became_active":"2024-09-13T14:58:10.511997+0000","last_became_peered":"2024-09-13T14:58:10.511997+0000","last_unstale":"2024-09-13T15:01:42.184487+0000","last_undegraded":"2024-09-13T15:01:42.184487+0000","last_fullsized":"2024-09-13T15:01:42.184487+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.415299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.415299+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:39.431834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"371'4664","reported_seq":4681,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.343786+0000","last_change":"2024-09-13T15:00:55.873173+0000","last_active":"2024-09-13T15:01:48.343786+0000","last_peered":"2024-09-13T15:01:48.343786+0000","last_clean":"2024-09-13T15:01:48.343786+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:01:48.343786+0000","last_undegraded":"2024-09-13T15:01:48.343786+0000","last_fullsized":"2024-09-13T15:01:48.343786+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:27.254765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305659904,"num_objects":4664,"num_object_clones":0,"num_object_copies":13992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":298496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":350,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.180104+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T15:01:42.180104+0000","last_peered":"2024-09-13T15:01:42.180104+0000","last_clean":"2024-09-13T15:01:42.180104+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T15:01:42.180104+0000","last_undegraded":"2024-09-13T15:01:42.180104+0000","last_fullsized":"2024-09-13T15:01:42.180104+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":354,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.179546+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T15:01:42.179546+0000","last_peered":"2024-09-13T15:01:42.179546+0000","last_clean":"2024-09-13T15:01:42.179546+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T15:01:42.179546+0000","last_undegraded":"2024-09-13T15:01:42.179546+0000","last_fullsized":"2024-09-13T15:01:42.179546+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":161,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.190917+0000","last_change":"2024-09-13T14:58:28.424238+0000","last_active":"2024-09-13T15:01:43.190917+0000","last_peered":"2024-09-13T15:01:43.190917+0000","last_clean":"2024-09-13T15:01:43.190917+0000","last_became_active":"2024-09-13T14:58:28.423844+0000","last_became_peered":"2024-09-13T14:58:28.423844+0000","last_unstale":"2024-09-13T15:01:43.190917+0000","last_undegraded":"2024-09-13T15:01:43.190917+0000","last_fullsized":"2024-09-13T15:01:43.190917+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:22.942492+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"371'4564","reported_seq":4581,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.361889+0000","last_change":"2024-09-13T15:00:55.407846+0000","last_active":"2024-09-13T15:01:48.361889+0000","last_peered":"2024-09-13T15:01:48.361889+0000","last_clean":"2024-09-13T15:01:48.361889+0000","last_became_active":"2024-09-13T15:00:55.407115+0000","last_became_peered":"2024-09-13T15:00:55.407115+0000","last_unstale":"2024-09-13T15:01:48.361889+0000","last_undegraded":"2024-09-13T15:01:48.361889+0000","last_fullsized":"2024-09-13T15:01:48.361889+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:29.545511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299106304,"num_objects":4564,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4564,"num_write_kb":292096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":333,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176071+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T15:01:42.176071+0000","last_peered":"2024-09-13T15:01:42.176071+0000","last_clean":"2024-09-13T15:01:42.176071+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T15:01:42.176071+0000","last_undegraded":"2024-09-13T15:01:42.176071+0000","last_fullsized":"2024-09-13T15:01:42.176071+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":593,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.193314+0000","last_change":"2024-09-13T14:58:10.522983+0000","last_active":"2024-09-13T15:01:43.193314+0000","last_peered":"2024-09-13T15:01:43.193314+0000","last_clean":"2024-09-13T15:01:43.193314+0000","last_became_active":"2024-09-13T14:58:10.513377+0000","last_became_peered":"2024-09-13T14:58:10.513377+0000","last_unstale":"2024-09-13T15:01:43.193314+0000","last_undegraded":"2024-09-13T15:01:43.193314+0000","last_fullsized":"2024-09-13T15:01:43.193314+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:12.579644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":174,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173303+0000","last_change":"2024-09-13T14:57:57.223728+0000","last_active":"2024-09-13T15:01:42.173303+0000","last_peered":"2024-09-13T15:01:42.173303+0000","last_clean":"2024-09-13T15:01:42.173303+0000","last_became_active":"2024-09-13T14:57:19.233883+0000","last_became_peered":"2024-09-13T14:57:19.233883+0000","last_unstale":"2024-09-13T15:01:42.173303+0000","last_undegraded":"2024-09-13T15:01:42.173303+0000","last_fullsized":"2024-09-13T15:01:42.173303+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:41.520060+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2","version":"371'4505","reported_seq":4533,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:47.704095+0000","last_change":"2024-09-13T15:00:55.410530+0000","last_active":"2024-09-13T15:01:47.704095+0000","last_peered":"2024-09-13T15:01:47.704095+0000","last_clean":"2024-09-13T15:01:47.704095+0000","last_became_active":"2024-09-13T15:00:55.410331+0000","last_became_peered":"2024-09-13T15:00:55.410331+0000","last_unstale":"2024-09-13T15:01:47.704095+0000","last_undegraded":"2024-09-13T15:01:47.704095+0000","last_fullsized":"2024-09-13T15:01:47.704095+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:24.475954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295239680,"num_objects":4505,"num_object_clones":0,"num_object_copies":13515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4505,"num_write_kb":288320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":102,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.175614+0000","last_change":"2024-09-13T14:58:10.513713+0000","last_active":"2024-09-13T15:01:42.175614+0000","last_peered":"2024-09-13T15:01:42.175614+0000","last_clean":"2024-09-13T15:01:42.175614+0000","last_became_active":"2024-09-13T14:58:10.513334+0000","last_became_peered":"2024-09-13T14:58:10.513334+0000","last_unstale":"2024-09-13T15:01:42.175614+0000","last_undegraded":"2024-09-13T15:01:42.175614+0000","last_fullsized":"2024-09-13T15:01:42.175614+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:41.326218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":184,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.192831+0000","last_change":"2024-09-13T14:58:28.428028+0000","last_active":"2024-09-13T15:01:43.192831+0000","last_peered":"2024-09-13T15:01:43.192831+0000","last_clean":"2024-09-13T15:01:43.192831+0000","last_became_active":"2024-09-13T14:58:28.427598+0000","last_became_peered":"2024-09-13T14:58:28.427598+0000","last_unstale":"2024-09-13T15:01:43.192831+0000","last_undegraded":"2024-09-13T15:01:43.192831+0000","last_fullsized":"2024-09-13T15:01:43.192831+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:32:50.930142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":174,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176158+0000","last_change":"2024-09-13T14:58:00.146439+0000","last_active":"2024-09-13T15:01:42.176158+0000","last_peered":"2024-09-13T15:01:42.176158+0000","last_clean":"2024-09-13T15:01:42.176158+0000","last_became_active":"2024-09-13T14:57:19.206097+0000","last_became_peered":"2024-09-13T14:57:19.206097+0000","last_unstale":"2024-09-13T15:01:42.176158+0000","last_undegraded":"2024-09-13T15:01:42.176158+0000","last_fullsized":"2024-09-13T15:01:42.176158+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:00.146384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:00.146384+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:09.697538+0000","scrub_duration":0.0014649660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"371'4479","reported_seq":4496,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.318564+0000","last_change":"2024-09-13T15:00:55.407630+0000","last_active":"2024-09-13T15:01:48.318564+0000","last_peered":"2024-09-13T15:01:48.318564+0000","last_clean":"2024-09-13T15:01:48.318564+0000","last_became_active":"2024-09-13T15:00:55.407307+0000","last_became_peered":"2024-09-13T15:00:55.407307+0000","last_unstale":"2024-09-13T15:01:48.318564+0000","last_undegraded":"2024-09-13T15:01:48.318564+0000","last_fullsized":"2024-09-13T15:01:48.318564+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:54.284626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293535744,"num_objects":4479,"num_object_clones":0,"num_object_copies":13437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4479,"num_write_kb":286656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":356,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.179400+0000","last_change":"2024-09-13T14:57:35.357982+0000","last_active":"2024-09-13T15:01:42.179400+0000","last_peered":"2024-09-13T15:01:42.179400+0000","last_clean":"2024-09-13T15:01:42.179400+0000","last_became_active":"2024-09-13T14:57:35.357618+0000","last_became_peered":"2024-09-13T14:57:35.357618+0000","last_unstale":"2024-09-13T15:01:42.179400+0000","last_undegraded":"2024-09-13T15:01:42.179400+0000","last_fullsized":"2024-09-13T15:01:42.179400+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:50.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":375,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173637+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:01:42.173637+0000","last_peered":"2024-09-13T15:01:42.173637+0000","last_clean":"2024-09-13T15:01:42.173637+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:01:42.173637+0000","last_undegraded":"2024-09-13T15:01:42.173637+0000","last_fullsized":"2024-09-13T15:01:42.173637+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":327,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.181107+0000","last_change":"2024-09-13T14:57:53.363216+0000","last_active":"2024-09-13T15:01:42.181107+0000","last_peered":"2024-09-13T15:01:42.181107+0000","last_clean":"2024-09-13T15:01:42.181107+0000","last_became_active":"2024-09-13T14:57:53.362679+0000","last_became_peered":"2024-09-13T14:57:53.362679+0000","last_unstale":"2024-09-13T15:01:42.181107+0000","last_undegraded":"2024-09-13T15:01:42.181107+0000","last_fullsized":"2024-09-13T15:01:42.181107+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.885757+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"371'4573","reported_seq":4590,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.388122+0000","last_change":"2024-09-13T15:00:55.873177+0000","last_active":"2024-09-13T15:01:48.388122+0000","last_peered":"2024-09-13T15:01:48.388122+0000","last_clean":"2024-09-13T15:01:48.388122+0000","last_became_active":"2024-09-13T15:00:55.872769+0000","last_became_peered":"2024-09-13T15:00:55.872769+0000","last_unstale":"2024-09-13T15:01:48.388122+0000","last_undegraded":"2024-09-13T15:01:48.388122+0000","last_fullsized":"2024-09-13T15:01:48.388122+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:22.869519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299696128,"num_objects":4573,"num_object_clones":0,"num_object_copies":13719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4573,"num_write_kb":292672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":360,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.177590+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T15:01:42.177590+0000","last_peered":"2024-09-13T15:01:42.177590+0000","last_clean":"2024-09-13T15:01:42.177590+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T15:01:42.177590+0000","last_undegraded":"2024-09-13T15:01:42.177590+0000","last_fullsized":"2024-09-13T15:01:42.177590+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":337,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.178752+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T15:01:42.178752+0000","last_peered":"2024-09-13T15:01:42.178752+0000","last_clean":"2024-09-13T15:01:42.178752+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T15:01:42.178752+0000","last_undegraded":"2024-09-13T15:01:42.178752+0000","last_fullsized":"2024-09-13T15:01:42.178752+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":352,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.192637+0000","last_change":"2024-09-13T14:58:10.531563+0000","last_active":"2024-09-13T15:01:43.192637+0000","last_peered":"2024-09-13T15:01:43.192637+0000","last_clean":"2024-09-13T15:01:43.192637+0000","last_became_active":"2024-09-13T14:58:10.515595+0000","last_became_peered":"2024-09-13T14:58:10.515595+0000","last_unstale":"2024-09-13T15:01:43.192637+0000","last_undegraded":"2024-09-13T15:01:43.192637+0000","last_fullsized":"2024-09-13T15:01:43.192637+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:33.797005+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"371'4547","reported_seq":4564,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:48.363666+0000","last_change":"2024-09-13T15:00:55.405838+0000","last_active":"2024-09-13T15:01:48.363666+0000","last_peered":"2024-09-13T15:01:48.363666+0000","last_clean":"2024-09-13T15:01:48.363666+0000","last_became_active":"2024-09-13T15:00:55.405095+0000","last_became_peered":"2024-09-13T15:00:55.405095+0000","last_unstale":"2024-09-13T15:01:48.363666+0000","last_undegraded":"2024-09-13T15:01:48.363666+0000","last_fullsized":"2024-09-13T15:01:48.363666+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:55.315749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297992192,"num_objects":4547,"num_object_clones":0,"num_object_copies":13641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4547,"num_write_kb":291008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":52,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.182957+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:01:42.182957+0000","last_peered":"2024-09-13T15:01:42.182957+0000","last_clean":"2024-09-13T15:01:42.182957+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:01:42.182957+0000","last_undegraded":"2024-09-13T15:01:42.182957+0000","last_fullsized":"2024-09-13T15:01:42.182957+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":146,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173577+0000","last_change":"2024-09-13T14:58:10.530803+0000","last_active":"2024-09-13T15:01:42.173577+0000","last_peered":"2024-09-13T15:01:42.173577+0000","last_clean":"2024-09-13T15:01:42.173577+0000","last_became_active":"2024-09-13T14:58:10.522078+0000","last_became_peered":"2024-09-13T14:58:10.522078+0000","last_unstale":"2024-09-13T15:01:42.173577+0000","last_undegraded":"2024-09-13T15:01:42.173577+0000","last_fullsized":"2024-09-13T15:01:42.173577+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:50.807687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":158,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173413+0000","last_change":"2024-09-13T14:57:55.208230+0000","last_active":"2024-09-13T15:01:42.173413+0000","last_peered":"2024-09-13T15:01:42.173413+0000","last_clean":"2024-09-13T15:01:42.173413+0000","last_became_active":"2024-09-13T14:57:19.211671+0000","last_became_peered":"2024-09-13T14:57:19.211671+0000","last_unstale":"2024-09-13T15:01:42.173413+0000","last_undegraded":"2024-09-13T15:01:42.173413+0000","last_fullsized":"2024-09-13T15:01:42.173413+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.066938+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":442,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.179696+0000","last_change":"2024-09-13T14:58:10.529854+0000","last_active":"2024-09-13T15:01:42.179696+0000","last_peered":"2024-09-13T15:01:42.179696+0000","last_clean":"2024-09-13T15:01:42.179696+0000","last_became_active":"2024-09-13T14:58:10.514813+0000","last_became_peered":"2024-09-13T14:58:10.514813+0000","last_unstale":"2024-09-13T15:01:42.179696+0000","last_undegraded":"2024-09-13T15:01:42.179696+0000","last_fullsized":"2024-09-13T15:01:42.179696+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:46.296441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":591,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.193213+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T15:01:43.193213+0000","last_peered":"2024-09-13T15:01:43.193213+0000","last_clean":"2024-09-13T15:01:43.193213+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T15:01:43.193213+0000","last_undegraded":"2024-09-13T15:01:43.193213+0000","last_fullsized":"2024-09-13T15:01:43.193213+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":52,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.183647+0000","last_change":"2024-09-13T14:57:53.356010+0000","last_active":"2024-09-13T15:01:42.183647+0000","last_peered":"2024-09-13T15:01:42.183647+0000","last_clean":"2024-09-13T15:01:42.183647+0000","last_became_active":"2024-09-13T14:57:53.354979+0000","last_became_peered":"2024-09-13T14:57:53.354979+0000","last_unstale":"2024-09-13T15:01:42.183647+0000","last_undegraded":"2024-09-13T15:01:42.183647+0000","last_fullsized":"2024-09-13T15:01:42.183647+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:38.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":338,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173442+0000","last_change":"2024-09-13T14:57:35.358990+0000","last_active":"2024-09-13T15:01:42.173442+0000","last_peered":"2024-09-13T15:01:42.173442+0000","last_clean":"2024-09-13T15:01:42.173442+0000","last_became_active":"2024-09-13T14:57:35.358547+0000","last_became_peered":"2024-09-13T14:57:35.358547+0000","last_unstale":"2024-09-13T15:01:42.173442+0000","last_undegraded":"2024-09-13T15:01:42.173442+0000","last_fullsized":"2024-09-13T15:01:42.173442+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:45.386301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":470,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.193932+0000","last_change":"2024-09-13T14:58:10.536776+0000","last_active":"2024-09-13T15:01:43.193932+0000","last_peered":"2024-09-13T15:01:43.193932+0000","last_clean":"2024-09-13T15:01:43.193932+0000","last_became_active":"2024-09-13T14:58:10.536376+0000","last_became_peered":"2024-09-13T14:58:10.536376+0000","last_unstale":"2024-09-13T15:01:43.193932+0000","last_undegraded":"2024-09-13T15:01:43.193932+0000","last_fullsized":"2024-09-13T15:01:43.193932+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:08.237256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":310,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T15:01:43.192935+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T15:01:43.192935+0000","last_peered":"2024-09-13T15:01:43.192935+0000","last_clean":"2024-09-13T15:01:43.192935+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T15:01:43.192935+0000","last_undegraded":"2024-09-13T15:01:43.192935+0000","last_fullsized":"2024-09-13T15:01:43.192935+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:57.647782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":439,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173529+0000","last_change":"2024-09-13T14:58:10.524047+0000","last_active":"2024-09-13T15:01:42.173529+0000","last_peered":"2024-09-13T15:01:42.173529+0000","last_clean":"2024-09-13T15:01:42.173529+0000","last_became_active":"2024-09-13T14:58:10.522007+0000","last_became_peered":"2024-09-13T14:58:10.522007+0000","last_unstale":"2024-09-13T15:01:42.173529+0000","last_undegraded":"2024-09-13T15:01:42.173529+0000","last_fullsized":"2024-09-13T15:01:42.173529+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:41.957759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":229,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.173489+0000","last_change":"2024-09-13T14:58:28.426617+0000","last_active":"2024-09-13T15:01:42.173489+0000","last_peered":"2024-09-13T15:01:42.173489+0000","last_clean":"2024-09-13T15:01:42.173489+0000","last_became_active":"2024-09-13T14:58:28.425773+0000","last_became_peered":"2024-09-13T14:58:28.425773+0000","last_unstale":"2024-09-13T15:01:42.173489+0000","last_undegraded":"2024-09-13T15:01:42.173489+0000","last_fullsized":"2024-09-13T15:01:42.173489+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:59.659871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":60,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.180236+0000","last_change":"2024-09-13T14:58:28.428896+0000","last_active":"2024-09-13T15:01:42.180236+0000","last_peered":"2024-09-13T15:01:42.180236+0000","last_clean":"2024-09-13T15:01:42.180236+0000","last_became_active":"2024-09-13T14:58:28.428522+0000","last_became_peered":"2024-09-13T14:58:28.428522+0000","last_unstale":"2024-09-13T15:01:42.180236+0000","last_undegraded":"2024-09-13T15:01:42.180236+0000","last_fullsized":"2024-09-13T15:01:42.180236+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:05.418586+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":353,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.176432+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T15:01:42.176432+0000","last_peered":"2024-09-13T15:01:42.176432+0000","last_clean":"2024-09-13T15:01:42.176432+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T15:01:42.176432+0000","last_undegraded":"2024-09-13T15:01:42.176432+0000","last_fullsized":"2024-09-13T15:01:42.176432+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.182771+0000","last_change":"2024-09-13T15:01:29.854791+0000","last_active":"2024-09-13T15:01:42.182771+0000","last_peered":"2024-09-13T15:01:42.182771+0000","last_clean":"2024-09-13T15:01:42.182771+0000","last_became_active":"2024-09-13T15:01:29.854455+0000","last_became_peered":"2024-09-13T15:01:29.854455+0000","last_unstale":"2024-09-13T15:01:42.182771+0000","last_undegraded":"2024-09-13T15:01:42.182771+0000","last_fullsized":"2024-09-13T15:01:42.182771+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:25.057250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":569,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T15:01:42.185477+0000","last_change":"2024-09-13T14:58:10.512480+0000","last_active":"2024-09-13T15:01:42.185477+0000","last_peered":"2024-09-13T15:01:42.185477+0000","last_clean":"2024-09-13T15:01:42.185477+0000","last_became_active":"2024-09-13T14:58:10.511589+0000","last_became_peered":"2024-09-13T14:58:10.511589+0000","last_unstale":"2024-09-13T15:01:42.185477+0000","last_undegraded":"2024-09-13T15:01:42.185477+0000","last_fullsized":"2024-09-13T15:01:42.185477+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:52.302190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":4826529792,"num_objects":73647,"num_object_clones":0,"num_object_copies":220941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73647,"num_write_kb":4713408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14523760640,"data_stored":14523760640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73647,"ondisk_log_size":73647,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":448,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365305,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2997312,"kb_used_data":1479672,"kb_used_omap":1,"kb_used_meta":1517630,"kb_avail":90747840,"statfs":{"total":95995035648,"available":92925788160,"internally_reserved":0,"allocated":1515184128,"data_stored":1512983700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1554053464},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561535,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3240952,"kb_used_data":1752880,"kb_used_omap":1,"kb_used_meta":1488062,"kb_avail":90504200,"statfs":{"total":95995035648,"available":92676300800,"internally_reserved":0,"allocated":1794949120,"data_stored":1792756733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1523775832},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790467,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4093804,"kb_used_data":2599076,"kb_used_omap":1,"kb_used_meta":1494718,"kb_avail":89651348,"statfs":{"total":95995035648,"available":91802980352,"internally_reserved":0,"allocated":2661453824,"data_stored":2659269273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1530591576},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986695,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3773944,"kb_used_data":2591152,"kb_used_omap":1,"kb_used_meta":1182782,"kb_avail":89971208,"statfs":{"total":95995035648,"available":92130516992,"internally_reserved":0,"allocated":2653339648,"data_stored":2651147029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1211169112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519850,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2376360,"kb_used_data":1432608,"kb_used_omap":4,"kb_used_meta":943739,"kb_avail":91368792,"statfs":{"total":95995035648,"available":93561643008,"internally_reserved":0,"allocated":1466990592,"data_stored":1464809603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":966389184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304766,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3690096,"kb_used_data":2751016,"kb_used_omap":6,"kb_used_meta":939065,"kb_avail":90055056,"statfs":{"total":95995035648,"available":92216377344,"internally_reserved":0,"allocated":2817040384,"data_stored":2814852261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":961603015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828069,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2646128,"kb_used_data":1482280,"kb_used_omap":5,"kb_used_meta":1163834,"kb_avail":91099024,"statfs":{"total":95995035648,"available":93285400576,"internally_reserved":0,"allocated":1517854720,"data_stored":1515665532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1191766252},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":356482285934,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1513750528,"data_stored":1513750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2827878400,"data_stored":2827878400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1474166784,"data_stored":1474166784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2688483328,"data_stored":2688483328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2692546560,"data_stored":2692546560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1811873792,"data_stored":1811873792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1515061248,"data_stored":1515061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:01:54.331 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T15:01:54.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-09-13T15:01:54.331 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:54.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.382 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:54.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:54 smithi019 ceph-mon[91296]: from='client.47015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:54.483 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:54.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:54.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:54.666 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:54.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:54 smithi003 ceph-mon[107278]: from='client.47015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:54 smithi003 ceph-mon[104007]: from='client.47015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:54.768 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:54.801 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:54.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:54.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:54.962 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 80051 80035 108.727 58.75 0.014142 0.00919211 2024-09-13T15:01:55.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:55.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:55.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:55.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:55.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:55.313 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:55 smithi019 ceph-mon[91296]: pgmap v1330: 119 pgs: 119 active+clean; 4.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T15:01:55.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:55.461 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:55.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:55.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:55 smithi003 ceph-mon[104007]: pgmap v1330: 119 pgs: 119 active+clean; 4.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T15:01:55.520 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:55 smithi003 ceph-mon[107278]: pgmap v1330: 119 pgs: 119 active+clean; 4.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T15:01:55.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:55.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:55.669 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:55.770 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:55.821 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:55.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:55.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:55.964 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 81156 81140 107.878 69.0625 0.023767 0.00926404 2024-09-13T15:01:56.052 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:01:56.053 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:01:56.053 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:01:56.053 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:01:56.053 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:01:56.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:56.209 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:56.254 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:56.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:56.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:56.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:56.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:56.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:56.716 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:01:56.716 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:56.742 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:01:56.743 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:56.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:56.779 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:01:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:56.808 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:01:56.809 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:56.813 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:56.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:56.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:01:56.843 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:01:56.844 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:56.874 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:01:56.874 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:56.900 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:01:56.900 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:56.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:56.930 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:01:56.931 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:56.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:56.965 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 82270 82254 107.081 69.625 0.0170286 0.00933398 2024-09-13T15:01:57.057 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:57.115 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:57.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:57.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:57.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:57.372 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:01:57.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:57 smithi003 ceph-mon[104007]: pgmap v1331: 119 pgs: 119 active+clean; 4.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T15:01:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:57 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4183627491' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:57 smithi003 ceph-mon[107278]: pgmap v1331: 119 pgs: 119 active+clean; 4.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T15:01:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:57 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4183627491' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:57.480 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:57.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:57.510 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:57 smithi019 ceph-mon[91296]: pgmap v1331: 119 pgs: 119 active+clean; 4.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T15:01:57.510 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:57 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4183627491' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:57.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:57.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:57.741 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:57.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:57.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:57.965 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 83316 83300 106.229 65.375 0.0118882 0.00940872 2024-09-13T15:01:58.031 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:58.071 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:58.173 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:58.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:58.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.363 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:58.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:58.464 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:58.505 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:58.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:58.508 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:58.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:58.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:58.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:58.878 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:58.922 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:58.965 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 84448 84432 105.52 70.75 0.00643472 0.00947162 2024-09-13T15:01:59.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:59.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:59.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:59.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:59.289 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:59.332 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:59.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:59.336 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:01:59 smithi019 ceph-mon[91296]: pgmap v1332: 119 pgs: 119 active+clean; 4.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:01:59.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:01:59 smithi003 ceph-mon[107278]: pgmap v1332: 119 pgs: 119 active+clean; 4.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:01:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:01:59 smithi003 ceph-mon[104007]: pgmap v1332: 119 pgs: 119 active+clean; 4.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:01:59.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:59.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:59.563 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.609 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:59.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:59.710 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.747 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:01:59.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:59.786 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:01:59.786 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:01:59.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:59.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:01:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:59.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:01:59.965 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 85518 85502 104.761 66.875 0.0165309 0.00954057 2024-09-13T15:01:59.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:00.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:00.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:00.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:00.282 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:02:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:00.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:00 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4215328688' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:00.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:00 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:00.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:02:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:02:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:00.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:00 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4215328688' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:00.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:00 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:00 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4215328688' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:00.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:00 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:00.450 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:00.486 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:02:00.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:00.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:00.687 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:00.740 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:00.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.890 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:00.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:00.965 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 86551 86535 103.988 64.5625 0.00910803 0.00961072 2024-09-13T15:02:00.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:01.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:01.019 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:01.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:01.174 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:01.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:01.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:01 smithi019 ceph-mon[91296]: pgmap v1333: 119 pgs: 119 active+clean; 5.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:02:01.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:01 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:01.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:01 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:02:01.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:01 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:01.415 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:01 smithi019 ceph-mon[91296]: osdmap e373: 8 total, 8 up, 7 in 2024-09-13T15:02:01.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:01.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:01.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[107278]: pgmap v1333: 119 pgs: 119 active+clean; 5.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:02:01.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:01.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:02:01.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:01.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[107278]: osdmap e373: 8 total, 8 up, 7 in 2024-09-13T15:02:01.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[104007]: pgmap v1333: 119 pgs: 119 active+clean; 5.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:02:01.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:01.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:02:01.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:01.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:01 smithi003 ceph-mon[104007]: osdmap e373: 8 total, 8 up, 7 in 2024-09-13T15:02:01.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:01.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:01.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:01.608 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:01.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:01.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:01.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:01.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:01.893 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-09-13T15:02:01.966 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 87614 87598 103.279 66.4375 0.011213 0.00967723 2024-09-13T15:02:01.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:01.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:02.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:02.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:02.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:02.279 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:02.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:02.450 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:02.540 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:02 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/216258291' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T15:02:02.540 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:02 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T15:02:02.552 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.590 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:02.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:02.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:02 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/216258291' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T15:02:02.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:02 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T15:02:02.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:02 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/216258291' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T15:02:02.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:02 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T15:02:02.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:02.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:02.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:02.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:02.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:02.965 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 88647 88631 102.562 64.5625 0.0146901 0.00974539 2024-09-13T15:02:02.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:03.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:03.123 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:03.225 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:03.265 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:03.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:02:03.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:03.406 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:03.508 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.554 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:03.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:03.594 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:03 smithi003 ceph-mon[104007]: pgmap v1335: 119 pgs: 119 active+clean; 5.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:02:03.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:03 smithi003 ceph-mon[107278]: pgmap v1335: 119 pgs: 119 active+clean; 5.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:02:03.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:03.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:03 smithi019 ceph-mon[91296]: pgmap v1335: 119 pgs: 119 active+clean; 5.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:02:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:03.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:03.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:03.918 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:02:03.918 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:02:03.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:03.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:03.966 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 89609 89593 101.791 60.125 0.0752274 0.00981795 2024-09-13T15:02:04.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:04.087 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:04.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:04.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:04.289 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:02:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:02:04] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-13T15:02:04.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:04.360 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:04.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:04.462 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.509 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:04.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:04.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:04.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:04.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:04 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:02:04.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:04 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:02:04.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:04 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:02:04.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:04.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:04.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:04.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:04.966 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 90665 90649 101.152 66 0.00527191 0.00988075 2024-09-13T15:02:05.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:05.072 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:05.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.211 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:05.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:02:05.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:05.349 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:05.374 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:05.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:05.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:05.580 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:05.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:05 smithi019 ceph-mon[91296]: pgmap v1336: 119 pgs: 119 active+clean; 5.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:02:05.619 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:05 smithi003 ceph-mon[107278]: pgmap v1336: 119 pgs: 119 active+clean; 5.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:02:05.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:05 smithi003 ceph-mon[104007]: pgmap v1336: 119 pgs: 119 active+clean; 5.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:02:05.625 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:05.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:05.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:05.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:05.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:05.922 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:02:05.922 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:05.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:02:05.944 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:02:05.944 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:05.962 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:02:05.962 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:05.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 91688 91672 100.497 63.9375 0.0148355 0.00994483 2024-09-13T15:02:05.984 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:02:05.984 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:05.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:06.018 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:02:06.018 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:06.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:06.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:02:06.049 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:06.080 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:02:06.080 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:06.112 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:02:06.112 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:06.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:06.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:06.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:06.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:06.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:06.405 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:06.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:06 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1778497090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:06.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:06 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1778497090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:06.451 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:06.556 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:06.591 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:06 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1778497090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.610 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:06.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:06.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:06.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:06.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:06.885 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:06.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 92622 92606 99.7708 58.375 0.0115386 0.0100179 2024-09-13T15:02:06.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:07.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:07.118 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:07.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:07.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:07.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:07.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:07.386 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.432 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:07.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:07.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.592 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:07.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:07.609 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:07 smithi019 ceph-mon[91296]: pgmap v1337: 119 pgs: 119 active+clean; 5.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:02:07.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:07 smithi003 ceph-mon[104007]: pgmap v1337: 119 pgs: 119 active+clean; 5.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:02:07.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:07 smithi003 ceph-mon[107278]: pgmap v1337: 119 pgs: 119 active+clean; 5.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:02:07.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:07.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:07.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:07.861 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:07.963 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:07.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 93484 93468 98.993 53.875 0.00536672 0.0100975 2024-09-13T15:02:08.005 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:08.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:08.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:08.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:08.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:08.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:08.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:08.508 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:08.552 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:08.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:08.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:08.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:08.838 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:08.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:08.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T15:02:08.966861+0000 min lat: 0.00155173 max lat: 0.263214 avg lat: 0.01016 2024-09-13T15:02:08.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:02:08.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 94473 94457 98.3733 61.8125 0.0122619 0.01016 2024-09-13T15:02:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:09.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:09.120 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:09.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:09.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:09.263 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:09.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:09.389 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:02:09.389 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:02:09.389 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:02:09.389 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:02:09.390 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:02:09.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:09.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:09.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:09.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:09.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:09 smithi019 ceph-mon[91296]: pgmap v1338: 119 pgs: 119 active+clean; 5.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:02:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:09 smithi003 ceph-mon[104007]: pgmap v1338: 119 pgs: 119 active+clean; 5.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:02:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:09 smithi003 ceph-mon[107278]: pgmap v1338: 119 pgs: 119 active+clean; 5.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:02:09.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:09.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:09.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:09.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:09.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 95477 95461 97.7893 62.75 0.0129581 0.010221 2024-09-13T15:02:10.030 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:10.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:10.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:10.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:02:10.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:10.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:10.309 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:02:10.309 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:02:10.309 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-13T15:02:10.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd in 0 2024-09-13T15:02:10.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:10.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:10.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:10.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:10.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:10.658 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:10.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:02:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:02:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:10.760 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:10.800 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:10.902 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:10.937 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:10.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 96485 96469 97.2281 63 0.0146954 0.0102798 2024-09-13T15:02:11.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:11.082 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:11.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:11.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:11.338 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:02:11.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:11 smithi003 ceph-mon[107278]: pgmap v1339: 119 pgs: 119 active+clean; 5.7 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T15:02:11.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3636737784' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:11.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:11 smithi003 ceph-mon[104007]: pgmap v1339: 119 pgs: 119 active+clean; 5.7 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T15:02:11.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3636737784' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:11.364 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:11.466 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:11.510 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:11.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:11.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:11.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:11 smithi019 ceph-mon[91296]: pgmap v1339: 119 pgs: 119 active+clean; 5.7 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T15:02:11.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3636737784' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:11.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:11.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:11.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:11.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:11.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 97416 97400 96.6084 58.1875 0.00861894 0.0103462 2024-09-13T15:02:12.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:12.044 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:12.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.191 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:12.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:12.246 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:02:12.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:12.368 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:12.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.546 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:12.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:12.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:12.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:12 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:12 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3636737784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:12 smithi003 ceph-mon[107278]: osdmap e374: 8 total, 8 up, 7 in 2024-09-13T15:02:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:12 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:12 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3636737784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:12 smithi003 ceph-mon[104007]: osdmap e374: 8 total, 8 up, 7 in 2024-09-13T15:02:12.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:12 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:12.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:12 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3636737784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:12.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:12 smithi019 ceph-mon[91296]: osdmap e374: 8 total, 8 up, 7 in 2024-09-13T15:02:12.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:12.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:12.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:12.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:12.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 15 98254 98239 95.9183 52.4375 0.015029 0.0104205 2024-09-13T15:02:13.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:13.080 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:13.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:13.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:13.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:13.349 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:13.359 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:13 smithi019 ceph-mon[91296]: pgmap v1341: 119 pgs: 119 active+clean; 5.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T15:02:13.359 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:13.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:13.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:13.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:13.654 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:13 smithi003 ceph-mon[104007]: pgmap v1341: 119 pgs: 119 active+clean; 5.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T15:02:13.654 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:13.655 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:13 smithi003 ceph-mon[107278]: pgmap v1341: 119 pgs: 119 active+clean; 5.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T15:02:13.655 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:13.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:13.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:13.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:13.887 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:13.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 99148 99132 95.3013 55.8125 0.0103521 0.0104868 2024-09-13T15:02:13.988 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:13.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:14.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:14.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:02:14.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:14.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:14.261 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:14.296 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:14.400 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:02:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:02:14] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-13T15:02:14.444 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:14.546 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.591 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:14.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:14.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:14.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:14.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:14.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:14.953 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:14.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 100023 100007 94.6859 54.6875 0.0166152 0.0105552 2024-09-13T15:02:14.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:15.100 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.154 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:15.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:15.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:15.292 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:15.394 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:15.437 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:15.539 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:15.589 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:15 smithi019 ceph-mon[91296]: pgmap v1342: 119 pgs: 119 active+clean; 6.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-13T15:02:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:15 smithi003 ceph-mon[107278]: pgmap v1342: 119 pgs: 119 active+clean; 6.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-13T15:02:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:15 smithi003 ceph-mon[104007]: pgmap v1342: 119 pgs: 119 active+clean; 6.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-13T15:02:15.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:15.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:15.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:15.859 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:15.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:15.967 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 100959 100943 94.1458 58.5 0.0335522 0.0106162 2024-09-13T15:02:16.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.036 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:16.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:16.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:16.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:16.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:16.282 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:16.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:16.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:16.418 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:16.444 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:16.488 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:16.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:16.590 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:16.633 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:16.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:16.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:16.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:16.874 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:02:16.874 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:02:16.874 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:02:16.874 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:02:16.874 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.874 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:206935 11.17 11:eb394a87:::benchmark_data_smithi003_229782_object84555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:00.480826+0000", 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 436.21539068700002, 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24575330300000001, 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 206935 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480826+0000", 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480826+0000", 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480824+0000", 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:16.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480830+0000", 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.19e-06 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480844+0000", 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3473e-05 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480852+0000", 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7149999999999998e-06 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.480912+0000", 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0019999999999998e-05 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.481041+0000", 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012961299999999999 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.481660+0000", 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00061894800000000002 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.485143+0000", 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0034825670000000002 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.485718+0000", 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00057511300000000003 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.726493+0000", 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24077558399999999 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.726540+0000", 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.655e-05 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.726579+0000", 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8927999999999997e-05 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:206941 11.17 11:eceecd00:::benchmark_data_smithi003_229782_object84561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:00.486544+0000", 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 436.20967191099999, 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25040380499999998, 2024-09-13T15:02:16.881 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 206941 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486544+0000", 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486544+0000", 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486543+0000", 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486549+0000", 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9009999999999999e-06 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.884 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.884 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.884 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486561+0000", 2024-09-13T15:02:16.884 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1782e-05 2024-09-13T15:02:16.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.884 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.884 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.884 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486568+0000", 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2189999999999998e-06 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486612+0000", 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4319e-05 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.486707+0000", 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.4728999999999994e-05 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.487203+0000", 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00049568500000000005 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.489672+0000", 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024689030000000002 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.490789+0000", 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0011174679999999999 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.886 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.736847+0000", 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24605777300000001 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.736902+0000", 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4867000000000002e-05 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.736948+0000", 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6579000000000002e-05 2024-09-13T15:02:16.887 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:206964 11.17 11:e838aa8e:::benchmark_data_smithi003_229782_object84584:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:00.511574+0000", 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 436.18464205399999, 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22550974500000001, 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T15:02:16.888 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 206964 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511574+0000", 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511574+0000", 2024-09-13T15:02:16.889 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511572+0000", 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511581+0000", 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8599999999999999e-06 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511597+0000", 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5214e-05 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511604+0000", 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0790000000000003e-06 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511660+0000", 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5927000000000002e-05 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.511792+0000", 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013238400000000001 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.512364+0000", 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00057206399999999995 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.515783+0000", 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003419329 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.516623+0000", 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00083948099999999999 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.737001+0000", 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.220378661 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.737048+0000", 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6328000000000001e-05 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.737084+0000", 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6198000000000001e-05 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:206983 11.17 11:ec4b1388:::benchmark_data_smithi003_229782_object84603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:00.546413+0000", 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 436.14980307899998, 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19765461000000001, 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 206983 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546413+0000", 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546413+0000", 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546412+0000", 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546418+0000", 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4670000000000001e-06 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546430+0000", 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.199e-05 2024-09-13T15:02:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546435+0000", 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9849999999999997e-06 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546472+0000", 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.663e-05 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.897 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.546560+0000", 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7880000000000006e-05 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.547044+0000", 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00048424699999999999 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.549255+0000", 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002211373 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.549881+0000", 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00062549199999999995 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.743981+0000", 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19410002600000001 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.744030+0000", 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.918e-05 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:00.744068+0000", 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7672000000000001e-05 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211651 11.7 11:e6ef7c31:::benchmark_data_smithi003_229782_object89271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.899729+0000", 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 430.796486899, 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.45350560400000001, 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.900 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211651 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899729+0000", 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899729+0000", 2024-09-13T15:02:16.901 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899728+0000", 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899736+0000", 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1620000000000008e-06 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899754+0000", 2024-09-13T15:02:16.902 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8436000000000001e-05 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899762+0000", 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4619999999999999e-06 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899813+0000", 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1476e-05 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899943+0000", 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012961900000000001 2024-09-13T15:02:16.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.900568+0000", 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00062517599999999997 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.903464+0000", 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028962689999999999 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.904 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.905949+0000", 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024846270000000001 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353141+0000", 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.447192647 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353193+0000", 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1270999999999998e-05 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.905 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353235+0000", 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2355999999999999e-05 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211672 11.7 11:e4d90238:::benchmark_data_smithi003_229782_object89292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.933359+0000", 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 430.762857747, 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.420019643, 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.906 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211672 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933359+0000", 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933359+0000", 2024-09-13T15:02:16.907 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933356+0000", 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933366+0000", 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7610000000000002e-06 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.908 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933383+0000", 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7181999999999999e-05 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933390+0000", 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0990000000000001e-06 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933445+0000", 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4756999999999998e-05 2024-09-13T15:02:16.909 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.933541+0000", 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6026000000000005e-05 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.934144+0000", 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00060253300000000004 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.936675+0000", 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002530942 2024-09-13T15:02:16.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.937326+0000", 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00065163500000000002 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353302+0000", 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.415976026 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353342+0000", 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9235e-05 2024-09-13T15:02:16.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353378+0000", 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6594000000000002e-05 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:235437 11.7 11:e4bc60de:::benchmark_data_smithi003_229782_object113057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:33.150334+0000", 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 403.54588267399998, 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.256778852, 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.912 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 235437 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150334+0000", 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.913 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150334+0000", 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150332+0000", 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150339+0000", 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3730000000000004e-06 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.914 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150356+0000", 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6878e-05 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150362+0000", 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.066e-06 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150413+0000", 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1004999999999997e-05 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.150560+0000", 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000146976 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.151193+0000", 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00063234200000000004 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.154946+0000", 2024-09-13T15:02:16.916 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0037530139999999998 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.155346+0000", 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00039992899999999999 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.407006+0000", 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25166077399999998 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.917 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.407065+0000", 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8998e-05 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:33.407112+0000", 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7182000000000003e-05 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:129077 12.c 12:3872950b:::benchmark_data_smithi003_259359_object13451:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:02:16.918 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:09.507607+0000", 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 247.18860905899999, 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1289123550000002, 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 129077 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507607+0000", 2024-09-13T15:02:16.919 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507607+0000", 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507606+0000", 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.920 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507612+0000", 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1920000000000002e-06 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507623+0000", 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1005999999999999e-05 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507629+0000", 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3430000000000001e-06 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507728+0000", 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8444999999999997e-05 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507737+0000", 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2250000000000006e-06 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502519+0000", 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.994782265 2024-09-13T15:02:16.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502536+0000", 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6582000000000001e-05 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.636437+0000", 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.133900761 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.636520+0000", 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3005999999999998e-05 2024-09-13T15:02:16.923 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:129078 12.c 12:30d637ad:::benchmark_data_smithi003_259359_object13452:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:09.507648+0000", 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 247.188568186, 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.128952795, 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 129078 2024-09-13T15:02:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507648+0000", 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507648+0000", 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507648+0000", 2024-09-13T15:02:16.925 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507651+0000", 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.016e-06 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507658+0000", 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.063e-06 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.926 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507662+0000", 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0330000000000002e-06 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507775+0000", 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011317899999999999 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507782+0000", 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5660000000000001e-06 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502564+0000", 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99478159799999999 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502572+0000", 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2989999999999997e-06 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.636544+0000", 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.13397163 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.636601+0000", 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6943e-05 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:129087 12.c 12:39a0c657:::benchmark_data_smithi003_259359_object13461:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:09.507685+0000", 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 247.188531168, 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1540226750000002, 2024-09-13T15:02:16.929 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 129087 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507685+0000", 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.930 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507685+0000", 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507684+0000", 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507689+0000", 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0119999999999996e-06 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.931 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507699+0000", 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8889999999999997e-06 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507706+0000", 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2969999999999998e-06 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507814+0000", 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000107311 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507821+0000", 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0659999999999998e-06 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502600+0000", 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99477939199999998 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502608+0000", 2024-09-13T15:02:16.933 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1289999999999996e-06 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.636623+0000", 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1340147840000001 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.636753+0000", 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013038000000000001 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2,7", 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.636872+0000", 2024-09-13T15:02:16.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011815 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.648817+0000", 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011945132000000001 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.648853+0000", 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6328000000000002e-05 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.935 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.649018+0000", 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000164606 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.661514+0000", 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012496027 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.661588+0000", 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.381e-05 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.661708+0000", 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012029200000000001 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:129105 12.c 12:36be9a40:::benchmark_data_smithi003_259359_object13479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:09.528831+0000", 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 247.16738503400001, 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1331466400000001, 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.937 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 129105 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.528831+0000", 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.528831+0000", 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.938 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.528830+0000", 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.528837+0000", 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.289e-06 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.528852+0000", 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5432000000000001e-05 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.939 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.528859+0000", 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.489e-06 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.528927+0000", 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8194999999999993e-05 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.528938+0000", 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1423999999999999e-05 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.940 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502636+0000", 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.973697217 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502643+0000", 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8189999999999992e-06 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.640485+0000", 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.13784153 2024-09-13T15:02:16.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.640616+0000", 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000130653 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2,7", 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.640739+0000", 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012334600000000001 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.649025+0000", 2024-09-13T15:02:16.942 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0082856479999999996 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.653012+0000", 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039874309999999996 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.653068+0000", 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5537e-05 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.661779+0000", 2024-09-13T15:02:16.943 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.008711234 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.661863+0000", 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4666999999999994e-05 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.661978+0000", 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000114468 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.944 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:129109 12.c 12:3ae60709:::benchmark_data_smithi003_259359_object13483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:09.531789+0000", 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 247.16442721000001, 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1304340430000002, 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 129109 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.945 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.531789+0000", 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.531789+0000", 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.531788+0000", 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.946 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.531794+0000", 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.959e-06 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.531809+0000", 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5277999999999998e-05 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.531816+0000", 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6980000000000001e-06 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.531881+0000", 2024-09-13T15:02:16.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5326000000000004e-05 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.531897+0000", 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5458999999999999e-05 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502674+0000", 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97077726399999997 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.502682+0000", 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9200000000000004e-06 2024-09-13T15:02:16.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.644469+0000", 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.141786798 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.644595+0000", 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012648800000000001 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2,7", 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.644739+0000", 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014339499999999999 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.649030+0000", 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0042917700000000003 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.653042+0000", 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0040112519999999999 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.653092+0000", 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0380000000000002e-05 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.662034+0000", 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0089417239999999999 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.662109+0000", 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5443000000000003e-05 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.662223+0000", 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000113995 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.46703.0:174745 12.c e348/339 12:3b1b4974:::benchmark_data_smithi003_259359_object59119:head v 348'11012, mlcod=348'11009)", 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:09.380077+0000", 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 187.316139279, 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30390309300000001, 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.380077+0000", 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.380077+0000", 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.380076+0000", 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.380083+0000", 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6370000000000002e-06 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.380092+0000", 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1940000000000004e-06 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.380097+0000", 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0359999999999997e-06 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.380140+0000", 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2605999999999998e-05 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.380164+0000", 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4403999999999999e-05 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.683953+0000", 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30378894400000001 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.683980+0000", 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7178e-05 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.955 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.46703.0:174761 12.c e348/339 12:3225c534:::benchmark_data_smithi003_259359_object59135:head v 348'11013, mlcod=348'11010)", 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:09.410361+0000", 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 187.28585535100001, 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27681715400000001, 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410361+0000", 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410361+0000", 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410359+0000", 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410371+0000", 2024-09-13T15:02:16.957 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1263e-05 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.678766+0000", 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26839568400000002 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.678778+0000", 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.168e-05 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.681248+0000", 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024696549999999999 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.681262+0000", 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4419e-05 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.687154+0000", 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0058915039999999997 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.687178+0000", 2024-09-13T15:02:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4462999999999999e-05 2024-09-13T15:02:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.960 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:76394 13.9 13:98d1d47d:::benchmark_data_smithi003_293363_object76393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:51.035510+0000", 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 25.660706179000002, 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13207186700000001, 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 76394 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.961 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035510+0000", 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035510+0000", 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035510+0000", 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035542+0000", 2024-09-13T15:02:16.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2233000000000001e-05 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035550+0000", 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4290000000000001e-06 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035555+0000", 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6929999999999998e-06 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.963 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035697+0000", 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014195399999999999 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035800+0000", 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000103602 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035933+0000", 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013245499999999999 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.038952+0000", 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00301943 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.041250+0000", 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022972800000000001 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.167469+0000", 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.126219205 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.167543+0000", 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4518999999999996e-05 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.167582+0000", 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8692000000000002e-05 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:76395 13.c 13:35117def:::benchmark_data_smithi003_293363_object76394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:51.035571+0000", 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 25.660644973, 2024-09-13T15:02:16.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13205798199999999, 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 76395 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035571+0000", 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035571+0000", 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035571+0000", 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035626+0000", 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5689000000000002e-05 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035635+0000", 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4009999999999997e-06 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035640+0000", 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3480000000000003e-06 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.038380+0000", 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027402529999999998 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.038491+0000", 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011061900000000001 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.038612+0000", 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120895 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.041106+0000", 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024941059999999998 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.042727+0000", 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0016204959999999999 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.164797+0000", 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.122070003 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.167598+0000", 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028010719999999999 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.167629+0000", 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1680000000000002e-05 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.36909.0:76747 13.3 e372/362 13:c4893afa:::benchmark_data_smithi003_293363_object76746:head v 372'4691, mlcod=372'4689)", 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:51.457522+0000", 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 25.238694439, 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.135021319, 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.972 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.457522+0000", 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.457522+0000", 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.457521+0000", 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.457545+0000", 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3693000000000002e-05 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.457556+0000", 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0774000000000001e-05 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.457561+0000", 2024-09-13T15:02:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6620000000000002e-06 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.457631+0000", 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0030000000000005e-05 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.457649+0000", 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7445e-05 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.592502+0000", 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13485374 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.592543+0000", 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0797999999999998e-05 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:76756 13.9 13:99446d09:::benchmark_data_smithi003_293363_object76755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T15:02:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:51.460503+0000", 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 25.235712989, 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.136802692, 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 76756 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.977 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.460503+0000", 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.460503+0000", 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.460502+0000", 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.460619+0000", 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011735000000000001 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.460631+0000", 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.223e-05 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.460637+0000", 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4909999999999996e-06 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.462527+0000", 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001890596 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.462693+0000", 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016583099999999999 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.583816+0000", 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.121122471 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.980 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.590135+0000", 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0063197629999999999 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.590278+0000", 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014253700000000001 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.597216+0000", 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0069378959999999998 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.597267+0000", 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0807999999999999e-05 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.597306+0000", 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9310999999999997e-05 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.982 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.36909.0:76753 13.6 e372/362 13:6b07ad2d:::benchmark_data_smithi003_293363_object76752:head v 372'4679, mlcod=372'4676)", 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:51.462952+0000", 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 25.233264066, 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13309354700000001, 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.462952+0000", 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.462952+0000", 2024-09-13T15:02:16.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.462952+0000", 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.462978+0000", 2024-09-13T15:02:16.984 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6424e-05 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.583816+0000", 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.120837867 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.583825+0000", 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7290000000000006e-06 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.584041+0000", 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00021681000000000001 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.584062+0000", 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0859000000000001e-05 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.595998+0000", 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011936271 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.596046+0000", 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7315999999999999e-05 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:79905 13.c 13:3ec89076:::benchmark_data_smithi003_293363_object79904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:54.740832+0000", 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 21.955384128999999, 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.15896549400000001, 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:16.987 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79905 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.740832+0000", 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.988 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.740832+0000", 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.740831+0000", 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.740903+0000", 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1135000000000002e-05 2024-09-13T15:02:16.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.740912+0000", 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3999999999999998e-06 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.740917+0000", 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2100000000000001e-06 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.740961+0000", 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3529000000000003e-05 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.741079+0000", 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000117849 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.741215+0000", 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000136985 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.744065+0000", 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028490709999999999 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.991 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.744121+0000", 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6650999999999997e-05 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.899707+0000", 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.15558528999999999 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.899758+0000", 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1953999999999998e-05 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:16.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.899798+0000", 2024-09-13T15:02:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9208999999999997e-05 2024-09-13T15:02:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:16.993 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:16.993 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:02:16.994 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 101857 101841 93.5867 56.125 0.0197687 0.0106802 2024-09-13T15:02:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.995 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:16.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:17.095 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.137 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:17.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:17.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:17.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:17.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:17.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:17.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:17 smithi019 ceph-mon[91296]: pgmap v1343: 119 pgs: 119 active+clean; 6.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-13T15:02:17.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:17.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:17.631 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:17.646 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:17 smithi003 ceph-mon[107278]: pgmap v1343: 119 pgs: 119 active+clean; 6.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-13T15:02:17.646 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:17 smithi003 ceph-mon[104007]: pgmap v1343: 119 pgs: 119 active+clean; 6.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-13T15:02:17.674 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:17.748 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:02:17.748 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:02:17.749 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:02:17.749 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:17.749 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:17.749 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:17.757 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:02:17.758 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:17.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:17.791 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:02:17.791 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:17.819 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:02:17.826 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:17.851 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:02:17.851 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:17.890 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:02:17.890 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:17.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:17.965 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:17.968 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 102876 102860 93.1533 63.6875 0.00884675 0.0107299 2024-09-13T15:02:18.067 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:18.117 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:18.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:18.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:18.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:18.385 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.0. 2024-09-13T15:02:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:18.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:18.488 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:18.512 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:18.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:18.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/43663197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:18.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1358204020' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:02:18.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:18 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:02:18.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:18 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:02:18.282+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e374 definitely_dead 0 2024-09-13T15:02:18.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/43663197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:18.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1358204020' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:02:18.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:18 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:02:18.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/43663197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1358204020' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:02:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:18 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:02:18.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:18.761 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:18.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:18.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:18.968 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 103348 103332 92.244 29.5 0.00404469 0.0107541 2024-09-13T15:02:18.988 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:19.014 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:19.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:19.116 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:19.142 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:19.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:19.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:19.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:19.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:19.429 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-13T15:02:19.520 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:19.545 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:19.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:19 smithi019 ceph-mon[91296]: pgmap v1344: 119 pgs: 119 active+clean; 6.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s 2024-09-13T15:02:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:19 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-13T15:02:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:19 smithi019 ceph-mon[91296]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T15:02:19.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:19 smithi003 ceph-mon[107278]: pgmap v1344: 119 pgs: 119 active+clean; 6.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s 2024-09-13T15:02:19.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:19 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-13T15:02:19.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:19 smithi003 ceph-mon[107278]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T15:02:19.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:19 smithi003 ceph-mon[104007]: pgmap v1344: 119 pgs: 119 active+clean; 6.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s 2024-09-13T15:02:19.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:19 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-13T15:02:19.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:19 smithi003 ceph-mon[104007]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T15:02:19.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:19.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:19.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:19.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:19.968 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 103348 103332 90.9448 0 - 0.0107541 2024-09-13T15:02:20.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:20.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:20.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:20.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:20.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.305 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:20.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:20.333 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:02:20.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:20.444 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:20.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:20.567 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:20 smithi003 ceph-mon[107278]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T15:02:20.567 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:20 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:20.567 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:20 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:20.567 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:20 smithi003 ceph-mon[107278]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T15:02:20.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:20 smithi003 ceph-mon[104007]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T15:02:20.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:20 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:20.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:20 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:20.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:20 smithi003 ceph-mon[104007]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T15:02:20.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:20.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:20.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:20 smithi019 ceph-mon[91296]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T15:02:20.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:20 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:20.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:20 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:20.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:20 smithi019 ceph-mon[91296]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T15:02:20.684 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:02:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:02:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:20.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:20.827 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:20.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.882 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:20.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:20.968 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 103680 103664 89.9699 10.375 0.0264172 0.0111088 2024-09-13T15:02:20.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:21.033 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:21.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.186 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:21.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:21.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:21.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:21.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:21.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:21.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:21.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:21 smithi019 ceph-mon[91296]: pgmap v1347: 119 pgs: 13 peering, 106 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T15:02:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:21 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T15:02:21.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:21 smithi019 ceph-mon[91296]: 13.3 starting backfill to osd.0 from (0'0,0'0] MAX to 374'6360 2024-09-13T15:02:21.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:21 smithi019 ceph-mon[91296]: 13.5 starting backfill to osd.0 from (0'0,0'0] MAX to 374'6434 2024-09-13T15:02:21.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:21.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:21.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:02:21.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:02:21.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[107278]: pgmap v1347: 119 pgs: 13 peering, 106 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[107278]: 13.3 starting backfill to osd.0 from (0'0,0'0] MAX to 374'6360 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[107278]: 13.5 starting backfill to osd.0 from (0'0,0'0] MAX to 374'6434 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[104007]: pgmap v1347: 119 pgs: 13 peering, 106 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T15:02:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T15:02:21.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[104007]: 13.3 starting backfill to osd.0 from (0'0,0'0] MAX to 374'6360 2024-09-13T15:02:21.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[104007]: 13.5 starting backfill to osd.0 from (0'0,0'0] MAX to 374'6434 2024-09-13T15:02:21.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:21.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:21.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:02:21.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:02:21.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:21.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:21.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:21.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:21.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:02:21.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:21.889 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:21.968 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 104590 104574 89.5165 56.875 0.01533 0.0111659 2024-09-13T15:02:21.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:22.023 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:22.124 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:22.170 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:22.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.316 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:22.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:22.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:22.448 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:02:22.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:22.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:22.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1157750303' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:22 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:22.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1157750303' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:22.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:22 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:22.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1157750303' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:22.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:22 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:22.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:22.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:22.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:22.869 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:22.911 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:02:22.911 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:02:22.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 105479 105463 89.0577 55.5625 0.0250801 0.0112216 2024-09-13T15:02:22.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:23.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:23.110 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:23.148 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:23.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:23.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:23.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:23.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:23.429 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:23.531 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:23.544 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:23 smithi019 ceph-mon[91296]: pgmap v1349: 119 pgs: 13 peering, 106 active+clean; 6.3 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 49 MiB/s wr, 784 op/s 2024-09-13T15:02:23.545 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:23 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:23.545 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:23 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:23.545 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:23 smithi019 ceph-mon[91296]: osdmap e378: 8 total, 8 up, 8 in 2024-09-13T15:02:23.569 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:23.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:23 smithi003 ceph-mon[104007]: pgmap v1349: 119 pgs: 13 peering, 106 active+clean; 6.3 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 49 MiB/s wr, 784 op/s 2024-09-13T15:02:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:23 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:23 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:23 smithi003 ceph-mon[104007]: osdmap e378: 8 total, 8 up, 8 in 2024-09-13T15:02:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:23 smithi003 ceph-mon[107278]: pgmap v1349: 119 pgs: 13 peering, 106 active+clean; 6.3 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 49 MiB/s wr, 784 op/s 2024-09-13T15:02:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:23 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:23 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:23 smithi003 ceph-mon[107278]: osdmap e378: 8 total, 8 up, 8 in 2024-09-13T15:02:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:23.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:23.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:23.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 106394 106378 88.6327 57.1875 0.0106757 0.0112774 2024-09-13T15:02:24.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:24.053 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:24.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:24.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:24.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:24.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:24.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:24.430 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:02:24.430 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:02:24.431 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T15:02:24.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:02:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:02:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:02:24] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-13T15:02:24.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:24.462 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:02:24.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:24.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:24.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:24.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:24.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:24.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:24.941 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:24.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:24.968 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 107221 107205 88.1466 51.6875 0.0133036 0.0113388 2024-09-13T15:02:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.022 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:25.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:25.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:25.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:25.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:25.306 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:25.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:25.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:25.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:25.489 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:25.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:25.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:25.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:25 smithi019 ceph-mon[91296]: pgmap v1351: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 6.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.9 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 19542/316614 objects misplaced (6.172%); 673 KiB/s, 11 objects/s recovering 2024-09-13T15:02:25.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:25.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:25.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:25 smithi003 ceph-mon[107278]: pgmap v1351: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 6.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.9 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 19542/316614 objects misplaced (6.172%); 673 KiB/s, 11 objects/s recovering 2024-09-13T15:02:25.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:25 smithi003 ceph-mon[104007]: pgmap v1351: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 6.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.9 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 19542/316614 objects misplaced (6.172%); 673 KiB/s, 11 objects/s recovering 2024-09-13T15:02:25.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:25.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:25.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 108080 108064 87.699 53.6875 0.0180924 0.0113973 2024-09-13T15:02:25.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:26.012 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:26.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:26.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:26.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:26.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:26.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:26.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:26.409 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:02:26.409 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:26.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:26.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:26.439 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:02:26.439 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:26.458 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/912840338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:26.463 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:02:26.463 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:26.492 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:02:26.492 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:26.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:26.521 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:02:26.521 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:26.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:26.556 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:02:26.556 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:26.584 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:02:26.584 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:26.609 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:02:26.609 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:26.610 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/912840338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:26.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/912840338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:26.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:26.696 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:26.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:02:26.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:26.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:26.930 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:02:26.930 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:02:26.930 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:02:26.930 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:02:26.930 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:02:26.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:26.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:26.968 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 108934 108918 87.259 53.375 0.0344218 0.0114553 2024-09-13T15:02:27.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:27.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:27.209 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:27.252 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:27.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:27.389 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:27.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:27.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:27.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:27.591 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:27 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T15:02:27.591 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:27 smithi019 ceph-mon[91296]: pgmap v1352: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 6.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.6 KiB/s rd, 27 MiB/s wr, 430 op/s; 19542/316614 objects misplaced (6.172%); 573 KiB/s, 9 objects/s recovering 2024-09-13T15:02:27.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:27.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:02:27.675 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:27.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:27.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:27.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:27.820 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:27 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T15:02:27.820 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:27 smithi003 ceph-mon[107278]: pgmap v1352: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 6.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.6 KiB/s rd, 27 MiB/s wr, 430 op/s; 19542/316614 objects misplaced (6.172%); 573 KiB/s, 9 objects/s recovering 2024-09-13T15:02:27.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:27 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T15:02:27.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:27 smithi003 ceph-mon[104007]: pgmap v1352: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 6.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.6 KiB/s rd, 27 MiB/s wr, 430 op/s; 19542/316614 objects misplaced (6.172%); 573 KiB/s, 9 objects/s recovering 2024-09-13T15:02:27.920 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:27.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 109819 109803 86.8545 55.3125 0.0161867 0.0115074 2024-09-13T15:02:27.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:28.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:28.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:28.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:28.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:28.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:28.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:28.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:28.441 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:28.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:28.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:28.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:28.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:28.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:28.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:28.800 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:02:28.802 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:02:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:28.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:28.917 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.961 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:28.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:28.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T15:02:28.968695+0000 min lat: 0.00155173 max lat: 2.25093 avg lat: 0.0115628 2024-09-13T15:02:28.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:02:28.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 110686 110670 86.4462 54.1875 0.0104496 0.0115628 2024-09-13T15:02:29.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:29.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:29.192 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:29.234 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:29.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:29.376 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:29.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:29.561 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:29 smithi019 ceph-mon[91296]: pgmap v1353: 119 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 33 MiB/s wr, 529 op/s; 19796/320664 objects misplaced (6.173%); 481 KiB/s, 8 objects/s recovering 2024-09-13T15:02:29.561 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:29 smithi019 ceph-mon[91296]: from='client.37020 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:02:29.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:29.655 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1353,"stamp":"2024-09-13T15:02:28.096505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7003912894,"num_objects":106888,"num_object_clones":0,"num_object_copies":320664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19796,"num_objects_unfound":0,"num_objects_dirty":106888,"num_whiteouts":0,"num_read":4,"num_read_kb":11,"num_write":106928,"num_write_kb":6839225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":4269844,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101321,"ondisk_log_size":101321,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":362,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":30593152,"kb_used_data":20742516,"kb_used_omap":29,"kb_used_meta":9850466,"kb_avail":719368064,"statfs":{"total":767960285184,"available":736632897536,"internally_reserved":0,"allocated":21240336384,"data_stored":21222408334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":10086877710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":277422833,"num_objects":4233,"num_object_clones":0,"num_object_copies":12699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19796,"num_objects_unfound":0,"num_objects_dirty":4233,"num_whiteouts":0,"num_read":4,"num_read_kb":11,"num_write":4235,"num_write_kb":270937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":3941697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002017"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":144,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.447004+0000","last_change":"2024-09-13T14:57:42.607827+0000","last_active":"2024-09-13T15:02:22.447004+0000","last_peered":"2024-09-13T15:02:22.447004+0000","last_clean":"2024-09-13T15:02:22.447004+0000","last_became_active":"2024-09-13T14:57:25.285562+0000","last_became_peered":"2024-09-13T14:57:25.285562+0000","last_unstale":"2024-09-13T15:02:22.447004+0000","last_undegraded":"2024-09-13T15:02:22.447004+0000","last_fullsized":"2024-09-13T15:02:22.447004+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.607753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.607753+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:13.317042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":22,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446945+0000","last_change":"2024-09-13T15:02:19.429607+0000","last_active":"2024-09-13T15:02:22.446945+0000","last_peered":"2024-09-13T15:02:22.446945+0000","last_clean":"2024-09-13T15:02:22.446945+0000","last_became_active":"2024-09-13T15:02:19.418793+0000","last_became_peered":"2024-09-13T15:02:19.418793+0000","last_unstale":"2024-09-13T15:02:22.446945+0000","last_undegraded":"2024-09-13T15:02:22.446945+0000","last_fullsized":"2024-09-13T15:02:22.446945+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:03.520208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":350,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.384774+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T15:02:20.384774+0000","last_peered":"2024-09-13T15:02:20.384774+0000","last_clean":"2024-09-13T15:02:20.384774+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T15:02:20.384774+0000","last_undegraded":"2024-09-13T15:02:20.384774+0000","last_fullsized":"2024-09-13T15:02:20.384774+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":76,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.492206+0000","last_change":"2024-09-13T14:57:58.200463+0000","last_active":"2024-09-13T15:02:20.492206+0000","last_peered":"2024-09-13T15:02:20.492206+0000","last_clean":"2024-09-13T15:02:20.492206+0000","last_became_active":"2024-09-13T14:57:25.279086+0000","last_became_peered":"2024-09-13T14:57:25.279086+0000","last_unstale":"2024-09-13T15:02:20.492206+0000","last_undegraded":"2024-09-13T15:02:20.492206+0000","last_fullsized":"2024-09-13T15:02:20.492206+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:58.986716+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":120,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446025+0000","last_change":"2024-09-13T14:57:35.356151+0000","last_active":"2024-09-13T15:02:22.446025+0000","last_peered":"2024-09-13T15:02:22.446025+0000","last_clean":"2024-09-13T15:02:22.446025+0000","last_became_active":"2024-09-13T14:57:35.355770+0000","last_became_peered":"2024-09-13T14:57:35.355770+0000","last_unstale":"2024-09-13T15:02:22.446025+0000","last_undegraded":"2024-09-13T15:02:22.446025+0000","last_fullsized":"2024-09-13T15:02:22.446025+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:37.887286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":612,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446659+0000","last_change":"2024-09-13T15:02:19.408393+0000","last_active":"2024-09-13T15:02:22.446659+0000","last_peered":"2024-09-13T15:02:22.446659+0000","last_clean":"2024-09-13T15:02:22.446659+0000","last_became_active":"2024-09-13T15:02:19.407871+0000","last_became_peered":"2024-09-13T15:02:19.407871+0000","last_unstale":"2024-09-13T15:02:22.446659+0000","last_undegraded":"2024-09-13T15:02:22.446659+0000","last_fullsized":"2024-09-13T15:02:22.446659+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:22.868137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":81,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.398504+0000","last_change":"2024-09-13T14:58:28.426763+0000","last_active":"2024-09-13T15:02:20.398504+0000","last_peered":"2024-09-13T15:02:20.398504+0000","last_clean":"2024-09-13T15:02:20.398504+0000","last_became_active":"2024-09-13T14:58:28.426270+0000","last_became_peered":"2024-09-13T14:58:28.426270+0000","last_unstale":"2024-09-13T15:02:20.398504+0000","last_undegraded":"2024-09-13T15:02:20.398504+0000","last_fullsized":"2024-09-13T15:02:20.398504+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.991576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":377,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385275+0000","last_change":"2024-09-13T15:02:19.407947+0000","last_active":"2024-09-13T15:02:20.385275+0000","last_peered":"2024-09-13T15:02:20.385275+0000","last_clean":"2024-09-13T15:02:20.385275+0000","last_became_active":"2024-09-13T15:02:19.405685+0000","last_became_peered":"2024-09-13T15:02:19.405685+0000","last_unstale":"2024-09-13T15:02:20.385275+0000","last_undegraded":"2024-09-13T15:02:20.385275+0000","last_fullsized":"2024-09-13T15:02:20.385275+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:46.971853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":146,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446395+0000","last_change":"2024-09-13T14:57:43.658933+0000","last_active":"2024-09-13T15:02:22.446395+0000","last_peered":"2024-09-13T15:02:22.446395+0000","last_clean":"2024-09-13T15:02:22.446395+0000","last_became_active":"2024-09-13T14:57:25.291285+0000","last_became_peered":"2024-09-13T14:57:25.291285+0000","last_unstale":"2024-09-13T15:02:22.446395+0000","last_undegraded":"2024-09-13T15:02:22.446395+0000","last_fullsized":"2024-09-13T15:02:22.446395+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:43.658833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:43.658833+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.888776+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":81,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385304+0000","last_change":"2024-09-13T14:58:28.429061+0000","last_active":"2024-09-13T15:02:20.385304+0000","last_peered":"2024-09-13T15:02:20.385304+0000","last_clean":"2024-09-13T15:02:20.385304+0000","last_became_active":"2024-09-13T14:58:28.428642+0000","last_became_peered":"2024-09-13T14:58:28.428642+0000","last_unstale":"2024-09-13T15:02:20.385304+0000","last_undegraded":"2024-09-13T15:02:20.385304+0000","last_fullsized":"2024-09-13T15:02:20.385304+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:03.197824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":387,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385477+0000","last_change":"2024-09-13T14:48:29.143709+0000","last_active":"2024-09-13T15:02:20.385477+0000","last_peered":"2024-09-13T15:02:20.385477+0000","last_clean":"2024-09-13T15:02:20.385477+0000","last_became_active":"2024-09-13T14:48:29.143196+0000","last_became_peered":"2024-09-13T14:48:29.143196+0000","last_unstale":"2024-09-13T15:02:20.385477+0000","last_undegraded":"2024-09-13T15:02:20.385477+0000","last_fullsized":"2024-09-13T15:02:20.385477+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.091028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":160,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446101+0000","last_change":"2024-09-13T15:02:19.407271+0000","last_active":"2024-09-13T15:02:22.446101+0000","last_peered":"2024-09-13T15:02:22.446101+0000","last_clean":"2024-09-13T15:02:22.446101+0000","last_became_active":"2024-09-13T15:02:19.406705+0000","last_became_peered":"2024-09-13T15:02:19.406705+0000","last_unstale":"2024-09-13T15:02:22.446101+0000","last_undegraded":"2024-09-13T15:02:22.446101+0000","last_fullsized":"2024-09-13T15:02:22.446101+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:59.732000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446553+0000","last_change":"2024-09-13T15:02:19.418023+0000","last_active":"2024-09-13T15:02:22.446553+0000","last_peered":"2024-09-13T15:02:22.446553+0000","last_clean":"2024-09-13T15:02:22.446553+0000","last_became_active":"2024-09-13T15:02:19.417503+0000","last_became_peered":"2024-09-13T15:02:19.417503+0000","last_unstale":"2024-09-13T15:02:22.446553+0000","last_undegraded":"2024-09-13T15:02:22.446553+0000","last_fullsized":"2024-09-13T15:02:22.446553+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:48.630767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":622,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.459196+0000","last_change":"2024-09-13T14:37:08.782351+0000","last_active":"2024-09-13T15:02:22.459196+0000","last_peered":"2024-09-13T15:02:22.459196+0000","last_clean":"2024-09-13T15:02:22.459196+0000","last_became_active":"2024-09-13T14:37:08.781918+0000","last_became_peered":"2024-09-13T14:37:08.781918+0000","last_unstale":"2024-09-13T15:02:22.459196+0000","last_undegraded":"2024-09-13T15:02:22.459196+0000","last_fullsized":"2024-09-13T15:02:22.459196+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:08.977498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":73,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.384007+0000","last_change":"2024-09-13T14:58:09.011981+0000","last_active":"2024-09-13T15:02:20.384007+0000","last_peered":"2024-09-13T15:02:20.384007+0000","last_clean":"2024-09-13T15:02:20.384007+0000","last_became_active":"2024-09-13T14:57:53.357674+0000","last_became_peered":"2024-09-13T14:57:53.357674+0000","last_unstale":"2024-09-13T15:02:20.384007+0000","last_undegraded":"2024-09-13T15:02:20.384007+0000","last_fullsized":"2024-09-13T15:02:20.384007+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.701912+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":359,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385428+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T15:02:20.385428+0000","last_peered":"2024-09-13T15:02:20.385428+0000","last_clean":"2024-09-13T15:02:20.385428+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T15:02:20.385428+0000","last_undegraded":"2024-09-13T15:02:20.385428+0000","last_fullsized":"2024-09-13T15:02:20.385428+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":565,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.447023+0000","last_change":"2024-09-13T14:37:08.758108+0000","last_active":"2024-09-13T15:02:22.447023+0000","last_peered":"2024-09-13T15:02:22.447023+0000","last_clean":"2024-09-13T15:02:22.447023+0000","last_became_active":"2024-09-13T14:37:08.757694+0000","last_became_peered":"2024-09-13T14:37:08.757694+0000","last_unstale":"2024-09-13T15:02:22.447023+0000","last_undegraded":"2024-09-13T15:02:22.447023+0000","last_fullsized":"2024-09-13T15:02:22.447023+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:58.142248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":89,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.384168+0000","last_change":"2024-09-13T15:02:19.406391+0000","last_active":"2024-09-13T15:02:20.384168+0000","last_peered":"2024-09-13T15:02:20.384168+0000","last_clean":"2024-09-13T15:02:20.384168+0000","last_became_active":"2024-09-13T15:02:19.406040+0000","last_became_peered":"2024-09-13T15:02:19.406040+0000","last_unstale":"2024-09-13T15:02:20.384168+0000","last_undegraded":"2024-09-13T15:02:20.384168+0000","last_fullsized":"2024-09-13T15:02:20.384168+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:45.665420+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":279,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.467422+0000","last_change":"2024-09-13T14:48:29.147306+0000","last_active":"2024-09-13T15:02:22.467422+0000","last_peered":"2024-09-13T15:02:22.467422+0000","last_clean":"2024-09-13T15:02:22.467422+0000","last_became_active":"2024-09-13T14:48:29.146739+0000","last_became_peered":"2024-09-13T14:48:29.146739+0000","last_unstale":"2024-09-13T15:02:22.467422+0000","last_undegraded":"2024-09-13T15:02:22.467422+0000","last_fullsized":"2024-09-13T15:02:22.467422+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:14.315221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446708+0000","last_change":"2024-09-13T15:02:19.412819+0000","last_active":"2024-09-13T15:02:22.446708+0000","last_peered":"2024-09-13T15:02:22.446708+0000","last_clean":"2024-09-13T15:02:22.446708+0000","last_became_active":"2024-09-13T15:02:19.412080+0000","last_became_peered":"2024-09-13T15:02:19.412080+0000","last_unstale":"2024-09-13T15:02:22.446708+0000","last_undegraded":"2024-09-13T15:02:22.446708+0000","last_fullsized":"2024-09-13T15:02:22.446708+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:20.712475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":76,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385065+0000","last_change":"2024-09-13T14:58:05.212101+0000","last_active":"2024-09-13T15:02:20.385065+0000","last_peered":"2024-09-13T15:02:20.385065+0000","last_clean":"2024-09-13T15:02:20.385065+0000","last_became_active":"2024-09-13T14:57:25.275734+0000","last_became_peered":"2024-09-13T14:57:25.275734+0000","last_unstale":"2024-09-13T15:02:20.385065+0000","last_undegraded":"2024-09-13T15:02:20.385065+0000","last_fullsized":"2024-09-13T15:02:20.385065+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.212008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.212008+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:01.720211+0000","scrub_duration":0.0048969690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":611,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.445880+0000","last_change":"2024-09-13T15:02:19.412786+0000","last_active":"2024-09-13T15:02:22.445880+0000","last_peered":"2024-09-13T15:02:22.445880+0000","last_clean":"2024-09-13T15:02:22.445880+0000","last_became_active":"2024-09-13T15:02:19.411949+0000","last_became_peered":"2024-09-13T15:02:19.411949+0000","last_unstale":"2024-09-13T15:02:22.445880+0000","last_undegraded":"2024-09-13T15:02:22.445880+0000","last_fullsized":"2024-09-13T15:02:22.445880+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:38.413012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":360,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.425237+0000","last_change":"2024-09-13T14:57:53.364039+0000","last_active":"2024-09-13T15:02:20.425237+0000","last_peered":"2024-09-13T15:02:20.425237+0000","last_clean":"2024-09-13T15:02:20.425237+0000","last_became_active":"2024-09-13T14:57:53.363550+0000","last_became_peered":"2024-09-13T14:57:53.363550+0000","last_unstale":"2024-09-13T15:02:20.425237+0000","last_undegraded":"2024-09-13T15:02:20.425237+0000","last_fullsized":"2024-09-13T15:02:20.425237+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:36.171089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":102,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.493330+0000","last_change":"2024-09-13T14:58:04.168950+0000","last_active":"2024-09-13T15:02:20.493330+0000","last_peered":"2024-09-13T15:02:20.493330+0000","last_clean":"2024-09-13T15:02:20.493330+0000","last_became_active":"2024-09-13T14:57:53.351301+0000","last_became_peered":"2024-09-13T14:57:53.351301+0000","last_unstale":"2024-09-13T15:02:20.493330+0000","last_undegraded":"2024-09-13T15:02:20.493330+0000","last_fullsized":"2024-09-13T15:02:20.493330+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:42.270600+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":565,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.445662+0000","last_change":"2024-09-13T14:37:08.779314+0000","last_active":"2024-09-13T15:02:22.445662+0000","last_peered":"2024-09-13T15:02:22.445662+0000","last_clean":"2024-09-13T15:02:22.445662+0000","last_became_active":"2024-09-13T14:37:08.778573+0000","last_became_peered":"2024-09-13T14:37:08.778573+0000","last_unstale":"2024-09-13T15:02:22.445662+0000","last_undegraded":"2024-09-13T15:02:22.445662+0000","last_fullsized":"2024-09-13T15:02:22.445662+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:30.896486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446837+0000","last_change":"2024-09-13T15:02:19.418083+0000","last_active":"2024-09-13T15:02:22.446837+0000","last_peered":"2024-09-13T15:02:22.446837+0000","last_clean":"2024-09-13T15:02:22.446837+0000","last_became_active":"2024-09-13T15:02:19.417727+0000","last_became_peered":"2024-09-13T15:02:19.417727+0000","last_unstale":"2024-09-13T15:02:22.446837+0000","last_undegraded":"2024-09-13T15:02:22.446837+0000","last_fullsized":"2024-09-13T15:02:22.446837+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:54.858638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":570,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446334+0000","last_change":"2024-09-13T14:49:05.828658+0000","last_active":"2024-09-13T15:02:22.446334+0000","last_peered":"2024-09-13T15:02:22.446334+0000","last_clean":"2024-09-13T15:02:22.446334+0000","last_became_active":"2024-09-13T14:49:05.828332+0000","last_became_peered":"2024-09-13T14:49:05.828332+0000","last_unstale":"2024-09-13T15:02:22.446334+0000","last_undegraded":"2024-09-13T15:02:22.446334+0000","last_fullsized":"2024-09-13T15:02:22.446334+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:43.984158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":59,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.384368+0000","last_change":"2024-09-13T14:57:53.350622+0000","last_active":"2024-09-13T15:02:20.384368+0000","last_peered":"2024-09-13T15:02:20.384368+0000","last_clean":"2024-09-13T15:02:20.384368+0000","last_became_active":"2024-09-13T14:57:53.350292+0000","last_became_peered":"2024-09-13T14:57:53.350292+0000","last_unstale":"2024-09-13T15:02:20.384368+0000","last_undegraded":"2024-09-13T15:02:20.384368+0000","last_fullsized":"2024-09-13T15:02:20.384368+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:16.088338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":324,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.458990+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T15:02:22.458990+0000","last_peered":"2024-09-13T15:02:22.458990+0000","last_clean":"2024-09-13T15:02:22.458990+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T15:02:22.458990+0000","last_undegraded":"2024-09-13T15:02:22.458990+0000","last_fullsized":"2024-09-13T15:02:22.458990+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:43:13.165444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":79,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.384321+0000","last_change":"2024-09-13T15:02:19.406264+0000","last_active":"2024-09-13T15:02:20.384321+0000","last_peered":"2024-09-13T15:02:20.384321+0000","last_clean":"2024-09-13T15:02:20.384321+0000","last_became_active":"2024-09-13T15:02:19.405876+0000","last_became_peered":"2024-09-13T15:02:19.405876+0000","last_unstale":"2024-09-13T15:02:20.384321+0000","last_undegraded":"2024-09-13T15:02:20.384321+0000","last_fullsized":"2024-09-13T15:02:20.384321+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:19.208307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":141,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.493049+0000","last_change":"2024-09-13T14:49:05.856566+0000","last_active":"2024-09-13T15:02:20.493049+0000","last_peered":"2024-09-13T15:02:20.493049+0000","last_clean":"2024-09-13T15:02:20.493049+0000","last_became_active":"2024-09-13T14:49:05.841449+0000","last_became_peered":"2024-09-13T14:49:05.841449+0000","last_unstale":"2024-09-13T15:02:20.493049+0000","last_undegraded":"2024-09-13T15:02:20.493049+0000","last_fullsized":"2024-09-13T15:02:20.493049+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:23.562187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":279,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.424440+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T15:02:20.424440+0000","last_peered":"2024-09-13T15:02:20.424440+0000","last_clean":"2024-09-13T15:02:20.424440+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T15:02:20.424440+0000","last_undegraded":"2024-09-13T15:02:20.424440+0000","last_fullsized":"2024-09-13T15:02:20.424440+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":158,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.458531+0000","last_change":"2024-09-13T15:02:19.421430+0000","last_active":"2024-09-13T15:02:22.458531+0000","last_peered":"2024-09-13T15:02:22.458531+0000","last_clean":"2024-09-13T15:02:22.458531+0000","last_became_active":"2024-09-13T15:02:19.420988+0000","last_became_peered":"2024-09-13T15:02:19.420988+0000","last_unstale":"2024-09-13T15:02:22.458531+0000","last_undegraded":"2024-09-13T15:02:22.458531+0000","last_fullsized":"2024-09-13T15:02:22.458531+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:27.445483+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":347,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385397+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T15:02:20.385397+0000","last_peered":"2024-09-13T15:02:20.385397+0000","last_clean":"2024-09-13T15:02:20.385397+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T15:02:20.385397+0000","last_undegraded":"2024-09-13T15:02:20.385397+0000","last_fullsized":"2024-09-13T15:02:20.385397+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":340,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385563+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T15:02:20.385563+0000","last_peered":"2024-09-13T15:02:20.385563+0000","last_clean":"2024-09-13T15:02:20.385563+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T15:02:20.385563+0000","last_undegraded":"2024-09-13T15:02:20.385563+0000","last_fullsized":"2024-09-13T15:02:20.385563+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"378'6704","reported_seq":6747,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:26.135699+0000","last_change":"2024-09-13T15:00:55.405989+0000","last_active":"2024-09-13T15:02:26.135699+0000","last_peered":"2024-09-13T15:02:26.135699+0000","last_clean":"2024-09-13T15:02:26.135699+0000","last_became_active":"2024-09-13T15:00:55.405741+0000","last_became_peered":"2024-09-13T15:00:55.405741+0000","last_unstale":"2024-09-13T15:02:26.135699+0000","last_undegraded":"2024-09-13T15:02:26.135699+0000","last_fullsized":"2024-09-13T15:02:26.135699+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6704,"log_dups_size":0,"ondisk_log_size":6704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:32.041720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439353344,"num_objects":6704,"num_object_clones":0,"num_object_copies":20112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6704,"num_write_kb":429056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":343,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.384897+0000","last_change":"2024-09-13T14:57:53.351374+0000","last_active":"2024-09-13T15:02:20.384897+0000","last_peered":"2024-09-13T15:02:20.384897+0000","last_clean":"2024-09-13T15:02:20.384897+0000","last_became_active":"2024-09-13T14:57:53.350803+0000","last_became_peered":"2024-09-13T14:57:53.350803+0000","last_unstale":"2024-09-13T15:02:20.384897+0000","last_undegraded":"2024-09-13T15:02:20.384897+0000","last_fullsized":"2024-09-13T15:02:20.384897+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:30.535837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":373,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.425245+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T15:02:20.425245+0000","last_peered":"2024-09-13T15:02:20.425245+0000","last_clean":"2024-09-13T15:02:20.425245+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T15:02:20.425245+0000","last_undegraded":"2024-09-13T15:02:20.425245+0000","last_fullsized":"2024-09-13T15:02:20.425245+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":20,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.447011+0000","last_change":"2024-09-13T15:02:19.432393+0000","last_active":"2024-09-13T15:02:22.447011+0000","last_peered":"2024-09-13T15:02:22.447011+0000","last_clean":"2024-09-13T15:02:22.447011+0000","last_became_active":"2024-09-13T15:02:19.419183+0000","last_became_peered":"2024-09-13T15:02:19.419183+0000","last_unstale":"2024-09-13T15:02:22.447011+0000","last_undegraded":"2024-09-13T15:02:22.447011+0000","last_fullsized":"2024-09-13T15:02:22.447011+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:51.143192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":346,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.445897+0000","last_change":"2024-09-13T14:57:53.347772+0000","last_active":"2024-09-13T15:02:22.445897+0000","last_peered":"2024-09-13T15:02:22.445897+0000","last_clean":"2024-09-13T15:02:22.445897+0000","last_became_active":"2024-09-13T14:57:53.347373+0000","last_became_peered":"2024-09-13T14:57:53.347373+0000","last_unstale":"2024-09-13T15:02:22.445897+0000","last_undegraded":"2024-09-13T15:02:22.445897+0000","last_fullsized":"2024-09-13T15:02:22.445897+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:51.119909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"378'6663","reported_seq":6762,"reported_epoch":378,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T15:02:23.416782+0000","last_change":"2024-09-13T15:02:20.700677+0000","last_active":"2024-09-13T15:02:23.416782+0000","last_peered":"2024-09-13T15:02:23.416782+0000","last_clean":"2024-09-13T15:02:18.361788+0000","last_became_active":"2024-09-13T15:02:20.574877+0000","last_became_peered":"2024-09-13T15:02:20.574877+0000","last_unstale":"2024-09-13T15:02:23.416782+0000","last_undegraded":"2024-09-13T15:02:23.416782+0000","last_fullsized":"2024-09-13T15:02:20.383180+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:14.477107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436666368,"num_objects":6663,"num_object_clones":0,"num_object_copies":19989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6466,"num_objects_unfound":0,"num_objects_dirty":6663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6663,"num_write_kb":426432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2162688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,2],"avail_no_missing":["5","2","6"],"object_location_counts":[{"shards":"2,5","objects":197},{"shards":"2,5,6","objects":6466}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":623,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.445758+0000","last_change":"2024-09-13T15:02:19.413114+0000","last_active":"2024-09-13T15:02:22.445758+0000","last_peered":"2024-09-13T15:02:22.445758+0000","last_clean":"2024-09-13T15:02:22.445758+0000","last_became_active":"2024-09-13T15:02:19.412380+0000","last_became_peered":"2024-09-13T15:02:19.412380+0000","last_unstale":"2024-09-13T15:02:22.445758+0000","last_undegraded":"2024-09-13T15:02:22.445758+0000","last_fullsized":"2024-09-13T15:02:22.445758+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:55.523795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":214,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.425067+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T15:02:20.425067+0000","last_peered":"2024-09-13T15:02:20.425067+0000","last_clean":"2024-09-13T15:02:20.425067+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T15:02:20.425067+0000","last_undegraded":"2024-09-13T15:02:20.425067+0000","last_fullsized":"2024-09-13T15:02:20.425067+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":619,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446277+0000","last_change":"2024-09-13T15:02:19.432796+0000","last_active":"2024-09-13T15:02:22.446277+0000","last_peered":"2024-09-13T15:02:22.446277+0000","last_clean":"2024-09-13T15:02:22.446277+0000","last_became_active":"2024-09-13T15:02:19.412358+0000","last_became_peered":"2024-09-13T15:02:19.412358+0000","last_unstale":"2024-09-13T15:02:22.446277+0000","last_undegraded":"2024-09-13T15:02:22.446277+0000","last_fullsized":"2024-09-13T15:02:22.446277+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:59.363215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":181,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.398431+0000","last_change":"2024-09-13T14:57:56.132062+0000","last_active":"2024-09-13T15:02:20.398431+0000","last_peered":"2024-09-13T15:02:20.398431+0000","last_clean":"2024-09-13T15:02:20.398431+0000","last_became_active":"2024-09-13T14:57:19.218181+0000","last_became_peered":"2024-09-13T14:57:19.218181+0000","last_unstale":"2024-09-13T15:02:20.398431+0000","last_undegraded":"2024-09-13T15:02:20.398431+0000","last_fullsized":"2024-09-13T15:02:20.398431+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:56.131976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:56.131976+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:31.359825+0000","scrub_duration":0.0061470450000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"378'6738","reported_seq":6781,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:26.167371+0000","last_change":"2024-09-13T15:00:55.873094+0000","last_active":"2024-09-13T15:02:26.167371+0000","last_peered":"2024-09-13T15:02:26.167371+0000","last_clean":"2024-09-13T15:02:26.167371+0000","last_became_active":"2024-09-13T15:00:55.872743+0000","last_became_peered":"2024-09-13T15:00:55.872743+0000","last_unstale":"2024-09-13T15:02:26.167371+0000","last_undegraded":"2024-09-13T15:02:26.167371+0000","last_fullsized":"2024-09-13T15:02:26.167371+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6738,"log_dups_size":0,"ondisk_log_size":6738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:46.267062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441581568,"num_objects":6738,"num_object_clones":0,"num_object_copies":20214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":431232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446649+0000","last_change":"2024-09-13T15:02:19.423088+0000","last_active":"2024-09-13T15:02:22.446649+0000","last_peered":"2024-09-13T15:02:22.446649+0000","last_clean":"2024-09-13T15:02:22.446649+0000","last_became_active":"2024-09-13T15:02:19.422675+0000","last_became_peered":"2024-09-13T15:02:19.422675+0000","last_unstale":"2024-09-13T15:02:22.446649+0000","last_undegraded":"2024-09-13T15:02:22.446649+0000","last_fullsized":"2024-09-13T15:02:22.446649+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:58.764834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":373,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.493397+0000","last_change":"2024-09-13T15:02:19.417107+0000","last_active":"2024-09-13T15:02:20.493397+0000","last_peered":"2024-09-13T15:02:20.493397+0000","last_clean":"2024-09-13T15:02:20.493397+0000","last_became_active":"2024-09-13T15:02:19.416741+0000","last_became_peered":"2024-09-13T15:02:19.416741+0000","last_unstale":"2024-09-13T15:02:20.493397+0000","last_undegraded":"2024-09-13T15:02:20.493397+0000","last_fullsized":"2024-09-13T15:02:20.493397+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:20.599091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":596,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.459151+0000","last_change":"2024-09-13T14:57:53.351358+0000","last_active":"2024-09-13T15:02:22.459151+0000","last_peered":"2024-09-13T15:02:22.459151+0000","last_clean":"2024-09-13T15:02:22.459151+0000","last_became_active":"2024-09-13T14:57:53.351021+0000","last_became_peered":"2024-09-13T14:57:53.351021+0000","last_unstale":"2024-09-13T15:02:22.459151+0000","last_undegraded":"2024-09-13T15:02:22.459151+0000","last_fullsized":"2024-09-13T15:02:22.459151+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.067202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":354,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.424882+0000","last_change":"2024-09-13T15:02:19.407067+0000","last_active":"2024-09-13T15:02:20.424882+0000","last_peered":"2024-09-13T15:02:20.424882+0000","last_clean":"2024-09-13T15:02:20.424882+0000","last_became_active":"2024-09-13T15:02:19.406685+0000","last_became_peered":"2024-09-13T15:02:19.406685+0000","last_unstale":"2024-09-13T15:02:20.424882+0000","last_undegraded":"2024-09-13T15:02:20.424882+0000","last_fullsized":"2024-09-13T15:02:20.424882+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:35.298283+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"378'6859","reported_seq":6902,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:27.691689+0000","last_change":"2024-09-13T15:00:55.405198+0000","last_active":"2024-09-13T15:02:27.691689+0000","last_peered":"2024-09-13T15:02:27.691689+0000","last_clean":"2024-09-13T15:02:27.691689+0000","last_became_active":"2024-09-13T15:00:55.404898+0000","last_became_peered":"2024-09-13T15:00:55.404898+0000","last_unstale":"2024-09-13T15:02:27.691689+0000","last_undegraded":"2024-09-13T15:02:27.691689+0000","last_fullsized":"2024-09-13T15:02:27.691689+0000","mapping_epoch":362,"log_start":"366'600","ondisk_log_start":"366'600","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:42.847939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449511424,"num_objects":6859,"num_object_clones":0,"num_object_copies":20577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6859,"num_write_kb":438976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":59,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.383876+0000","last_change":"2024-09-13T14:57:53.351087+0000","last_active":"2024-09-13T15:02:20.383876+0000","last_peered":"2024-09-13T15:02:20.383876+0000","last_clean":"2024-09-13T15:02:20.383876+0000","last_became_active":"2024-09-13T14:57:53.350744+0000","last_became_peered":"2024-09-13T14:57:53.350744+0000","last_unstale":"2024-09-13T15:02:20.383876+0000","last_undegraded":"2024-09-13T15:02:20.383876+0000","last_fullsized":"2024-09-13T15:02:20.383876+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:43.300957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446464+0000","last_change":"2024-09-13T15:02:19.410671+0000","last_active":"2024-09-13T15:02:22.446464+0000","last_peered":"2024-09-13T15:02:22.446464+0000","last_clean":"2024-09-13T15:02:22.446464+0000","last_became_active":"2024-09-13T15:02:19.410330+0000","last_became_peered":"2024-09-13T15:02:19.410330+0000","last_unstale":"2024-09-13T15:02:22.446464+0000","last_undegraded":"2024-09-13T15:02:22.446464+0000","last_fullsized":"2024-09-13T15:02:22.446464+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:30.403982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":59,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.383820+0000","last_change":"2024-09-13T14:57:53.347982+0000","last_active":"2024-09-13T15:02:20.383820+0000","last_peered":"2024-09-13T15:02:20.383820+0000","last_clean":"2024-09-13T15:02:20.383820+0000","last_became_active":"2024-09-13T14:57:53.347623+0000","last_became_peered":"2024-09-13T14:57:53.347623+0000","last_unstale":"2024-09-13T15:02:20.383820+0000","last_undegraded":"2024-09-13T15:02:20.383820+0000","last_fullsized":"2024-09-13T15:02:20.383820+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:15.734246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":354,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.425167+0000","last_change":"2024-09-13T14:58:05.197923+0000","last_active":"2024-09-13T15:02:20.425167+0000","last_peered":"2024-09-13T15:02:20.425167+0000","last_clean":"2024-09-13T15:02:20.425167+0000","last_became_active":"2024-09-13T14:57:19.208023+0000","last_became_peered":"2024-09-13T14:57:19.208023+0000","last_unstale":"2024-09-13T15:02:20.425167+0000","last_undegraded":"2024-09-13T15:02:20.425167+0000","last_fullsized":"2024-09-13T15:02:20.425167+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:06.994746+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"378'6677","reported_seq":6700,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:23.383374+0000","last_change":"2024-09-13T15:00:55.874092+0000","last_active":"2024-09-13T15:02:23.383374+0000","last_peered":"2024-09-13T15:02:23.383374+0000","last_clean":"2024-09-13T15:02:23.383374+0000","last_became_active":"2024-09-13T15:00:55.873709+0000","last_became_peered":"2024-09-13T15:00:55.873709+0000","last_unstale":"2024-09-13T15:02:23.383374+0000","last_undegraded":"2024-09-13T15:02:23.383374+0000","last_fullsized":"2024-09-13T15:02:23.383374+0000","mapping_epoch":362,"log_start":"366'1000","ondisk_log_start":"366'1000","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5677,"log_dups_size":0,"ondisk_log_size":5677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.813547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437583872,"num_objects":6677,"num_object_clones":0,"num_object_copies":20031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":427328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":494,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.459309+0000","last_change":"2024-09-13T14:45:44.442625+0000","last_active":"2024-09-13T15:02:22.459309+0000","last_peered":"2024-09-13T15:02:22.459309+0000","last_clean":"2024-09-13T15:02:22.459309+0000","last_became_active":"2024-09-13T14:45:44.442322+0000","last_became_peered":"2024-09-13T14:45:44.442322+0000","last_unstale":"2024-09-13T15:02:22.459309+0000","last_undegraded":"2024-09-13T15:02:22.459309+0000","last_fullsized":"2024-09-13T15:02:22.459309+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:38.221420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":361,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385652+0000","last_change":"2024-09-13T15:02:19.406489+0000","last_active":"2024-09-13T15:02:20.385652+0000","last_peered":"2024-09-13T15:02:20.385652+0000","last_clean":"2024-09-13T15:02:20.385652+0000","last_became_active":"2024-09-13T15:02:19.405872+0000","last_became_peered":"2024-09-13T15:02:19.405872+0000","last_unstale":"2024-09-13T15:02:20.385652+0000","last_undegraded":"2024-09-13T15:02:20.385652+0000","last_fullsized":"2024-09-13T15:02:20.385652+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:15.343020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":127,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.458192+0000","last_change":"2024-09-13T14:58:28.428183+0000","last_active":"2024-09-13T15:02:22.458192+0000","last_peered":"2024-09-13T15:02:22.458192+0000","last_clean":"2024-09-13T15:02:22.458192+0000","last_became_active":"2024-09-13T14:58:28.427553+0000","last_became_peered":"2024-09-13T14:58:28.427553+0000","last_unstale":"2024-09-13T15:02:22.458192+0000","last_undegraded":"2024-09-13T15:02:22.458192+0000","last_fullsized":"2024-09-13T15:02:22.458192+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:59.108147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.445428+0000","last_change":"2024-09-13T15:02:19.420193+0000","last_active":"2024-09-13T15:02:22.445428+0000","last_peered":"2024-09-13T15:02:22.445428+0000","last_clean":"2024-09-13T15:02:22.445428+0000","last_became_active":"2024-09-13T15:02:19.419807+0000","last_became_peered":"2024-09-13T15:02:19.419807+0000","last_unstale":"2024-09-13T15:02:22.445428+0000","last_undegraded":"2024-09-13T15:02:22.445428+0000","last_fullsized":"2024-09-13T15:02:22.445428+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:13.929284+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"378'6604","reported_seq":6628,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:23.414736+0000","last_change":"2024-09-13T15:00:55.408720+0000","last_active":"2024-09-13T15:02:23.414736+0000","last_peered":"2024-09-13T15:02:23.414736+0000","last_clean":"2024-09-13T15:02:23.414736+0000","last_became_active":"2024-09-13T15:00:55.408379+0000","last_became_peered":"2024-09-13T15:00:55.408379+0000","last_unstale":"2024-09-13T15:02:23.414736+0000","last_undegraded":"2024-09-13T15:02:23.414736+0000","last_fullsized":"2024-09-13T15:02:23.414736+0000","mapping_epoch":362,"log_start":"366'900","ondisk_log_start":"366'900","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:24.296460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432799744,"num_objects":6604,"num_object_clones":0,"num_object_copies":19812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":422656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":349,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.459006+0000","last_change":"2024-09-13T15:02:19.420231+0000","last_active":"2024-09-13T15:02:22.459006+0000","last_peered":"2024-09-13T15:02:22.459006+0000","last_clean":"2024-09-13T15:02:22.459006+0000","last_became_active":"2024-09-13T15:02:19.419627+0000","last_became_peered":"2024-09-13T15:02:19.419627+0000","last_unstale":"2024-09-13T15:02:22.459006+0000","last_undegraded":"2024-09-13T15:02:22.459006+0000","last_fullsized":"2024-09-13T15:02:22.459006+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:21.403387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":434,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.457445+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T15:02:22.457445+0000","last_peered":"2024-09-13T15:02:22.457445+0000","last_clean":"2024-09-13T15:02:22.457445+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T15:02:22.457445+0000","last_undegraded":"2024-09-13T15:02:22.457445+0000","last_fullsized":"2024-09-13T15:02:22.457445+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":638,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.461829+0000","last_change":"2024-09-13T15:02:19.407739+0000","last_active":"2024-09-13T15:02:22.461829+0000","last_peered":"2024-09-13T15:02:22.461829+0000","last_clean":"2024-09-13T15:02:22.461829+0000","last_became_active":"2024-09-13T15:02:19.407290+0000","last_became_peered":"2024-09-13T15:02:19.407290+0000","last_unstale":"2024-09-13T15:02:22.461829+0000","last_undegraded":"2024-09-13T15:02:22.461829+0000","last_fullsized":"2024-09-13T15:02:22.461829+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:33.303592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":155,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.491715+0000","last_change":"2024-09-13T15:02:19.406171+0000","last_active":"2024-09-13T15:02:20.491715+0000","last_peered":"2024-09-13T15:02:20.491715+0000","last_clean":"2024-09-13T15:02:20.491715+0000","last_became_active":"2024-09-13T15:02:19.405713+0000","last_became_peered":"2024-09-13T15:02:19.405713+0000","last_unstale":"2024-09-13T15:02:20.491715+0000","last_undegraded":"2024-09-13T15:02:20.491715+0000","last_fullsized":"2024-09-13T15:02:20.491715+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:13.657811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.f","version":"378'6668","reported_seq":6691,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:23.399552+0000","last_change":"2024-09-13T15:00:55.408205+0000","last_active":"2024-09-13T15:02:23.399552+0000","last_peered":"2024-09-13T15:02:23.399552+0000","last_clean":"2024-09-13T15:02:23.399552+0000","last_became_active":"2024-09-13T15:00:55.407801+0000","last_became_peered":"2024-09-13T15:00:55.407801+0000","last_unstale":"2024-09-13T15:02:23.399552+0000","last_undegraded":"2024-09-13T15:02:23.399552+0000","last_fullsized":"2024-09-13T15:02:23.399552+0000","mapping_epoch":362,"log_start":"366'400","ondisk_log_start":"366'400","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6268,"log_dups_size":0,"ondisk_log_size":6268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:36.295309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436994048,"num_objects":6668,"num_object_clones":0,"num_object_copies":20004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6668,"num_write_kb":426752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446803+0000","last_change":"2024-09-13T15:02:19.419571+0000","last_active":"2024-09-13T15:02:22.446803+0000","last_peered":"2024-09-13T15:02:22.446803+0000","last_clean":"2024-09-13T15:02:22.446803+0000","last_became_active":"2024-09-13T15:02:19.419205+0000","last_became_peered":"2024-09-13T15:02:19.419205+0000","last_unstale":"2024-09-13T15:02:22.446803+0000","last_undegraded":"2024-09-13T15:02:22.446803+0000","last_fullsized":"2024-09-13T15:02:22.446803+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:04.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.444701+0000","last_change":"2024-09-13T15:02:19.411527+0000","last_active":"2024-09-13T15:02:22.444701+0000","last_peered":"2024-09-13T15:02:22.444701+0000","last_clean":"2024-09-13T15:02:22.444701+0000","last_became_active":"2024-09-13T15:02:19.411206+0000","last_became_peered":"2024-09-13T15:02:19.411206+0000","last_unstale":"2024-09-13T15:02:22.444701+0000","last_undegraded":"2024-09-13T15:02:22.444701+0000","last_fullsized":"2024-09-13T15:02:22.444701+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:10.707413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":410,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.494053+0000","last_change":"2024-09-13T15:02:19.408677+0000","last_active":"2024-09-13T15:02:20.494053+0000","last_peered":"2024-09-13T15:02:20.494053+0000","last_clean":"2024-09-13T15:02:20.494053+0000","last_became_active":"2024-09-13T15:02:19.408262+0000","last_became_peered":"2024-09-13T15:02:19.408262+0000","last_unstale":"2024-09-13T15:02:20.494053+0000","last_undegraded":"2024-09-13T15:02:20.494053+0000","last_fullsized":"2024-09-13T15:02:20.494053+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:47:22.461507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":314,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.424291+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T15:02:20.424291+0000","last_peered":"2024-09-13T15:02:20.424291+0000","last_clean":"2024-09-13T15:02:20.424291+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T15:02:20.424291+0000","last_undegraded":"2024-09-13T15:02:20.424291+0000","last_fullsized":"2024-09-13T15:02:20.424291+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.c","version":"378'6657","reported_seq":6681,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:23.424687+0000","last_change":"2024-09-13T15:00:55.873540+0000","last_active":"2024-09-13T15:02:23.424687+0000","last_peered":"2024-09-13T15:02:23.424687+0000","last_clean":"2024-09-13T15:02:23.424687+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:02:23.424687+0000","last_undegraded":"2024-09-13T15:02:23.424687+0000","last_fullsized":"2024-09-13T15:02:23.424687+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6657,"log_dups_size":0,"ondisk_log_size":6657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:49.820345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436273152,"num_objects":6657,"num_object_clones":0,"num_object_copies":19971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6657,"num_write_kb":426048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":323,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.424738+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T15:02:20.424738+0000","last_peered":"2024-09-13T15:02:20.424738+0000","last_clean":"2024-09-13T15:02:20.424738+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T15:02:20.424738+0000","last_undegraded":"2024-09-13T15:02:20.424738+0000","last_fullsized":"2024-09-13T15:02:20.424738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":571,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.445983+0000","last_change":"2024-09-13T14:57:53.351036+0000","last_active":"2024-09-13T15:02:22.445983+0000","last_peered":"2024-09-13T15:02:22.445983+0000","last_clean":"2024-09-13T15:02:22.445983+0000","last_became_active":"2024-09-13T14:57:53.350553+0000","last_became_peered":"2024-09-13T14:57:53.350553+0000","last_unstale":"2024-09-13T15:02:22.445983+0000","last_undegraded":"2024-09-13T15:02:22.445983+0000","last_fullsized":"2024-09-13T15:02:22.445983+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:17.354351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":63,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.384242+0000","last_change":"2024-09-13T14:57:53.360296+0000","last_active":"2024-09-13T15:02:20.384242+0000","last_peered":"2024-09-13T15:02:20.384242+0000","last_clean":"2024-09-13T15:02:20.384242+0000","last_became_active":"2024-09-13T14:57:53.347710+0000","last_became_peered":"2024-09-13T14:57:53.347710+0000","last_unstale":"2024-09-13T15:02:20.384242+0000","last_undegraded":"2024-09-13T15:02:20.384242+0000","last_fullsized":"2024-09-13T15:02:20.384242+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:43.099884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":155,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.493782+0000","last_change":"2024-09-13T15:02:19.407022+0000","last_active":"2024-09-13T15:02:20.493782+0000","last_peered":"2024-09-13T15:02:20.493782+0000","last_clean":"2024-09-13T15:02:20.493782+0000","last_became_active":"2024-09-13T15:02:19.406620+0000","last_became_peered":"2024-09-13T15:02:19.406620+0000","last_unstale":"2024-09-13T15:02:20.493782+0000","last_undegraded":"2024-09-13T15:02:20.493782+0000","last_fullsized":"2024-09-13T15:02:20.493782+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:37.100188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"378'6498","reported_seq":6522,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:23.330156+0000","last_change":"2024-09-13T15:00:55.409383+0000","last_active":"2024-09-13T15:02:23.330156+0000","last_peered":"2024-09-13T15:02:23.330156+0000","last_clean":"2024-09-13T15:02:23.330156+0000","last_became_active":"2024-09-13T15:00:55.407034+0000","last_became_peered":"2024-09-13T15:00:55.407034+0000","last_unstale":"2024-09-13T15:02:23.330156+0000","last_undegraded":"2024-09-13T15:02:23.330156+0000","last_fullsized":"2024-09-13T15:02:23.330156+0000","mapping_epoch":362,"log_start":"366'800","ondisk_log_start":"366'800","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5698,"log_dups_size":0,"ondisk_log_size":5698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:26.464719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425852928,"num_objects":6498,"num_object_clones":0,"num_object_copies":19494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6498,"num_write_kb":415872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":280,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.458523+0000","last_change":"2024-09-13T14:48:29.157322+0000","last_active":"2024-09-13T15:02:22.458523+0000","last_peered":"2024-09-13T15:02:22.458523+0000","last_clean":"2024-09-13T15:02:22.458523+0000","last_became_active":"2024-09-13T14:48:29.153619+0000","last_became_peered":"2024-09-13T14:48:29.153619+0000","last_unstale":"2024-09-13T15:02:22.458523+0000","last_undegraded":"2024-09-13T15:02:22.458523+0000","last_fullsized":"2024-09-13T15:02:22.458523+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:57.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":567,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.445976+0000","last_change":"2024-09-13T14:37:08.778933+0000","last_active":"2024-09-13T15:02:22.445976+0000","last_peered":"2024-09-13T15:02:22.445976+0000","last_clean":"2024-09-13T15:02:22.445976+0000","last_became_active":"2024-09-13T14:37:08.777516+0000","last_became_peered":"2024-09-13T14:37:08.777516+0000","last_unstale":"2024-09-13T15:02:22.445976+0000","last_undegraded":"2024-09-13T15:02:22.445976+0000","last_fullsized":"2024-09-13T15:02:22.445976+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:23.750208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":121,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.457103+0000","last_change":"2024-09-13T14:58:28.428608+0000","last_active":"2024-09-13T15:02:22.457103+0000","last_peered":"2024-09-13T15:02:22.457103+0000","last_clean":"2024-09-13T15:02:22.457103+0000","last_became_active":"2024-09-13T14:58:28.428282+0000","last_became_peered":"2024-09-13T14:58:28.428282+0000","last_unstale":"2024-09-13T15:02:22.457103+0000","last_undegraded":"2024-09-13T15:02:22.457103+0000","last_fullsized":"2024-09-13T15:02:22.457103+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.603606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.603606+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:10.695417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.7","version":"378'6641","reported_seq":6665,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:23.380650+0000","last_change":"2024-09-13T15:00:55.411034+0000","last_active":"2024-09-13T15:02:23.380650+0000","last_peered":"2024-09-13T15:02:23.380650+0000","last_clean":"2024-09-13T15:02:23.380650+0000","last_became_active":"2024-09-13T15:00:55.410636+0000","last_became_peered":"2024-09-13T15:00:55.410636+0000","last_unstale":"2024-09-13T15:02:23.380650+0000","last_undegraded":"2024-09-13T15:02:23.380650+0000","last_fullsized":"2024-09-13T15:02:23.380650+0000","mapping_epoch":362,"log_start":"366'1000","ondisk_log_start":"366'1000","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5641,"log_dups_size":0,"ondisk_log_size":5641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:42.688532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435224576,"num_objects":6641,"num_object_clones":0,"num_object_copies":19923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6641,"num_write_kb":425024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":360,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.386047+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T15:02:20.386047+0000","last_peered":"2024-09-13T15:02:20.386047+0000","last_clean":"2024-09-13T15:02:20.386047+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T15:02:20.386047+0000","last_undegraded":"2024-09-13T15:02:20.386047+0000","last_fullsized":"2024-09-13T15:02:20.386047+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":341,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385352+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T15:02:20.385352+0000","last_peered":"2024-09-13T15:02:20.385352+0000","last_clean":"2024-09-13T15:02:20.385352+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T15:02:20.385352+0000","last_undegraded":"2024-09-13T15:02:20.385352+0000","last_fullsized":"2024-09-13T15:02:20.385352+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":22,"reported_epoch":378,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T15:02:26.067138+0000","last_change":"2024-09-13T15:02:26.066996+0000","last_active":"2024-09-13T15:02:26.067138+0000","last_peered":"2024-09-13T15:02:26.067138+0000","last_clean":"2024-09-13T15:02:26.067138+0000","last_became_active":"2024-09-13T15:02:19.422885+0000","last_became_peered":"2024-09-13T15:02:19.422885+0000","last_unstale":"2024-09-13T15:02:26.067138+0000","last_undegraded":"2024-09-13T15:02:26.067138+0000","last_fullsized":"2024-09-13T15:02:26.067138+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:42.415299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.415299+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":11,"num_write":2,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":8945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.4","version":"378'6689","reported_seq":6713,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:23.372149+0000","last_change":"2024-09-13T15:00:55.873173+0000","last_active":"2024-09-13T15:02:23.372149+0000","last_peered":"2024-09-13T15:02:23.372149+0000","last_clean":"2024-09-13T15:02:23.372149+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:02:23.372149+0000","last_undegraded":"2024-09-13T15:02:23.372149+0000","last_fullsized":"2024-09-13T15:02:23.372149+0000","mapping_epoch":362,"log_start":"366'400","ondisk_log_start":"366'400","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:27.254765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438370304,"num_objects":6689,"num_object_clones":0,"num_object_copies":20067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6689,"num_write_kb":428096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":357,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.398599+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T15:02:20.398599+0000","last_peered":"2024-09-13T15:02:20.398599+0000","last_clean":"2024-09-13T15:02:20.398599+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T15:02:20.398599+0000","last_undegraded":"2024-09-13T15:02:20.398599+0000","last_fullsized":"2024-09-13T15:02:20.398599+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":361,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.424446+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T15:02:20.424446+0000","last_peered":"2024-09-13T15:02:20.424446+0000","last_clean":"2024-09-13T15:02:20.424446+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T15:02:20.424446+0000","last_undegraded":"2024-09-13T15:02:20.424446+0000","last_fullsized":"2024-09-13T15:02:20.424446+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":176,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.457353+0000","last_change":"2024-09-13T14:58:28.424238+0000","last_active":"2024-09-13T15:02:22.457353+0000","last_peered":"2024-09-13T15:02:22.457353+0000","last_clean":"2024-09-13T15:02:22.457353+0000","last_became_active":"2024-09-13T14:58:28.423844+0000","last_became_peered":"2024-09-13T14:58:28.423844+0000","last_unstale":"2024-09-13T15:02:22.457353+0000","last_undegraded":"2024-09-13T15:02:22.457353+0000","last_fullsized":"2024-09-13T15:02:22.457353+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:22.942492+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"378'6598","reported_seq":6679,"reported_epoch":378,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:02:23.335669+0000","last_change":"2024-09-13T15:02:20.660837+0000","last_active":"2024-09-13T15:02:23.335669+0000","last_peered":"2024-09-13T15:02:23.335669+0000","last_clean":"2024-09-13T15:02:18.352546+0000","last_became_active":"2024-09-13T15:02:20.474569+0000","last_became_peered":"2024-09-13T15:02:20.474569+0000","last_unstale":"2024-09-13T15:02:23.335669+0000","last_undegraded":"2024-09-13T15:02:23.335669+0000","last_fullsized":"2024-09-13T15:02:23.335669+0000","mapping_epoch":376,"log_start":"366'900","ondisk_log_start":"366'900","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5698,"log_dups_size":0,"ondisk_log_size":5698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:04.599223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432406528,"num_objects":6598,"num_object_clones":0,"num_object_copies":19794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6577,"num_objects_unfound":0,"num_objects_dirty":6598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6598,"num_write_kb":422272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,5,3],"avail_no_missing":["4","3","5"],"object_location_counts":[{"shards":"3,4,5","objects":6598}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":340,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385211+0000","last_change":"2024-09-13T14:37:08.762309+0000","last_active":"2024-09-13T15:02:20.385211+0000","last_peered":"2024-09-13T15:02:20.385211+0000","last_clean":"2024-09-13T15:02:20.385211+0000","last_became_active":"2024-09-13T14:37:08.762002+0000","last_became_peered":"2024-09-13T14:37:08.762002+0000","last_unstale":"2024-09-13T15:02:20.385211+0000","last_undegraded":"2024-09-13T15:02:20.385211+0000","last_fullsized":"2024-09-13T15:02:20.385211+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:57.986862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":619,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446500+0000","last_change":"2024-09-13T15:02:19.428580+0000","last_active":"2024-09-13T15:02:22.446500+0000","last_peered":"2024-09-13T15:02:22.446500+0000","last_clean":"2024-09-13T15:02:22.446500+0000","last_became_active":"2024-09-13T15:02:19.409871+0000","last_became_peered":"2024-09-13T15:02:19.409871+0000","last_unstale":"2024-09-13T15:02:22.446500+0000","last_undegraded":"2024-09-13T15:02:22.446500+0000","last_fullsized":"2024-09-13T15:02:22.446500+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:34.238752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":181,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.491297+0000","last_change":"2024-09-13T14:57:57.223728+0000","last_active":"2024-09-13T15:02:20.491297+0000","last_peered":"2024-09-13T15:02:20.491297+0000","last_clean":"2024-09-13T15:02:20.491297+0000","last_became_active":"2024-09-13T14:57:19.233883+0000","last_became_peered":"2024-09-13T14:57:19.233883+0000","last_unstale":"2024-09-13T15:02:20.491297+0000","last_undegraded":"2024-09-13T15:02:20.491297+0000","last_fullsized":"2024-09-13T15:02:20.491297+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:41.520060+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2","version":"378'6864","reported_seq":6907,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:27.752717+0000","last_change":"2024-09-13T15:00:55.410530+0000","last_active":"2024-09-13T15:02:27.752717+0000","last_peered":"2024-09-13T15:02:27.752717+0000","last_clean":"2024-09-13T15:02:27.752717+0000","last_became_active":"2024-09-13T15:00:55.410331+0000","last_became_peered":"2024-09-13T15:00:55.410331+0000","last_unstale":"2024-09-13T15:02:27.752717+0000","last_undegraded":"2024-09-13T15:02:27.752717+0000","last_fullsized":"2024-09-13T15:02:27.752717+0000","mapping_epoch":362,"log_start":"366'500","ondisk_log_start":"366'500","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:24.475954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449839104,"num_objects":6864,"num_object_clones":0,"num_object_copies":20592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6864,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446454+0000","last_change":"2024-09-13T15:02:19.417296+0000","last_active":"2024-09-13T15:02:22.446454+0000","last_peered":"2024-09-13T15:02:22.446454+0000","last_clean":"2024-09-13T15:02:22.446454+0000","last_became_active":"2024-09-13T15:02:19.416930+0000","last_became_peered":"2024-09-13T15:02:19.416930+0000","last_unstale":"2024-09-13T15:02:22.446454+0000","last_undegraded":"2024-09-13T15:02:22.446454+0000","last_fullsized":"2024-09-13T15:02:22.446454+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:18.684193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":199,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.447333+0000","last_change":"2024-09-13T14:58:28.428028+0000","last_active":"2024-09-13T15:02:22.447333+0000","last_peered":"2024-09-13T15:02:22.447333+0000","last_clean":"2024-09-13T15:02:22.447333+0000","last_became_active":"2024-09-13T14:58:28.427598+0000","last_became_peered":"2024-09-13T14:58:28.427598+0000","last_unstale":"2024-09-13T15:02:22.447333+0000","last_undegraded":"2024-09-13T15:02:22.447333+0000","last_fullsized":"2024-09-13T15:02:22.447333+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:32:50.930142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":181,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385841+0000","last_change":"2024-09-13T14:58:00.146439+0000","last_active":"2024-09-13T15:02:20.385841+0000","last_peered":"2024-09-13T15:02:20.385841+0000","last_clean":"2024-09-13T15:02:20.385841+0000","last_became_active":"2024-09-13T14:57:19.206097+0000","last_became_peered":"2024-09-13T14:57:19.206097+0000","last_unstale":"2024-09-13T15:02:20.385841+0000","last_undegraded":"2024-09-13T15:02:20.385841+0000","last_fullsized":"2024-09-13T15:02:20.385841+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:00.146384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:00.146384+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:09.697538+0000","scrub_duration":0.0014649660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"378'6753","reported_seq":6785,"reported_epoch":378,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:02:27.752174+0000","last_change":"2024-09-13T15:02:20.487504+0000","last_active":"2024-09-13T15:02:27.752174+0000","last_peered":"2024-09-13T15:02:27.752174+0000","last_clean":"2024-09-13T15:02:18.361205+0000","last_became_active":"2024-09-13T15:02:20.477786+0000","last_became_peered":"2024-09-13T15:02:20.477786+0000","last_unstale":"2024-09-13T15:02:27.752174+0000","last_undegraded":"2024-09-13T15:02:27.752174+0000","last_fullsized":"2024-09-13T15:02:27.752174+0000","mapping_epoch":376,"log_start":"366'900","ondisk_log_start":"366'900","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442564608,"num_objects":6753,"num_object_clones":0,"num_object_copies":20259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6753,"num_objects_unfound":0,"num_objects_dirty":6753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6753,"num_write_kb":432192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[3,4,5],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":6753}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":363,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.424332+0000","last_change":"2024-09-13T14:57:35.357982+0000","last_active":"2024-09-13T15:02:20.424332+0000","last_peered":"2024-09-13T15:02:20.424332+0000","last_clean":"2024-09-13T15:02:20.424332+0000","last_became_active":"2024-09-13T14:57:35.357618+0000","last_became_peered":"2024-09-13T14:57:35.357618+0000","last_unstale":"2024-09-13T15:02:20.424332+0000","last_undegraded":"2024-09-13T15:02:20.424332+0000","last_fullsized":"2024-09-13T15:02:20.424332+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:50.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":382,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.493255+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:02:20.493255+0000","last_peered":"2024-09-13T15:02:20.493255+0000","last_clean":"2024-09-13T15:02:20.493255+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:02:20.493255+0000","last_undegraded":"2024-09-13T15:02:20.493255+0000","last_fullsized":"2024-09-13T15:02:20.493255+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":334,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.425161+0000","last_change":"2024-09-13T14:57:53.363216+0000","last_active":"2024-09-13T15:02:20.425161+0000","last_peered":"2024-09-13T15:02:20.425161+0000","last_clean":"2024-09-13T15:02:20.425161+0000","last_became_active":"2024-09-13T14:57:53.362679+0000","last_became_peered":"2024-09-13T14:57:53.362679+0000","last_unstale":"2024-09-13T15:02:20.425161+0000","last_undegraded":"2024-09-13T15:02:20.425161+0000","last_fullsized":"2024-09-13T15:02:20.425161+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.885757+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"378'6680","reported_seq":6704,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:23.404603+0000","last_change":"2024-09-13T15:00:55.873177+0000","last_active":"2024-09-13T15:02:23.404603+0000","last_peered":"2024-09-13T15:02:23.404603+0000","last_clean":"2024-09-13T15:02:23.404603+0000","last_became_active":"2024-09-13T15:00:55.872769+0000","last_became_peered":"2024-09-13T15:00:55.872769+0000","last_unstale":"2024-09-13T15:02:23.404603+0000","last_undegraded":"2024-09-13T15:02:23.404603+0000","last_fullsized":"2024-09-13T15:02:23.404603+0000","mapping_epoch":362,"log_start":"366'1000","ondisk_log_start":"366'1000","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:22.869519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437780480,"num_objects":6680,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":427520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":367,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.385209+0000","last_change":"2024-09-13T14:47:29.241375+0000","last_active":"2024-09-13T15:02:20.385209+0000","last_peered":"2024-09-13T15:02:20.385209+0000","last_clean":"2024-09-13T15:02:20.385209+0000","last_became_active":"2024-09-13T14:47:29.240940+0000","last_became_peered":"2024-09-13T14:47:29.240940+0000","last_unstale":"2024-09-13T15:02:20.385209+0000","last_undegraded":"2024-09-13T15:02:20.385209+0000","last_fullsized":"2024-09-13T15:02:20.385209+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:22.782323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":344,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.492758+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T15:02:20.492758+0000","last_peered":"2024-09-13T15:02:20.492758+0000","last_clean":"2024-09-13T15:02:20.492758+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T15:02:20.492758+0000","last_undegraded":"2024-09-13T15:02:20.492758+0000","last_fullsized":"2024-09-13T15:02:20.492758+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":377,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.447205+0000","last_change":"2024-09-13T15:02:19.434917+0000","last_active":"2024-09-13T15:02:22.447205+0000","last_peered":"2024-09-13T15:02:22.447205+0000","last_clean":"2024-09-13T15:02:22.447205+0000","last_became_active":"2024-09-13T15:02:19.413020+0000","last_became_peered":"2024-09-13T15:02:19.413020+0000","last_unstale":"2024-09-13T15:02:22.447205+0000","last_undegraded":"2024-09-13T15:02:22.447205+0000","last_fullsized":"2024-09-13T15:02:22.447205+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:46.970485+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"378'6569","reported_seq":6593,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:23.367049+0000","last_change":"2024-09-13T15:00:55.405838+0000","last_active":"2024-09-13T15:02:23.367049+0000","last_peered":"2024-09-13T15:02:23.367049+0000","last_clean":"2024-09-13T15:02:23.367049+0000","last_became_active":"2024-09-13T15:00:55.405095+0000","last_became_peered":"2024-09-13T15:00:55.405095+0000","last_unstale":"2024-09-13T15:02:23.367049+0000","last_undegraded":"2024-09-13T15:02:23.367049+0000","last_fullsized":"2024-09-13T15:02:23.367049+0000","mapping_epoch":362,"log_start":"366'300","ondisk_log_start":"366'300","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6269,"log_dups_size":0,"ondisk_log_size":6269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:55.315749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430505984,"num_objects":6569,"num_object_clones":0,"num_object_copies":19707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6569,"num_write_kb":420416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":59,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.384444+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:02:20.384444+0000","last_peered":"2024-09-13T15:02:20.384444+0000","last_clean":"2024-09-13T15:02:20.384444+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:02:20.384444+0000","last_undegraded":"2024-09-13T15:02:20.384444+0000","last_fullsized":"2024-09-13T15:02:20.384444+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":167,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.491785+0000","last_change":"2024-09-13T15:02:19.430869+0000","last_active":"2024-09-13T15:02:20.491785+0000","last_peered":"2024-09-13T15:02:20.491785+0000","last_clean":"2024-09-13T15:02:20.491785+0000","last_became_active":"2024-09-13T15:02:19.417063+0000","last_became_peered":"2024-09-13T15:02:19.417063+0000","last_unstale":"2024-09-13T15:02:20.491785+0000","last_undegraded":"2024-09-13T15:02:20.491785+0000","last_fullsized":"2024-09-13T15:02:20.491785+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:25.391998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":165,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.493127+0000","last_change":"2024-09-13T14:57:55.208230+0000","last_active":"2024-09-13T15:02:20.493127+0000","last_peered":"2024-09-13T15:02:20.493127+0000","last_clean":"2024-09-13T15:02:20.493127+0000","last_became_active":"2024-09-13T14:57:19.211671+0000","last_became_peered":"2024-09-13T14:57:19.211671+0000","last_unstale":"2024-09-13T15:02:20.493127+0000","last_undegraded":"2024-09-13T15:02:20.493127+0000","last_fullsized":"2024-09-13T15:02:20.493127+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.066938+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":469,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.424194+0000","last_change":"2024-09-13T15:02:19.426511+0000","last_active":"2024-09-13T15:02:20.424194+0000","last_peered":"2024-09-13T15:02:20.424194+0000","last_clean":"2024-09-13T15:02:20.424194+0000","last_became_active":"2024-09-13T15:02:19.391448+0000","last_became_peered":"2024-09-13T15:02:19.391448+0000","last_unstale":"2024-09-13T15:02:20.424194+0000","last_undegraded":"2024-09-13T15:02:20.424194+0000","last_fullsized":"2024-09-13T15:02:20.424194+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:40.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":605,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446106+0000","last_change":"2024-09-13T14:49:05.841111+0000","last_active":"2024-09-13T15:02:22.446106+0000","last_peered":"2024-09-13T15:02:22.446106+0000","last_clean":"2024-09-13T15:02:22.446106+0000","last_became_active":"2024-09-13T14:49:05.840608+0000","last_became_peered":"2024-09-13T14:49:05.840608+0000","last_unstale":"2024-09-13T15:02:22.446106+0000","last_undegraded":"2024-09-13T15:02:22.446106+0000","last_fullsized":"2024-09-13T15:02:22.446106+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:17.596925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":59,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.383740+0000","last_change":"2024-09-13T14:57:53.356010+0000","last_active":"2024-09-13T15:02:20.383740+0000","last_peered":"2024-09-13T15:02:20.383740+0000","last_clean":"2024-09-13T15:02:20.383740+0000","last_became_active":"2024-09-13T14:57:53.354979+0000","last_became_peered":"2024-09-13T14:57:53.354979+0000","last_unstale":"2024-09-13T15:02:20.383740+0000","last_undegraded":"2024-09-13T15:02:20.383740+0000","last_fullsized":"2024-09-13T15:02:20.383740+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:38.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":345,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.491443+0000","last_change":"2024-09-13T14:57:35.358990+0000","last_active":"2024-09-13T15:02:20.491443+0000","last_peered":"2024-09-13T15:02:20.491443+0000","last_clean":"2024-09-13T15:02:20.491443+0000","last_became_active":"2024-09-13T14:57:35.358547+0000","last_became_peered":"2024-09-13T14:57:35.358547+0000","last_unstale":"2024-09-13T15:02:20.491443+0000","last_undegraded":"2024-09-13T15:02:20.491443+0000","last_fullsized":"2024-09-13T15:02:20.491443+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:45.386301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446902+0000","last_change":"2024-09-13T15:02:19.413127+0000","last_active":"2024-09-13T15:02:22.446902+0000","last_peered":"2024-09-13T15:02:22.446902+0000","last_clean":"2024-09-13T15:02:22.446902+0000","last_became_active":"2024-09-13T15:02:19.412502+0000","last_became_peered":"2024-09-13T15:02:19.412502+0000","last_unstale":"2024-09-13T15:02:22.446902+0000","last_undegraded":"2024-09-13T15:02:22.446902+0000","last_fullsized":"2024-09-13T15:02:22.446902+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:36.015849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":324,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.446338+0000","last_change":"2024-09-13T14:47:07.242912+0000","last_active":"2024-09-13T15:02:22.446338+0000","last_peered":"2024-09-13T15:02:22.446338+0000","last_clean":"2024-09-13T15:02:22.446338+0000","last_became_active":"2024-09-13T14:47:07.242539+0000","last_became_peered":"2024-09-13T14:47:07.242539+0000","last_unstale":"2024-09-13T15:02:22.446338+0000","last_undegraded":"2024-09-13T15:02:22.446338+0000","last_fullsized":"2024-09-13T15:02:22.446338+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:57.647782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.445469+0000","last_change":"2024-09-13T15:02:19.414100+0000","last_active":"2024-09-13T15:02:22.445469+0000","last_peered":"2024-09-13T15:02:22.445469+0000","last_clean":"2024-09-13T15:02:22.445469+0000","last_became_active":"2024-09-13T15:02:19.413703+0000","last_became_peered":"2024-09-13T15:02:19.413703+0000","last_unstale":"2024-09-13T15:02:22.445469+0000","last_undegraded":"2024-09-13T15:02:22.445469+0000","last_fullsized":"2024-09-13T15:02:22.445469+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:44.727621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":236,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.492854+0000","last_change":"2024-09-13T14:58:28.426617+0000","last_active":"2024-09-13T15:02:20.492854+0000","last_peered":"2024-09-13T15:02:20.492854+0000","last_clean":"2024-09-13T15:02:20.492854+0000","last_became_active":"2024-09-13T14:58:28.425773+0000","last_became_peered":"2024-09-13T14:58:28.425773+0000","last_unstale":"2024-09-13T15:02:20.492854+0000","last_undegraded":"2024-09-13T15:02:20.492854+0000","last_fullsized":"2024-09-13T15:02:20.492854+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:59.659871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":67,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.493491+0000","last_change":"2024-09-13T14:58:28.428896+0000","last_active":"2024-09-13T15:02:20.493491+0000","last_peered":"2024-09-13T15:02:20.493491+0000","last_clean":"2024-09-13T15:02:20.493491+0000","last_became_active":"2024-09-13T14:58:28.428522+0000","last_became_peered":"2024-09-13T14:58:28.428522+0000","last_unstale":"2024-09-13T15:02:20.493491+0000","last_undegraded":"2024-09-13T15:02:20.493491+0000","last_fullsized":"2024-09-13T15:02:20.493491+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:05.418586+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":360,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.491943+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T15:02:20.491943+0000","last_peered":"2024-09-13T15:02:20.491943+0000","last_clean":"2024-09-13T15:02:20.491943+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T15:02:20.491943+0000","last_undegraded":"2024-09-13T15:02:20.491943+0000","last_fullsized":"2024-09-13T15:02:20.491943+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T15:02:20.384447+0000","last_change":"2024-09-13T15:01:29.854791+0000","last_active":"2024-09-13T15:02:20.384447+0000","last_peered":"2024-09-13T15:02:20.384447+0000","last_clean":"2024-09-13T15:02:20.384447+0000","last_became_active":"2024-09-13T15:01:29.854455+0000","last_became_peered":"2024-09-13T15:01:29.854455+0000","last_unstale":"2024-09-13T15:02:20.384447+0000","last_undegraded":"2024-09-13T15:02:20.384447+0000","last_fullsized":"2024-09-13T15:02:20.384447+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:25.057250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T15:02:22.445445+0000","last_change":"2024-09-13T15:02:19.420296+0000","last_active":"2024-09-13T15:02:22.445445+0000","last_peered":"2024-09-13T15:02:22.445445+0000","last_clean":"2024-09-13T15:02:22.445445+0000","last_became_active":"2024-09-13T15:02:19.419838+0000","last_became_peered":"2024-09-13T15:02:19.419838+0000","last_unstale":"2024-09-13T15:02:22.445445+0000","last_undegraded":"2024-09-13T15:02:22.445445+0000","last_fullsized":"2024-09-13T15:02:22.445445+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:50.598055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":7003308032,"num_objects":106862,"num_object_clones":0,"num_object_copies":320586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19796,"num_objects_unfound":0,"num_objects_dirty":106862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106862,"num_write_kb":6839168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":3604480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21073690624,"data_stored":21073690624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98162,"ondisk_log_size":98162,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1040,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":599313,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":11,"num_write":2,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":664305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365312,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3617444,"kb_used_data":2116440,"kb_used_omap":1,"kb_used_meta":1500990,"kb_avail":90127708,"statfs":{"total":95995035648,"available":92290772992,"internally_reserved":0,"allocated":2167234560,"data_stored":2164987784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1537014104},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561542,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4022120,"kb_used_data":2525724,"kb_used_omap":1,"kb_used_meta":1496382,"kb_avail":89723032,"statfs":{"total":95995035648,"available":91876384768,"internally_reserved":0,"allocated":2586341376,"data_stored":2584098329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1532295512},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790474,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5268448,"kb_used_data":3777876,"kb_used_omap":1,"kb_used_meta":1490558,"kb_avail":88476704,"statfs":{"total":95995035648,"available":90600144896,"internally_reserved":0,"allocated":3868545024,"data_stored":3866306287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1526331736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986702,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4980264,"kb_used_data":3801632,"kb_used_omap":1,"kb_used_meta":1178622,"kb_avail":88764888,"statfs":{"total":95995035648,"available":90895245312,"internally_reserved":0,"allocated":3892871168,"data_stored":3890624363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1206909272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519858,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3152984,"kb_used_data":2177232,"kb_used_omap":4,"kb_used_meta":975675,"kb_avail":90592168,"statfs":{"total":95995035648,"available":92766380032,"internally_reserved":0,"allocated":2229485568,"data_stored":2227250393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":999091648},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304773,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4875424,"kb_used_data":3900248,"kb_used_omap":6,"kb_used_meta":975161,"kb_avail":88869728,"statfs":{"total":95995035648,"available":91002601472,"internally_reserved":0,"allocated":3993853952,"data_stored":3991611643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":998565319},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828077,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3415452,"kb_used_data":2205780,"kb_used_omap":5,"kb_used_meta":1209658,"kb_avail":90329700,"statfs":{"total":95995035648,"available":92497612800,"internally_reserved":0,"allocated":2258718720,"data_stored":2256479420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1238690028},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285942,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1261016,"kb_used_data":237584,"kb_used_omap":5,"kb_used_meta":1023418,"kb_avail":92484136,"statfs":{"total":95995035648,"available":94703755264,"internally_reserved":0,"allocated":243286016,"data_stored":241050115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":1047980091},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34144256,"data_stored":34144256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2254241792,"data_stored":2254241792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3998482432,"data_stored":3998482432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2230452224,"data_stored":2230452224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3887726592,"data_stored":3887726592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3891658752,"data_stored":3891658752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2595880960,"data_stored":2595880960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2181103616,"data_stored":2181103616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:02:29.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 41 2024-09-13T15:02:29.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:29.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:29.768 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:29 smithi003 ceph-mon[107278]: pgmap v1353: 119 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 33 MiB/s wr, 529 op/s; 19796/320664 objects misplaced (6.173%); 481 KiB/s, 8 objects/s recovering 2024-09-13T15:02:29.769 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:29 smithi003 ceph-mon[107278]: from='client.37020 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:02:29.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:29 smithi003 ceph-mon[104007]: pgmap v1353: 119 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 33 MiB/s wr, 529 op/s; 19796/320664 objects misplaced (6.173%); 481 KiB/s, 8 objects/s recovering 2024-09-13T15:02:29.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:29 smithi003 ceph-mon[104007]: from='client.37020 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:29.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:29.910 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:29.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:29.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 111551 111535 86.0463 54.0625 0.0249029 0.0116163 2024-09-13T15:02:30.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:30.108 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:30.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:02:30.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:30.255 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:30.292 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:02:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:02:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:02:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:02:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211667 11.3 11:c090de96:::benchmark_data_smithi003_229782_object89287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.920496+0000", 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 444.36772062400001, 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.41771117699999999, 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211667 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920496+0000", 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920496+0000", 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920494+0000", 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920502+0000", 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3189999999999995e-06 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920517+0000", 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4761e-05 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920524+0000", 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7159999999999997e-06 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920587+0000", 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3699000000000001e-05 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.920724+0000", 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013690299999999999 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.921260+0000", 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00053544300000000003 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.924130+0000", 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028703309999999998 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.924568+0000", 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00043812400000000002 2024-09-13T15:02:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.338055+0000", 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.41348705299999999 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.338147+0000", 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1294999999999999e-05 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.338207+0000", 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0582999999999999e-05 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:153232 12.5 12:af4c6e80:::benchmark_data_smithi003_259359_object37606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:43.439281+0000", 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 226.848935541, 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.23204219800000001, 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 153232 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439281+0000", 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439281+0000", 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439279+0000", 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439287+0000", 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5909999999999998e-06 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439305+0000", 2024-09-13T15:02:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8099999999999999e-05 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439312+0000", 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5830000000000003e-06 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439435+0000", 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012348 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439566+0000", 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000131073 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.439695+0000", 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000128961 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.446857+0000", 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0071618189999999998 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.451096+0000", 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0042388520000000004 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671214+0000", 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.22011769 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671279+0000", 2024-09-13T15:02:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5673000000000006e-05 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:43.671323+0000", 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4024999999999999e-05 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:167621 12.1 12:8edd8664:::benchmark_data_smithi003_259359_object51995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:00.658417+0000", 2024-09-13T15:02:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 209.62979923200001, 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24238083799999999, 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 167621 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.658417+0000", 2024-09-13T15:02:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.658417+0000", 2024-09-13T15:02:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.658416+0000", 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.658422+0000", 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8320000000000002e-06 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.658436+0000", 2024-09-13T15:02:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3711e-05 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.658442+0000", 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1240000000000002e-06 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.658491+0000", 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9291e-05 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.658602+0000", 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011113800000000001 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.658713+0000", 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011097199999999999 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.661965+0000", 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003252296 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.662176+0000", 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00021054999999999999 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.900736+0000", 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.23855977 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.900773+0000", 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6854999999999999e-05 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:00.900798+0000", 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5561000000000001e-05 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:174765 12.f 12:f0a86859:::benchmark_data_smithi003_259359_object59139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:09.410588+0000", 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 200.87762849500001, 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.279047816, 2024-09-13T15:02:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 174765 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410588+0000", 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410588+0000", 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410587+0000", 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410592+0000", 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3739999999999997e-06 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410603+0000", 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0682999999999999e-05 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410634+0000", 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1080000000000001e-05 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410710+0000", 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5748999999999996e-05 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410833+0000", 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012261800000000001 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410951+0000", 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118779 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.416669+0000", 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057180470000000004 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.422507+0000", 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0058372049999999998 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689544+0000", 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26703726700000002 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689593+0000", 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9254e-05 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689636+0000", 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2843999999999998e-05 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:174767 12.f 12:f1a47245:::benchmark_data_smithi003_259359_object59141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:09.412636+0000", 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 200.875580512, 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27712137100000001, 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 174767 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412636+0000", 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412636+0000", 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412635+0000", 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412641+0000", 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6579999999999999e-06 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412658+0000", 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7682000000000001e-05 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412664+0000", 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.818e-06 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.413722+0000", 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001057793 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.413846+0000", 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000123569 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.413969+0000", 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012367699999999999 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.416764+0000", 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027944110000000001 2024-09-13T15:02:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.422532+0000", 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057681219999999997 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689683+0000", 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26715064700000002 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689722+0000", 2024-09-13T15:02:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9638000000000002e-05 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689757+0000", 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5339999999999997e-05 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:174785 12.f 12:f8732d2b:::benchmark_data_smithi003_259359_object59159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:09.438051+0000", 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 200.85016515800001, 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25179602099999998, 2024-09-13T15:02:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 174785 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.438051+0000", 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.438051+0000", 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.438051+0000", 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.438053+0000", 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.34e-06 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.438059+0000", 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.032e-06 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.438063+0000", 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5039999999999998e-06 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.438101+0000", 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8411999999999997e-05 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.438180+0000", 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8343000000000005e-05 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.438246+0000", 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6097000000000001e-05 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.443955+0000", 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057087639999999999 2024-09-13T15:02:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.445613+0000", 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001658865 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689795+0000", 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24418166399999999 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689820+0000", 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4935000000000001e-05 2024-09-13T15:02:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689847+0000", 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7492e-05 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:174789 12.f 12:f2fb2149:::benchmark_data_smithi003_259359_object59163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:09.439170+0000", 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 200.84904615400001, 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25081042100000001, 2024-09-13T15:02:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 174789 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.439170+0000", 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.439170+0000", 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.439169+0000", 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.439175+0000", 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8239999999999996e-06 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.439191+0000", 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6039999999999999e-05 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.439198+0000", 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7839999999999998e-06 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.441014+0000", 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0018158180000000001 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.441130+0000", 2024-09-13T15:02:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011613299999999999 2024-09-13T15:02:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:02:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.441252+0000", 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012222200000000001 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.444129+0000", 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028771220000000002 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.445762+0000", 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0016327290000000001 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689905+0000", 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.244142635 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689947+0000", 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2120999999999999e-05 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689981+0000", 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4044999999999997e-05 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:76390 13.1 13:8bd6f4c6:::benchmark_data_smithi003_293363_object76389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:51.035126+0000", 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 39.253090286999999, 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13343565299999999, 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 76390 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035126+0000", 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035126+0000", 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035125+0000", 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035155+0000", 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9655e-05 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035167+0000", 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2381999999999999e-05 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035173+0000", 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8300000000000001e-06 2024-09-13T15:02:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035234+0000", 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1587000000000005e-05 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035356+0000", 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012108300000000001 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,4", 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.035494+0000", 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013872900000000001 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.040873+0000", 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00537916 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.040879+0000", 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8499999999999999e-06 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.168419+0000", 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.127540187 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.168522+0000", 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010275 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.168562+0000", 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9786000000000002e-05 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:76400 13.f 13:f7b7998a:::benchmark_data_smithi003_293363_object76399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:51.041645+0000", 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 39.246571492999998, 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.133654626, 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:30.345 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 76400 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.041645+0000", 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.041645+0000", 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.346 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.347 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.041644+0000", 2024-09-13T15:02:30.347 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:30.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.347 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.347 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.041693+0000", 2024-09-13T15:02:30.347 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9450000000000003e-05 2024-09-13T15:02:30.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.041705+0000", 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1939e-05 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.041711+0000", 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8939999999999998e-06 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.164862+0000", 2024-09-13T15:02:30.348 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.123151205 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.165060+0000", 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00019728200000000001 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.165224+0000", 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016410799999999999 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.174044+0000", 2024-09-13T15:02:30.349 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0088198219999999997 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.174256+0000", 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00021275800000000001 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.175209+0000", 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00095315999999999999 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.175262+0000", 2024-09-13T15:02:30.350 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3053000000000002e-05 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:51.175300+0000", 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7258999999999997e-05 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:77491 13.4 13:2716fcbd:::benchmark_data_smithi003_293363_object77490:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:52.282958+0000", 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 38.005258417999997, 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25187820300000002, 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.351 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 77491 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.282958+0000", 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.352 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.282958+0000", 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.282949+0000", 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999905 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.282989+0000", 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0160999999999997e-05 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.353 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283005+0000", 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6155999999999999e-05 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283012+0000", 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1670000000000002e-06 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283065+0000", 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2516000000000003e-05 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.354 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283183+0000", 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011841 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283309+0000", 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012641099999999999 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.355 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287025+0000", 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0037159749999999998 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534707+0000", 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24768168700000001 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534745+0000", 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7808999999999997e-05 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534797+0000", 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1719999999999999e-05 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534836+0000", 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9771999999999998e-05 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.357 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.36909.0:77488 13.7 e372/362 13:eeaf4936:::benchmark_data_smithi003_293363_object77487:head v 372'4874, mlcod=372'4872)", 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:52.283562+0000", 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 38.004654639999998, 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25382753200000002, 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283562+0000", 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.358 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283562+0000", 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283561+0000", 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283587+0000", 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6443000000000001e-05 2024-09-13T15:02:30.359 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283601+0000", 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3518e-05 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283608+0000", 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8159999999999996e-06 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.360 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283652+0000", 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4401999999999999e-05 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283672+0000", 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9782000000000001e-05 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537344+0000", 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25367196800000003 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.361 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537390+0000", 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5686000000000003e-05 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.36909.0:77494 13.7 e372/362 13:e3fb757b:::benchmark_data_smithi003_293363_object77493:head v 372'4875, mlcod=372'4872)", 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:52.286365+0000", 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 38.001851942999998, 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25109604800000002, 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.362 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286365+0000", 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286365+0000", 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286364+0000", 2024-09-13T15:02:30.363 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286388+0000", 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.393e-05 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286400+0000", 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2193e-05 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.364 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286406+0000", 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9320000000000001e-06 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286467+0000", 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0798000000000003e-05 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286485+0000", 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8448e-05 2024-09-13T15:02:30.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537436+0000", 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25095098700000001 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537461+0000", 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4785000000000001e-05 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.366 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:77498 13.f 13:fc5c02f9:::benchmark_data_smithi003_293363_object77497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:52.286932+0000", 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 38.001284261000002, 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.250623335, 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 77498 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.367 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286932+0000", 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286932+0000", 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286931+0000", 2024-09-13T15:02:30.368 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286959+0000", 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8075000000000001e-05 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286969+0000", 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0227000000000001e-05 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.286974+0000", 2024-09-13T15:02:30.369 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0189999999999997e-06 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287125+0000", 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015082299999999999 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287237+0000", 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000112565 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:02:30.370 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287362+0000", 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012497800000000001 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534725+0000", 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.247362422 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537434+0000", 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027090080000000002 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537468+0000", 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3865999999999997e-05 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537513+0000", 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5423999999999997e-05 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537556+0000", 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2658999999999999e-05 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.372 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop_reply(client.36909.0:77491 13.4 e372/362 ondisk, result = 0)", 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:52.287054+0000", 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 38.001162569999998, 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24765585400000001, 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "started", 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287054+0000", 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.373 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287054+0000", 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287054+0000", 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287058+0000", 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9459999999999997e-06 2024-09-13T15:02:30.374 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287066+0000", 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9060000000000002e-06 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.287074+0000", 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2139999999999999e-06 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534686+0000", 2024-09-13T15:02:30.375 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24761233999999999 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534698+0000", 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2276000000000001e-05 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534710+0000", 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1721e-05 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.36909.0:77497 13.7 e372/362 13:e94a3da8:::benchmark_data_smithi003_293363_object77496:head v 372'4876, mlcod=372'4872)", 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:52.289980+0000", 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 37.998236775000002, 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25818336600000003, 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.289980+0000", 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.377 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.378 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.289980+0000", 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.289979+0000", 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.290006+0000", 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6775999999999999e-05 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534627+0000", 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.244621595 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534647+0000", 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9950999999999999e-05 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534686+0000", 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9085999999999998e-05 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534710+0000", 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3839999999999999e-05 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.547839+0000", 2024-09-13T15:02:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013128862999999999 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.548163+0000", 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000324075 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop_reply(client.36909.0:77498 13.f e372/362 ondisk, result = 0)", 2024-09-13T15:02:30.382 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:52.292801+0000", 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 37.995415903999998, 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.241926168, 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "started", 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.292801+0000", 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.292801+0000", 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.292800+0000", 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.292807+0000", 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.311e-06 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.384 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534654+0000", 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24184741200000001 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534663+0000", 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6989999999999992e-06 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534713+0000", 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9913999999999997e-05 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.385 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534719+0000", 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5200000000000003e-06 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534727+0000", 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4159999999999998e-06 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:79901 13.4 13:27ac56b7:::benchmark_data_smithi003_293363_object79900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:54.738607+0000", 2024-09-13T15:02:30.386 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 35.549610088999998, 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.16119814700000001, 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 79901 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.738607+0000", 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.738607+0000", 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.738605+0000", 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.738643+0000", 2024-09-13T15:02:30.388 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8223000000000003e-05 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.738656+0000", 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.289e-05 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.738662+0000", 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4540000000000004e-06 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.389 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.738708+0000", 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5797999999999998e-05 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.738894+0000", 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000186279 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.739028+0000", 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013362700000000001 2024-09-13T15:02:30.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.742512+0000", 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0034835510000000001 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.743889+0000", 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001377769 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.899691+0000", 2024-09-13T15:02:30.391 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.155801255 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.899759+0000", 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7983000000000003e-05 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:54.899805+0000", 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6097999999999998e-05 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:93217 13.f 13:f1306830:::benchmark_data_smithi003_293363_object93216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:07.617399+0000", 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 22.670817369000002, 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13420901800000001, 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 93217 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.617399+0000", 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.617399+0000", 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.617399+0000", 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:30.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.617427+0000", 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8521999999999999e-05 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.617434+0000", 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0489999999999998e-06 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.617440+0000", 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7080000000000002e-06 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.623373+0000", 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0059328790000000003 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.623493+0000", 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000119765 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.623635+0000", 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000142392 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.630992+0000", 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00735712 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.635859+0000", 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0048663339999999999 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.751481+0000", 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.115622893 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.751536+0000", 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4321999999999999e-05 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.751608+0000", 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2503999999999996e-05 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.398 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:93220 13.4 13:247c9847:::benchmark_data_smithi003_293363_object93219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:07.622026+0000", 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 22.666190968999999, 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13147029299999999, 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 93220 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.399 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.622026+0000", 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.622026+0000", 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.622025+0000", 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.400 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.622056+0000", 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1307999999999999e-05 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.622064+0000", 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.763e-06 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.622068+0000", 2024-09-13T15:02:30.401 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9879999999999998e-06 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.622131+0000", 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2384000000000004e-05 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.622267+0000", 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013602699999999999 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.402 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.622406+0000", 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013911699999999999 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.631847+0000", 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0094410810000000005 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.636174+0000", 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0043267560000000002 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.403 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.753411+0000", 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.117236959 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.753457+0000", 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6672000000000003e-05 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.753496+0000", 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8733000000000003e-05 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.404 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:93223 13.f 13:fd129008:::benchmark_data_smithi003_293363_object93222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:07.626925+0000", 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 22.661291372000001, 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13169562600000001, 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 93223 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:30.405 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.626925+0000", 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.626925+0000", 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.626924+0000", 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.406 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.626958+0000", 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4167e-05 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.626971+0000", 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2979000000000001e-05 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.626977+0000", 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.054e-06 2024-09-13T15:02:30.407 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.627195+0000", 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000217796 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.627370+0000", 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017529700000000001 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.627494+0000", 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000124382 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.631931+0000", 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0044367629999999998 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.635895+0000", 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003963884 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.758536+0000", 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.122641242 2024-09-13T15:02:30.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.758582+0000", 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6388999999999999e-05 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:07.758621+0000", 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8420000000000001e-05 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:30.410 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:30.411 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:02:30.411 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:02:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:02:30.217Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:30.447 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:30.549 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:30.590 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:30 smithi019 ceph-mon[91296]: pgmap v1354: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_wait, 115 active+clean; 6.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 743 op/s; 19982/329628 objects misplaced (6.062%); 837 KiB/s, 13 objects/s recovering 2024-09-13T15:02:30.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:30.737 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:30.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:30.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:30.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:30 smithi003 ceph-mon[107278]: pgmap v1354: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_wait, 115 active+clean; 6.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 743 op/s; 19982/329628 objects misplaced (6.062%); 837 KiB/s, 13 objects/s recovering 2024-09-13T15:02:30.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:30 smithi003 ceph-mon[104007]: pgmap v1354: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_wait, 115 active+clean; 6.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 743 op/s; 19982/329628 objects misplaced (6.062%); 837 KiB/s, 13 objects/s recovering 2024-09-13T15:02:30.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 112384 112368 85.6319 52.0625 0.00852416 0.0116712 2024-09-13T15:02:30.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:31.026 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:31.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:31.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:31.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:31.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:31.297 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:31.348 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:02:31.399 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:31.443 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:31.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:31.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:31.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:31.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:31.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:31.778 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:31.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:31.839 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:31 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T15:02:31.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:31 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T15:02:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:31.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:31.929 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:31 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T15:02:31.964 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:31.965 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:31.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 113176 113160 85.1965 49.5 0.00510985 0.0117301 2024-09-13T15:02:31.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:32.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:32.140 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:32.180 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:32.282 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:32.323 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:32.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:32.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:32.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:32.582 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:02:32.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:32.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:32 smithi003 ceph-mon[104007]: pgmap v1355: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_wait, 115 active+clean; 6.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 743 op/s; 19982/331290 objects misplaced (6.032%); 813 KiB/s, 13 objects/s recovering 2024-09-13T15:02:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2547088391' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:32 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:32.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:32 smithi003 ceph-mon[107278]: pgmap v1355: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_wait, 115 active+clean; 6.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 743 op/s; 19982/331290 objects misplaced (6.032%); 813 KiB/s, 13 objects/s recovering 2024-09-13T15:02:32.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2547088391' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:32.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:32 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:32.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:32.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:32.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:32.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:32 smithi019 ceph-mon[91296]: pgmap v1355: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_wait, 115 active+clean; 6.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 743 op/s; 19982/331290 objects misplaced (6.032%); 813 KiB/s, 13 objects/s recovering 2024-09-13T15:02:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2547088391' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:32 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:32.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 114032 114016 84.8192 53.5 0.0105373 0.0117847 2024-09-13T15:02:32.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:33.018 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:33.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:33.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:33.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:33.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:33.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:33.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:33.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:33.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:33.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:33.652 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.697 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:33.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:33.757 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:33 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:33.757 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:33 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:02:33.757 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:33 smithi019 ceph-mon[91296]: osdmap e379: 8 total, 8 up, 8 in 2024-09-13T15:02:33.758 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:33 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:33.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:33 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:33.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:33 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:02:33.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:33 smithi003 ceph-mon[104007]: osdmap e379: 8 total, 8 up, 8 in 2024-09-13T15:02:33.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:33 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:33.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:33 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:33.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:33 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:02:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:33 smithi003 ceph-mon[107278]: osdmap e379: 8 total, 8 up, 8 in 2024-09-13T15:02:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:33 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:33.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:33.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:33.925 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:02:33.925 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:02:33.925 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.925 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:114849 13.0 13:0d3aa3b3:::benchmark_data_smithi003_293363_object114848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-13T15:02:33.925 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:33.917603+0000", 2024-09-13T15:02:33.925 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.0056339290000000002, 2024-09-13T15:02:33.925 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0056688340000000002, 2024-09-13T15:02:33.925 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 114849 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.917603+0000", 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.926 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.917603+0000", 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.917601+0000", 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.917629+0000", 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8058000000000001e-05 2024-09-13T15:02:33.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.917639+0000", 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.9699999999999994e-06 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.917644+0000", 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0830000000000003e-06 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.918829+0000", 2024-09-13T15:02:33.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0011851089999999999 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.918985+0000", 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015584299999999999 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.919148+0000", 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016309199999999999 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.929 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.923173+0000", 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0040250190000000003 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.923182+0000", 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7800000000000006e-06 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:114850 13.e 13:7319442e:::benchmark_data_smithi003_293363_object114849:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-13T15:02:33.930 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:33.918016+0000", 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.0052204390000000003, 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0053479570000000004, 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 114850 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.918016+0000", 2024-09-13T15:02:33.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.918016+0000", 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.918015+0000", 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:33.932 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.918080+0000", 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4380000000000004e-05 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.918090+0000", 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.9909999999999997e-06 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.918095+0000", 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2089999999999996e-06 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:33.933 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.920030+0000", 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001935062 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.920316+0000", 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00028652000000000001 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:33.920766+0000", 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00044915500000000002 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:33.934 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:33.935 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:33.935 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-09-13T15:02:33.935 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 2 2024-09-13T15:02:33.935 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:02:33.935 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:33.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 114892 114876 84.4536 53.75 0.0432525 0.0118361 2024-09-13T15:02:33.979 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:34.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:34.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:34.215 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:34.254 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:34.356 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:34.392 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:34.398 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:02:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:02:34] "GET /metrics HTTP/1.1" 200 36562 "" "Prometheus/2.51.0" 2024-09-13T15:02:34.494 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:34.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:34.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:34.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:34 smithi003 ceph-mon[107278]: pgmap v1357: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 54 MiB/s wr, 857 op/s; 20446/342351 objects misplaced (5.972%); 845 KiB/s, 13 objects/s recovering 2024-09-13T15:02:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:34 smithi003 ceph-mon[104007]: pgmap v1357: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 54 MiB/s wr, 857 op/s; 20446/342351 objects misplaced (5.972%); 845 KiB/s, 13 objects/s recovering 2024-09-13T15:02:34.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:34.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:34.896 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:34 smithi019 ceph-mon[91296]: pgmap v1357: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 54 MiB/s wr, 857 op/s; 20446/342351 objects misplaced (5.972%); 845 KiB/s, 13 objects/s recovering 2024-09-13T15:02:34.911 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:34.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:02:34.970 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 115770 115754 84.1096 54.875 0.00850788 0.0118832 2024-09-13T15:02:34.973 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:35.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.132 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:35.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:35.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:35.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:35.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:35.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:35.454 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:35.503 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:35.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:35.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:35.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:35.719 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:02:35.719 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:35.755 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:02:35.756 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:35.782 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:02:35.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:35.795 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:35.809 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:02:35.810 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:35.839 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:35.841 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:02:35.841 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:35.875 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:02:35.875 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:35.892 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:35 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1473330862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:35.909 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:02:35.909 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:35.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:35.945 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:02:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:35.970 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 116598 116582 83.7377 51.75 0.0128338 0.0119369 2024-09-13T15:02:35.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:35.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:35 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1473330862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:35.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:35 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1473330862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:36.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:36.136 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:36.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:36.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:36.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:36.419 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:36.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:36.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:36.662 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.703 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:36.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:36.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:36.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:36.923 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:36 smithi019 ceph-mon[91296]: pgmap v1358: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 54 MiB/s wr, 857 op/s; 20446/342351 objects misplaced (5.972%); 845 KiB/s, 13 objects/s recovering 2024-09-13T15:02:36.923 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:36 smithi019 ceph-mon[91296]: 1.b scrub starts 2024-09-13T15:02:36.923 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:36 smithi019 ceph-mon[91296]: 1.b scrub ok 2024-09-13T15:02:36.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:36.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 117498 117482 83.4253 56.25 0.0176339 0.0119815 2024-09-13T15:02:36.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:37.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:37.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:36 smithi003 ceph-mon[107278]: pgmap v1358: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 54 MiB/s wr, 857 op/s; 20446/342351 objects misplaced (5.972%); 845 KiB/s, 13 objects/s recovering 2024-09-13T15:02:37.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:36 smithi003 ceph-mon[107278]: 1.b scrub starts 2024-09-13T15:02:37.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:36 smithi003 ceph-mon[107278]: 1.b scrub ok 2024-09-13T15:02:37.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:36 smithi003 ceph-mon[104007]: pgmap v1358: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 54 MiB/s wr, 857 op/s; 20446/342351 objects misplaced (5.972%); 845 KiB/s, 13 objects/s recovering 2024-09-13T15:02:37.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:36 smithi003 ceph-mon[104007]: 1.b scrub starts 2024-09-13T15:02:37.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:36 smithi003 ceph-mon[104007]: 1.b scrub ok 2024-09-13T15:02:37.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:37.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:37.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:37.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.416 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:37.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:37.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:02:37.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:37.577 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:37.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:37.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:37.818 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:37.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:37.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: specified pgp_num 41 > pg_num 31 2024-09-13T15:02:37.911 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:02:37.911 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:02:37.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:37.969 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 118347 118331 83.0841 53.0625 0.00841929 0.0120308 2024-09-13T15:02:37.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:38.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:38.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:38.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:38.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2558921997' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T15:02:38.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:37 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T15:02:38.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2558921997' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T15:02:38.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:37 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T15:02:38.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2558921997' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T15:02:38.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:37 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T15:02:38.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:38.270 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:38.343 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:38.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:38.409 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:38.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:38.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:38.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:02:38.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:38.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:38.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:38.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:38.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:38.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:38.969 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:38.970 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0093 2024-09-13T15:02:38.970 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 119209 2024-09-13T15:02:38.970 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T15:02:38.970 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T15:02:38.970 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 82.7755 2024-09-13T15:02:38.970 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 40.6642 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 229.188 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1324 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 650.628 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3667 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0120767 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0263109 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.25093 2024-09-13T15:02:38.971 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00155173 2024-09-13T15:02:38.978 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T15:02:38.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:02:39.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:39.086 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:39.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:39.200 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:39.226 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:38 smithi019 ceph-mon[91296]: pgmap v1359: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 54 MiB/s wr, 856 op/s; 20736/346365 objects misplaced (5.987%); 844 KiB/s, 13 objects/s recovering 2024-09-13T15:02:39.241 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:39.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:39.366 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:39.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:38 smithi003 ceph-mon[104007]: pgmap v1359: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 54 MiB/s wr, 856 op/s; 20736/346365 objects misplaced (5.987%); 844 KiB/s, 13 objects/s recovering 2024-09-13T15:02:39.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:38 smithi003 ceph-mon[107278]: pgmap v1359: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 54 MiB/s wr, 856 op/s; 20736/346365 objects misplaced (5.987%); 844 KiB/s, 13 objects/s recovering 2024-09-13T15:02:39.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:39.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:39.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:39.592 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.637 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:39.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:39.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:39.775 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:39.877 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:39.925 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:40.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:40.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:02:40.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:40.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:40.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:40.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:40.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:40.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:40.611 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:40.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:02:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:02:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:40.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:40.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:40.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:40.844 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:40.877 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:40.879 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:40.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:41.019 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:41.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:41.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:41.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:41.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:41.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:41.304 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:41.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:41.417 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:41 smithi003 ceph-mon[104007]: pgmap v1360: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.2 GiB data, 32 GiB used, 684 GiB / 715 GiB avail; 53 MiB/s wr, 855 op/s; 20931/355302 objects misplaced (5.891%); 812 KiB/s, 12 objects/s recovering 2024-09-13T15:02:41.417 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:41 smithi003 ceph-mon[107278]: pgmap v1360: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.2 GiB data, 32 GiB used, 684 GiB / 715 GiB avail; 53 MiB/s wr, 855 op/s; 20931/355302 objects misplaced (5.891%); 812 KiB/s, 12 objects/s recovering 2024-09-13T15:02:41.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:41 smithi019 ceph-mon[91296]: pgmap v1360: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.2 GiB data, 32 GiB used, 684 GiB / 715 GiB avail; 53 MiB/s wr, 855 op/s; 20931/355302 objects misplaced (5.891%); 812 KiB/s, 12 objects/s recovering 2024-09-13T15:02:41.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:41.545 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:41.578 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:41.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:41.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:41.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:41.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:41.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:41.991 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:42.093 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:42.131 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:42.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:42.233 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:42.275 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:42.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:42.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:42.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:42.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:42.649 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.683 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:42.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:42.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:42.915 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:42.953 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:43.055 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:43.099 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:43 smithi003 ceph-mon[107278]: pgmap v1361: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.2 GiB data, 32 GiB used, 684 GiB / 715 GiB avail; 52 MiB/s wr, 853 op/s; 20931/355476 objects misplaced (5.888%); 812 KiB/s, 12 objects/s recovering 2024-09-13T15:02:43.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:43.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:43 smithi003 ceph-mon[104007]: pgmap v1361: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.2 GiB data, 32 GiB used, 684 GiB / 715 GiB avail; 52 MiB/s wr, 853 op/s; 20931/355476 objects misplaced (5.888%); 812 KiB/s, 12 objects/s recovering 2024-09-13T15:02:43.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:43.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:43.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:43.337 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:43.376 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:43 smithi019 ceph-mon[91296]: pgmap v1361: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.2 GiB data, 32 GiB used, 684 GiB / 715 GiB avail; 52 MiB/s wr, 853 op/s; 20931/355476 objects misplaced (5.888%); 812 KiB/s, 12 objects/s recovering 2024-09-13T15:02:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:43.479 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.522 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:43.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:43.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:43.671 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:02:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:43.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.675 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:43.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:43.703 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:02:43.703 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:43.723 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:02:43.723 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:43.742 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:02:43.742 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:43.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:43.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:43.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:43.836 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:02:43.836 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:43.867 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:02:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:43.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:43.905 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:02:43.905 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:43.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:43.935 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:02:43.936 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:44.036 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:44.087 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:44.176 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:02:44.189 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:44.232 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:44.248 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/779997093' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:44.248 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2840813013' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:44.248 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:44 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:44.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:44.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/779997093' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2840813013' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:44.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:44 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:44.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:02:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:02:44] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T15:02:44.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/779997093' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:44.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2840813013' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:44.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:44 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:44.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:44.561 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:02:44.561 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:02:44.561 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:02:44.561 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:02:44.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.561 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36438.0:211645 11.f 11:fd3da10e:::benchmark_data_smithi003_229782_object89265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:02:44.561 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:55:05.896332+0000", 2024-09-13T15:02:44.561 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 458.56019999799997, 2024-09-13T15:02:44.561 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.45682966899999999, 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36438", 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211645 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.896332+0000", 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.562 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.896332+0000", 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.896330+0000", 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.896336+0000", 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3740000000000003e-06 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.563 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.896352+0000", 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5184000000000001e-05 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.896359+0000", 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0149999999999998e-06 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.896397+0000", 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8744e-05 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.564 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.896483+0000", 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5277000000000004e-05 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.896555+0000", 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2418999999999999e-05 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.899075+0000", 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0025197040000000002 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:05.908022+0000", 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0089468199999999994 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353063+0000", 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.44504137399999999 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353120+0000", 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7243000000000001e-05 2024-09-13T15:02:44.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:55:06.353161+0000", 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1230000000000003e-05 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:129079 12.3 12:c082c005:::benchmark_data_smithi003_259359_object13453:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:09.506844+0000", 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 274.949688073, 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.121197574, 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.567 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 129079 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.506844+0000", 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.506844+0000", 2024-09-13T15:02:44.568 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.506842+0000", 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.506849+0000", 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4339999999999997e-06 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.506861+0000", 2024-09-13T15:02:44.569 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2329e-05 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507118+0000", 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00025668899999999999 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507309+0000", 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00019124300000000001 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507319+0000", 2024-09-13T15:02:44.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0226e-05 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.518820+0000", 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.011500579 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.518841+0000", 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0896000000000001e-05 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.625270+0000", 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1064291550000001 2024-09-13T15:02:44.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.628041+0000", 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027715320000000002 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:129094 12.3 12:cc721e61:::benchmark_data_smithi003_259359_object13468:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:09.507133+0000", 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 274.94939861300003, 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.156844939, 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.572 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 129094 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507133+0000", 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507133+0000", 2024-09-13T15:02:44.573 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507132+0000", 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507137+0000", 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5369999999999999e-06 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507146+0000", 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.346e-06 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507152+0000", 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5439999999999998e-06 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507417+0000", 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00026501900000000002 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.507425+0000", 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.79e-06 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.519501+0000", 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.012076239 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.519510+0000", 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9649999999999997e-06 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.576 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.628362+0000", 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1088520850000001 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.629133+0000", 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00077132000000000004 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,3,4", 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.629454+0000", 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00032073300000000002 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.577 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.649352+0000", 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.019898563000000001 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.654251+0000", 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0048984429999999997 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.656625+0000", 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0023737110000000001 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.663819+0000", 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0071945669999999998 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.663872+0000", 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3236000000000003e-05 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.663978+0000", 2024-09-13T15:02:44.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010562 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:129097 12.3 12:c0b5bf8c:::benchmark_data_smithi003_259359_object13471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:09.511977+0000", 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 274.94455488400001, 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1522716370000001, 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 129097 2024-09-13T15:02:44.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.511977+0000", 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.511977+0000", 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.511976+0000", 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.511981+0000", 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0830000000000003e-06 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.511991+0000", 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.9729999999999992e-06 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.511998+0000", 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2910000000000002e-06 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.512049+0000", 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1298999999999998e-05 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:09.512061+0000", 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1411e-05 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.519535+0000", 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.007474615 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:10.519542+0000", 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7309999999999996e-06 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.645839+0000", 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1262967100000001 2024-09-13T15:02:44.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.645959+0000", 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012053900000000001 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,3,4", 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.646096+0000", 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000136959 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.654283+0000", 2024-09-13T15:02:44.585 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0081862689999999995 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.655865+0000", 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0015818270000000001 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.664016+0000", 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0081516060000000005 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.586 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.664034+0000", 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.766e-05 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.664100+0000", 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6000999999999994e-05 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:11.664248+0000", 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000148679 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.587 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144141 12.3 12:c49e4e56:::benchmark_data_smithi003_259359_object28515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.825373+0000", 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.631158782, 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0041502980000001, 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.588 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144141 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.825373+0000", 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.825373+0000", 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.825372+0000", 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.825377+0000", 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.091e-06 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.825388+0000", 2024-09-13T15:02:44.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0766e-05 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.825394+0000", 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6679999999999998e-06 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.825437+0000", 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3668999999999998e-05 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.825444+0000", 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6150000000000002e-06 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.821241+0000", 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99579691000000004 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.821337+0000", 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6599000000000001e-05 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.592 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.821417+0000", 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9377999999999998e-05 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.823916+0000", 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024996279999999998 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.825828+0000", 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0019119689999999999 2024-09-13T15:02:44.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829459+0000", 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003630561 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829498+0000", 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9264000000000002e-05 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829523+0000", 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5068000000000001e-05 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144142 12.3 12:c7012d3b:::benchmark_data_smithi003_259359_object28516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.827678+0000", 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.628854104, 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0019622239999999, 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144142 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.595 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.827678+0000", 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.827678+0000", 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.827676+0000", 2024-09-13T15:02:44.596 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.827683+0000", 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.404e-06 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.827693+0000", 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0390999999999999e-05 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.827699+0000", 2024-09-13T15:02:44.597 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4820000000000002e-06 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.827745+0000", 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6105e-05 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.827752+0000", 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3819999999999998e-06 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.598 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.823953+0000", 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99620104399999998 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.824073+0000", 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000119796 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.824204+0000", 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000131235 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.599 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829426+0000", 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0052221029999999996 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829532+0000", 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000105832 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829598+0000", 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5921999999999997e-05 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.600 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829622+0000", 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3725000000000001e-05 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829640+0000", 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8142999999999999e-05 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144151 12.3 12:c3cf5978:::benchmark_data_smithi003_259359_object28525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.601 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.842166+0000", 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.61436557600001, 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99186849099999996, 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144151 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.842166+0000", 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.842166+0000", 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.842165+0000", 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.842170+0000", 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9960000000000001e-06 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.842179+0000", 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9900000000000003e-06 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.842183+0000", 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6399999999999999e-06 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.842225+0000", 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2403e-05 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.842233+0000", 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0849999999999997e-06 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.825869+0000", 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98363529599999999 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.825975+0000", 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010676200000000001 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.605 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.826094+0000", 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118649 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829538+0000", 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0034443999999999998 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829568+0000", 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9825e-05 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.606 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.833898+0000", 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0043298850000000003 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.833981+0000", 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3081e-05 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.834035+0000", 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3343000000000001e-05 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144175 12.3 12:c9ba940f:::benchmark_data_smithi003_259359_object28549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.872808+0000", 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.583723991, 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96673017999999999, 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144175 2024-09-13T15:02:44.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.872808+0000", 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.872808+0000", 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.609 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.872807+0000", 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.872812+0000", 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5759999999999996e-06 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.872822+0000", 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0156999999999999e-05 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.610 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.872827+0000", 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2930000000000002e-06 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.872876+0000", 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9465999999999997e-05 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.872896+0000", 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9789e-05 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.827781+0000", 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95488556499999999 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.827861+0000", 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9699999999999999e-05 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.827928+0000", 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7223000000000003e-05 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.612 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829647+0000", 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0017187389999999999 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829678+0000", 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0729999999999999e-05 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839477+0000", 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0097993490000000006 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.613 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839515+0000", 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7628e-05 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839538+0000", 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3056000000000001e-05 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144180 12.3 12:c46ba274:::benchmark_data_smithi003_259359_object28554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.614 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.881588+0000", 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.57494377699999, 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95804958100000004, 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144180 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.615 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.881588+0000", 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.881588+0000", 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.881587+0000", 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:44.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.881593+0000", 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.003e-06 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.881605+0000", 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2397e-05 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.617 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.881610+0000", 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8879999999999996e-06 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.881654+0000", 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4475999999999999e-05 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.881676+0000", 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2072000000000001e-05 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.618 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829704+0000", 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.94802708300000005 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829776+0000", 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2012000000000001e-05 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.829861+0000", 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5339000000000004e-05 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.619 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.833856+0000", 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039955340000000002 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839445+0000", 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0055882270000000003 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839593+0000", 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014881000000000001 2024-09-13T15:02:44.620 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839618+0000", 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4779000000000001e-05 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839638+0000", 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9261000000000001e-05 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.621 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144184 12.3 12:c85eb269:::benchmark_data_smithi003_259359_object28558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.887211+0000", 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.56932072699999, 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95252954400000001, 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144184 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.622 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.887211+0000", 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.887211+0000", 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.887209+0000", 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.623 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.887221+0000", 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2197e-05 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.887248+0000", 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6682999999999998e-05 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.887258+0000", 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0916e-05 2024-09-13T15:02:44.624 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.887351+0000", 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2579999999999998e-05 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.887363+0000", 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2403e-05 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.625 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.831799+0000", 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.944435574 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.831899+0000", 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.9646000000000001e-05 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.831970+0000", 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1327000000000003e-05 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.626 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839432+0000", 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0074624849999999996 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839449+0000", 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6356000000000001e-05 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839700+0000", 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000251029 2024-09-13T15:02:44.627 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839721+0000", 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1364000000000001e-05 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839741+0000", 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9301e-05 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.628 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144187 12.3 12:c56765ba:::benchmark_data_smithi003_259359_object28561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.889203+0000", 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.56732880999999, 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96404044799999999, 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144187 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.629 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.889203+0000", 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.889203+0000", 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.889202+0000", 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.630 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.889205+0000", 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.038e-06 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.889221+0000", 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5389000000000001e-05 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.631 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.889227+0000", 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6599999999999998e-06 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.889273+0000", 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5221e-05 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.889297+0000", 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4264e-05 2024-09-13T15:02:44.632 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.834072+0000", 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.94477480000000003 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.834179+0000", 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010678799999999999 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.633 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.834270+0000", 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1804000000000004e-05 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839568+0000", 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0052974329999999998 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839572+0000", 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1289999999999998e-06 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.634 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.853160+0000", 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.01358797 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.853203+0000", 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3132e-05 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.853243+0000", 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.036e-05 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.635 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144190 12.3 12:c8c97e14:::benchmark_data_smithi003_259359_object28564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.893588+0000", 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.56294389300001, 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96256075600000002, 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.636 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144190 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.893588+0000", 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.893588+0000", 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.637 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.893587+0000", 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.893592+0000", 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7829999999999998e-06 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.893602+0000", 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8300000000000008e-06 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.638 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.893607+0000", 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8680000000000004e-06 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.893641+0000", 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3946000000000003e-05 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.893646+0000", 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6909999999999996e-06 2024-09-13T15:02:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.836820+0000", 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.94317351999999999 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.836922+0000", 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000102853 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.837030+0000", 2024-09-13T15:02:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000107229 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839669+0000", 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002639668 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.843485+0000", 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0038157199999999999 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.641 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856063+0000", 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012577715999999999 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856111+0000", 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8565e-05 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856149+0000", 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7218000000000002e-05 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.642 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144216 12.3 12:caab1751:::benchmark_data_smithi003_259359_object28590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.937214+0000", 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.519317524, 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.91911119100000005, 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144216 2024-09-13T15:02:44.643 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.937214+0000", 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.937214+0000", 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.644 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.937214+0000", 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.937217+0000", 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.675e-06 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.937226+0000", 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1649999999999995e-06 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.645 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.937232+0000", 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4020000000000001e-06 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.937282+0000", 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9975000000000002e-05 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.937296+0000", 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4219000000000001e-05 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.646 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839766+0000", 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.90247010299999997 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839836+0000", 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9800000000000003e-05 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.647 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.839914+0000", 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7856999999999997e-05 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.847644+0000", 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0077302810000000003 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856159+0000", 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0085150710000000008 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.648 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856256+0000", 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6979999999999996e-05 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856294+0000", 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7589000000000002e-05 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856325+0000", 2024-09-13T15:02:44.649 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1603000000000002e-05 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144224 12.3 12:cba5c09e:::benchmark_data_smithi003_259359_object28598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.943745+0000", 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.512786672, 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.91270554599999998, 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.650 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144224 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.943745+0000", 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.943745+0000", 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.651 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.943743+0000", 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.943750+0000", 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5359999999999998e-06 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.943765+0000", 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5115e-05 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.652 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.943771+0000", 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1820000000000003e-06 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.943819+0000", 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7670999999999998e-05 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.943831+0000", 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2021000000000001e-05 2024-09-13T15:02:44.653 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.843528+0000", 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.89969712000000002 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.843634+0000", 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106388 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.843747+0000", 2024-09-13T15:02:44.654 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011301799999999999 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.847675+0000", 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039272170000000002 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856334+0000", 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0086594570000000006 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856404+0000", 2024-09-13T15:02:44.655 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0178000000000005e-05 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856431+0000", 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6937999999999999e-05 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856451+0000", 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9440999999999999e-05 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.656 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144238 12.3 12:cdf0e37c:::benchmark_data_smithi003_259359_object28612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.970231+0000", 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.48630103299999, 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.89482389699999998, 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144238 2024-09-13T15:02:44.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.970231+0000", 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.970231+0000", 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.658 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.970229+0000", 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.970236+0000", 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.195e-06 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.970250+0000", 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4790000000000001e-05 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.970257+0000", 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.392e-06 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.970303+0000", 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6684000000000002e-05 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.970309+0000", 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2349999999999999e-06 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.845945+0000", 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.87563683999999997 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.846016+0000", 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0140000000000003e-05 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.846080+0000", 2024-09-13T15:02:44.661 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3965000000000002e-05 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.853126+0000", 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0070467710000000003 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856338+0000", 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0032119309999999999 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.864977+0000", 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.008638405 2024-09-13T15:02:44.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.865016+0000", 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9650000000000002e-05 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.865055+0000", 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8256000000000001e-05 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.663 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144249 12.3 12:c1057144:::benchmark_data_smithi003_259359_object28623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.991302+0000", 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.465229862, 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.87801961900000003, 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144249 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.991302+0000", 2024-09-13T15:02:44.664 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.991302+0000", 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.991301+0000", 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.665 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.991307+0000", 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2820000000000002e-06 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.991320+0000", 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3379e-05 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.991326+0000", 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9009999999999999e-06 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.991376+0000", 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9944000000000002e-05 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.991382+0000", 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3119999999999999e-06 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.847694+0000", 2024-09-13T15:02:44.667 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.85631209699999999 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.847783+0000", 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8800000000000004e-05 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.847881+0000", 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7924000000000006e-05 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.668 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856220+0000", 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0083392650000000002 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856381+0000", 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016017900000000001 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.869195+0000", 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012814344 2024-09-13T15:02:44.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.670 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.670 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.670 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.869281+0000", 2024-09-13T15:02:44.672 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5844e-05 2024-09-13T15:02:44.672 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.672 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.672 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.672 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.869321+0000", 2024-09-13T15:02:44.672 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0651000000000001e-05 2024-09-13T15:02:44.672 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.672 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144250 12.3 12:c89bef3e:::benchmark_data_smithi003_259359_object28624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:31.994434+0000", 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.46209784800001, 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.87504256800000002, 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144250 2024-09-13T15:02:44.673 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.994434+0000", 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.994434+0000", 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.674 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.994433+0000", 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.994438+0000", 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5290000000000002e-06 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.994448+0000", 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0829000000000001e-05 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.675 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.994453+0000", 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5129999999999998e-06 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.994506+0000", 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3426e-05 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:31.994511+0000", 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7700000000000001e-06 2024-09-13T15:02:44.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.850137+0000", 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.85562629700000004 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.850375+0000", 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00023722500000000001 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.850491+0000", 2024-09-13T15:02:44.677 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000116589 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856366+0000", 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0058749830000000003 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856387+0000", 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0239999999999999e-05 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.678 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.869355+0000", 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012968822 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.869442+0000", 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7058999999999994e-05 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.869476+0000", 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4026000000000003e-05 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.679 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144271 12.3 12:c75bbf6a:::benchmark_data_smithi003_259359_object28645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:32.063793+0000", 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.39273890000001, 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.80593484299999996, 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.680 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144271 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.063793+0000", 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.063793+0000", 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.681 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.063791+0000", 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.063798+0000", 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4209999999999998e-06 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.063813+0000", 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4423000000000001e-05 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.682 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.063820+0000", 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3950000000000003e-06 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.063865+0000", 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4709999999999997e-05 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.063872+0000", 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1799999999999999e-06 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.683 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.853271+0000", 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.78939926000000005 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.853377+0000", 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000105239 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.853486+0000", 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000109244 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.684 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.859221+0000", 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057346560000000003 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.861791+0000", 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0025699830000000001 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.869649+0000", 2024-09-13T15:02:44.685 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0078589560000000003 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.869691+0000", 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1464999999999997e-05 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.869728+0000", 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6693000000000003e-05 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.686 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144284 12.3 12:c5fcd1de:::benchmark_data_smithi003_259359_object28658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:32.090252+0000", 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.36627974999999, 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.78379003000000003, 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144284 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.687 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.090252+0000", 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.090252+0000", 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.688 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.090251+0000", 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.090257+0000", 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6080000000000003e-06 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.090271+0000", 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3899e-05 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.689 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.090277+0000", 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6120000000000004e-06 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.090315+0000", 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7460999999999999e-05 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.090319+0000", 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7670000000000003e-06 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.690 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856471+0000", 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.76615125500000003 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856531+0000", 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9973999999999999e-05 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.856595+0000", 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4784999999999997e-05 2024-09-13T15:02:44.691 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.861834+0000", 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0052382510000000002 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.861841+0000", 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6839999999999996e-06 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.873953+0000", 2024-09-13T15:02:44.692 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012111960999999999 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.874004+0000", 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0667000000000001e-05 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.874042+0000", 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8012000000000001e-05 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.693 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:144291 12.3 12:c4bf8cc0:::benchmark_data_smithi003_259359_object28665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:58:32.132458+0000", 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 252.32407377300001, 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.74168973699999996, 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 144291 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.694 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.132458+0000", 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.132458+0000", 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.132456+0000", 2024-09-13T15:02:44.695 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.132464+0000", 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9880000000000004e-06 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.132479+0000", 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5034999999999999e-05 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:02:44.696 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.132485+0000", 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1140000000000002e-06 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.132535+0000", 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0047999999999999e-05 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.132542+0000", 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7519999999999999e-06 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.697 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.859274+0000", 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.72673185399999995 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.859400+0000", 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012646000000000001 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.859508+0000", 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000107518 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.698 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.699 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:02:44.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:44.700 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.861935+0000", 2024-09-13T15:02:44.700 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0024271330000000002 2024-09-13T15:02:44.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.700 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.700 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.700 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.862663+0000", 2024-09-13T15:02:44.700 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000728047 2024-09-13T15:02:44.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.874079+0000", 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011416387 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.874118+0000", 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8458999999999999e-05 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:58:32.874148+0000", 2024-09-13T15:02:44.701 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9751e-05 2024-09-13T15:02:44.702 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.702 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.702 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.702 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:02:44.702 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:02:44.702 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:02:44.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:44.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:44.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:44.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:45.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:45.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:45.136 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:02:45.137 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:02:45.137 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [1, 2, 3, 4, 6, 7, 5, 0] 2024-09-13T15:02:45.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd out 3 2024-09-13T15:02:45.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:45.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:45.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:45.372 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:45.424 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:45 smithi019 ceph-mon[91296]: pgmap v1362: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.1 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 48 MiB/s wr, 1.07k op/s; 20518/347133 objects misplaced (5.911%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T15:02:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:45 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:45 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:02:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:45 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:45.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:45 smithi019 ceph-mon[91296]: osdmap e380: 8 total, 8 up, 8 in 2024-09-13T15:02:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[104007]: pgmap v1362: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.1 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 48 MiB/s wr, 1.07k op/s; 20518/347133 objects misplaced (5.911%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T15:02:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:02:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:45.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[104007]: osdmap e380: 8 total, 8 up, 8 in 2024-09-13T15:02:45.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[107278]: pgmap v1362: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.1 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 48 MiB/s wr, 1.07k op/s; 20518/347133 objects misplaced (5.911%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T15:02:45.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:45.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:02:45.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:45.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:45 smithi003 ceph-mon[107278]: osdmap e380: 8 total, 8 up, 8 in 2024-09-13T15:02:45.526 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:45.554 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:45.564 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:45.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.709 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:45.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:45.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:45.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:45.948 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:45.981 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:46.083 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.121 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:46.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:46.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:46.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:46.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:46.362 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:46.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T15:02:46.408 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:46.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:46.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:46.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:46.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:46.805 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:46.841 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:46.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:02:46.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:47.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:47.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:47.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:47.173 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:47 smithi003 ceph-mon[104007]: pgmap v1364: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.1 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 32 MiB/s wr, 863 op/s; 20518/347133 objects misplaced (5.911%); 863 KiB/s, 13 objects/s recovering 2024-09-13T15:02:47.252 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:47.288 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:47.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:47 smithi019 ceph-mon[91296]: pgmap v1364: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.1 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 32 MiB/s wr, 863 op/s; 20518/347133 objects misplaced (5.911%); 863 KiB/s, 13 objects/s recovering 2024-09-13T15:02:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:47 smithi003 ceph-mon[107278]: pgmap v1364: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.1 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 32 MiB/s wr, 863 op/s; 20518/347133 objects misplaced (5.911%); 863 KiB/s, 13 objects/s recovering 2024-09-13T15:02:47.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:47.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:47.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:02:47.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:47.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:47.665 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:47.698 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:47.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:47.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:47.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:47.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:48.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:48.125 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:48.195 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.3. 2024-09-13T15:02:48.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:48.262 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:48.324 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/238348148' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-13T15:02:48.324 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:48 smithi003 ceph-mon[107278]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-13T15:02:48.324 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:47 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:02:47.982+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e380 definitely_dead 0 2024-09-13T15:02:48.324 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/238348148' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-13T15:02:48.324 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:48 smithi003 ceph-mon[104007]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-13T15:02:48.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:48.387 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/238348148' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-13T15:02:48.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:48 smithi019 ceph-mon[91296]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-13T15:02:48.471 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:48.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:48.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:48.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:48.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:48.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:48.760 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:48.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:48.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:48.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:49.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:49.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:49.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:49.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:49.259 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:49.361 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:49.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:49.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:49 smithi003 ceph-mon[107278]: pgmap v1365: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 102 B/s rd, 24 MiB/s wr, 862 op/s; 20261/343176 objects misplaced (5.904%); 864 KiB/s, 13 objects/s recovering 2024-09-13T15:02:49.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/238348148' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-13T15:02:49.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:49 smithi003 ceph-mon[107278]: osdmap e381: 8 total, 8 up, 7 in 2024-09-13T15:02:49.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:49 smithi003 ceph-mon[104007]: pgmap v1365: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 102 B/s rd, 24 MiB/s wr, 862 op/s; 20261/343176 objects misplaced (5.904%); 864 KiB/s, 13 objects/s recovering 2024-09-13T15:02:49.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/238348148' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-13T15:02:49.472 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:49 smithi003 ceph-mon[104007]: osdmap e381: 8 total, 8 up, 7 in 2024-09-13T15:02:49.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:49.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:49.615 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:49.638 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:49 smithi019 ceph-mon[91296]: pgmap v1365: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 7.0 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 102 B/s rd, 24 MiB/s wr, 862 op/s; 20261/343176 objects misplaced (5.904%); 864 KiB/s, 13 objects/s recovering 2024-09-13T15:02:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/238348148' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-13T15:02:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:49 smithi019 ceph-mon[91296]: osdmap e381: 8 total, 8 up, 7 in 2024-09-13T15:02:49.741 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:49.763 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:49.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:49.887 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:49.914 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:49.989 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:50.012 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:50.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:50.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:50.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:50.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:50.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:50.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:50 smithi003 ceph-mon[107278]: osdmap e382: 8 total, 8 up, 7 in 2024-09-13T15:02:50.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:50 smithi003 ceph-mon[107278]: 13.3 starting backfill to osd.0 from (366'1400,380'7865] MIN to 380'7866 2024-09-13T15:02:50.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:50 smithi003 ceph-mon[107278]: osdmap e383: 8 total, 8 up, 7 in 2024-09-13T15:02:50.307 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:50 smithi003 ceph-mon[104007]: osdmap e382: 8 total, 8 up, 7 in 2024-09-13T15:02:50.307 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:50 smithi003 ceph-mon[104007]: 13.3 starting backfill to osd.0 from (366'1400,380'7865] MIN to 380'7866 2024-09-13T15:02:50.307 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:50 smithi003 ceph-mon[104007]: osdmap e383: 8 total, 8 up, 7 in 2024-09-13T15:02:50.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:50.411 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:50.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:50.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:50.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:50 smithi019 ceph-mon[91296]: osdmap e382: 8 total, 8 up, 7 in 2024-09-13T15:02:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:50 smithi019 ceph-mon[91296]: 13.3 starting backfill to osd.0 from (366'1400,380'7865] MIN to 380'7866 2024-09-13T15:02:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:50 smithi019 ceph-mon[91296]: osdmap e383: 8 total, 8 up, 7 in 2024-09-13T15:02:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:02:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:02:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:50.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:50.791 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:50.931 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.970 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:50.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:51.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:51.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:51.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:51.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:51.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:51.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[107278]: pgmap v1368: 119 pgs: 28 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 88 active+clean; 6.8 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 0 B/s wr, 690 op/s; 174/334695 objects degraded (0.052%); 19740/334695 objects misplaced (5.898%); 960 KiB/s, 14 objects/s recovering 2024-09-13T15:02:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[107278]: 13.2 starting backfill to osd.6 from (0'0,0'0] MAX to 380'7910 2024-09-13T15:02:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[107278]: 13.b starting backfill to osd.4 from (0'0,0'0] MAX to 380'7943 2024-09-13T15:02:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[107278]: 13.6 starting backfill to osd.2 from (0'0,0'0] MAX to 380'7848 2024-09-13T15:02:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 11 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-09-13T15:02:51.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 174/334695 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:02:51.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[104007]: pgmap v1368: 119 pgs: 28 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 88 active+clean; 6.8 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 0 B/s wr, 690 op/s; 174/334695 objects degraded (0.052%); 19740/334695 objects misplaced (5.898%); 960 KiB/s, 14 objects/s recovering 2024-09-13T15:02:51.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[104007]: 13.2 starting backfill to osd.6 from (0'0,0'0] MAX to 380'7910 2024-09-13T15:02:51.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[104007]: 13.b starting backfill to osd.4 from (0'0,0'0] MAX to 380'7943 2024-09-13T15:02:51.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[104007]: 13.6 starting backfill to osd.2 from (0'0,0'0] MAX to 380'7848 2024-09-13T15:02:51.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 11 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-09-13T15:02:51.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:51 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 174/334695 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:02:51.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:51.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:51.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.664 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:51.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:51 smithi019 ceph-mon[91296]: pgmap v1368: 119 pgs: 28 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 88 active+clean; 6.8 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 0 B/s wr, 690 op/s; 174/334695 objects degraded (0.052%); 19740/334695 objects misplaced (5.898%); 960 KiB/s, 14 objects/s recovering 2024-09-13T15:02:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:51 smithi019 ceph-mon[91296]: 13.2 starting backfill to osd.6 from (0'0,0'0] MAX to 380'7910 2024-09-13T15:02:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:51 smithi019 ceph-mon[91296]: 13.b starting backfill to osd.4 from (0'0,0'0] MAX to 380'7943 2024-09-13T15:02:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:51 smithi019 ceph-mon[91296]: 13.6 starting backfill to osd.2 from (0'0,0'0] MAX to 380'7848 2024-09-13T15:02:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:51 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 11 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-09-13T15:02:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:51 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 174/334695 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:02:51.766 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:51.806 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:51.908 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.950 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:51.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:52.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:52.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:52.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:52.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:52.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:52.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:52.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:52.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:52.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:52.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:52.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:52.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:52.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:52.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:52.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:52.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:53.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:53.047 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:53.149 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:53.183 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:53.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:53.301 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:02:53.302 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:02:53.302 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-13T15:02:53.302 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:02:53.302 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:53 smithi003 ceph-mon[107278]: pgmap v1370: 119 pgs: 1 active+remapped+backfilling, 28 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 87 active+clean; 6.8 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 0 B/s wr, 761 op/s; 174/333423 objects degraded (0.052%); 26499/333423 objects misplaced (7.948%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T15:02:53.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:53 smithi003 ceph-mon[104007]: pgmap v1370: 119 pgs: 1 active+remapped+backfilling, 28 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 87 active+clean; 6.8 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 0 B/s wr, 761 op/s; 174/333423 objects degraded (0.052%); 26499/333423 objects misplaced (7.948%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T15:02:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.325 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:53.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:53.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:53.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:53.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:53.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:53 smithi019 ceph-mon[91296]: pgmap v1370: 119 pgs: 1 active+remapped+backfilling, 28 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 87 active+clean; 6.8 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 0 B/s wr, 761 op/s; 174/333423 objects degraded (0.052%); 26499/333423 objects misplaced (7.948%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T15:02:53.686 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:53.733 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:53.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:53.882 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:53.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:54.019 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:54.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:54.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:54.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:54.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:54.256 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 5, 0] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:02:54.256 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:02:54.256 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-13T15:02:54.256 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:02:54.256 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:02:54.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-13T15:02:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.297 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:54.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:54.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:54.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:54 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 14 pgs peering) 2024-09-13T15:02:54.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:54 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 14 pgs peering) 2024-09-13T15:02:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:02:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:02:54] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T15:02:54.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:54 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 14 pgs peering) 2024-09-13T15:02:54.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:54.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:54.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:54.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:54.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:54.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:55.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:55.037 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:55.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:55.139 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:55.163 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:55.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:55.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:55.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:55.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:55.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:55.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:55.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:55.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:55 smithi019 ceph-mon[91296]: pgmap v1371: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 202/326061 objects degraded (0.062%); 39377/326061 objects misplaced (12.077%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T15:02:55.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:55 smithi003 ceph-mon[107278]: pgmap v1371: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 202/326061 objects degraded (0.062%); 39377/326061 objects misplaced (12.077%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T15:02:55.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:55 smithi003 ceph-mon[104007]: pgmap v1371: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 202/326061 objects degraded (0.062%); 39377/326061 objects misplaced (12.077%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T15:02:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:55.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:55.850 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:55.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:55.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:56.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:56.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:56.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:56.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:56.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T15:02:56.378 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:56.416 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:56.518 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:56.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:56 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T15:02:56.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 202/326061 objects degraded (0.062%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:02:56.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:56 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T15:02:56.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 202/326061 objects degraded (0.062%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:02:56.565 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:56.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:56.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:56 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T15:02:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 202/326061 objects degraded (0.062%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:02:56.697 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:56.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:56.834 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:56.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:56.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:57.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:57.118 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:57.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:57.262 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:57.277 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:57.300 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:02:57.300 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:02:57.345 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:02:57.345 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:57.365 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:57.382 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:02:57.382 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:57.399 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:57.413 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:02:57.413 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:57.413 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:57 smithi003 ceph-mon[104007]: pgmap v1372: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 202/326061 objects degraded (0.062%); 39377/326061 objects misplaced (12.077%); 2.9 MiB/s, 47 objects/s recovering 2024-09-13T15:02:57.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:57 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1561755306' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:57.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:57 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/51634411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:57.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:57 smithi003 ceph-mon[107278]: pgmap v1372: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 202/326061 objects degraded (0.062%); 39377/326061 objects misplaced (12.077%); 2.9 MiB/s, 47 objects/s recovering 2024-09-13T15:02:57.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:57 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1561755306' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:57.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:57 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/51634411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:57.446 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:02:57.447 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:57.480 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:02:57.480 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:57.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:57.507 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:02:57.507 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:57.543 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:02:57.544 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:57.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:57.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:57 smithi019 ceph-mon[91296]: pgmap v1372: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 202/326061 objects degraded (0.062%); 39377/326061 objects misplaced (12.077%); 2.9 MiB/s, 47 objects/s recovering 2024-09-13T15:02:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:57 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1561755306' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:57 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/51634411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:57.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:57.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:57.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:57.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:57.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:58.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:58.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:58.199 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:58.239 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:58.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:58.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:58.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:58.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1561755306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:58.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[104007]: osdmap e384: 8 total, 8 up, 7 in 2024-09-13T15:02:58.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:58.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:58.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:58.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1561755306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:58.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[107278]: osdmap e384: 8 total, 8 up, 7 in 2024-09-13T15:02:58.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:58.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:58.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:58.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:58.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:58.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:58 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:58 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1561755306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:58 smithi019 ceph-mon[91296]: osdmap e384: 8 total, 8 up, 7 in 2024-09-13T15:02:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:02:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:58.742 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:58.776 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:58.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:58.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:59.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:59.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:59.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:59.198 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:59.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:59.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:59.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:59.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:59.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:59.540 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:02:59 smithi003 ceph-mon[107278]: pgmap v1374: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 504 op/s; 202/322590 objects degraded (0.063%); 38227/322590 objects misplaced (11.850%); 6.8 MiB/s, 110 objects/s recovering 2024-09-13T15:02:59.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:02:59 smithi003 ceph-mon[104007]: pgmap v1374: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 504 op/s; 202/322590 objects degraded (0.063%); 38227/322590 objects misplaced (11.850%); 6.8 MiB/s, 110 objects/s recovering 2024-09-13T15:02:59.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:59.611 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:02:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:59.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:02:59 smithi019 ceph-mon[91296]: pgmap v1374: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 504 op/s; 202/322590 objects degraded (0.063%); 38227/322590 objects misplaced (11.850%); 6.8 MiB/s, 110 objects/s recovering 2024-09-13T15:02:59.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:59.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:59.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:59.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:02:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:59.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:00.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:00.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:00.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:00.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:00.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:00.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:00.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:00.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:00.614 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:00.665 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:03:00.665 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T15:03:00.665 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:03:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:03:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:03:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:00.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:00.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:00.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:00.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:01.021 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:01.123 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:01.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:01.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:01.401 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:01.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:01.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:01.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:01.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:03:01.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:01 smithi003 ceph-mon[107278]: pgmap v1375: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 756 op/s; 414/312198 objects degraded (0.133%); 37087/312198 objects misplaced (11.879%); 6.0 MiB/s, 97 objects/s recovering 2024-09-13T15:03:01.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:01 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 414/312198 objects degraded (0.133%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:01.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:01 smithi003 ceph-mon[104007]: pgmap v1375: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 756 op/s; 414/312198 objects degraded (0.133%); 37087/312198 objects misplaced (11.879%); 6.0 MiB/s, 97 objects/s recovering 2024-09-13T15:03:01.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:01 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 414/312198 objects degraded (0.133%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:01.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:01.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:01 smithi019 ceph-mon[91296]: pgmap v1375: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 756 op/s; 414/312198 objects degraded (0.133%); 37087/312198 objects misplaced (11.879%); 6.0 MiB/s, 97 objects/s recovering 2024-09-13T15:03:01.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:01 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 414/312198 objects degraded (0.133%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:01.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:01.815 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:01.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:01.953 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:01.994 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:02.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:03:02.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:02.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:02.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:02.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:02.414 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:02.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:02.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:02.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:02.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:02.690 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:02.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:02.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:02.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:02.963 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:03.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:03.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:03.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:03.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:03.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:03.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:03.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:03.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:03.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:03.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:03 smithi003 ceph-mon[107278]: pgmap v1376: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 798 op/s; 414/309468 objects degraded (0.134%); 36307/309468 objects misplaced (11.732%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T15:03:03.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:03 smithi003 ceph-mon[104007]: pgmap v1376: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 798 op/s; 414/309468 objects degraded (0.134%); 36307/309468 objects misplaced (11.732%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T15:03:03.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:03 smithi019 ceph-mon[91296]: pgmap v1376: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 798 op/s; 414/309468 objects degraded (0.134%); 36307/309468 objects misplaced (11.732%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T15:03:03.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:03.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:03.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:03.901 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:03.945 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:04.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:04.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:04.196 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.236 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:04.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:04.311 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:03:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:03:04] "GET /metrics HTTP/1.1" 200 36553 "" "Prometheus/2.51.0" 2024-09-13T15:03:04.323 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:04.326 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:04.337 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:04.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:04.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:04.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:04.612 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:04.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:04.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:04.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:04.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:04.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:04.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:05.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:05.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:05.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:05.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:05.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:05.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:05.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:05.492 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:05.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:05.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:05 smithi003 ceph-mon[104007]: pgmap v1377: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 619/298188 objects degraded (0.208%); 34895/298188 objects misplaced (11.702%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:03:05.600 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:05 smithi003 ceph-mon[107278]: pgmap v1377: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 619/298188 objects degraded (0.208%); 34895/298188 objects misplaced (11.702%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:03:05.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:05 smithi019 ceph-mon[91296]: pgmap v1377: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 619/298188 objects degraded (0.208%); 34895/298188 objects misplaced (11.702%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:03:05.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:05.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:05.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:05.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:05.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:05.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:06.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:06.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:03:06.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:06.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:06.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:06.324 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:06.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:06.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:06.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 619/298188 objects degraded (0.208%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:06.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 619/298188 objects degraded (0.208%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 619/298188 objects degraded (0.208%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:06.697 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:06.734 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:06.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:06.870 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:06.972 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:07.017 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:07.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:07.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:07.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:07.288 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:07.348 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:07 smithi003 ceph-mon[107278]: pgmap v1378: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 619/298188 objects degraded (0.208%); 34895/298188 objects misplaced (11.702%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:03:07.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:07 smithi003 ceph-mon[104007]: pgmap v1378: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 619/298188 objects degraded (0.208%); 34895/298188 objects misplaced (11.702%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:03:07.389 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:07.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:07.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:07.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:07 smithi019 ceph-mon[91296]: pgmap v1378: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 619/298188 objects degraded (0.208%); 34895/298188 objects misplaced (11.702%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:03:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.709 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:07.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:07.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:07.885 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 5, 0] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:03:07.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:03:07.885 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-13T15:03:07.885 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:03:07.885 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:03:07.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-13T15:03:07.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:07.993 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:08.095 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:08.133 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:08.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:08.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:08.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:08.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:08.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:08.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:08.670 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:08.712 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:08.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:08.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:08.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:08.962 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:03:08.962 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:09.002 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:09.003 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:03:09.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:09.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:09.043 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:03:09.043 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:09.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:03:09.071 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:09.100 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:03:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:09.129 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:09.130 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:03:09.130 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:09.166 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:03:09.166 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:09.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:09.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:09.194 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:03:09.195 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:09.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:09.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:09.346 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:03:09.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:09.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:09.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.621 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:09.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:09 smithi019 ceph-mon[91296]: pgmap v1379: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 619/294717 objects degraded (0.210%); 33878/294717 objects misplaced (11.495%); 6.8 MiB/s, 108 objects/s recovering 2024-09-13T15:03:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/623676508' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:09 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1262849458' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:09 smithi003 ceph-mon[107278]: pgmap v1379: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 619/294717 objects degraded (0.210%); 33878/294717 objects misplaced (11.495%); 6.8 MiB/s, 108 objects/s recovering 2024-09-13T15:03:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/623676508' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:09 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1262849458' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:09 smithi003 ceph-mon[104007]: pgmap v1379: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 619/294717 objects degraded (0.210%); 33878/294717 objects misplaced (11.495%); 6.8 MiB/s, 108 objects/s recovering 2024-09-13T15:03:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/623676508' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:09 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1262849458' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:09.722 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:09.769 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:09.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:09.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:09.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:10.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:10.050 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:10.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:10.193 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:10.295 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:10.332 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:10.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:10.473 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:10.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:10.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:03:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:03:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:10.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:10 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:10.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:10 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:10.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:10 smithi019 ceph-mon[91296]: osdmap e385: 8 total, 8 up, 7 in 2024-09-13T15:03:10.588 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:10 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:10.588 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:10 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:10.589 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:10 smithi003 ceph-mon[107278]: osdmap e385: 8 total, 8 up, 7 in 2024-09-13T15:03:10.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:10 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:10.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:10 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:10.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:10 smithi003 ceph-mon[104007]: osdmap e385: 8 total, 8 up, 7 in 2024-09-13T15:03:10.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:10.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:10.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:10.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:10.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:10.885 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:10.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:11.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:11.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:11.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:11.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:11.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:11.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:11.436 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:11.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:11.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:11 smithi019 ceph-mon[91296]: pgmap v1381: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 793/283869 objects degraded (0.279%); 32676/283869 objects misplaced (11.511%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T15:03:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 793/283869 objects degraded (0.279%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:11.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:11 smithi003 ceph-mon[107278]: pgmap v1381: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 793/283869 objects degraded (0.279%); 32676/283869 objects misplaced (11.511%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T15:03:11.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 793/283869 objects degraded (0.279%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:11.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:11 smithi003 ceph-mon[104007]: pgmap v1381: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 793/283869 objects degraded (0.279%); 32676/283869 objects misplaced (11.511%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T15:03:11.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 793/283869 objects degraded (0.279%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:11.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:11.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:11.852 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:11.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:12.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:12.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:12.216 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.259 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:12.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:12.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:12.396 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:12.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:03:12.498 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.535 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:12.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:12.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:12.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:12.787 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:12.830 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:12.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:12.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:12.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:13.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:13.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:13.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:13.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:13.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:13.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:13.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:13.474 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:13.509 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:13.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:13.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:13 smithi003 ceph-mon[107278]: pgmap v1382: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 793/281136 objects degraded (0.282%); 31958/281136 objects misplaced (11.367%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T15:03:13.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:13.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:13 smithi003 ceph-mon[104007]: pgmap v1382: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 793/281136 objects degraded (0.282%); 31958/281136 objects misplaced (11.367%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T15:03:13.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:13.655 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:13 smithi019 ceph-mon[91296]: pgmap v1382: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 793/281136 objects degraded (0.282%); 31958/281136 objects misplaced (11.367%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T15:03:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:13.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:13.773 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:03:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "" 2024-09-13T15:03:13.774 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:03:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:13.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:13.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:13.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:14.031 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:14.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:03:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:14.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:14.178 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:14.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:14.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:03:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:03:14] "GET /metrics HTTP/1.1" 200 36553 "" "Prometheus/2.51.0" 2024-09-13T15:03:14.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:14.509 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:14.611 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:14.654 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:14.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:14.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:14.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:14.929 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:15.031 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:15.070 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:15.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:15.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:15.210 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:15.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:15.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:15.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:15.502 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:15.583 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:15 smithi003 ceph-mon[107278]: pgmap v1383: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 985/270291 objects degraded (0.364%); 30564/270291 objects misplaced (11.308%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T15:03:15.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:15 smithi003 ceph-mon[104007]: pgmap v1383: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 985/270291 objects degraded (0.364%); 30564/270291 objects misplaced (11.308%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T15:03:15.603 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:15.631 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:15 smithi019 ceph-mon[91296]: pgmap v1383: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 985/270291 objects degraded (0.364%); 30564/270291 objects misplaced (11.308%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T15:03:15.733 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:15.753 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:03:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196647 11.10 11:0f02dab9:::benchmark_data_smithi003_229782_object74267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.362025+0000", 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.386706893, 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96367963199999995, 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:03:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196647 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362025+0000", 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362025+0000", 2024-09-13T15:03:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362026+0000", 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.575e-06 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362032+0000", 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5759999999999996e-06 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362050+0000", 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7706999999999998e-05 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362058+0000", 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1510000000000004e-06 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362184+0000", 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012584599999999999 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.362194+0000", 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7780000000000002e-06 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.316597+0000", 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95440383100000004 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.316735+0000", 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013800100000000001 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.317290+0000", 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00055507099999999997 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325534+0000", 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082431039999999994 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325589+0000", 2024-09-13T15:03:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5556000000000001e-05 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325643+0000", 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3569999999999997e-05 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325680+0000", 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7002999999999998e-05 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325705+0000", 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4862999999999999e-05 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196656 11.10 11:0bb599c9:::benchmark_data_smithi003_229782_object74276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.372318+0000", 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.37641354599998, 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95649141100000001, 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196656 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372318+0000", 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372318+0000", 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372320+0000", 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4610000000000001e-06 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372324+0000", 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9949999999999999e-06 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372334+0000", 2024-09-13T15:03:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0366999999999999e-05 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372340+0000", 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6620000000000002e-06 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372429+0000", 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9018999999999996e-05 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.372440+0000", 2024-09-13T15:03:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1080000000000001e-05 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.320065+0000", 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94762556499999995 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.320193+0000", 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012741900000000001 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.320554+0000", 2024-09-13T15:03:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036128799999999997 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325909+0000", 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053550749999999999 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325950+0000", 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0527e-05 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:03:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328705+0000", 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027551720000000002 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328766+0000", 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1408000000000006e-05 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328810+0000", 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3373e-05 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196725 11.10 11:0a049c51:::benchmark_data_smithi003_229782_object74345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.454725+0000", 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.29400697099999, 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87791044399999996, 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196725 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454725+0000", 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454725+0000", 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454726+0000", 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5850000000000001e-06 2024-09-13T15:03:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454732+0000", 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1869999999999996e-06 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454748+0000", 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6378000000000001e-05 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454757+0000", 2024-09-13T15:03:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5699999999999993e-06 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454827+0000", 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0030000000000005e-05 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.454837+0000", 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0190999999999999e-05 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.322756+0000", 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86791948200000002 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.322884+0000", 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012783 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.323283+0000", 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039857400000000001 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.325986+0000", 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027037799999999998 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328880+0000", 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028937870000000001 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332304+0000", 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003423929 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332354+0000", 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9883999999999999e-05 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332635+0000", 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028123699999999999 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196757 11.10 11:0c19dfc2:::benchmark_data_smithi003_229782_object74377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.484186+0000", 2024-09-13T15:03:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.26454548599997, 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.85628883899999997, 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196757 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484186+0000", 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484186+0000", 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484188+0000", 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2580000000000001e-06 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484192+0000", 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7929999999999997e-06 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484203+0000", 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0227000000000001e-05 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484209+0000", 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4999999999999996e-06 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484269+0000", 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0273999999999999e-05 2024-09-13T15:03:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.484279+0000", 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0049e-05 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.326015+0000", 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.84173542999999995 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.326152+0000", 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013716800000000001 2024-09-13T15:03:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.326568+0000", 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041612499999999999 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332646+0000", 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006077743 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332686+0000", 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0349000000000003e-05 2024-09-13T15:03:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340381+0000", 2024-09-13T15:03:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076943139999999998 2024-09-13T15:03:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340431+0000", 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0798999999999999e-05 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340475+0000", 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3810000000000002e-05 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36438.0:196769 11.10 11:086d7a4f:::benchmark_data_smithi003_229782_object74389:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:54:48.495957+0000", 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.25277440299999, 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.84475226599999997, 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36438", 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3685350852", 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196769 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495957+0000", 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495957+0000", 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495958+0000", 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2900000000000002e-07 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495964+0000", 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3850000000000001e-06 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495975+0000", 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1039999999999999e-05 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.495982+0000", 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.605e-06 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.496055+0000", 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2558999999999994e-05 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:48.496066+0000", 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0643e-05 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.328942+0000", 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.83287688400000004 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.329070+0000", 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012754500000000001 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.329470+0000", 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039961100000000002 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332697+0000", 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032277400000000002 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.332735+0000", 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7824000000000002e-05 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340569+0000", 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078336359999999997 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340669+0000", 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100529 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:54:49.340710+0000", 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0336000000000001e-05 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:144754 13.9 e384/376 13:9c3a3660:::benchmark_data_smithi003_293363_object25542:head v 384'9133, mlcod=366'164)", 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:09.035490+0000", 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.7132413980000001, 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.049242753, 2024-09-13T15:03:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.035490+0000", 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.035490+0000", 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.035491+0000", 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4799999999999997e-07 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.035503+0000", 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1518999999999999e-05 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.035521+0000", 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8566000000000001e-05 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.035530+0000", 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3780000000000001e-06 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.041305+0000", 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057748310000000002 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.041329+0000", 2024-09-13T15:03:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4927999999999999e-05 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.084661+0000", 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.043331620000000001 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.084733+0000", 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2063000000000004e-05 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:144771 13.9 e384/376 13:942f5597:::benchmark_data_smithi003_293363_object25559:head v 384'9134, mlcod=366'164)", 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:09.041202+0000", 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.7075294249999997, 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.059922178999999999, 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.041202+0000", 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.041202+0000", 2024-09-13T15:03:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.794 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.041203+0000", 2024-09-13T15:03:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0809999999999999e-06 2024-09-13T15:03:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.041224+0000", 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0732000000000001e-05 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.041240+0000", 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5693999999999999e-05 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.041248+0000", 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2120000000000003e-06 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.053877+0000", 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012628485 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.053905+0000", 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7991999999999999e-05 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.100940+0000", 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.047035479999999998 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.101125+0000", 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018450300000000001 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:144776 13.9 e384/376 13:9b0c466e:::benchmark_data_smithi003_293363_object25564:head v 384'9135, mlcod=366'164)", 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:09.042404+0000", 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.7063273370000003, 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.058869998999999999, 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.042404+0000", 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.042404+0000", 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.042405+0000", 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9999999999999996e-07 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.042415+0000", 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5780000000000005e-06 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.042431+0000", 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6290000000000002e-05 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.042439+0000", 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8900000000000007e-06 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.062004+0000", 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019564703999999999 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.062055+0000", 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1002999999999999e-05 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.101131+0000", 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039075912999999997 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.101274+0000", 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000143721 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:144778 13.9 e384/376 13:95ad0746:::benchmark_data_smithi003_293363_object25566:head v 384'9136, mlcod=366'164)", 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:09.051088+0000", 2024-09-13T15:03:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.6976435030000001, 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.050305313999999997, 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.051088+0000", 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.051088+0000", 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.051090+0000", 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7069999999999999e-06 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.051099+0000", 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3309999999999993e-06 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.051120+0000", 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1135e-05 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.051130+0000", 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3440000000000007e-06 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.069211+0000", 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018080718999999999 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.069228+0000", 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7444000000000001e-05 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.101278+0000", 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032050083 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.101394+0000", 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011555100000000001 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:144784 13.9 e384/376 13:957a734a:::benchmark_data_smithi003_293363_object25572:head v 384'9138, mlcod=366'164)", 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:09.058885+0000", 2024-09-13T15:03:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.689847103, 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.059027591999999997, 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.058885+0000", 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.058885+0000", 2024-09-13T15:03:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.058886+0000", 2024-09-13T15:03:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0780000000000001e-06 2024-09-13T15:03:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.808 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:15.809 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:15.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.058901+0000", 2024-09-13T15:03:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5149e-05 2024-09-13T15:03:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.058915+0000", 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.366e-05 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.058923+0000", 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9920000000000007e-06 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.085100+0000", 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026177679999999998 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.085128+0000", 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7310000000000001e-05 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.117766+0000", 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032638400999999997 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.117912+0000", 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000146322 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:145040 13.9 e384/376 13:9d521074:::benchmark_data_smithi003_293363_object25828:head v 384'9154, mlcod=366'164)", 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:09.347004+0000", 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.4017281979999998, 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.049662137000000002, 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.347004+0000", 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.347004+0000", 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.347005+0000", 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2279999999999999e-06 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.347013+0000", 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3960000000000006e-06 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.347032+0000", 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8800999999999998e-05 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.347042+0000", 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9150000000000008e-06 2024-09-13T15:03:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.347120+0000", 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7804000000000002e-05 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.347171+0000", 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1542000000000002e-05 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.396606+0000", 2024-09-13T15:03:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.049434964999999997 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.396666+0000", 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9485999999999999e-05 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:145058 13.9 e384/376 13:9db8ce47:::benchmark_data_smithi003_293363_object25846:head v 384'9156, mlcod=366'164)", 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:09.358656+0000", 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.3900761140000002, 2024-09-13T15:03:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.048976538999999999, 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.358656+0000", 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.358656+0000", 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.358657+0000", 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7599999999999996e-07 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.358664+0000", 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9949999999999999e-06 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.358678+0000", 2024-09-13T15:03:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4482e-05 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.358685+0000", 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2540000000000003e-06 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.374535+0000", 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015849878000000001 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.374563+0000", 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.815e-05 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.407566+0000", 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033002778000000003 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.407632+0000", 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6125999999999997e-05 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.820 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPush(13.9 385/376 [PushOp(13:97a370b4:::benchmark_data_smithi003_293363_object26004:head, version: 367'1657, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(13:97a370b4:::benchmark_data_smithi003_293363_object26004:head@367'1657, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:09.525580+0000", 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.2231521049999996, 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.050379079, 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.525580+0000", 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.525580+0000", 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.525580+0000", 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0499999999999999e-07 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.525620+0000", 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0061999999999997e-05 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.525637+0000", 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7244999999999998e-05 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.525652+0000", 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4408e-05 2024-09-13T15:03:15.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.573527+0000", 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.047875474000000001 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.573541+0000", 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4136e-05 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.575959+0000", 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024174489999999999 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.824 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:147859 13.9 e385/376 13:994dc2e2:::benchmark_data_smithi003_293363_object28647:head v 385'9330, mlcod=366'164)", 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:12.490879+0000", 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2578526800000001, 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.050573424999999998, 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.490879+0000", 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.490879+0000", 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.490880+0000", 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9999999999999996e-07 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.490888+0000", 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7789999999999996e-06 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.490904+0000", 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5957999999999999e-05 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.490911+0000", 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4209999999999998e-06 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.504106+0000", 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013194943000000001 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.504134+0000", 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8140000000000002e-05 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.541383+0000", 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.037249152000000001 2024-09-13T15:03:15.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:12.541453+0000", 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9132000000000002e-05 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:150421 13.9 e385/376 13:9773010f:::benchmark_data_smithi003_293363_object31209:head v 385'9497, mlcod=366'164)", 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:15.333865+0000", 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.41486649199999998, 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.052252139000000003, 2024-09-13T15:03:15.829 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.333865+0000", 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.333865+0000", 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.333866+0000", 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.006e-06 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.333874+0000", 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4610000000000002e-06 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.333892+0000", 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7903000000000001e-05 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.333900+0000", 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4449999999999996e-06 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.350469+0000", 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016568877999999999 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.350492+0000", 2024-09-13T15:03:15.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3141000000000001e-05 2024-09-13T15:03:15.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.385922+0000", 2024-09-13T15:03:15.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035429527000000002 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.386117+0000", 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019577799999999999 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:150597 13.9 e385/376 13:9cf45759:::benchmark_data_smithi003_293363_object31385:head v 385'9514, mlcod=366'164)", 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:15.573239+0000", 2024-09-13T15:03:15.834 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.175493235, 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.064323669, 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.573239+0000", 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.573239+0000", 2024-09-13T15:03:15.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.573240+0000", 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6899999999999996e-07 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.573252+0000", 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2393e-05 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.573275+0000", 2024-09-13T15:03:15.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3425999999999999e-05 2024-09-13T15:03:15.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.573284+0000", 2024-09-13T15:03:15.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8310000000000007e-06 2024-09-13T15:03:15.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.573396+0000", 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011173199999999999 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.573424+0000", 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8048999999999999e-05 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.637467+0000", 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.064043467000000007 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.637562+0000", 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4802000000000005e-05 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:150601 13.9 e385/376 13:9e46c945:::benchmark_data_smithi003_293363_object31389:head v 385'9515, mlcod=366'164)", 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:15.574836+0000", 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.17389590399999999, 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.062814405000000004, 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.839 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.574836+0000", 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.574836+0000", 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.574837+0000", 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.133e-06 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.574845+0000", 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3690000000000007e-06 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.574876+0000", 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1096000000000002e-05 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.574885+0000", 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3240000000000003e-06 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.582491+0000", 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076060640000000001 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.582592+0000", 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100645 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.637568+0000", 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.054976880999999998 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.637650+0000", 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1892999999999997e-05 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:150622 13.9 e385/376 13:9a701476:::benchmark_data_smithi003_293363_object31410:head v 385'9516, mlcod=366'164)", 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:15.579476+0000", 2024-09-13T15:03:15.843 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.169255978, 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.05826257, 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.579476+0000", 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.579476+0000", 2024-09-13T15:03:15.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.579477+0000", 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3099999999999996e-07 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.579483+0000", 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1240000000000002e-06 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.579498+0000", 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4946000000000001e-05 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.579505+0000", 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.024e-06 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.598062+0000", 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018556939000000001 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.598150+0000", 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8337000000000005e-05 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.637655+0000", 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039505108999999997 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.637738+0000", 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3259999999999999e-05 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.847 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:150627 13.9 e385/376 13:9062383f:::benchmark_data_smithi003_293363_object31415:head v 385'9517, mlcod=366'164)", 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:15.582160+0000", 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.16657155300000001, 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.055660139999999997, 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.582160+0000", 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.582160+0000", 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.582162+0000", 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.376e-06 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.582173+0000", 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1622e-05 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.582196+0000", 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2487999999999999e-05 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.582207+0000", 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1141999999999999e-05 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.608694+0000", 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026487607999999999 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.608726+0000", 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1776000000000002e-05 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.637748+0000", 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029022088000000001 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.637820+0000", 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2039999999999995e-05 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPush(13.9 385/376 [PushOp(13:94163008:::benchmark_data_smithi003_293363_object31438:head, version: 367'2009, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(13:94163008:::benchmark_data_smithi003_293363_object31438:head@367'2009, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:15.598143+0000", 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.15058848699999999, 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.047602972, 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:15.852 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.598143+0000", 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.598143+0000", 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.598145+0000", 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.578e-06 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.598201+0000", 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6543999999999999e-05 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.598238+0000", 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6319000000000002e-05 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.598260+0000", 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2413e-05 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.638079+0000", 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039818576000000001 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.638092+0000", 2024-09-13T15:03:15.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3012e-05 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.645746+0000", 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076545299999999997 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:15.856 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:03:15.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:15.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:16.055 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.087 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:16.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:16.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:16.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:16.327 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:16.361 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:16.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:16.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:16.600 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:16.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 985/270291 objects degraded (0.364%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:16.617 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 985/270291 objects degraded (0.364%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:16.631 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 985/270291 objects degraded (0.364%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:16.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:16.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:16.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:16.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:17.017 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:17.061 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:17.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:17.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.337 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:17.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:17.438 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:17.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:17.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:17.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:17 smithi019 ceph-mon[91296]: pgmap v1384: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 985/270291 objects degraded (0.364%); 30564/270291 objects misplaced (11.308%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T15:03:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:17 smithi003 ceph-mon[104007]: pgmap v1384: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 985/270291 objects degraded (0.364%); 30564/270291 objects misplaced (11.308%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T15:03:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:17 smithi003 ceph-mon[107278]: pgmap v1384: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 985/270291 objects degraded (0.364%); 30564/270291 objects misplaced (11.308%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T15:03:17.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:17.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:17.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:17.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:17.989 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.033 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:18.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:18.134 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:18.171 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:18.274 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:18.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:18.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:18.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:18.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:18.553 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:18.598 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:18.700 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:18.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:18.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T15:03:18.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:18.876 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:18.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:19.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:19.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:19.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:19.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:19.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:19.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:19.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:19.576 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:19.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:19 smithi019 ceph-mon[91296]: pgmap v1385: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 985/267027 objects degraded (0.369%); 29656/267027 objects misplaced (11.106%); 7.2 MiB/s, 114 objects/s recovering 2024-09-13T15:03:19.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:19 smithi003 ceph-mon[107278]: pgmap v1385: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 985/267027 objects degraded (0.369%); 29656/267027 objects misplaced (11.106%); 7.2 MiB/s, 114 objects/s recovering 2024-09-13T15:03:19.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:19 smithi003 ceph-mon[104007]: pgmap v1385: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 985/267027 objects degraded (0.369%); 29656/267027 objects misplaced (11.106%); 7.2 MiB/s, 114 objects/s recovering 2024-09-13T15:03:19.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:19.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:19.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:19.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:19.971 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 5, 0] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:03:19.972 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:03:19.972 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-13T15:03:19.972 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:03:19.972 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:03:19.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-13T15:03:20.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:20.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:20.139 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:03:20.237 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:20.237 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:03:20.237 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:20.238 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:03:20.238 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:20.238 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:03:20.238 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.239 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:20.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:20.268 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:03:20.269 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:03:20.314 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:20.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:20.356 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:03:20.357 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:20.374 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:20.396 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:03:20.397 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:20.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:20 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2261866734' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:20.398 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:20 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2261866734' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:20.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:20.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:20.609 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:20.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.647 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:20.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:20 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2261866734' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:03:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:03:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:20.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:20.873 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:20.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:21.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:21.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:21.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:21.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:21.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:21.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:21.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:21.392 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:03:21.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:21.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:21.592 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.641 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:21.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:21 smithi019 ceph-mon[91296]: pgmap v1386: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 1226/256656 objects degraded (0.478%); 28528/256656 objects misplaced (11.115%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T15:03:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1226/256656 objects degraded (0.478%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4137927051' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:03:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:21 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:03:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:21 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:03:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[104007]: pgmap v1386: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 1226/256656 objects degraded (0.478%); 28528/256656 objects misplaced (11.115%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T15:03:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1226/256656 objects degraded (0.478%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4137927051' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:03:21.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:03:21.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:03:21.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[107278]: pgmap v1386: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 1226/256656 objects degraded (0.478%); 28528/256656 objects misplaced (11.115%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T15:03:21.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1226/256656 objects degraded (0.478%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:21.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4137927051' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:03:21.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:03:21.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:21 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:03:21.685 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:21.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:21.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:21.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:21.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:22.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:22.059 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:22.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:22.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:22.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:22.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:22.430 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:22.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:22.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:22.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:22 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:22 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:03:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:22 smithi019 ceph-mon[91296]: osdmap e386: 8 total, 8 up, 7 in 2024-09-13T15:03:22.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:22 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:22.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:22 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:03:22.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:22 smithi003 ceph-mon[107278]: osdmap e386: 8 total, 8 up, 7 in 2024-09-13T15:03:22.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:22 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:22.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:22 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:03:22.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:22 smithi003 ceph-mon[104007]: osdmap e386: 8 total, 8 up, 7 in 2024-09-13T15:03:22.710 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.759 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:22.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:22.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:22.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:22.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:23.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:23.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:23.123 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:23.172 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:23.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:23.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:23.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:23.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:23.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:03:23.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:23.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:23.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:23 smithi003 ceph-mon[107278]: pgmap v1388: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 1226/254124 objects degraded (0.482%); 27906/254124 objects misplaced (10.981%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T15:03:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:23 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T15:03:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:23 smithi003 ceph-mon[104007]: pgmap v1388: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 1226/254124 objects degraded (0.482%); 27906/254124 objects misplaced (10.981%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T15:03:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:23 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T15:03:23.685 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:23 smithi019 ceph-mon[91296]: pgmap v1388: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 1226/254124 objects degraded (0.482%); 27906/254124 objects misplaced (10.981%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T15:03:23.685 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:23 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T15:03:23.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:23.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:23.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:23.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:23.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:23.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:24.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:24.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.138 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:24.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:24.239 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.293 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:24.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:24.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:24.430 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:03:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:03:24] "GET /metrics HTTP/1.1" 200 36553 "" "Prometheus/2.51.0" 2024-09-13T15:03:24.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:24.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:24.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:24.608 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:24 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T15:03:24.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:24 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T15:03:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:24 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T15:03:24.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:24.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:24.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:24.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:24.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.904 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:24.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:25.005 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.029 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:25.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:25.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:25.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:25.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:25.298 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:25.400 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:25.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:25.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:25 smithi003 ceph-mon[107278]: pgmap v1389: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 1381/243345 objects degraded (0.568%); 26545/243345 objects misplaced (10.908%); 7.2 MiB/s, 114 objects/s recovering 2024-09-13T15:03:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:25 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T15:03:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:25 smithi003 ceph-mon[104007]: pgmap v1389: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 1381/243345 objects degraded (0.568%); 26545/243345 objects misplaced (10.908%); 7.2 MiB/s, 114 objects/s recovering 2024-09-13T15:03:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:25 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T15:03:25.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:25.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:25.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:25.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:25.887 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:25 smithi019 ceph-mon[91296]: pgmap v1389: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 1381/243345 objects degraded (0.568%); 26545/243345 objects misplaced (10.908%); 7.2 MiB/s, 114 objects/s recovering 2024-09-13T15:03:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:25 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T15:03:25.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:26.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:26.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:26.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:26.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:26.292 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:26.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:26.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:26.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:26.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:26.653 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:26.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1381/243345 objects degraded (0.568%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:26.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1381/243345 objects degraded (0.568%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:26.694 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:26.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:26.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1381/243345 objects degraded (0.568%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:26.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:26.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:27.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:27.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:27.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:27.259 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:27.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:27.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:27.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:27.632 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:27.676 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:27 smithi019 ceph-mon[91296]: pgmap v1390: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 1381/243345 objects degraded (0.568%); 26545/243345 objects misplaced (10.908%); 7.2 MiB/s, 114 objects/s recovering 2024-09-13T15:03:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:27 smithi003 ceph-mon[107278]: pgmap v1390: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 1381/243345 objects degraded (0.568%); 26545/243345 objects misplaced (10.908%); 7.2 MiB/s, 114 objects/s recovering 2024-09-13T15:03:27.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:27 smithi003 ceph-mon[104007]: pgmap v1390: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 1381/243345 objects degraded (0.568%); 26545/243345 objects misplaced (10.908%); 7.2 MiB/s, 114 objects/s recovering 2024-09-13T15:03:27.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:27.813 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:27.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:27.916 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:27.959 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:28.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:28.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:28.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:28.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:28.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:28.479 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.540 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:28.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:28.590 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:28.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:28.595 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:28.641 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:28.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.687 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:28.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:28.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:28.833 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:28.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:28.960 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:03:28.961 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "" 2024-09-13T15:03:28.961 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:03:28.973 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:29.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:29.212 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:29.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:29.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:29.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:29.441 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:29.441 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:29.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.540 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:29.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:29.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:03:29.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:29.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:29.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:29.783 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:29.824 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:29.835 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:29 smithi003 ceph-mon[107278]: pgmap v1391: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 893 op/s; 1381/240219 objects degraded (0.575%); 25753/240219 objects misplaced (10.721%); 6.9 MiB/s, 110 objects/s recovering 2024-09-13T15:03:29.835 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:29 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T15:03:29.836 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:29.836 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:29.836 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:29 smithi003 ceph-mon[104007]: pgmap v1391: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 893 op/s; 1381/240219 objects degraded (0.575%); 25753/240219 objects misplaced (10.721%); 6.9 MiB/s, 110 objects/s recovering 2024-09-13T15:03:29.836 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:29 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T15:03:29.836 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:29.836 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:29.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:29.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:29 smithi019 ceph-mon[91296]: pgmap v1391: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 893 op/s; 1381/240219 objects degraded (0.575%); 25753/240219 objects misplaced (10.721%); 6.9 MiB/s, 110 objects/s recovering 2024-09-13T15:03:29.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:29 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T15:03:29.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:29.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:29.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:30.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:30.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:30.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:30.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:30.356 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:30.396 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:30.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:30.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:30.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:30.679 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:03:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:03:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:30.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:03:30.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:30.820 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:30.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:31.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.109 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:31.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:31.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:31.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:31.355 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:31.392 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:31.494 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:31.529 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:31.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:31.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:31.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:31.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:31 smithi003 ceph-mon[104007]: pgmap v1392: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 1516/230739 objects degraded (0.657%); 24716/230739 objects misplaced (10.712%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T15:03:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1516/230739 objects degraded (0.657%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:31.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:31 smithi003 ceph-mon[107278]: pgmap v1392: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 1516/230739 objects degraded (0.657%); 24716/230739 objects misplaced (10.712%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T15:03:31.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1516/230739 objects degraded (0.657%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:31.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:31.919 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:31 smithi019 ceph-mon[91296]: pgmap v1392: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 1516/230739 objects degraded (0.657%); 24716/230739 objects misplaced (10.712%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T15:03:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1516/230739 objects degraded (0.657%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:31.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:32.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:32.096 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:32.198 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:32.244 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:32.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:32.387 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:32.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:32.624 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:32.661 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:32.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:32.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:32.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:32.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:33.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:33.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:33.187 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:33.223 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:33.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:33.359 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:33.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:33.497 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:33.560 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:33.564 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:33.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.670 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:33.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:33.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:33 smithi003 ceph-mon[104007]: pgmap v1393: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 1516/228276 objects degraded (0.664%); 24194/228276 objects misplaced (10.599%); 6.3 MiB/s, 101 objects/s recovering 2024-09-13T15:03:33.712 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:33 smithi003 ceph-mon[107278]: pgmap v1393: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 1516/228276 objects degraded (0.664%); 24194/228276 objects misplaced (10.599%); 6.3 MiB/s, 101 objects/s recovering 2024-09-13T15:03:33.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:33.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:33.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:33 smithi019 ceph-mon[91296]: pgmap v1393: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 1516/228276 objects degraded (0.664%); 24194/228276 objects misplaced (10.599%); 6.3 MiB/s, 101 objects/s recovering 2024-09-13T15:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:33.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:34.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:34.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:34.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:34.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:34.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:34.357 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:34.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:34.413 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:03:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:03:34] "GET /metrics HTTP/1.1" 200 36554 "" "Prometheus/2.51.0" 2024-09-13T15:03:34.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:34.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:34.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:34.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:34.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:34.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:34.914 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:35.016 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:35.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:35.157 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:35.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.205 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:35.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:35.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:35.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:35.441 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:35.482 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:35.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:35.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:35.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:35.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:35 smithi003 ceph-mon[107278]: pgmap v1394: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 1680/217443 objects degraded (0.773%); 22848/217443 objects misplaced (10.508%); 6.2 MiB/s, 98 objects/s recovering 2024-09-13T15:03:35.728 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:35 smithi003 ceph-mon[104007]: pgmap v1394: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 1680/217443 objects degraded (0.773%); 22848/217443 objects misplaced (10.508%); 6.2 MiB/s, 98 objects/s recovering 2024-09-13T15:03:35.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:35.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:35.899 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:35 smithi019 ceph-mon[91296]: pgmap v1394: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 1680/217443 objects degraded (0.773%); 22848/217443 objects misplaced (10.508%); 6.2 MiB/s, 98 objects/s recovering 2024-09-13T15:03:36.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:36.051 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:36.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:36.170 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 5, 0] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:03:36.170 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:03:36.170 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-13T15:03:36.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd in 3 2024-09-13T15:03:36.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:36.290 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:36.330 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:36.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:36.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:36.575 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:36.613 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:36.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:36.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:36.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:36.885 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:36.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:36 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1680/217443 objects degraded (0.773%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:36.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:36 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1680/217443 objects degraded (0.773%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:36.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:36 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1680/217443 objects degraded (0.773%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:36.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:37.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:37.130 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:37.166 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:37.268 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:37.302 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:37.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:37.442 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:37.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:37.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:37.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:37.719 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:37.799 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:03:37.799 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:03:37.800 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:03:37.800 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:03:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:37.800 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:03:37.801 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:03:37.801 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:03:37.801 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:03:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:37.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:37 smithi003 ceph-mon[107278]: pgmap v1395: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 1680/217443 objects degraded (0.773%); 22848/217443 objects misplaced (10.508%); 5.7 MiB/s, 90 objects/s recovering 2024-09-13T15:03:37.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1144710623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:37.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:37 smithi003 ceph-mon[104007]: pgmap v1395: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 1680/217443 objects degraded (0.773%); 22848/217443 objects misplaced (10.508%); 5.7 MiB/s, 90 objects/s recovering 2024-09-13T15:03:37.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1144710623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:37.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:37.870 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:37.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:37 smithi019 ceph-mon[91296]: pgmap v1395: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 1680/217443 objects degraded (0.773%); 22848/217443 objects misplaced (10.508%); 5.7 MiB/s, 90 objects/s recovering 2024-09-13T15:03:37.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1144710623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:37.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:38.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:38.112 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:38.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.163 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:38.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:38.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:38.299 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:38.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:38.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:38.534 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:38.581 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:38.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:38.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:38.814 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.857 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:38.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:38.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:38 smithi003 ceph-mon[107278]: pgmap v1396: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 1680/214206 objects degraded (0.784%); 22046/214206 objects misplaced (10.292%); 8.2 MiB/s, 130 objects/s recovering 2024-09-13T15:03:38.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:38 smithi003 ceph-mon[104007]: pgmap v1396: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 1680/214206 objects degraded (0.784%); 22046/214206 objects misplaced (10.292%); 8.2 MiB/s, 130 objects/s recovering 2024-09-13T15:03:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:38 smithi019 ceph-mon[91296]: pgmap v1396: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 1680/214206 objects degraded (0.784%); 22046/214206 objects misplaced (10.292%); 8.2 MiB/s, 130 objects/s recovering 2024-09-13T15:03:38.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:38.993 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:39.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:39.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:39.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:39.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:39.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:39.379 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:39.413 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:39.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:39.558 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:39.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:03:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:39.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:39.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:39.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1362434840' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:39.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:39 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:39.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1362434840' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:39.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:39 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:39.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:39.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:39.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:40.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.138 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:40.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:40.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1362434840' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:40.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:39 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:40.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:40.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:40.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:40.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:40.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:40.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:40.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:40 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:03:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:40 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:40 smithi003 ceph-mon[107278]: osdmap e387: 8 total, 8 up, 7 in 2024-09-13T15:03:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:40 smithi003 ceph-mon[107278]: pgmap v1398: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 1866/203496 objects degraded (0.917%); 20868/203496 objects misplaced (10.255%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T15:03:40.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:40 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:03:40.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:40 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:40.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:40 smithi003 ceph-mon[104007]: osdmap e387: 8 total, 8 up, 7 in 2024-09-13T15:03:40.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:40 smithi003 ceph-mon[104007]: pgmap v1398: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 1866/203496 objects degraded (0.917%); 20868/203496 objects misplaced (10.255%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T15:03:40.856 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:03:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:03:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:40.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:41.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:41.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:41.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:40 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:03:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:40 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:41.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:40 smithi019 ceph-mon[91296]: osdmap e387: 8 total, 8 up, 7 in 2024-09-13T15:03:41.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:40 smithi019 ceph-mon[91296]: pgmap v1398: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 1866/203496 objects degraded (0.917%); 20868/203496 objects misplaced (10.255%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T15:03:41.207 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:41.309 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:41.349 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:41.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:41.411 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:03:41.411 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:03:41.411 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:03:41.411 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:03:41.411 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:03:41.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:41.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:41.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:41.631 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:41.733 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:41.776 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:41.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:41.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:41.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:41 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T15:03:41.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1866/203496 objects degraded (0.917%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:42.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:42.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:42.077 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:42.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:03:42.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:41 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T15:03:42.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1866/203496 objects degraded (0.917%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:42.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:41 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T15:03:42.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1866/203496 objects degraded (0.917%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:42.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:42.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:42.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:42.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:42.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:42.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:42.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.612 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:42.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:42.713 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:42.752 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:42.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:03:42.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:42.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:03:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:42.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:42.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:42 smithi003 ceph-mon[104007]: pgmap v1399: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 908 op/s; 1866/200994 objects degraded (0.928%); 20343/200994 objects misplaced (10.121%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T15:03:42.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:42 smithi003 ceph-mon[107278]: pgmap v1399: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 908 op/s; 1866/200994 objects degraded (0.928%); 20343/200994 objects misplaced (10.121%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T15:03:43.017 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.058 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:43.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:43.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:43.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:42 smithi019 ceph-mon[91296]: pgmap v1399: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 908 op/s; 1866/200994 objects degraded (0.928%); 20343/200994 objects misplaced (10.121%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T15:03:43.202 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:43.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:43.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:43.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:43.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:43.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:43.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:43.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:43.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:43.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:43.877 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:43.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:43.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:44.016 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:44.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:44.038 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:44.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:44.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:44.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:44.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:44.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:44.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:44.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:44.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:44.293 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:44.359 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:03:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:03:44] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T15:03:44.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:44.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:44.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:44.571 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:44.673 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:44.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:44.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:44.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:44.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:44 smithi003 ceph-mon[104007]: pgmap v1400: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 904 op/s; 2027/190284 objects degraded (1.065%); 18982/190284 objects misplaced (9.976%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T15:03:44.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:44 smithi003 ceph-mon[107278]: pgmap v1400: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 904 op/s; 2027/190284 objects degraded (1.065%); 18982/190284 objects misplaced (9.976%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T15:03:44.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:45.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:45.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:45.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:45.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:44 smithi019 ceph-mon[91296]: pgmap v1400: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 904 op/s; 2027/190284 objects degraded (1.065%); 18982/190284 objects misplaced (9.976%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T15:03:45.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:45 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:03:45.127+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e388 definitely_dead 0 2024-09-13T15:03:45.197 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:45.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:45.259 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:45.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:45.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:45.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:45.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:45.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:45.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:45.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:45.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:45.757 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:45.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:45.842 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.3. 2024-09-13T15:03:45.859 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:45.886 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:45.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:46.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:46.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:45 smithi003 ceph-mon[104007]: osdmap e388: 8 total, 8 up, 7 in 2024-09-13T15:03:46.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:45 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3549731180' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-13T15:03:46.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:45 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-13T15:03:46.054 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:45 smithi003 ceph-mon[107278]: osdmap e388: 8 total, 8 up, 7 in 2024-09-13T15:03:46.054 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:45 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3549731180' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-13T15:03:46.054 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:45 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-13T15:03:46.112 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:46.134 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:45 smithi019 ceph-mon[91296]: osdmap e388: 8 total, 8 up, 7 in 2024-09-13T15:03:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:45 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3549731180' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-13T15:03:46.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:45 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-13T15:03:46.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:46.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:46.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:46.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:46.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:46.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:46.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:46.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:46.735 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:46.758 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:46.860 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:46.892 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:46.994 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:47.033 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:47.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:47.167 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:46 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-13T15:03:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:46 smithi019 ceph-mon[91296]: osdmap e389: 8 total, 8 up, 8 in 2024-09-13T15:03:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2027/190284 objects degraded (1.065%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:46 smithi019 ceph-mon[91296]: pgmap v1403: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 682 op/s; 2027/190284 objects degraded (1.065%); 18982/190284 objects misplaced (9.976%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T15:03:47.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:46 smithi019 ceph-mon[91296]: osdmap e390: 8 total, 8 up, 8 in 2024-09-13T15:03:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-13T15:03:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[107278]: osdmap e389: 8 total, 8 up, 8 in 2024-09-13T15:03:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2027/190284 objects degraded (1.065%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[107278]: pgmap v1403: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 682 op/s; 2027/190284 objects degraded (1.065%); 18982/190284 objects misplaced (9.976%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T15:03:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[107278]: osdmap e390: 8 total, 8 up, 8 in 2024-09-13T15:03:47.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-13T15:03:47.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[104007]: osdmap e389: 8 total, 8 up, 8 in 2024-09-13T15:03:47.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2027/190284 objects degraded (1.065%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:47.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[104007]: pgmap v1403: 119 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 682 op/s; 2027/190284 objects degraded (1.065%); 18982/190284 objects misplaced (9.976%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T15:03:47.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:46 smithi003 ceph-mon[104007]: osdmap e390: 8 total, 8 up, 8 in 2024-09-13T15:03:47.269 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:47.298 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:47.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:47.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:47.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:47.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:47.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:47.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:47.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:47.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:47.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:47.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:47.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:48.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:48.150 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:47 smithi003 ceph-mon[107278]: 13.3 continuing backfill to osd.0 from (366'1400,388'10869] MIN to 388'10869 2024-09-13T15:03:48.150 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:47 smithi003 ceph-mon[104007]: 13.3 continuing backfill to osd.0 from (366'1400,388'10869] MIN to 388'10869 2024-09-13T15:03:48.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:48.181 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:47 smithi019 ceph-mon[91296]: 13.3 continuing backfill to osd.0 from (366'1400,388'10869] MIN to 388'10869 2024-09-13T15:03:48.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:48.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:48.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:48.437 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:48.475 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:48.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:48.617 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:48.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.764 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:48.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:48.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:48.892 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:03:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:49.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:49.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:49.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:49.178 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:48 smithi019 ceph-mon[91296]: 1.1e scrub starts 2024-09-13T15:03:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:48 smithi019 ceph-mon[91296]: 1.1e scrub ok 2024-09-13T15:03:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:48 smithi019 ceph-mon[91296]: osdmap e391: 8 total, 8 up, 8 in 2024-09-13T15:03:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:48 smithi019 ceph-mon[91296]: pgmap v1406: 119 pgs: 1 active+recovery_wait, 7 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 107 active+clean; 3.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 212 op/s; 2027/187761 objects degraded (1.080%); 14640/187761 objects misplaced (7.797%); 3.9 MiB/s, 61 objects/s recovering 2024-09-13T15:03:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:48 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/246694241' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:49.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:48 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:49.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[107278]: 1.1e scrub starts 2024-09-13T15:03:49.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[107278]: 1.1e scrub ok 2024-09-13T15:03:49.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[107278]: osdmap e391: 8 total, 8 up, 8 in 2024-09-13T15:03:49.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[107278]: pgmap v1406: 119 pgs: 1 active+recovery_wait, 7 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 107 active+clean; 3.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 212 op/s; 2027/187761 objects degraded (1.080%); 14640/187761 objects misplaced (7.797%); 3.9 MiB/s, 61 objects/s recovering 2024-09-13T15:03:49.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/246694241' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:49.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[104007]: 1.1e scrub starts 2024-09-13T15:03:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[104007]: 1.1e scrub ok 2024-09-13T15:03:49.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[104007]: osdmap e391: 8 total, 8 up, 8 in 2024-09-13T15:03:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[104007]: pgmap v1406: 119 pgs: 1 active+recovery_wait, 7 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 107 active+clean; 3.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 212 op/s; 2027/187761 objects degraded (1.080%); 14640/187761 objects misplaced (7.797%); 3.9 MiB/s, 61 objects/s recovering 2024-09-13T15:03:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/246694241' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:48 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:49.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:49.320 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:49.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:49.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:49.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:49.595 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:49.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:49.731 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:49.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:49.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:49.965 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:50.005 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:50.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:50.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:50.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:50.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:49 smithi019 ceph-mon[91296]: 1.1d scrub starts 2024-09-13T15:03:50.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:49 smithi019 ceph-mon[91296]: 1.1d scrub ok 2024-09-13T15:03:50.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:49 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:50.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:49 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:50.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:49 smithi019 ceph-mon[91296]: osdmap e392: 8 total, 8 up, 8 in 2024-09-13T15:03:50.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[107278]: 1.1d scrub starts 2024-09-13T15:03:50.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[107278]: 1.1d scrub ok 2024-09-13T15:03:50.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:50.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:50.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[107278]: osdmap e392: 8 total, 8 up, 8 in 2024-09-13T15:03:50.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[104007]: 1.1d scrub starts 2024-09-13T15:03:50.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[104007]: 1.1d scrub ok 2024-09-13T15:03:50.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:50.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:50.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:49 smithi003 ceph-mon[104007]: osdmap e392: 8 total, 8 up, 8 in 2024-09-13T15:03:50.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:50.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:50.297 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-13T15:03:50.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:50.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:50.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:50.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:50.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:50.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:03:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:03:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:50.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:50.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:50.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:50 smithi003 ceph-mon[107278]: pgmap v1408: 119 pgs: 1 active+recovery_wait, 7 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 108 active+clean; 3.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 4.0 KiB/s rd, 8.6 KiB/s wr, 597 op/s; 2085/182643 objects degraded (1.142%); 10266/182643 objects misplaced (5.621%); 7.6 MiB/s, 121 objects/s recovering 2024-09-13T15:03:50.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:50 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2027/187761 objects degraded (1.080%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:50.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:50 smithi003 ceph-mon[104007]: pgmap v1408: 119 pgs: 1 active+recovery_wait, 7 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 108 active+clean; 3.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 4.0 KiB/s rd, 8.6 KiB/s wr, 597 op/s; 2085/182643 objects degraded (1.142%); 10266/182643 objects misplaced (5.621%); 7.6 MiB/s, 121 objects/s recovering 2024-09-13T15:03:50.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:50 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2027/187761 objects degraded (1.080%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:50.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:50.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:51.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:51.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:50 smithi019 ceph-mon[91296]: pgmap v1408: 119 pgs: 1 active+recovery_wait, 7 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 108 active+clean; 3.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 4.0 KiB/s rd, 8.6 KiB/s wr, 597 op/s; 2085/182643 objects degraded (1.142%); 10266/182643 objects misplaced (5.621%); 7.6 MiB/s, 121 objects/s recovering 2024-09-13T15:03:51.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:50 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2027/187761 objects degraded (1.080%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:51.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:51.263 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:51.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:51.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:51.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:51.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:51.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:51.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:51.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:51.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:51.904 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:51.943 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:52.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:52.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:52.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:52.223 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:52.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:52.353 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:52.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:52.492 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:03:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T15:03:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T15:03:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:03:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129065 12.6 12:6bd62a78:::benchmark_data_smithi003_259359_object13439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-13T15:03:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.462455+0000", 2024-09-13T15:03:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 342.96503750400001, 2024-09-13T15:03:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2997067840000001, 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129065 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462455+0000", 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462455+0000", 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462456+0000", 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5859999999999999e-06 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462462+0000", 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3380000000000004e-06 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462477+0000", 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5766000000000001e-05 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.462484+0000", 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1770000000000001e-06 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.469103+0000", 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066184740000000001 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.469401+0000", 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029828199999999997 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.469826+0000", 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042470499999999999 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481700+0000", 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011874312 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.483337+0000", 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016369069999999999 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.522366+0000", 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039028411999999998 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.522380+0000", 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4521999999999999e-05 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498225+0000", 2024-09-13T15:03:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97584474499999996 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498239+0000", 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3729e-05 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762079+0000", 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.26384074 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762161+0000", 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2089000000000006e-05 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129080 12.6 12:6497ebd7:::benchmark_data_smithi003_259359_object13454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.476318+0000", 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 342.95117369100001, 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.286167098, 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129080 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476318+0000", 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476318+0000", 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476320+0000", 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7290000000000001e-06 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476326+0000", 2024-09-13T15:03:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6239999999999999e-06 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476388+0000", 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2374999999999998e-05 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.476396+0000", 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0919999999999998e-06 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.483366+0000", 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069699829999999999 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.483476+0000", 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010993399999999999 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.483590+0000", 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113804 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.497960+0000", 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01436978 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.500120+0000", 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002160655 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523174+0000", 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023053361000000001 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523184+0000", 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0451e-05 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498273+0000", 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97508908500000002 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498284+0000", 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0513000000000001e-05 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762447+0000", 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.26416269 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762486+0000", 2024-09-13T15:03:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9022e-05 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129083 12.6 12:68209590:::benchmark_data_smithi003_259359_object13457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.481116+0000", 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 342.946375762, 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.28149063, 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129083 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481116+0000", 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481116+0000", 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481118+0000", 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6789999999999999e-06 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481124+0000", 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1569999999999997e-06 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481136+0000", 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2269000000000001e-05 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.481144+0000", 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.79e-06 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.487171+0000", 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060264100000000003 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.487298+0000", 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012718399999999999 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.487424+0000", 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012619500000000001 2024-09-13T15:03:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.500581+0000", 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013157294999999999 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.501814+0000", 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012331410000000001 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523215+0000", 2024-09-13T15:03:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021400395999999999 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523224+0000", 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1989999999999995e-06 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498314+0000", 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97509020000000002 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498325+0000", 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.024e-05 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762534+0000", 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2642099520000001 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762607+0000", 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2522999999999997e-05 2024-09-13T15:03:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129089 12.6 12:667fb162:::benchmark_data_smithi003_259359_object13463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.484895+0000", 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 342.94259703500001, 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3123250710000001, 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:03:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129089 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484895+0000", 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484895+0000", 2024-09-13T15:03:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484897+0000", 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5340000000000001e-06 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484901+0000", 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4630000000000003e-06 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484914+0000", 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2524000000000001e-05 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.484921+0000", 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5079999999999999e-06 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.494354+0000", 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094326810000000001 2024-09-13T15:03:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.494480+0000", 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000126456 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.494602+0000", 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121773 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:03:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.501835+0000", 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072334019999999999 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523645+0000", 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021809151999999998 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.523654+0000", 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1889999999999996e-06 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498354+0000", 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97470043299999998 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.498364+0000", 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0203e-05 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762734+0000", 2024-09-13T15:03:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2643698400000001 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for missing object", 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.762922+0000", 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000187519 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.796756+0000", 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033834649000000001 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.797220+0000", 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000463745 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:145051 13.a e384/362 13:510271f1:::benchmark_data_smithi003_293363_object25839:head v 384'9032, mlcod=384'9028)", 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:09.343768+0000", 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.083723755000001, 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023501395000000001, 2024-09-13T15:03:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.343768+0000", 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.343768+0000", 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.343769+0000", 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1299999999999998e-07 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.343777+0000", 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6410000000000002e-06 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.343788+0000", 2024-09-13T15:03:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1129e-05 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.343801+0000", 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2469e-05 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.361647+0000", 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017846194999999999 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.361709+0000", 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2106999999999993e-05 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.366748+0000", 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050393139999999996 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:09.367270+0000", 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052162699999999998 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:150664 13.c e385/362 13:3ffee1c6:::benchmark_data_smithi003_293363_object31452:head v 385'9410, mlcod=385'9408)", 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:15.615342+0000", 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 36.812150324999998, 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024206346, 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.615342+0000", 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.615342+0000", 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.615343+0000", 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7499999999999999e-07 2024-09-13T15:03:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.615352+0000", 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5950000000000005e-06 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.615369+0000", 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6799999999999998e-05 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.615377+0000", 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3720000000000005e-06 2024-09-13T15:03:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.620970+0000", 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005592227 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.621042+0000", 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2339e-05 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.639463+0000", 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018420700000000002 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:15.639548+0000", 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5438000000000004e-05 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:152101 13.4 e385/362 13:2af5f7b3:::benchmark_data_smithi003_293363_object32889:head v 385'9616, mlcod=385'9613)", 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:17.230344+0000", 2024-09-13T15:03:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 35.197147653999998, 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025315810000000001, 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.230344+0000", 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.230344+0000", 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.230345+0000", 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2799999999999995e-07 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.230350+0000", 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5909999999999998e-06 2024-09-13T15:03:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.230391+0000", 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1180000000000002e-05 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.230399+0000", 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0229999999999992e-06 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.234277+0000", 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038783429999999998 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.234330+0000", 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2540000000000002e-05 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.255435+0000", 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021104666000000001 2024-09-13T15:03:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:17.255660+0000", 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000225739 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:153789 13.a e385/362 13:50698e2c:::benchmark_data_smithi003_293363_object34577:head v 385'9588, mlcod=385'9583)", 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:19.044399+0000", 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 33.383092734000002, 2024-09-13T15:03:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.02577016, 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.044399+0000", 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.044399+0000", 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.044400+0000", 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.051e-06 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.044408+0000", 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7340000000000008e-06 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.044428+0000", 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9848999999999999e-05 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.044437+0000", 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9979999999999992e-06 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.045180+0000", 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00074341999999999995 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.045246+0000", 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5420999999999996e-05 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.070090+0000", 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024843791 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:19.070170+0000", 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9895999999999995e-05 2024-09-13T15:03:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:176525 13.a e387/362 13:51dc1406:::benchmark_data_smithi003_293363_object57313:head v 387'10997, mlcod=387'10994)", 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.834511+0000", 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.5929812229999998, 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023246401999999999, 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.834511+0000", 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.834511+0000", 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.834512+0000", 2024-09-13T15:03:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2900000000000003e-07 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.834517+0000", 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5160000000000002e-06 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.834531+0000", 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3740000000000001e-05 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.834537+0000", 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3439999999999997e-06 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.838003+0000", 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034661399999999999 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.838025+0000", 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1189999999999999e-05 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.850132+0000", 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012106982000000001 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.857757+0000", 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007625761 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:178133 13.4 e391/362 13:2a851a14:::benchmark_data_smithi003_293363_object58921:head v 391'11286, mlcod=391'11283)", 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:48.837876+0000", 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.5896159980000002, 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023458357999999999, 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.837876+0000", 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.837876+0000", 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.837877+0000", 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0699999999999996e-07 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.837886+0000", 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8570000000000001e-06 2024-09-13T15:03:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.837901+0000", 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5000999999999999e-05 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.837907+0000", 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6839999999999999e-06 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.838290+0000", 2024-09-13T15:03:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00038277199999999997 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.838339+0000", 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8689000000000001e-05 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.861109+0000", 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022770578999999999 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:48.861334+0000", 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000225069 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181372 13.4 e392/362 13:2768a73e:::benchmark_data_smithi003_293363_object62160:head v 392'11472, mlcod=392'11469)", 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.044322+0000", 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.38316995700000001, 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028073019000000001, 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044322+0000", 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044322+0000", 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044323+0000", 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1399999999999996e-07 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044331+0000", 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8760000000000005e-06 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044345+0000", 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4234e-05 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044351+0000", 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0510000000000002e-06 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.046318+0000", 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019670820000000002 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.046407+0000", 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9167999999999997e-05 2024-09-13T15:03:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.072327+0000", 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025920097 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.072395+0000", 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7797000000000002e-05 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181371 13.c e392/362 13:347e662f:::benchmark_data_smithi003_293363_object62159:head v 392'11343, mlcod=392'11341)", 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.044610+0000", 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.38288216800000002, 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.030884864000000001, 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044610+0000", 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044610+0000", 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044611+0000", 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5000000000000002e-07 2024-09-13T15:03:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044616+0000", 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9590000000000003e-06 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044629+0000", 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3713000000000001e-05 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.044635+0000", 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8209999999999998e-06 2024-09-13T15:03:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.046335+0000", 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016996190000000001 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.046357+0000", 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2609999999999999e-05 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.075387+0000", 2024-09-13T15:03:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029029816999999999 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.075495+0000", 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107675 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181375 13.c e392/362 13:36017507:::benchmark_data_smithi003_293363_object62163:head v 392'11344, mlcod=392'11341)", 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.047926+0000", 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.37956617599999998, 2024-09-13T15:03:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027629542, 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.047926+0000", 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.047926+0000", 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.047927+0000", 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3600000000000003e-07 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.047936+0000", 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0100000000000001e-06 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.047950+0000", 2024-09-13T15:03:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4214e-05 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.047958+0000", 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.622e-06 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.051341+0000", 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033838710000000001 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.051392+0000", 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0596000000000001e-05 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.075499+0000", 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024107144 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.075556+0000", 2024-09-13T15:03:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6348999999999999e-05 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181374 13.4 e392/362 13:27d62793:::benchmark_data_smithi003_293363_object62162:head v 392'11473, mlcod=392'11469)", 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.049118+0000", 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.37837377700000002, 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029132046000000002, 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.049118+0000", 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.049118+0000", 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.049119+0000", 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5700000000000001e-07 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.049128+0000", 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5220000000000007e-06 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.049144+0000", 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6036999999999999e-05 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.049151+0000", 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2509999999999997e-06 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.051193+0000", 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020414980000000001 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.051212+0000", 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9031000000000001e-05 2024-09-13T15:03:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.078191+0000", 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.02697942 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.078250+0000", 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9429999999999999e-05 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181376 13.4 e392/362 13:24fcdb2e:::benchmark_data_smithi003_293363_object62164:head v 392'11474, mlcod=392'11469)", 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.053949+0000", 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.37354336599999999, 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024344461000000001, 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.053949+0000", 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.053949+0000", 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.053950+0000", 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.049e-06 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.053961+0000", 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0728e-05 2024-09-13T15:03:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.053977+0000", 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5964000000000002e-05 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.053985+0000", 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4919999999999993e-06 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.058252+0000", 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042665089999999999 2024-09-13T15:03:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.058270+0000", 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8618e-05 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.078257+0000", 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019986995 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.078293+0000", 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6106000000000003e-05 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181707 13.c e392/362 13:33a63bfd:::benchmark_data_smithi003_293363_object62495:head v 392'11363, mlcod=392'11361)", 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.386950+0000", 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.040541765, 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026179442000000001, 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.386950+0000", 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.386950+0000", 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.386951+0000", 2024-09-13T15:03:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5799999999999998e-07 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.386957+0000", 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8409999999999996e-06 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.386972+0000", 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5164000000000001e-05 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.386979+0000", 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1879999999999996e-06 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393013+0000", 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060332420000000003 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393038+0000", 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5358e-05 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.412967+0000", 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019929483000000001 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.413130+0000", 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016240800000000001 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181706 13.4 e392/362 13:26cc2274:::benchmark_data_smithi003_293363_object62494:head v 392'11500, mlcod=392'11495)", 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.389593+0000", 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.037899001000000002, 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023905853000000001, 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.389593+0000", 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.389593+0000", 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.389594+0000", 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0869999999999999e-06 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.389601+0000", 2024-09-13T15:03:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0269999999999998e-06 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.389618+0000", 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6475000000000001e-05 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.389626+0000", 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.687e-06 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393191+0000", 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035649129999999998 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393386+0000", 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000194883 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.413136+0000", 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019750276000000001 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.413499+0000", 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000362505 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181709 13.c e392/362 13:341e554d:::benchmark_data_smithi003_293363_object62497:head v 392'11364, mlcod=392'11361)", 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.390882+0000", 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.036610350999999999, 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029432656000000001, 2024-09-13T15:03:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.390882+0000", 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.390882+0000", 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.390883+0000", 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1999999999999998e-07 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.390889+0000", 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5950000000000004e-06 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.390905+0000", 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5976000000000001e-05 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.390912+0000", 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0740000000000004e-06 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.397331+0000", 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064188149999999996 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.397351+0000", 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9967999999999999e-05 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.420285+0000", 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022933795 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.420314+0000", 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9513e-05 2024-09-13T15:03:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181713 13.4 e392/362 13:2fcc3a02:::benchmark_data_smithi003_293363_object62501:head v 392'11501, mlcod=392'11495)", 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.393615+0000", 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.033877193999999999, 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026719597000000001, 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393615+0000", 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393615+0000", 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393616+0000", 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0100000000000001e-06 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393625+0000", 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9560000000000003e-06 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393635+0000", 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0387999999999999e-05 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.393643+0000", 2024-09-13T15:03:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5089999999999996e-06 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.397866+0000", 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042235320000000003 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.397885+0000", 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8947999999999999e-05 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.420318+0000", 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022433043999999999 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.420335+0000", 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6209999999999999e-05 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:181711 13.c e392/362 13:320b460c:::benchmark_data_smithi003_293363_object62499:head v 392'11365, mlcod=392'11361)", 2024-09-13T15:03:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:52.394887+0000", 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.032605108000000001, 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025461182999999998, 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.394887+0000", 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:03:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.394887+0000", 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.394888+0000", 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8599999999999997e-07 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.394897+0000", 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3060000000000003e-06 2024-09-13T15:03:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.394908+0000", 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0465000000000001e-05 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.394914+0000", 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7789999999999998e-06 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.401052+0000", 2024-09-13T15:03:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061377840000000003 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.401110+0000", 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7633000000000002e-05 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.420336+0000", 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019226166999999999 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:52.420348+0000", 2024-09-13T15:03:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2262999999999999e-05 2024-09-13T15:03:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:03:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:03:52.598 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:03:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.599 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:52.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:52.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:52.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:52.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:52.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:52.975 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:53.011 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:53.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:53.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:53.195 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:03:53.195 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:53.230 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:03:53.231 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:53.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:53.268 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:03:53.269 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:53.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:03:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:53.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:53.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:53 smithi003 ceph-mon[104007]: pgmap v1409: 119 pgs: 1 active+recovery_wait, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 114 active+clean; 3.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.8 KiB/s rd, 6.2 KiB/s wr, 504 op/s; 2085/181242 objects degraded (1.150%); 10266/181242 objects misplaced (5.664%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T15:03:53.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1057538347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:53.309 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:53 smithi003 ceph-mon[107278]: pgmap v1409: 119 pgs: 1 active+recovery_wait, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 114 active+clean; 3.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.8 KiB/s rd, 6.2 KiB/s wr, 504 op/s; 2085/181242 objects degraded (1.150%); 10266/181242 objects misplaced (5.664%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T15:03:53.309 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1057538347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:53.309 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:03:53.309 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:53.345 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:03:53.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:53.385 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:03:53.385 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:53.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:53.419 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:03:53.419 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:53.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:53.454 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:03:53.455 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:53.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:53.567 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:53.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:53 smithi019 ceph-mon[91296]: pgmap v1409: 119 pgs: 1 active+recovery_wait, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 114 active+clean; 3.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.8 KiB/s rd, 6.2 KiB/s wr, 504 op/s; 2085/181242 objects degraded (1.150%); 10266/181242 objects misplaced (5.664%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T15:03:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1057538347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:53.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:53.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:53.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:53.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:53.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:54.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:54.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:54.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:54.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:03:54] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T15:03:54.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:54.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:54.420 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:54.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:54.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:54.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:54.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:54.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:54.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:54 smithi019 ceph-mon[91296]: pgmap v1410: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 959 op/s; 2159/168078 objects degraded (1.285%); 9459/168078 objects misplaced (5.628%); 5.4 MiB/s, 85 objects/s recovering 2024-09-13T15:03:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:03:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:03:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[107278]: pgmap v1410: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 959 op/s; 2159/168078 objects degraded (1.285%); 9459/168078 objects misplaced (5.628%); 5.4 MiB/s, 85 objects/s recovering 2024-09-13T15:03:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:03:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:03:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:54.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:54.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[104007]: pgmap v1410: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 959 op/s; 2159/168078 objects degraded (1.285%); 9459/168078 objects misplaced (5.628%); 5.4 MiB/s, 85 objects/s recovering 2024-09-13T15:03:54.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:03:54.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:03:54.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:54.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:54.987 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:55.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.136 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:55.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:55.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:55.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:55.298 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 5, 0, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:03:55.300 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:03:55.301 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_8 2024-09-13T15:03:55.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:03:55.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:55.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:55.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:55.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:55.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:03:55.705 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:55.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:55.785 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:55.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:55.865 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:55.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:56.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:56.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:56.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:56.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:55 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2159/168078 objects degraded (1.285%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:56.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:55 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2159/168078 objects degraded (1.285%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:56.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:56.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:56.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:55 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2159/168078 objects degraded (1.285%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:03:56.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:56.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:56.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:03:56.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:56.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:56.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:56.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:56.679 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:56.712 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:56.814 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.852 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:56.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:56.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:56.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:57.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:57.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:57.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:56 smithi019 ceph-mon[91296]: pgmap v1411: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 793 op/s; 2159/168078 objects degraded (1.285%); 9459/168078 objects misplaced (5.628%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T15:03:57.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:56 smithi003 ceph-mon[107278]: pgmap v1411: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 793 op/s; 2159/168078 objects degraded (1.285%); 9459/168078 objects misplaced (5.628%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T15:03:57.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:56 smithi003 ceph-mon[104007]: pgmap v1411: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 793 op/s; 2159/168078 objects degraded (1.285%); 9459/168078 objects misplaced (5.628%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T15:03:57.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:57.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:57.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:57.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:57.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:57.515 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:57.552 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:57.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:57.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:57.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:57.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:57.926 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.973 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:57.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:58.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:58.108 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:58.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:58.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:58.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:58.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:58.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:58.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:58.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:58.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:58.894 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:58.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:58.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:58.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:58 smithi003 ceph-mon[104007]: pgmap v1412: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 804 op/s; 2159/163599 objects degraded (1.320%); 9160/163599 objects misplaced (5.599%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T15:03:58.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:58.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:58.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:58 smithi003 ceph-mon[107278]: pgmap v1412: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 804 op/s; 2159/163599 objects degraded (1.320%); 9160/163599 objects misplaced (5.599%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T15:03:58.938 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:59.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:59.080 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:59.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:03:59.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:59.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:58 smithi019 ceph-mon[91296]: pgmap v1412: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 804 op/s; 2159/163599 objects degraded (1.320%); 9160/163599 objects misplaced (5.599%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T15:03:59.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:59.322 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:59.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:59.457 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:59.462 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:59.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:59.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:59.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:59.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:59.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:59.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:59.809 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:03:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:59.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:59.942 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:03:59.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:03:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:00.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1630294877' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:00.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:03:59 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:00.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1630294877' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:00.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:03:59 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:00.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:00.100 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:00.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1630294877' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:00.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:03:59 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:00.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:00.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:00.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:00.378 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:00.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:00.518 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:00.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:00.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:04:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:04:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:00.775 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:00.811 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:00.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:00.961 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:00.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:01.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:01.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:00 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:00 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:00 smithi019 ceph-mon[91296]: osdmap e393: 8 total, 8 up, 8 in 2024-09-13T15:04:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:00 smithi019 ceph-mon[91296]: pgmap v1414: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 2366/153687 objects degraded (1.539%); 8545/153687 objects misplaced (5.560%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T15:04:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:00 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2159/163599 objects degraded (1.320%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[104007]: osdmap e393: 8 total, 8 up, 8 in 2024-09-13T15:04:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[104007]: pgmap v1414: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 2366/153687 objects degraded (1.539%); 8545/153687 objects misplaced (5.560%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T15:04:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2159/163599 objects degraded (1.320%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[107278]: osdmap e393: 8 total, 8 up, 8 in 2024-09-13T15:04:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[107278]: pgmap v1414: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 2366/153687 objects degraded (1.539%); 8545/153687 objects misplaced (5.560%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T15:04:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:00 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2159/163599 objects degraded (1.320%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:01.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:01.240 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:01.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:01.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:01.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:01.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:01.613 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:01.644 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:01.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:01.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:01.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:01.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:01.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:01.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:02.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:02.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:02.162 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:02.207 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:02.309 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:02.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:02.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:02.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:01 smithi019 ceph-mon[91296]: from='client.37104 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:04:02.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:01 smithi003 ceph-mon[107278]: from='client.37104 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:04:02.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:01 smithi003 ceph-mon[104007]: from='client.37104 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:04:02.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:02.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:02.592 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:02.634 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:02.736 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:02.773 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:02.803 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:04:02.808 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:04:02.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:02.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:03.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:03.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:03.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:03.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:02 smithi003 ceph-mon[107278]: pgmap v1415: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 2366/151941 objects degraded (1.557%); 8545/151941 objects misplaced (5.624%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T15:04:03.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:02 smithi003 ceph-mon[104007]: pgmap v1415: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 2366/151941 objects degraded (1.557%); 8545/151941 objects misplaced (5.624%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T15:04:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.185 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:03.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:03.286 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:03.334 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:03.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:02 smithi019 ceph-mon[91296]: pgmap v1415: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 3.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 2366/151941 objects degraded (1.557%); 8545/151941 objects misplaced (5.624%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T15:04:03.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:03.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:03.539 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1414,"stamp":"2024-09-13T15:04:00.131197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3356170213,"num_objects":51229,"num_object_clones":0,"num_object_copies":153687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2366,"num_objects_misplaced":8545,"num_objects_unfound":0,"num_objects_dirty":51229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187281,"num_write_kb":7629409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7323,"num_bytes_recovered":477103319,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103670,"ondisk_log_size":103670,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":362,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20856016,"kb_used_data":10538448,"kb_used_omap":29,"kb_used_meta":10317090,"kb_avail":729105200,"statfs":{"total":767960285184,"available":746603724800,"internally_reserved":0,"allocated":10791370752,"data_stored":10772563896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":10564700686},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-632566890,"num_objects":-9652,"num_object_clones":0,"num_object_copies":-28956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":281,"num_objects_misplaced":-1721,"num_objects_unfound":0,"num_objects_dirty":-9652,"num_whiteouts":0,"num_read":-7,"num_read_kb":-17,"num_write":9648,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":13814678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003611"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":24,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899196+0000","last_change":"2024-09-13T15:03:48.428392+0000","last_active":"2024-09-13T15:03:48.899196+0000","last_peered":"2024-09-13T15:03:48.899196+0000","last_clean":"2024-09-13T15:03:48.899196+0000","last_became_active":"2024-09-13T15:03:46.867925+0000","last_became_peered":"2024-09-13T15:03:46.867925+0000","last_unstale":"2024-09-13T15:03:48.899196+0000","last_undegraded":"2024-09-13T15:03:48.899196+0000","last_fullsized":"2024-09-13T15:03:48.899196+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:48.428315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:48.428315+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:45.557387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":50,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.889186+0000","last_change":"2024-09-13T15:02:19.429607+0000","last_active":"2024-09-13T15:03:48.889186+0000","last_peered":"2024-09-13T15:03:48.889186+0000","last_clean":"2024-09-13T15:03:48.889186+0000","last_became_active":"2024-09-13T15:02:19.418793+0000","last_became_peered":"2024-09-13T15:02:19.418793+0000","last_unstale":"2024-09-13T15:03:48.889186+0000","last_undegraded":"2024-09-13T15:03:48.889186+0000","last_fullsized":"2024-09-13T15:03:48.889186+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:03.520208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":364,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.866974+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T15:03:47.866974+0000","last_peered":"2024-09-13T15:03:47.866974+0000","last_clean":"2024-09-13T15:03:47.866974+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T15:03:47.866974+0000","last_undegraded":"2024-09-13T15:03:47.866974+0000","last_fullsized":"2024-09-13T15:03:47.866974+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":90,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906242+0000","last_change":"2024-09-13T14:57:58.200463+0000","last_active":"2024-09-13T15:03:47.906242+0000","last_peered":"2024-09-13T15:03:47.906242+0000","last_clean":"2024-09-13T15:03:47.906242+0000","last_became_active":"2024-09-13T14:57:25.279086+0000","last_became_peered":"2024-09-13T14:57:25.279086+0000","last_unstale":"2024-09-13T15:03:47.906242+0000","last_undegraded":"2024-09-13T15:03:47.906242+0000","last_fullsized":"2024-09-13T15:03:47.906242+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:58.986716+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899902+0000","last_change":"2024-09-13T15:03:46.871160+0000","last_active":"2024-09-13T15:03:48.899902+0000","last_peered":"2024-09-13T15:03:48.899902+0000","last_clean":"2024-09-13T15:03:48.899902+0000","last_became_active":"2024-09-13T15:03:46.870825+0000","last_became_peered":"2024-09-13T15:03:46.870825+0000","last_unstale":"2024-09-13T15:03:48.899902+0000","last_undegraded":"2024-09-13T15:03:48.899902+0000","last_fullsized":"2024-09-13T15:03:48.899902+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:54.702996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899115+0000","last_change":"2024-09-13T15:03:46.873412+0000","last_active":"2024-09-13T15:03:48.899115+0000","last_peered":"2024-09-13T15:03:48.899115+0000","last_clean":"2024-09-13T15:03:48.899115+0000","last_became_active":"2024-09-13T15:03:46.873094+0000","last_became_peered":"2024-09-13T15:03:46.873094+0000","last_unstale":"2024-09-13T15:03:48.899115+0000","last_undegraded":"2024-09-13T15:03:48.899115+0000","last_fullsized":"2024-09-13T15:03:48.899115+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:45.283467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":95,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867969+0000","last_change":"2024-09-13T14:58:28.426763+0000","last_active":"2024-09-13T15:03:47.867969+0000","last_peered":"2024-09-13T15:03:47.867969+0000","last_clean":"2024-09-13T15:03:47.867969+0000","last_became_active":"2024-09-13T14:58:28.426270+0000","last_became_peered":"2024-09-13T14:58:28.426270+0000","last_unstale":"2024-09-13T15:03:47.867969+0000","last_undegraded":"2024-09-13T15:03:47.867969+0000","last_fullsized":"2024-09-13T15:03:47.867969+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.991576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":411,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867342+0000","last_change":"2024-09-13T15:03:46.867138+0000","last_active":"2024-09-13T15:03:47.867342+0000","last_peered":"2024-09-13T15:03:47.867342+0000","last_clean":"2024-09-13T15:03:47.867342+0000","last_became_active":"2024-09-13T15:03:46.866870+0000","last_became_peered":"2024-09-13T15:03:46.866870+0000","last_unstale":"2024-09-13T15:03:47.867342+0000","last_undegraded":"2024-09-13T15:03:47.867342+0000","last_fullsized":"2024-09-13T15:03:47.867342+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:44.031064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":24,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899829+0000","last_change":"2024-09-13T15:03:47.469254+0000","last_active":"2024-09-13T15:03:48.899829+0000","last_peered":"2024-09-13T15:03:48.899829+0000","last_clean":"2024-09-13T15:03:48.899829+0000","last_became_active":"2024-09-13T15:03:46.854242+0000","last_became_peered":"2024-09-13T15:03:46.854242+0000","last_unstale":"2024-09-13T15:03:48.899829+0000","last_undegraded":"2024-09-13T15:03:48.899829+0000","last_fullsized":"2024-09-13T15:03:48.899829+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:47.469211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:47.469211+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:59.563316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":95,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867520+0000","last_change":"2024-09-13T14:58:28.429061+0000","last_active":"2024-09-13T15:03:47.867520+0000","last_peered":"2024-09-13T15:03:47.867520+0000","last_clean":"2024-09-13T15:03:47.867520+0000","last_became_active":"2024-09-13T14:58:28.428642+0000","last_became_peered":"2024-09-13T14:58:28.428642+0000","last_unstale":"2024-09-13T15:03:47.867520+0000","last_undegraded":"2024-09-13T15:03:47.867520+0000","last_fullsized":"2024-09-13T15:03:47.867520+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:03.197824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":421,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867580+0000","last_change":"2024-09-13T15:03:46.868016+0000","last_active":"2024-09-13T15:03:47.867580+0000","last_peered":"2024-09-13T15:03:47.867580+0000","last_clean":"2024-09-13T15:03:47.867580+0000","last_became_active":"2024-09-13T15:03:46.867643+0000","last_became_peered":"2024-09-13T15:03:46.867643+0000","last_unstale":"2024-09-13T15:03:47.867580+0000","last_undegraded":"2024-09-13T15:03:47.867580+0000","last_fullsized":"2024-09-13T15:03:47.867580+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:26.304605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899216+0000","last_change":"2024-09-13T15:03:46.864596+0000","last_active":"2024-09-13T15:03:48.899216+0000","last_peered":"2024-09-13T15:03:48.899216+0000","last_clean":"2024-09-13T15:03:48.899216+0000","last_became_active":"2024-09-13T15:03:46.863107+0000","last_became_peered":"2024-09-13T15:03:46.863107+0000","last_unstale":"2024-09-13T15:03:48.899216+0000","last_undegraded":"2024-09-13T15:03:48.899216+0000","last_fullsized":"2024-09-13T15:03:48.899216+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:07.821841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":44,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.890899+0000","last_change":"2024-09-13T15:02:19.418023+0000","last_active":"2024-09-13T15:03:48.890899+0000","last_peered":"2024-09-13T15:03:48.890899+0000","last_clean":"2024-09-13T15:03:48.890899+0000","last_became_active":"2024-09-13T15:02:19.417503+0000","last_became_peered":"2024-09-13T15:02:19.417503+0000","last_unstale":"2024-09-13T15:03:48.890899+0000","last_undegraded":"2024-09-13T15:03:48.890899+0000","last_fullsized":"2024-09-13T15:03:48.890899+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:48.630767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":665,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.906843+0000","last_change":"2024-09-13T15:03:46.871823+0000","last_active":"2024-09-13T15:03:48.906843+0000","last_peered":"2024-09-13T15:03:48.906843+0000","last_clean":"2024-09-13T15:03:48.906843+0000","last_became_active":"2024-09-13T15:03:46.868753+0000","last_became_peered":"2024-09-13T15:03:46.868753+0000","last_unstale":"2024-09-13T15:03:48.906843+0000","last_undegraded":"2024-09-13T15:03:48.906843+0000","last_fullsized":"2024-09-13T15:03:48.906843+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:23.680514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":87,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.868557+0000","last_change":"2024-09-13T14:58:09.011981+0000","last_active":"2024-09-13T15:03:47.868557+0000","last_peered":"2024-09-13T15:03:47.868557+0000","last_clean":"2024-09-13T15:03:47.868557+0000","last_became_active":"2024-09-13T14:57:53.357674+0000","last_became_peered":"2024-09-13T14:57:53.357674+0000","last_unstale":"2024-09-13T15:03:47.868557+0000","last_undegraded":"2024-09-13T15:03:47.868557+0000","last_fullsized":"2024-09-13T15:03:47.868557+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.701912+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":373,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867471+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T15:03:47.867471+0000","last_peered":"2024-09-13T15:03:47.867471+0000","last_clean":"2024-09-13T15:03:47.867471+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T15:03:47.867471+0000","last_undegraded":"2024-09-13T15:03:47.867471+0000","last_fullsized":"2024-09-13T15:03:47.867471+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899076+0000","last_change":"2024-09-13T15:03:46.868133+0000","last_active":"2024-09-13T15:03:48.899076+0000","last_peered":"2024-09-13T15:03:48.899076+0000","last_clean":"2024-09-13T15:03:48.899076+0000","last_became_active":"2024-09-13T15:03:46.866551+0000","last_became_peered":"2024-09-13T15:03:46.866551+0000","last_unstale":"2024-09-13T15:03:48.899076+0000","last_undegraded":"2024-09-13T15:03:48.899076+0000","last_fullsized":"2024-09-13T15:03:48.899076+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:00.313559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":103,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.869020+0000","last_change":"2024-09-13T15:02:19.406391+0000","last_active":"2024-09-13T15:03:47.869020+0000","last_peered":"2024-09-13T15:03:47.869020+0000","last_clean":"2024-09-13T15:03:47.869020+0000","last_became_active":"2024-09-13T15:02:19.406040+0000","last_became_peered":"2024-09-13T15:02:19.406040+0000","last_unstale":"2024-09-13T15:03:47.869020+0000","last_undegraded":"2024-09-13T15:03:47.869020+0000","last_fullsized":"2024-09-13T15:03:47.869020+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:45.665420+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":322,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.901375+0000","last_change":"2024-09-13T15:03:46.870303+0000","last_active":"2024-09-13T15:03:48.901375+0000","last_peered":"2024-09-13T15:03:48.901375+0000","last_clean":"2024-09-13T15:03:48.901375+0000","last_became_active":"2024-09-13T15:03:46.869984+0000","last_became_peered":"2024-09-13T15:03:46.869984+0000","last_unstale":"2024-09-13T15:03:48.901375+0000","last_undegraded":"2024-09-13T15:03:48.901375+0000","last_fullsized":"2024-09-13T15:03:48.901375+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:38.720142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":58,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.888618+0000","last_change":"2024-09-13T15:03:46.864512+0000","last_active":"2024-09-13T15:03:48.888618+0000","last_peered":"2024-09-13T15:03:48.888618+0000","last_clean":"2024-09-13T15:03:48.888618+0000","last_became_active":"2024-09-13T15:03:46.864120+0000","last_became_peered":"2024-09-13T15:03:46.864120+0000","last_unstale":"2024-09-13T15:03:48.888618+0000","last_undegraded":"2024-09-13T15:03:48.888618+0000","last_fullsized":"2024-09-13T15:03:48.888618+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:36.450217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":90,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867160+0000","last_change":"2024-09-13T14:58:05.212101+0000","last_active":"2024-09-13T15:03:47.867160+0000","last_peered":"2024-09-13T15:03:47.867160+0000","last_clean":"2024-09-13T15:03:47.867160+0000","last_became_active":"2024-09-13T14:57:25.275734+0000","last_became_peered":"2024-09-13T14:57:25.275734+0000","last_unstale":"2024-09-13T15:03:47.867160+0000","last_undegraded":"2024-09-13T15:03:47.867160+0000","last_fullsized":"2024-09-13T15:03:47.867160+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.212008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.212008+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:01.720211+0000","scrub_duration":0.0048969690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.898659+0000","last_change":"2024-09-13T15:03:46.867411+0000","last_active":"2024-09-13T15:03:48.898659+0000","last_peered":"2024-09-13T15:03:48.898659+0000","last_clean":"2024-09-13T15:03:48.898659+0000","last_became_active":"2024-09-13T15:03:46.866890+0000","last_became_peered":"2024-09-13T15:03:46.866890+0000","last_unstale":"2024-09-13T15:03:48.898659+0000","last_undegraded":"2024-09-13T15:03:48.898659+0000","last_fullsized":"2024-09-13T15:03:48.898659+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:59.143319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":374,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900923+0000","last_change":"2024-09-13T14:57:53.364039+0000","last_active":"2024-09-13T15:03:47.900923+0000","last_peered":"2024-09-13T15:03:47.900923+0000","last_clean":"2024-09-13T15:03:47.900923+0000","last_became_active":"2024-09-13T14:57:53.363550+0000","last_became_peered":"2024-09-13T14:57:53.363550+0000","last_unstale":"2024-09-13T15:03:47.900923+0000","last_undegraded":"2024-09-13T15:03:47.900923+0000","last_fullsized":"2024-09-13T15:03:47.900923+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:36.171089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":116,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906777+0000","last_change":"2024-09-13T14:58:04.168950+0000","last_active":"2024-09-13T15:03:47.906777+0000","last_peered":"2024-09-13T15:03:47.906777+0000","last_clean":"2024-09-13T15:03:47.906777+0000","last_became_active":"2024-09-13T14:57:53.351301+0000","last_became_peered":"2024-09-13T14:57:53.351301+0000","last_unstale":"2024-09-13T15:03:47.906777+0000","last_undegraded":"2024-09-13T15:03:47.906777+0000","last_fullsized":"2024-09-13T15:03:47.906777+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:42.270600+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.898842+0000","last_change":"2024-09-13T15:03:46.855014+0000","last_active":"2024-09-13T15:03:48.898842+0000","last_peered":"2024-09-13T15:03:48.898842+0000","last_clean":"2024-09-13T15:03:48.898842+0000","last_became_active":"2024-09-13T15:03:46.854333+0000","last_became_peered":"2024-09-13T15:03:46.854333+0000","last_unstale":"2024-09-13T15:03:48.898842+0000","last_undegraded":"2024-09-13T15:03:48.898842+0000","last_fullsized":"2024-09-13T15:03:48.898842+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:30.681891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":51,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.889093+0000","last_change":"2024-09-13T15:03:46.871342+0000","last_active":"2024-09-13T15:03:48.889093+0000","last_peered":"2024-09-13T15:03:48.889093+0000","last_clean":"2024-09-13T15:03:48.889093+0000","last_became_active":"2024-09-13T15:03:46.863406+0000","last_became_peered":"2024-09-13T15:03:46.863406+0000","last_unstale":"2024-09-13T15:03:48.889093+0000","last_undegraded":"2024-09-13T15:03:48.889093+0000","last_fullsized":"2024-09-13T15:03:48.889093+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:53.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.898919+0000","last_change":"2024-09-13T15:03:46.859350+0000","last_active":"2024-09-13T15:03:48.898919+0000","last_peered":"2024-09-13T15:03:48.898919+0000","last_clean":"2024-09-13T15:03:48.898919+0000","last_became_active":"2024-09-13T15:03:46.853202+0000","last_became_peered":"2024-09-13T15:03:46.853202+0000","last_unstale":"2024-09-13T15:03:48.898919+0000","last_undegraded":"2024-09-13T15:03:48.898919+0000","last_fullsized":"2024-09-13T15:03:48.898919+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:42.594537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":73,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.869301+0000","last_change":"2024-09-13T14:57:53.350622+0000","last_active":"2024-09-13T15:03:47.869301+0000","last_peered":"2024-09-13T15:03:47.869301+0000","last_clean":"2024-09-13T15:03:47.869301+0000","last_became_active":"2024-09-13T14:57:53.350292+0000","last_became_peered":"2024-09-13T14:57:53.350292+0000","last_unstale":"2024-09-13T15:03:47.869301+0000","last_undegraded":"2024-09-13T15:03:47.869301+0000","last_fullsized":"2024-09-13T15:03:47.869301+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:16.088338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":353,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.906743+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T15:03:48.906743+0000","last_peered":"2024-09-13T15:03:48.906743+0000","last_clean":"2024-09-13T15:03:48.906743+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T15:03:48.906743+0000","last_undegraded":"2024-09-13T15:03:48.906743+0000","last_fullsized":"2024-09-13T15:03:48.906743+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:43:13.165444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":93,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.869554+0000","last_change":"2024-09-13T15:02:19.406264+0000","last_active":"2024-09-13T15:03:47.869554+0000","last_peered":"2024-09-13T15:03:47.869554+0000","last_clean":"2024-09-13T15:03:47.869554+0000","last_became_active":"2024-09-13T15:02:19.405876+0000","last_became_peered":"2024-09-13T15:02:19.405876+0000","last_unstale":"2024-09-13T15:03:47.869554+0000","last_undegraded":"2024-09-13T15:03:47.869554+0000","last_fullsized":"2024-09-13T15:03:47.869554+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:19.208307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":187,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906548+0000","last_change":"2024-09-13T15:03:46.877994+0000","last_active":"2024-09-13T15:03:47.906548+0000","last_peered":"2024-09-13T15:03:47.906548+0000","last_clean":"2024-09-13T15:03:47.906548+0000","last_became_active":"2024-09-13T15:03:46.859821+0000","last_became_peered":"2024-09-13T15:03:46.859821+0000","last_unstale":"2024-09-13T15:03:47.906548+0000","last_undegraded":"2024-09-13T15:03:47.906548+0000","last_fullsized":"2024-09-13T15:03:47.906548+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:44.527820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":293,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900022+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T15:03:47.900022+0000","last_peered":"2024-09-13T15:03:47.900022+0000","last_clean":"2024-09-13T15:03:47.900022+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T15:03:47.900022+0000","last_undegraded":"2024-09-13T15:03:47.900022+0000","last_fullsized":"2024-09-13T15:03:47.900022+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":200,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.904603+0000","last_change":"2024-09-13T15:03:46.857574+0000","last_active":"2024-09-13T15:03:48.904603+0000","last_peered":"2024-09-13T15:03:48.904603+0000","last_clean":"2024-09-13T15:03:48.904603+0000","last_became_active":"2024-09-13T15:03:46.857142+0000","last_became_peered":"2024-09-13T15:03:46.857142+0000","last_unstale":"2024-09-13T15:03:48.904603+0000","last_undegraded":"2024-09-13T15:03:48.904603+0000","last_fullsized":"2024-09-13T15:03:48.904603+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:12.884559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":361,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867437+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T15:03:47.867437+0000","last_peered":"2024-09-13T15:03:47.867437+0000","last_clean":"2024-09-13T15:03:47.867437+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T15:03:47.867437+0000","last_undegraded":"2024-09-13T15:03:47.867437+0000","last_fullsized":"2024-09-13T15:03:47.867437+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":354,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867594+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T15:03:47.867594+0000","last_peered":"2024-09-13T15:03:47.867594+0000","last_clean":"2024-09-13T15:03:47.867594+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T15:03:47.867594+0000","last_undegraded":"2024-09-13T15:03:47.867594+0000","last_fullsized":"2024-09-13T15:03:47.867594+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"392'11457","reported_seq":11529,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:56.289648+0000","last_change":"2024-09-13T15:00:55.405989+0000","last_active":"2024-09-13T15:03:56.289648+0000","last_peered":"2024-09-13T15:03:56.289648+0000","last_clean":"2024-09-13T15:03:56.289648+0000","last_became_active":"2024-09-13T15:00:55.405741+0000","last_became_peered":"2024-09-13T15:00:55.405741+0000","last_unstale":"2024-09-13T15:03:56.289648+0000","last_undegraded":"2024-09-13T15:03:56.289648+0000","last_fullsized":"2024-09-13T15:03:56.289648+0000","mapping_epoch":362,"log_start":"369'4051","ondisk_log_start":"369'4051","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":7406,"log_dups_size":0,"ondisk_log_size":7406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:32.041720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215547904,"num_objects":3289,"num_object_clones":0,"num_object_copies":9867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":357,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867064+0000","last_change":"2024-09-13T14:57:53.351374+0000","last_active":"2024-09-13T15:03:47.867064+0000","last_peered":"2024-09-13T15:03:47.867064+0000","last_clean":"2024-09-13T15:03:47.867064+0000","last_became_active":"2024-09-13T14:57:53.350803+0000","last_became_peered":"2024-09-13T14:57:53.350803+0000","last_unstale":"2024-09-13T15:03:47.867064+0000","last_undegraded":"2024-09-13T15:03:47.867064+0000","last_fullsized":"2024-09-13T15:03:47.867064+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:30.535837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":387,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900988+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T15:03:47.900988+0000","last_peered":"2024-09-13T15:03:47.900988+0000","last_clean":"2024-09-13T15:03:47.900988+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T15:03:47.900988+0000","last_undegraded":"2024-09-13T15:03:47.900988+0000","last_fullsized":"2024-09-13T15:03:47.900988+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":48,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.888789+0000","last_change":"2024-09-13T15:02:19.432393+0000","last_active":"2024-09-13T15:03:48.888789+0000","last_peered":"2024-09-13T15:03:48.888789+0000","last_clean":"2024-09-13T15:03:48.888789+0000","last_became_active":"2024-09-13T15:02:19.419183+0000","last_became_peered":"2024-09-13T15:02:19.419183+0000","last_unstale":"2024-09-13T15:03:48.888789+0000","last_undegraded":"2024-09-13T15:03:48.888789+0000","last_fullsized":"2024-09-13T15:03:48.888789+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:51.143192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899521+0000","last_change":"2024-09-13T15:03:46.868597+0000","last_active":"2024-09-13T15:03:48.899521+0000","last_peered":"2024-09-13T15:03:48.899521+0000","last_clean":"2024-09-13T15:03:48.899521+0000","last_became_active":"2024-09-13T15:03:46.866939+0000","last_became_peered":"2024-09-13T15:03:46.866939+0000","last_unstale":"2024-09-13T15:03:48.899521+0000","last_undegraded":"2024-09-13T15:03:48.899521+0000","last_fullsized":"2024-09-13T15:03:48.899521+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.060483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"392'11835","reported_seq":15812,"reported_epoch":392,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T15:03:58.775855+0000","last_change":"2024-09-13T15:03:46.891007+0000","last_active":"2024-09-13T15:03:58.775855+0000","last_peered":"2024-09-13T15:03:58.775855+0000","last_clean":"2024-09-13T15:02:18.361788+0000","last_became_active":"2024-09-13T15:02:20.574877+0000","last_became_peered":"2024-09-13T15:02:20.574877+0000","last_unstale":"2024-09-13T15:03:58.775855+0000","last_undegraded":"2024-09-13T15:02:44.391781+0000","last_fullsized":"2024-09-13T15:02:20.383180+0000","mapping_epoch":376,"log_start":"371'4809","ondisk_log_start":"371'4809","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":7026,"log_dups_size":0,"ondisk_log_size":7026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:14.477107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208601088,"num_objects":3183,"num_object_clones":0,"num_object_copies":9549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2366,"num_objects_misplaced":2172,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1961,"num_bytes_recovered":128516096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,2],"avail_no_missing":["5","2","6"],"object_location_counts":[{"shards":"2,5","objects":-1355},{"shards":"2,5,6","objects":2172}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.898560+0000","last_change":"2024-09-13T15:03:46.865877+0000","last_active":"2024-09-13T15:03:48.898560+0000","last_peered":"2024-09-13T15:03:48.898560+0000","last_clean":"2024-09-13T15:03:48.898560+0000","last_became_active":"2024-09-13T15:03:46.865029+0000","last_became_peered":"2024-09-13T15:03:46.865029+0000","last_unstale":"2024-09-13T15:03:48.898560+0000","last_undegraded":"2024-09-13T15:03:48.898560+0000","last_fullsized":"2024-09-13T15:03:48.898560+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:28.871714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":228,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900556+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T15:03:47.900556+0000","last_peered":"2024-09-13T15:03:47.900556+0000","last_clean":"2024-09-13T15:03:47.900556+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T15:03:47.900556+0000","last_undegraded":"2024-09-13T15:03:47.900556+0000","last_fullsized":"2024-09-13T15:03:47.900556+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":20,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.900138+0000","last_change":"2024-09-13T15:03:46.879528+0000","last_active":"2024-09-13T15:03:48.900138+0000","last_peered":"2024-09-13T15:03:48.900138+0000","last_clean":"2024-09-13T15:03:48.900138+0000","last_became_active":"2024-09-13T15:03:46.860753+0000","last_became_peered":"2024-09-13T15:03:46.860753+0000","last_unstale":"2024-09-13T15:03:48.900138+0000","last_undegraded":"2024-09-13T15:03:48.900138+0000","last_fullsized":"2024-09-13T15:03:48.900138+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:28.561501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":195,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867871+0000","last_change":"2024-09-13T14:57:56.132062+0000","last_active":"2024-09-13T15:03:47.867871+0000","last_peered":"2024-09-13T15:03:47.867871+0000","last_clean":"2024-09-13T15:03:47.867871+0000","last_became_active":"2024-09-13T14:57:19.218181+0000","last_became_peered":"2024-09-13T14:57:19.218181+0000","last_unstale":"2024-09-13T15:03:47.867871+0000","last_undegraded":"2024-09-13T15:03:47.867871+0000","last_fullsized":"2024-09-13T15:03:47.867871+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:56.131976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:56.131976+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:31.359825+0000","scrub_duration":0.0061470450000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"392'11543","reported_seq":11615,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:56.281592+0000","last_change":"2024-09-13T15:00:55.873094+0000","last_active":"2024-09-13T15:03:56.281592+0000","last_peered":"2024-09-13T15:03:56.281592+0000","last_clean":"2024-09-13T15:03:56.281592+0000","last_became_active":"2024-09-13T15:00:55.872743+0000","last_became_peered":"2024-09-13T15:00:55.872743+0000","last_unstale":"2024-09-13T15:03:56.281592+0000","last_undegraded":"2024-09-13T15:03:56.281592+0000","last_fullsized":"2024-09-13T15:03:56.281592+0000","mapping_epoch":362,"log_start":"370'4122","ondisk_log_start":"370'4122","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":7421,"log_dups_size":0,"ondisk_log_size":7421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:46.267062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219348992,"num_objects":3347,"num_object_clones":0,"num_object_copies":10041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11543,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":44,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.890999+0000","last_change":"2024-09-13T15:02:19.423088+0000","last_active":"2024-09-13T15:03:48.890999+0000","last_peered":"2024-09-13T15:03:48.890999+0000","last_clean":"2024-09-13T15:03:48.890999+0000","last_became_active":"2024-09-13T15:02:19.422675+0000","last_became_peered":"2024-09-13T15:02:19.422675+0000","last_unstale":"2024-09-13T15:03:48.890999+0000","last_undegraded":"2024-09-13T15:03:48.890999+0000","last_fullsized":"2024-09-13T15:03:48.890999+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:58.764834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":387,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906853+0000","last_change":"2024-09-13T15:02:19.417107+0000","last_active":"2024-09-13T15:03:47.906853+0000","last_peered":"2024-09-13T15:03:47.906853+0000","last_clean":"2024-09-13T15:03:47.906853+0000","last_became_active":"2024-09-13T15:02:19.416741+0000","last_became_peered":"2024-09-13T15:02:19.416741+0000","last_unstale":"2024-09-13T15:03:47.906853+0000","last_undegraded":"2024-09-13T15:03:47.906853+0000","last_fullsized":"2024-09-13T15:03:47.906853+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:20.599091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":624,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.906810+0000","last_change":"2024-09-13T14:57:53.351358+0000","last_active":"2024-09-13T15:03:48.906810+0000","last_peered":"2024-09-13T15:03:48.906810+0000","last_clean":"2024-09-13T15:03:48.906810+0000","last_became_active":"2024-09-13T14:57:53.351021+0000","last_became_peered":"2024-09-13T14:57:53.351021+0000","last_unstale":"2024-09-13T15:03:48.906810+0000","last_undegraded":"2024-09-13T15:03:48.906810+0000","last_fullsized":"2024-09-13T15:03:48.906810+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.067202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":388,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900371+0000","last_change":"2024-09-13T15:03:46.869628+0000","last_active":"2024-09-13T15:03:47.900371+0000","last_peered":"2024-09-13T15:03:47.900371+0000","last_clean":"2024-09-13T15:03:47.900371+0000","last_became_active":"2024-09-13T15:03:46.869274+0000","last_became_peered":"2024-09-13T15:03:46.869274+0000","last_unstale":"2024-09-13T15:03:47.900371+0000","last_undegraded":"2024-09-13T15:03:47.900371+0000","last_fullsized":"2024-09-13T15:03:47.900371+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:21.999130+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"392'11743","reported_seq":22378,"reported_epoch":392,"state":"active+recovery_wait","last_fresh":"2024-09-13T15:03:57.808038+0000","last_change":"2024-09-13T15:03:46.962262+0000","last_active":"2024-09-13T15:03:57.808038+0000","last_peered":"2024-09-13T15:03:57.808038+0000","last_clean":"2024-09-13T15:03:44.825017+0000","last_became_active":"2024-09-13T15:03:46.901821+0000","last_became_peered":"2024-09-13T15:03:46.901821+0000","last_unstale":"2024-09-13T15:03:57.808038+0000","last_undegraded":"2024-09-13T15:03:57.808038+0000","last_fullsized":"2024-09-13T15:03:57.808038+0000","mapping_epoch":389,"log_start":"373'5539","ondisk_log_start":"373'5539","created":362,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204275712,"num_objects":3117,"num_object_clones":0,"num_object_copies":9351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11743,"num_write_kb":475520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5275,"num_bytes_recovered":345636864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":["3","2","4","5"],"object_location_counts":[{"shards":"2,3,5","objects":3117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":73,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.868819+0000","last_change":"2024-09-13T14:57:53.351087+0000","last_active":"2024-09-13T15:03:47.868819+0000","last_peered":"2024-09-13T15:03:47.868819+0000","last_clean":"2024-09-13T15:03:47.868819+0000","last_became_active":"2024-09-13T14:57:53.350744+0000","last_became_peered":"2024-09-13T14:57:53.350744+0000","last_unstale":"2024-09-13T15:03:47.868819+0000","last_undegraded":"2024-09-13T15:03:47.868819+0000","last_fullsized":"2024-09-13T15:03:47.868819+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:43.300957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.888663+0000","last_change":"2024-09-13T15:02:19.410671+0000","last_active":"2024-09-13T15:03:48.888663+0000","last_peered":"2024-09-13T15:03:48.888663+0000","last_clean":"2024-09-13T15:03:48.888663+0000","last_became_active":"2024-09-13T15:02:19.410330+0000","last_became_peered":"2024-09-13T15:02:19.410330+0000","last_unstale":"2024-09-13T15:03:48.888663+0000","last_undegraded":"2024-09-13T15:03:48.888663+0000","last_fullsized":"2024-09-13T15:03:48.888663+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:30.403982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":93,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.869243+0000","last_change":"2024-09-13T15:03:46.866800+0000","last_active":"2024-09-13T15:03:47.869243+0000","last_peered":"2024-09-13T15:03:47.869243+0000","last_clean":"2024-09-13T15:03:47.869243+0000","last_became_active":"2024-09-13T15:03:46.866557+0000","last_became_peered":"2024-09-13T15:03:46.866557+0000","last_unstale":"2024-09-13T15:03:47.869243+0000","last_undegraded":"2024-09-13T15:03:47.869243+0000","last_fullsized":"2024-09-13T15:03:47.869243+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:28.181919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":388,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900766+0000","last_change":"2024-09-13T15:03:46.851975+0000","last_active":"2024-09-13T15:03:47.900766+0000","last_peered":"2024-09-13T15:03:47.900766+0000","last_clean":"2024-09-13T15:03:47.900766+0000","last_became_active":"2024-09-13T15:03:46.851616+0000","last_became_peered":"2024-09-13T15:03:46.851616+0000","last_unstale":"2024-09-13T15:03:47.900766+0000","last_undegraded":"2024-09-13T15:03:47.900766+0000","last_fullsized":"2024-09-13T15:03:47.900766+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:01.819924+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"392'11876","reported_seq":11913,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:58.571635+0000","last_change":"2024-09-13T15:00:55.874092+0000","last_active":"2024-09-13T15:03:58.571635+0000","last_peered":"2024-09-13T15:03:58.571635+0000","last_clean":"2024-09-13T15:03:58.571635+0000","last_became_active":"2024-09-13T15:00:55.873709+0000","last_became_peered":"2024-09-13T15:00:55.873709+0000","last_unstale":"2024-09-13T15:03:58.571635+0000","last_undegraded":"2024-09-13T15:03:58.571635+0000","last_fullsized":"2024-09-13T15:03:58.571635+0000","mapping_epoch":362,"log_start":"373'6055","ondisk_log_start":"373'6055","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.813547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206897176,"num_objects":3158,"num_object_clones":0,"num_object_copies":9474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11876,"num_write_kb":481025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":536,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.906946+0000","last_change":"2024-09-13T15:03:46.872633+0000","last_active":"2024-09-13T15:03:48.906946+0000","last_peered":"2024-09-13T15:03:48.906946+0000","last_clean":"2024-09-13T15:03:48.906946+0000","last_became_active":"2024-09-13T15:03:46.872123+0000","last_became_peered":"2024-09-13T15:03:46.872123+0000","last_unstale":"2024-09-13T15:03:48.906946+0000","last_undegraded":"2024-09-13T15:03:48.906946+0000","last_fullsized":"2024-09-13T15:03:48.906946+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:47.174643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":375,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867684+0000","last_change":"2024-09-13T15:02:19.406489+0000","last_active":"2024-09-13T15:03:47.867684+0000","last_peered":"2024-09-13T15:03:47.867684+0000","last_clean":"2024-09-13T15:03:47.867684+0000","last_became_active":"2024-09-13T15:02:19.405872+0000","last_became_peered":"2024-09-13T15:02:19.405872+0000","last_unstale":"2024-09-13T15:03:47.867684+0000","last_undegraded":"2024-09-13T15:03:47.867684+0000","last_fullsized":"2024-09-13T15:03:47.867684+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:15.343020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":156,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.904514+0000","last_change":"2024-09-13T14:58:28.428183+0000","last_active":"2024-09-13T15:03:48.904514+0000","last_peered":"2024-09-13T15:03:48.904514+0000","last_clean":"2024-09-13T15:03:48.904514+0000","last_became_active":"2024-09-13T14:58:28.427553+0000","last_became_peered":"2024-09-13T14:58:28.427553+0000","last_unstale":"2024-09-13T15:03:48.904514+0000","last_undegraded":"2024-09-13T15:03:48.904514+0000","last_fullsized":"2024-09-13T15:03:48.904514+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:59.108147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":44,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.889680+0000","last_change":"2024-09-13T15:02:19.420193+0000","last_active":"2024-09-13T15:03:48.889680+0000","last_peered":"2024-09-13T15:03:48.889680+0000","last_clean":"2024-09-13T15:03:48.889680+0000","last_became_active":"2024-09-13T15:02:19.419807+0000","last_became_peered":"2024-09-13T15:02:19.419807+0000","last_unstale":"2024-09-13T15:03:48.889680+0000","last_undegraded":"2024-09-13T15:03:48.889680+0000","last_fullsized":"2024-09-13T15:03:48.889680+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:13.929284+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"392'11696","reported_seq":11734,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:58.497193+0000","last_change":"2024-09-13T15:00:55.408720+0000","last_active":"2024-09-13T15:03:58.497193+0000","last_peered":"2024-09-13T15:03:58.497193+0000","last_clean":"2024-09-13T15:03:58.497193+0000","last_became_active":"2024-09-13T15:00:55.408379+0000","last_became_peered":"2024-09-13T15:00:55.408379+0000","last_unstale":"2024-09-13T15:03:58.497193+0000","last_undegraded":"2024-09-13T15:03:58.497193+0000","last_fullsized":"2024-09-13T15:03:58.497193+0000","mapping_epoch":362,"log_start":"373'5632","ondisk_log_start":"373'5632","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6064,"log_dups_size":0,"ondisk_log_size":6064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:24.296460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207880192,"num_objects":3172,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11696,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":377,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.904618+0000","last_change":"2024-09-13T15:02:19.420231+0000","last_active":"2024-09-13T15:03:48.904618+0000","last_peered":"2024-09-13T15:03:48.904618+0000","last_clean":"2024-09-13T15:03:48.904618+0000","last_became_active":"2024-09-13T15:02:19.419627+0000","last_became_peered":"2024-09-13T15:02:19.419627+0000","last_unstale":"2024-09-13T15:03:48.904618+0000","last_undegraded":"2024-09-13T15:03:48.904618+0000","last_fullsized":"2024-09-13T15:03:48.904618+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:21.403387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":463,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.901513+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T15:03:48.901513+0000","last_peered":"2024-09-13T15:03:48.901513+0000","last_clean":"2024-09-13T15:03:48.901513+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T15:03:48.901513+0000","last_undegraded":"2024-09-13T15:03:48.901513+0000","last_fullsized":"2024-09-13T15:03:48.901513+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":681,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.901567+0000","last_change":"2024-09-13T15:03:46.871298+0000","last_active":"2024-09-13T15:03:48.901567+0000","last_peered":"2024-09-13T15:03:48.901567+0000","last_clean":"2024-09-13T15:03:48.901567+0000","last_became_active":"2024-09-13T15:03:46.870966+0000","last_became_peered":"2024-09-13T15:03:46.870966+0000","last_unstale":"2024-09-13T15:03:48.901567+0000","last_undegraded":"2024-09-13T15:03:48.901567+0000","last_fullsized":"2024-09-13T15:03:48.901567+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:22.463434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":169,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906159+0000","last_change":"2024-09-13T15:02:19.406171+0000","last_active":"2024-09-13T15:03:47.906159+0000","last_peered":"2024-09-13T15:03:47.906159+0000","last_clean":"2024-09-13T15:03:47.906159+0000","last_became_active":"2024-09-13T15:02:19.405713+0000","last_became_peered":"2024-09-13T15:02:19.405713+0000","last_unstale":"2024-09-13T15:03:47.906159+0000","last_undegraded":"2024-09-13T15:03:47.906159+0000","last_fullsized":"2024-09-13T15:03:47.906159+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:13.657811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.f","version":"392'11844","reported_seq":11882,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:58.498027+0000","last_change":"2024-09-13T15:00:55.408205+0000","last_active":"2024-09-13T15:03:58.498027+0000","last_peered":"2024-09-13T15:03:58.498027+0000","last_clean":"2024-09-13T15:03:58.498027+0000","last_became_active":"2024-09-13T15:00:55.407801+0000","last_became_peered":"2024-09-13T15:00:55.407801+0000","last_unstale":"2024-09-13T15:03:58.498027+0000","last_undegraded":"2024-09-13T15:03:58.498027+0000","last_fullsized":"2024-09-13T15:03:58.498027+0000","mapping_epoch":362,"log_start":"373'5843","ondisk_log_start":"373'5843","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6001,"log_dups_size":0,"ondisk_log_size":6001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:36.295309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208535552,"num_objects":3182,"num_object_clones":0,"num_object_copies":9546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":480832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":44,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.888701+0000","last_change":"2024-09-13T15:02:19.419571+0000","last_active":"2024-09-13T15:03:48.888701+0000","last_peered":"2024-09-13T15:03:48.888701+0000","last_clean":"2024-09-13T15:03:48.888701+0000","last_became_active":"2024-09-13T15:02:19.419205+0000","last_became_peered":"2024-09-13T15:02:19.419205+0000","last_unstale":"2024-09-13T15:03:48.888701+0000","last_undegraded":"2024-09-13T15:03:48.888701+0000","last_fullsized":"2024-09-13T15:03:48.888701+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:04.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":58,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.889593+0000","last_change":"2024-09-13T15:03:46.866278+0000","last_active":"2024-09-13T15:03:48.889593+0000","last_peered":"2024-09-13T15:03:48.889593+0000","last_clean":"2024-09-13T15:03:48.889593+0000","last_became_active":"2024-09-13T15:03:46.865838+0000","last_became_peered":"2024-09-13T15:03:46.865838+0000","last_unstale":"2024-09-13T15:03:48.889593+0000","last_undegraded":"2024-09-13T15:03:48.889593+0000","last_fullsized":"2024-09-13T15:03:48.889593+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:02.073646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":424,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.905942+0000","last_change":"2024-09-13T15:02:19.408677+0000","last_active":"2024-09-13T15:03:47.905942+0000","last_peered":"2024-09-13T15:03:47.905942+0000","last_clean":"2024-09-13T15:03:47.905942+0000","last_became_active":"2024-09-13T15:02:19.408262+0000","last_became_peered":"2024-09-13T15:02:19.408262+0000","last_unstale":"2024-09-13T15:03:47.905942+0000","last_undegraded":"2024-09-13T15:03:47.905942+0000","last_fullsized":"2024-09-13T15:03:47.905942+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:47:22.461507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":328,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.899772+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T15:03:47.899772+0000","last_peered":"2024-09-13T15:03:47.899772+0000","last_clean":"2024-09-13T15:03:47.899772+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T15:03:47.899772+0000","last_undegraded":"2024-09-13T15:03:47.899772+0000","last_fullsized":"2024-09-13T15:03:47.899772+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.c","version":"392'11764","reported_seq":11802,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:58.798350+0000","last_change":"2024-09-13T15:00:55.873540+0000","last_active":"2024-09-13T15:03:58.798350+0000","last_peered":"2024-09-13T15:03:58.798350+0000","last_clean":"2024-09-13T15:03:58.798350+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:03:58.798350+0000","last_undegraded":"2024-09-13T15:03:58.798350+0000","last_fullsized":"2024-09-13T15:03:58.798350+0000","mapping_epoch":362,"log_start":"371'4724","ondisk_log_start":"371'4724","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":7040,"log_dups_size":0,"ondisk_log_size":7040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:49.820345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207880192,"num_objects":3172,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":477952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":337,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900157+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T15:03:47.900157+0000","last_peered":"2024-09-13T15:03:47.900157+0000","last_clean":"2024-09-13T15:03:47.900157+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T15:03:47.900157+0000","last_undegraded":"2024-09-13T15:03:47.900157+0000","last_fullsized":"2024-09-13T15:03:47.900157+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.898680+0000","last_change":"2024-09-13T15:03:46.875410+0000","last_active":"2024-09-13T15:03:48.898680+0000","last_peered":"2024-09-13T15:03:48.898680+0000","last_clean":"2024-09-13T15:03:48.898680+0000","last_became_active":"2024-09-13T15:03:46.875103+0000","last_became_peered":"2024-09-13T15:03:46.875103+0000","last_unstale":"2024-09-13T15:03:48.898680+0000","last_undegraded":"2024-09-13T15:03:48.898680+0000","last_fullsized":"2024-09-13T15:03:48.898680+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:53.941854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":77,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.869013+0000","last_change":"2024-09-13T14:57:53.360296+0000","last_active":"2024-09-13T15:03:47.869013+0000","last_peered":"2024-09-13T15:03:47.869013+0000","last_clean":"2024-09-13T15:03:47.869013+0000","last_became_active":"2024-09-13T14:57:53.347710+0000","last_became_peered":"2024-09-13T14:57:53.347710+0000","last_unstale":"2024-09-13T15:03:47.869013+0000","last_undegraded":"2024-09-13T15:03:47.869013+0000","last_fullsized":"2024-09-13T15:03:47.869013+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:43.099884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":169,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906010+0000","last_change":"2024-09-13T15:02:19.407022+0000","last_active":"2024-09-13T15:03:47.906010+0000","last_peered":"2024-09-13T15:03:47.906010+0000","last_clean":"2024-09-13T15:03:47.906010+0000","last_became_active":"2024-09-13T15:02:19.406620+0000","last_became_peered":"2024-09-13T15:02:19.406620+0000","last_unstale":"2024-09-13T15:03:47.906010+0000","last_undegraded":"2024-09-13T15:03:47.906010+0000","last_fullsized":"2024-09-13T15:03:47.906010+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:37.100188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"392'11530","reported_seq":11590,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:58.552077+0000","last_change":"2024-09-13T15:03:46.903789+0000","last_active":"2024-09-13T15:03:58.552077+0000","last_peered":"2024-09-13T15:03:58.552077+0000","last_clean":"2024-09-13T15:03:58.552077+0000","last_became_active":"2024-09-13T15:03:46.868002+0000","last_became_peered":"2024-09-13T15:03:46.868002+0000","last_unstale":"2024-09-13T15:03:58.552077+0000","last_undegraded":"2024-09-13T15:03:58.552077+0000","last_fullsized":"2024-09-13T15:03:58.552077+0000","mapping_epoch":389,"log_start":"373'5764","ondisk_log_start":"373'5764","created":362,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:18.708394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208666624,"num_objects":3184,"num_object_clones":0,"num_object_copies":9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11530,"num_write_kb":470848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":322,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.901003+0000","last_change":"2024-09-13T15:03:46.856441+0000","last_active":"2024-09-13T15:03:48.901003+0000","last_peered":"2024-09-13T15:03:48.901003+0000","last_clean":"2024-09-13T15:03:48.901003+0000","last_became_active":"2024-09-13T15:03:46.856111+0000","last_became_peered":"2024-09-13T15:03:46.856111+0000","last_unstale":"2024-09-13T15:03:48.901003+0000","last_undegraded":"2024-09-13T15:03:48.901003+0000","last_fullsized":"2024-09-13T15:03:48.901003+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:44.991274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.898753+0000","last_change":"2024-09-13T15:03:46.867861+0000","last_active":"2024-09-13T15:03:48.898753+0000","last_peered":"2024-09-13T15:03:48.898753+0000","last_clean":"2024-09-13T15:03:48.898753+0000","last_became_active":"2024-09-13T15:03:46.867422+0000","last_became_peered":"2024-09-13T15:03:46.867422+0000","last_unstale":"2024-09-13T15:03:48.898753+0000","last_undegraded":"2024-09-13T15:03:48.898753+0000","last_fullsized":"2024-09-13T15:03:48.898753+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:19.172678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":149,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.900588+0000","last_change":"2024-09-13T14:58:28.428608+0000","last_active":"2024-09-13T15:03:48.900588+0000","last_peered":"2024-09-13T15:03:48.900588+0000","last_clean":"2024-09-13T15:03:48.900588+0000","last_became_active":"2024-09-13T14:58:28.428282+0000","last_became_peered":"2024-09-13T14:58:28.428282+0000","last_unstale":"2024-09-13T15:03:48.900588+0000","last_undegraded":"2024-09-13T15:03:48.900588+0000","last_fullsized":"2024-09-13T15:03:48.900588+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.603606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.603606+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:10.695417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.7","version":"392'11861","reported_seq":11899,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:58.453763+0000","last_change":"2024-09-13T15:00:55.411034+0000","last_active":"2024-09-13T15:03:58.453763+0000","last_peered":"2024-09-13T15:03:58.453763+0000","last_clean":"2024-09-13T15:03:58.453763+0000","last_became_active":"2024-09-13T15:00:55.410636+0000","last_became_peered":"2024-09-13T15:00:55.410636+0000","last_unstale":"2024-09-13T15:03:58.453763+0000","last_undegraded":"2024-09-13T15:03:58.453763+0000","last_fullsized":"2024-09-13T15:03:58.453763+0000","mapping_epoch":362,"log_start":"373'5858","ondisk_log_start":"373'5858","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6003,"log_dups_size":0,"ondisk_log_size":6003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:42.688532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208863232,"num_objects":3187,"num_object_clones":0,"num_object_copies":9561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":374,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867958+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T15:03:47.867958+0000","last_peered":"2024-09-13T15:03:47.867958+0000","last_clean":"2024-09-13T15:03:47.867958+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T15:03:47.867958+0000","last_undegraded":"2024-09-13T15:03:47.867958+0000","last_fullsized":"2024-09-13T15:03:47.867958+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":355,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867188+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T15:03:47.867188+0000","last_peered":"2024-09-13T15:03:47.867188+0000","last_clean":"2024-09-13T15:03:47.867188+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T15:03:47.867188+0000","last_undegraded":"2024-09-13T15:03:47.867188+0000","last_fullsized":"2024-09-13T15:03:47.867188+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":61,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.890686+0000","last_change":"2024-09-13T15:02:36.128841+0000","last_active":"2024-09-13T15:03:48.890686+0000","last_peered":"2024-09-13T15:03:48.890686+0000","last_clean":"2024-09-13T15:03:48.890686+0000","last_became_active":"2024-09-13T15:02:19.422885+0000","last_became_peered":"2024-09-13T15:02:19.422885+0000","last_unstale":"2024-09-13T15:03:48.890686+0000","last_undegraded":"2024-09-13T15:03:48.890686+0000","last_fullsized":"2024-09-13T15:03:48.890686+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:02:36.128767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:02:36.128767+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:32.092793+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.4","version":"392'11868","reported_seq":11906,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:58.497875+0000","last_change":"2024-09-13T15:00:55.873173+0000","last_active":"2024-09-13T15:03:58.497875+0000","last_peered":"2024-09-13T15:03:58.497875+0000","last_clean":"2024-09-13T15:03:58.497875+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:03:58.497875+0000","last_undegraded":"2024-09-13T15:03:58.497875+0000","last_fullsized":"2024-09-13T15:03:58.497875+0000","mapping_epoch":362,"log_start":"373'5876","ondisk_log_start":"373'5876","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5992,"log_dups_size":0,"ondisk_log_size":5992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:27.254765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":3200,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11868,"num_write_kb":482176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":371,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.868055+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T15:03:47.868055+0000","last_peered":"2024-09-13T15:03:47.868055+0000","last_clean":"2024-09-13T15:03:47.868055+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T15:03:47.868055+0000","last_undegraded":"2024-09-13T15:03:47.868055+0000","last_fullsized":"2024-09-13T15:03:47.868055+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":375,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900101+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T15:03:47.900101+0000","last_peered":"2024-09-13T15:03:47.900101+0000","last_clean":"2024-09-13T15:03:47.900101+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T15:03:47.900101+0000","last_undegraded":"2024-09-13T15:03:47.900101+0000","last_fullsized":"2024-09-13T15:03:47.900101+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":218,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.901427+0000","last_change":"2024-09-13T15:03:46.869405+0000","last_active":"2024-09-13T15:03:48.901427+0000","last_peered":"2024-09-13T15:03:48.901427+0000","last_clean":"2024-09-13T15:03:48.901427+0000","last_became_active":"2024-09-13T15:03:46.868913+0000","last_became_peered":"2024-09-13T15:03:46.868913+0000","last_unstale":"2024-09-13T15:03:48.901427+0000","last_undegraded":"2024-09-13T15:03:48.901427+0000","last_fullsized":"2024-09-13T15:03:48.901427+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:02.098981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"392'11678","reported_seq":11798,"reported_epoch":392,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:03:58.561912+0000","last_change":"2024-09-13T15:03:46.903406+0000","last_active":"2024-09-13T15:03:58.561912+0000","last_peered":"2024-09-13T15:03:58.561912+0000","last_clean":"2024-09-13T15:02:18.352546+0000","last_became_active":"2024-09-13T15:02:20.474569+0000","last_became_peered":"2024-09-13T15:02:20.474569+0000","last_unstale":"2024-09-13T15:03:58.561912+0000","last_undegraded":"2024-09-13T15:03:58.561912+0000","last_fullsized":"2024-09-13T15:03:58.561912+0000","mapping_epoch":376,"log_start":"373'5832","ondisk_log_start":"373'5832","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:04.599223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204865536,"num_objects":3126,"num_object_clones":0,"num_object_copies":9378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3111,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":473728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,5,3],"avail_no_missing":["4","3","5"],"object_location_counts":[{"shards":"3,4,5","objects":3126}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":374,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867314+0000","last_change":"2024-09-13T15:03:46.866312+0000","last_active":"2024-09-13T15:03:47.867314+0000","last_peered":"2024-09-13T15:03:47.867314+0000","last_clean":"2024-09-13T15:03:47.867314+0000","last_became_active":"2024-09-13T15:03:46.866038+0000","last_became_peered":"2024-09-13T15:03:46.866038+0000","last_unstale":"2024-09-13T15:03:47.867314+0000","last_undegraded":"2024-09-13T15:03:47.867314+0000","last_fullsized":"2024-09-13T15:03:47.867314+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:05.770254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":20,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.902949+0000","last_change":"2024-09-13T15:03:46.881258+0000","last_active":"2024-09-13T15:03:48.902949+0000","last_peered":"2024-09-13T15:03:48.902949+0000","last_clean":"2024-09-13T15:03:48.902949+0000","last_became_active":"2024-09-13T15:03:46.866883+0000","last_became_peered":"2024-09-13T15:03:46.866883+0000","last_unstale":"2024-09-13T15:03:48.902949+0000","last_undegraded":"2024-09-13T15:03:48.902949+0000","last_fullsized":"2024-09-13T15:03:48.902949+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:12.364139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":215,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.905820+0000","last_change":"2024-09-13T15:03:46.863264+0000","last_active":"2024-09-13T15:03:47.905820+0000","last_peered":"2024-09-13T15:03:47.905820+0000","last_clean":"2024-09-13T15:03:47.905820+0000","last_became_active":"2024-09-13T15:03:46.862850+0000","last_became_peered":"2024-09-13T15:03:46.862850+0000","last_unstale":"2024-09-13T15:03:47.905820+0000","last_undegraded":"2024-09-13T15:03:47.905820+0000","last_fullsized":"2024-09-13T15:03:47.905820+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:57.707268+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2","version":"392'11618","reported_seq":11700,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:57.815271+0000","last_change":"2024-09-13T15:03:47.891333+0000","last_active":"2024-09-13T15:03:57.815271+0000","last_peered":"2024-09-13T15:03:57.815271+0000","last_clean":"2024-09-13T15:03:57.815271+0000","last_became_active":"2024-09-13T15:03:47.870054+0000","last_became_peered":"2024-09-13T15:03:47.870054+0000","last_unstale":"2024-09-13T15:03:57.815271+0000","last_undegraded":"2024-09-13T15:03:57.815271+0000","last_fullsized":"2024-09-13T15:03:57.815271+0000","mapping_epoch":390,"log_start":"373'5826","ondisk_log_start":"373'5826","created":362,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:12.911319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213254144,"num_objects":3254,"num_object_clones":0,"num_object_copies":9762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11618,"num_write_kb":475904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":44,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.890789+0000","last_change":"2024-09-13T15:02:19.417296+0000","last_active":"2024-09-13T15:03:48.890789+0000","last_peered":"2024-09-13T15:03:48.890789+0000","last_clean":"2024-09-13T15:03:48.890789+0000","last_became_active":"2024-09-13T15:02:19.416930+0000","last_became_peered":"2024-09-13T15:02:19.416930+0000","last_unstale":"2024-09-13T15:03:48.890789+0000","last_undegraded":"2024-09-13T15:03:48.890789+0000","last_fullsized":"2024-09-13T15:03:48.890789+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:18.684193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899427+0000","last_change":"2024-09-13T15:03:46.869406+0000","last_active":"2024-09-13T15:03:48.899427+0000","last_peered":"2024-09-13T15:03:48.899427+0000","last_clean":"2024-09-13T15:03:48.899427+0000","last_became_active":"2024-09-13T15:03:46.866230+0000","last_became_peered":"2024-09-13T15:03:46.866230+0000","last_unstale":"2024-09-13T15:03:48.899427+0000","last_undegraded":"2024-09-13T15:03:48.899427+0000","last_fullsized":"2024-09-13T15:03:48.899427+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:40.914289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":215,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867793+0000","last_change":"2024-09-13T15:03:46.855303+0000","last_active":"2024-09-13T15:03:47.867793+0000","last_peered":"2024-09-13T15:03:47.867793+0000","last_clean":"2024-09-13T15:03:47.867793+0000","last_became_active":"2024-09-13T15:03:46.854880+0000","last_became_peered":"2024-09-13T15:03:46.854880+0000","last_unstale":"2024-09-13T15:03:47.867793+0000","last_undegraded":"2024-09-13T15:03:47.867793+0000","last_fullsized":"2024-09-13T15:03:47.867793+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:00.146384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:00.146384+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:30.662335+0000","scrub_duration":0.0014649660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"392'11508","reported_seq":11581,"reported_epoch":392,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:03:57.804573+0000","last_change":"2024-09-13T15:03:46.922564+0000","last_active":"2024-09-13T15:03:57.804573+0000","last_peered":"2024-09-13T15:03:57.804573+0000","last_clean":"2024-09-13T15:02:18.361205+0000","last_became_active":"2024-09-13T15:03:46.889747+0000","last_became_peered":"2024-09-13T15:03:46.889747+0000","last_unstale":"2024-09-13T15:03:57.804573+0000","last_undegraded":"2024-09-13T15:03:57.804573+0000","last_fullsized":"2024-09-13T15:03:57.804573+0000","mapping_epoch":389,"log_start":"373'5347","ondisk_log_start":"373'5347","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213778432,"num_objects":3262,"num_object_clones":0,"num_object_copies":9786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3262,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11508,"num_write_kb":472640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[3,4,5],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":3262}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":377,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.899942+0000","last_change":"2024-09-13T14:57:35.357982+0000","last_active":"2024-09-13T15:03:47.899942+0000","last_peered":"2024-09-13T15:03:47.899942+0000","last_clean":"2024-09-13T15:03:47.899942+0000","last_became_active":"2024-09-13T14:57:35.357618+0000","last_became_peered":"2024-09-13T14:57:35.357618+0000","last_unstale":"2024-09-13T15:03:47.899942+0000","last_undegraded":"2024-09-13T15:03:47.899942+0000","last_fullsized":"2024-09-13T15:03:47.899942+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:50.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":396,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906774+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:03:47.906774+0000","last_peered":"2024-09-13T15:03:47.906774+0000","last_clean":"2024-09-13T15:03:47.906774+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:03:47.906774+0000","last_undegraded":"2024-09-13T15:03:47.906774+0000","last_fullsized":"2024-09-13T15:03:47.906774+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":348,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900844+0000","last_change":"2024-09-13T14:57:53.363216+0000","last_active":"2024-09-13T15:03:47.900844+0000","last_peered":"2024-09-13T15:03:47.900844+0000","last_clean":"2024-09-13T15:03:47.900844+0000","last_became_active":"2024-09-13T14:57:53.362679+0000","last_became_peered":"2024-09-13T14:57:53.362679+0000","last_unstale":"2024-09-13T15:03:47.900844+0000","last_undegraded":"2024-09-13T15:03:47.900844+0000","last_fullsized":"2024-09-13T15:03:47.900844+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.885757+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"392'11739","reported_seq":11776,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:58.482673+0000","last_change":"2024-09-13T15:00:55.873177+0000","last_active":"2024-09-13T15:03:58.482673+0000","last_peered":"2024-09-13T15:03:58.482673+0000","last_clean":"2024-09-13T15:03:58.482673+0000","last_became_active":"2024-09-13T15:00:55.872769+0000","last_became_peered":"2024-09-13T15:00:55.872769+0000","last_unstale":"2024-09-13T15:03:58.482673+0000","last_undegraded":"2024-09-13T15:03:58.482673+0000","last_fullsized":"2024-09-13T15:03:58.482673+0000","mapping_epoch":362,"log_start":"373'5699","ondisk_log_start":"373'5699","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:22.869519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210436096,"num_objects":3211,"num_object_clones":0,"num_object_copies":9633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11739,"num_write_kb":478400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":401,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.867100+0000","last_change":"2024-09-13T15:03:46.866745+0000","last_active":"2024-09-13T15:03:47.867100+0000","last_peered":"2024-09-13T15:03:47.867100+0000","last_clean":"2024-09-13T15:03:47.867100+0000","last_became_active":"2024-09-13T15:03:46.866320+0000","last_became_peered":"2024-09-13T15:03:46.866320+0000","last_unstale":"2024-09-13T15:03:47.867100+0000","last_undegraded":"2024-09-13T15:03:47.867100+0000","last_fullsized":"2024-09-13T15:03:47.867100+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:36.155505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":358,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906667+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T15:03:47.906667+0000","last_peered":"2024-09-13T15:03:47.906667+0000","last_clean":"2024-09-13T15:03:47.906667+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T15:03:47.906667+0000","last_undegraded":"2024-09-13T15:03:47.906667+0000","last_fullsized":"2024-09-13T15:03:47.906667+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":20,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899626+0000","last_change":"2024-09-13T15:03:46.882026+0000","last_active":"2024-09-13T15:03:48.899626+0000","last_peered":"2024-09-13T15:03:48.899626+0000","last_clean":"2024-09-13T15:03:48.899626+0000","last_became_active":"2024-09-13T15:03:46.859557+0000","last_became_peered":"2024-09-13T15:03:46.859557+0000","last_unstale":"2024-09-13T15:03:48.899626+0000","last_undegraded":"2024-09-13T15:03:48.899626+0000","last_fullsized":"2024-09-13T15:03:48.899626+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:00.933755+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"392'11657","reported_seq":11695,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:58.487310+0000","last_change":"2024-09-13T15:00:55.405838+0000","last_active":"2024-09-13T15:03:58.487310+0000","last_peered":"2024-09-13T15:03:58.487310+0000","last_clean":"2024-09-13T15:03:58.487310+0000","last_became_active":"2024-09-13T15:00:55.405095+0000","last_became_peered":"2024-09-13T15:00:55.405095+0000","last_unstale":"2024-09-13T15:03:58.487310+0000","last_undegraded":"2024-09-13T15:03:58.487310+0000","last_fullsized":"2024-09-13T15:03:58.487310+0000","mapping_epoch":362,"log_start":"373'5729","ondisk_log_start":"373'5729","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:55.315749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207028224,"num_objects":3159,"num_object_clones":0,"num_object_copies":9477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11657,"num_write_kb":474112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":73,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.868488+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:03:47.868488+0000","last_peered":"2024-09-13T15:03:47.868488+0000","last_clean":"2024-09-13T15:03:47.868488+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:03:47.868488+0000","last_undegraded":"2024-09-13T15:03:47.868488+0000","last_fullsized":"2024-09-13T15:03:47.868488+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":181,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.907019+0000","last_change":"2024-09-13T15:02:19.430869+0000","last_active":"2024-09-13T15:03:47.907019+0000","last_peered":"2024-09-13T15:03:47.907019+0000","last_clean":"2024-09-13T15:03:47.907019+0000","last_became_active":"2024-09-13T15:02:19.417063+0000","last_became_peered":"2024-09-13T15:02:19.417063+0000","last_unstale":"2024-09-13T15:03:47.907019+0000","last_undegraded":"2024-09-13T15:03:47.907019+0000","last_fullsized":"2024-09-13T15:03:47.907019+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:25.391998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":179,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906622+0000","last_change":"2024-09-13T14:57:55.208230+0000","last_active":"2024-09-13T15:03:47.906622+0000","last_peered":"2024-09-13T15:03:47.906622+0000","last_clean":"2024-09-13T15:03:47.906622+0000","last_became_active":"2024-09-13T14:57:19.211671+0000","last_became_peered":"2024-09-13T14:57:19.211671+0000","last_unstale":"2024-09-13T15:03:47.906622+0000","last_undegraded":"2024-09-13T15:03:47.906622+0000","last_fullsized":"2024-09-13T15:03:47.906622+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.066938+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":483,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.900067+0000","last_change":"2024-09-13T15:02:19.426511+0000","last_active":"2024-09-13T15:03:47.900067+0000","last_peered":"2024-09-13T15:03:47.900067+0000","last_clean":"2024-09-13T15:03:47.900067+0000","last_became_active":"2024-09-13T15:02:19.391448+0000","last_became_peered":"2024-09-13T15:02:19.391448+0000","last_unstale":"2024-09-13T15:03:47.900067+0000","last_undegraded":"2024-09-13T15:03:47.900067+0000","last_fullsized":"2024-09-13T15:03:47.900067+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:40.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.900285+0000","last_change":"2024-09-13T15:03:46.870536+0000","last_active":"2024-09-13T15:03:48.900285+0000","last_peered":"2024-09-13T15:03:48.900285+0000","last_clean":"2024-09-13T15:03:48.900285+0000","last_became_active":"2024-09-13T15:03:46.869736+0000","last_became_peered":"2024-09-13T15:03:46.869736+0000","last_unstale":"2024-09-13T15:03:48.900285+0000","last_undegraded":"2024-09-13T15:03:48.900285+0000","last_fullsized":"2024-09-13T15:03:48.900285+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:45.432825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":73,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.868757+0000","last_change":"2024-09-13T14:57:53.356010+0000","last_active":"2024-09-13T15:03:47.868757+0000","last_peered":"2024-09-13T15:03:47.868757+0000","last_clean":"2024-09-13T15:03:47.868757+0000","last_became_active":"2024-09-13T14:57:53.354979+0000","last_became_peered":"2024-09-13T14:57:53.354979+0000","last_unstale":"2024-09-13T15:03:47.868757+0000","last_undegraded":"2024-09-13T15:03:47.868757+0000","last_fullsized":"2024-09-13T15:03:47.868757+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:38.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":359,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.905971+0000","last_change":"2024-09-13T14:57:35.358990+0000","last_active":"2024-09-13T15:03:47.905971+0000","last_peered":"2024-09-13T15:03:47.905971+0000","last_clean":"2024-09-13T15:03:47.905971+0000","last_became_active":"2024-09-13T14:57:35.358547+0000","last_became_peered":"2024-09-13T14:57:35.358547+0000","last_unstale":"2024-09-13T15:03:47.905971+0000","last_undegraded":"2024-09-13T15:03:47.905971+0000","last_fullsized":"2024-09-13T15:03:47.905971+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:45.386301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":58,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.888763+0000","last_change":"2024-09-13T15:03:46.864291+0000","last_active":"2024-09-13T15:03:48.888763+0000","last_peered":"2024-09-13T15:03:48.888763+0000","last_clean":"2024-09-13T15:03:48.888763+0000","last_became_active":"2024-09-13T15:03:46.863671+0000","last_became_peered":"2024-09-13T15:03:46.863671+0000","last_unstale":"2024-09-13T15:03:48.888763+0000","last_undegraded":"2024-09-13T15:03:48.888763+0000","last_fullsized":"2024-09-13T15:03:48.888763+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:28.672041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.902843+0000","last_change":"2024-09-13T15:03:46.867845+0000","last_active":"2024-09-13T15:03:48.902843+0000","last_peered":"2024-09-13T15:03:48.902843+0000","last_clean":"2024-09-13T15:03:48.902843+0000","last_became_active":"2024-09-13T15:03:46.867336+0000","last_became_peered":"2024-09-13T15:03:46.867336+0000","last_unstale":"2024-09-13T15:03:48.902843+0000","last_undegraded":"2024-09-13T15:03:48.902843+0000","last_fullsized":"2024-09-13T15:03:48.902843+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:46.862346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":58,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.888432+0000","last_change":"2024-09-13T15:03:46.863848+0000","last_active":"2024-09-13T15:03:48.888432+0000","last_peered":"2024-09-13T15:03:48.888432+0000","last_clean":"2024-09-13T15:03:48.888432+0000","last_became_active":"2024-09-13T15:03:46.863523+0000","last_became_peered":"2024-09-13T15:03:46.863523+0000","last_unstale":"2024-09-13T15:03:48.888432+0000","last_undegraded":"2024-09-13T15:03:48.888432+0000","last_fullsized":"2024-09-13T15:03:48.888432+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:39.303427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":270,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906681+0000","last_change":"2024-09-13T15:03:46.860138+0000","last_active":"2024-09-13T15:03:47.906681+0000","last_peered":"2024-09-13T15:03:47.906681+0000","last_clean":"2024-09-13T15:03:47.906681+0000","last_became_active":"2024-09-13T15:03:46.859819+0000","last_became_peered":"2024-09-13T15:03:46.859819+0000","last_unstale":"2024-09-13T15:03:47.906681+0000","last_undegraded":"2024-09-13T15:03:47.906681+0000","last_fullsized":"2024-09-13T15:03:47.906681+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:28.081805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":81,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.905755+0000","last_change":"2024-09-13T14:58:28.428896+0000","last_active":"2024-09-13T15:03:47.905755+0000","last_peered":"2024-09-13T15:03:47.905755+0000","last_clean":"2024-09-13T15:03:47.905755+0000","last_became_active":"2024-09-13T14:58:28.428522+0000","last_became_peered":"2024-09-13T14:58:28.428522+0000","last_unstale":"2024-09-13T15:03:47.905755+0000","last_undegraded":"2024-09-13T15:03:47.905755+0000","last_fullsized":"2024-09-13T15:03:47.905755+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:05.418586+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":374,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-13T15:03:47.906189+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T15:03:47.906189+0000","last_peered":"2024-09-13T15:03:47.906189+0000","last_clean":"2024-09-13T15:03:47.906189+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T15:03:47.906189+0000","last_undegraded":"2024-09-13T15:03:47.906189+0000","last_fullsized":"2024-09-13T15:03:47.906189+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.899727+0000","last_change":"2024-09-13T15:03:46.871926+0000","last_active":"2024-09-13T15:03:48.899727+0000","last_peered":"2024-09-13T15:03:48.899727+0000","last_clean":"2024-09-13T15:03:48.899727+0000","last_became_active":"2024-09-13T15:03:46.868103+0000","last_became_peered":"2024-09-13T15:03:46.868103+0000","last_unstale":"2024-09-13T15:03:48.899727+0000","last_undegraded":"2024-09-13T15:03:48.899727+0000","last_fullsized":"2024-09-13T15:03:48.899727+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:43.957131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":58,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-13T15:03:48.889794+0000","last_change":"2024-09-13T15:03:46.864604+0000","last_active":"2024-09-13T15:03:48.889794+0000","last_peered":"2024-09-13T15:03:48.889794+0000","last_clean":"2024-09-13T15:03:48.889794+0000","last_became_active":"2024-09-13T15:03:46.864085+0000","last_became_peered":"2024-09-13T15:03:46.864085+0000","last_unstale":"2024-09-13T15:03:48.889794+0000","last_undegraded":"2024-09-13T15:03:48.889794+0000","last_fullsized":"2024-09-13T15:03:48.889794+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:32.524052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":3355574296,"num_objects":51203,"num_object_clones":0,"num_object_copies":153609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2366,"num_objects_misplaced":8545,"num_objects_unfound":0,"num_objects_dirty":51203,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187217,"num_write_kb":7629377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7268,"num_bytes_recovered":476119040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10627133440,"data_stored":10627121224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100511,"ondisk_log_size":100511,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":1220,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365331,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2558804,"kb_used_data":1041164,"kb_used_omap":1,"kb_used_meta":1517630,"kb_avail":91186348,"statfs":{"total":95995035648,"available":93374820352,"internally_reserved":0,"allocated":1066151936,"data_stored":1063785906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1554053464},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561561,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2930500,"kb_used_data":1438012,"kb_used_omap":1,"kb_used_meta":1492222,"kb_avail":90814652,"statfs":{"total":95995035648,"available":92994203648,"internally_reserved":0,"allocated":1472524288,"data_stored":1470173897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1528035672},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790493,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3333884,"kb_used_data":1847348,"kb_used_omap":1,"kb_used_meta":1486398,"kb_avail":90411268,"statfs":{"total":95995035648,"available":92581138432,"internally_reserved":0,"allocated":1891684352,"data_stored":1889338271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1522071896},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986721,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3606796,"kb_used_data":2104516,"kb_used_omap":1,"kb_used_meta":1502270,"kb_avail":90138356,"statfs":{"total":95995035648,"available":92301676544,"internally_reserved":0,"allocated":2155024384,"data_stored":2152666163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1538324824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519876,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2044152,"kb_used_data":1036528,"kb_used_omap":4,"kb_used_meta":1007611,"kb_avail":91701000,"statfs":{"total":95995035648,"available":93901824000,"internally_reserved":0,"allocated":1061404672,"data_stored":1059062153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":1031794112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304791,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2942336,"kb_used_data":1917176,"kb_used_omap":6,"kb_used_meta":1025145,"kb_avail":90802816,"statfs":{"total":95995035648,"available":92982083584,"internally_reserved":0,"allocated":1963188224,"data_stored":1960838571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":1049748935},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828095,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2292800,"kb_used_data":1052600,"kb_used_omap":5,"kb_used_meta":1240186,"kb_avail":91452352,"statfs":{"total":95995035648,"available":93647208448,"internally_reserved":0,"allocated":1077862400,"data_stored":1075511684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1269950700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285960,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1146744,"kb_used_data":101104,"kb_used_omap":5,"kb_used_meta":1045626,"kb_avail":92598408,"statfs":{"total":95995035648,"available":94820769792,"internally_reserved":0,"allocated":103530496,"data_stored":101187251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":1070721083},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67371008,"data_stored":67371008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1066078208,"data_stored":1066074136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1951793152,"data_stored":1951793152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1049952256,"data_stored":1049952256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2123632640,"data_stored":2123628568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1858994176,"data_stored":1858994176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1466433536,"data_stored":1466433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1042878464,"data_stored":1042874392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:04:03.545 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T15:04:03.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_8 pg_num 26 2024-09-13T15:04:03.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:03.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:03.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:03.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:03.771 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:03.873 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:03.914 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:04.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:04.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:04.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:04.184 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:04.286 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:04.322 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:04.410 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:04:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:04:04] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T15:04:04.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:04.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:04.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:04.621 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:04:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:04.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:04.629 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:04:04.629 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:04.664 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:04:04.665 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:04.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:04:04.702 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:04:04.702 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:04.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:04.733 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:04:04.734 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:04.767 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:04:04.768 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:04.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:04.801 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:04:04.801 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:04.840 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:04:04.840 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:04.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:04.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:05.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:05.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:05.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:05.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:05.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:05.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:05.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:05.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:05.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:05.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:05.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:05.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:05.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:05 smithi003 ceph-mon[104007]: pgmap v1416: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 2534/139842 objects degraded (1.812%); 7697/139842 objects misplaced (5.504%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:05.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3681096312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:05.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:05 smithi003 ceph-mon[107278]: pgmap v1416: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 2534/139842 objects degraded (1.812%); 7697/139842 objects misplaced (5.504%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:05.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3681096312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:05 smithi019 ceph-mon[91296]: pgmap v1416: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 2534/139842 objects degraded (1.812%); 7697/139842 objects misplaced (5.504%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3681096312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:05.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:05.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:05.837 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:05.871 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:05.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:05.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:06.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:06.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:06.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:06.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:06.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:06.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2534/139842 objects degraded (1.812%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:06.468 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2534/139842 objects degraded (1.812%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:06.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:06.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:06.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2534/139842 objects degraded (1.812%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:06.709 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:06.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:06.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:06.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:06.986 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:07.088 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:07.129 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:07.231 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.272 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:07.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:07.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:07.412 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:07.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:07.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:07.646 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:07 smithi019 ceph-mon[91296]: pgmap v1417: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 2534/139842 objects degraded (1.812%); 7697/139842 objects misplaced (5.504%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:07.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:07 smithi003 ceph-mon[104007]: pgmap v1417: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 2534/139842 objects degraded (1.812%); 7697/139842 objects misplaced (5.504%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:07.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:07 smithi003 ceph-mon[107278]: pgmap v1417: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 2534/139842 objects degraded (1.812%); 7697/139842 objects misplaced (5.504%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.694 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:07.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:07.795 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:07.835 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:07.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:07.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:07.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:08.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:08.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:08.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:08.174 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:04:08.174 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:04:08.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.174 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:196591 13.8 13:1a3a2872:::benchmark_data_smithi003_293363_object77379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:08.160585+0000", 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0095963579999999993, 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096270069999999999, 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196591 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.160585+0000", 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.160585+0000", 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.160587+0000", 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6079999999999998e-06 2024-09-13T15:04:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.160598+0000", 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0531e-05 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.160617+0000", 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9602999999999998e-05 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.160629+0000", 2024-09-13T15:04:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1438e-05 2024-09-13T15:04:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.161747+0000", 2024-09-13T15:04:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001118085 2024-09-13T15:04:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.161930+0000", 2024-09-13T15:04:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018273700000000001 2024-09-13T15:04:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:04:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.162062+0000", 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013246300000000001 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167537+0000", 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054753709999999997 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:196593 13.b e393/389 13:d26aa44a:::benchmark_data_smithi003_293363_object77381:head v 393'12353, mlcod=393'12351)", 2024-09-13T15:04:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:08.166883+0000", 2024-09-13T15:04:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0032984199999999998, 2024-09-13T15:04:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034068420000000002, 2024-09-13T15:04:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:04:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.166883+0000", 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.166883+0000", 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.166884+0000", 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3009999999999999e-06 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.166893+0000", 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1109999999999997e-06 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.166915+0000", 2024-09-13T15:04:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2259000000000002e-05 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.166927+0000", 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1805e-05 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167029+0000", 2024-09-13T15:04:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102168 2024-09-13T15:04:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167058+0000", 2024-09-13T15:04:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9198999999999999e-05 2024-09-13T15:04:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:196597 13.9 e393/376 13:962d3779:::benchmark_data_smithi003_293363_object77385:head v 393'12397, mlcod=366'164)", 2024-09-13T15:04:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:08.167732+0000", 2024-09-13T15:04:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0024492810000000002, 2024-09-13T15:04:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026087829999999999, 2024-09-13T15:04:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:08.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:08.187 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:04:08.187 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:08.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:08.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167732+0000", 2024-09-13T15:04:08.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:08.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167732+0000", 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167732+0000", 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6799999999999999e-07 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167740+0000", 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3490000000000003e-06 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167753+0000", 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3709e-05 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167760+0000", 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7009999999999999e-06 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167865+0000", 2024-09-13T15:04:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104421 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.167888+0000", 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3031999999999999e-05 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36909.0:196591 13.8 e393/362)", 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:08.169635+0000", 2024-09-13T15:04:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00054606200000000004, 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00075023899999999998, 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169635+0000", 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169635+0000", 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169636+0000", 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6300000000000005e-07 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169641+0000", 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0359999999999997e-06 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169650+0000", 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8819999999999996e-06 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169657+0000", 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3779999999999998e-06 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36909.0:196591 13.8 e393/362)", 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:08.169986+0000", 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00019469800000000001, 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00043577500000000001, 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169986+0000", 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169986+0000", 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169987+0000", 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2799999999999996e-07 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.169989+0000", 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.565e-06 2024-09-13T15:04:08.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:08.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.170003+0000", 2024-09-13T15:04:08.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3454e-05 2024-09-13T15:04:08.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:08.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:08.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:08.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:08.170009+0000", 2024-09-13T15:04:08.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7799999999999997e-06 2024-09-13T15:04:08.198 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.198 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:08.198 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.198 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:08.198 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-13T15:04:08.199 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 5 2024-09-13T15:04:08.199 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:04:08.207 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:08.236 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:08.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:08.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:08.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:08.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:08.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:08.790 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:08.892 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:09.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:09.042 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:09.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:09.228 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:09.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:09.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:09.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:09.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:09.495 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:09.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:09 smithi003 ceph-mon[104007]: pgmap v1418: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 2534/135375 objects degraded (1.872%); 7403/135375 objects misplaced (5.469%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:09.500 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:09 smithi003 ceph-mon[107278]: pgmap v1418: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 2534/135375 objects degraded (1.872%); 7403/135375 objects misplaced (5.469%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:09.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:04:09.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:09.637 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:09 smithi019 ceph-mon[91296]: pgmap v1418: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 2534/135375 objects degraded (1.872%); 7403/135375 objects misplaced (5.469%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:09.739 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.786 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:09.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:09.792 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:09.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:09.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:10.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.079 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:10.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:10.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:10.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:10.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:10.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:10.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:10.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:04:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:04:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:10.589 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:10.629 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:10.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:10.778 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:10.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:10.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:04:10.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:10.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:11.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:11.074 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:11.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:11.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:11.311 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:11.358 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:11 smithi019 ceph-mon[91296]: pgmap v1419: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 2730/125592 objects degraded (2.174%); 6815/125592 objects misplaced (5.426%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T15:04:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2730/125592 objects degraded (2.174%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:11 smithi003 ceph-mon[107278]: pgmap v1419: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 2730/125592 objects degraded (2.174%); 6815/125592 objects misplaced (5.426%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T15:04:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2730/125592 objects degraded (2.174%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:11 smithi003 ceph-mon[104007]: pgmap v1419: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 2730/125592 objects degraded (2.174%); 6815/125592 objects misplaced (5.426%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T15:04:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2730/125592 objects degraded (2.174%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:11.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:11.589 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:11.636 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:11.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:11.777 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:11.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:11.925 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:11.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:12.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:12.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:12.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:12.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:12.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:12.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:12.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:12.593 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:12.630 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:12.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:12.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:12.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:12.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:13.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:13.051 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:13.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:13.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:13.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:13.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:13.425 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:13.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:13.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:13.520 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:13 smithi003 ceph-mon[104007]: pgmap v1420: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 2730/123840 objects degraded (2.204%); 6815/123840 objects misplaced (5.503%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:13.520 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:13.520 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:13 smithi003 ceph-mon[107278]: pgmap v1420: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 2730/123840 objects degraded (2.204%); 6815/123840 objects misplaced (5.503%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:13.520 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:13.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:13.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:13 smithi019 ceph-mon[91296]: pgmap v1420: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 2730/123840 objects degraded (2.204%); 6815/123840 objects misplaced (5.503%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:13.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:13.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:13.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.923 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:13.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:14.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:14.164 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:14.209 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:14.229 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:04:14.240 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 13 pg_num to 26 2024-09-13T15:04:14.311 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:14.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:04:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:04:14] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-09-13T15:04:14.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3281706975' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T15:04:14.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:14 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T15:04:14.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1192726523' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:14.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3281706975' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T15:04:14.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:14 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T15:04:14.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1192726523' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:14.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.500 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:14.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:14.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:14.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3281706975' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T15:04:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:14 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T15:04:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1192726523' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:14.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:14.780 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:14.881 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:14.925 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:15.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:15.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:15.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:15.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:15.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:15.334 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:15.437 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.474 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:15.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:15.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:15.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:15 smithi019 ceph-mon[91296]: pgmap v1421: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 2906/112281 objects degraded (2.588%); 5992/112281 objects misplaced (5.337%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T15:04:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:15 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:15 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "26"}]': finished 2024-09-13T15:04:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1192726523' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:04:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:15 smithi019 ceph-mon[91296]: osdmap e394: 8 total, 8 up, 8 in 2024-09-13T15:04:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[107278]: pgmap v1421: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 2906/112281 objects degraded (2.588%); 5992/112281 objects misplaced (5.337%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T15:04:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "26"}]': finished 2024-09-13T15:04:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1192726523' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:04:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[107278]: osdmap e394: 8 total, 8 up, 8 in 2024-09-13T15:04:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[104007]: pgmap v1421: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 2906/112281 objects degraded (2.588%); 5992/112281 objects misplaced (5.337%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T15:04:15.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:15.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "26"}]': finished 2024-09-13T15:04:15.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1192726523' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:04:15.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:15 smithi003 ceph-mon[104007]: osdmap e394: 8 total, 8 up, 8 in 2024-09-13T15:04:15.711 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:15.756 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:15.857 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:15.892 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:15.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:16.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:16.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:16.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:16.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:16.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:16.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:16.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:16.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:16.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:16.624 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2906/112281 objects degraded (2.588%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:16 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T15:04:16.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2906/112281 objects degraded (2.588%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:16.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:16 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T15:04:16.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2906/112281 objects degraded (2.588%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:04:16.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:16 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T15:04:16.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:16.748 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:16.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.873 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:16.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:16.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:16.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:17.099 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:17.122 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:17.224 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:17.247 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:17.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:17.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:17.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:17 smithi003 ceph-mon[104007]: pgmap v1423: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 2906/112281 objects degraded (2.588%); 5992/112281 objects misplaced (5.337%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T15:04:17.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:17 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T15:04:17.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:17 smithi003 ceph-mon[104007]: osdmap e395: 8 total, 8 up, 8 in 2024-09-13T15:04:17.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:17 smithi003 ceph-mon[107278]: pgmap v1423: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 2906/112281 objects degraded (2.588%); 5992/112281 objects misplaced (5.337%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T15:04:17.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:17 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T15:04:17.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:17 smithi003 ceph-mon[107278]: osdmap e395: 8 total, 8 up, 8 in 2024-09-13T15:04:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:17.486 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:17.513 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:17.615 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:17.639 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:17 smithi019 ceph-mon[91296]: pgmap v1423: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 2.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 2906/112281 objects degraded (2.588%); 5992/112281 objects misplaced (5.337%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T15:04:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:17 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T15:04:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:17 smithi019 ceph-mon[91296]: osdmap e395: 8 total, 8 up, 8 in 2024-09-13T15:04:17.741 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:17.765 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:17.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:17.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:17.993 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:18.016 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:18.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:18.140 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:18.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:18.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:18.282 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:04:18.283 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:04:18.283 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:04:18.283 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:18.283 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:18.283 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:18.283 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:04:18.283 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:04:18.284 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:18.284 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:18.294 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:04:18.295 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:18.310 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:04:18.311 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:18.327 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:04:18.327 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[107278]: 13.3 continuing backfill to osd.0 from (374'6347,394'12557] MIN to 394'12557 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[107278]: osdmap e396: 8 total, 8 up, 8 in 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[107278]: 13.5 continuing backfill to osd.0 from (378'6832,394'12651] MIN to 394'12651 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[107278]: 13.2 scrub starts 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[107278]: 13.2 scrub ok 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/473642984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[107278]: osdmap e397: 8 total, 8 up, 8 in 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[104007]: 13.3 continuing backfill to osd.0 from (374'6347,394'12557] MIN to 394'12557 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[104007]: osdmap e396: 8 total, 8 up, 8 in 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[104007]: 13.5 continuing backfill to osd.0 from (378'6832,394'12651] MIN to 394'12651 2024-09-13T15:04:18.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[104007]: 13.2 scrub starts 2024-09-13T15:04:18.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[104007]: 13.2 scrub ok 2024-09-13T15:04:18.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/473642984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:18.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:18 smithi003 ceph-mon[104007]: osdmap e397: 8 total, 8 up, 8 in 2024-09-13T15:04:18.379 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:18.401 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:18.503 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:18.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:18.627 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:18.650 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:18 smithi019 ceph-mon[91296]: 13.3 continuing backfill to osd.0 from (374'6347,394'12557] MIN to 394'12557 2024-09-13T15:04:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:18 smithi019 ceph-mon[91296]: osdmap e396: 8 total, 8 up, 8 in 2024-09-13T15:04:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:18 smithi019 ceph-mon[91296]: 13.5 continuing backfill to osd.0 from (378'6832,394'12651] MIN to 394'12651 2024-09-13T15:04:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:18 smithi019 ceph-mon[91296]: 13.2 scrub starts 2024-09-13T15:04:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:18 smithi019 ceph-mon[91296]: 13.2 scrub ok 2024-09-13T15:04:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/473642984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:18 smithi019 ceph-mon[91296]: osdmap e397: 8 total, 8 up, 8 in 2024-09-13T15:04:18.752 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:18.775 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:18.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:18.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:18.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:19.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:19.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:19.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:19.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:19.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:19.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:19.340 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[107278]: pgmap v1426: 129 pgs: 1 peering, 9 unknown, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2906/102204 objects degraded (2.843%); 4673/102204 objects misplaced (4.572%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:19.340 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[107278]: 13.8 scrub starts 2024-09-13T15:04:19.340 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[107278]: 13.8 scrub ok 2024-09-13T15:04:19.340 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[107278]: 13.12 scrub starts 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[107278]: 13.12 scrub ok 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[107278]: 13.1 scrub starts 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[107278]: osdmap e398: 8 total, 8 up, 8 in 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[104007]: pgmap v1426: 129 pgs: 1 peering, 9 unknown, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2906/102204 objects degraded (2.843%); 4673/102204 objects misplaced (4.572%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[104007]: 13.8 scrub starts 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[104007]: 13.8 scrub ok 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[104007]: 13.12 scrub starts 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[104007]: 13.12 scrub ok 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[104007]: 13.1 scrub starts 2024-09-13T15:04:19.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:19 smithi003 ceph-mon[104007]: osdmap e398: 8 total, 8 up, 8 in 2024-09-13T15:04:19.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:19.428 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:19.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:04:19.531 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:19.566 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:19.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:19 smithi019 ceph-mon[91296]: pgmap v1426: 129 pgs: 1 peering, 9 unknown, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 2.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2906/102204 objects degraded (2.843%); 4673/102204 objects misplaced (4.572%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:19 smithi019 ceph-mon[91296]: 13.8 scrub starts 2024-09-13T15:04:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:19 smithi019 ceph-mon[91296]: 13.8 scrub ok 2024-09-13T15:04:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:19 smithi019 ceph-mon[91296]: 13.12 scrub starts 2024-09-13T15:04:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:19 smithi019 ceph-mon[91296]: 13.12 scrub ok 2024-09-13T15:04:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:19 smithi019 ceph-mon[91296]: 13.1 scrub starts 2024-09-13T15:04:19.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:19 smithi019 ceph-mon[91296]: osdmap e398: 8 total, 8 up, 8 in 2024-09-13T15:04:19.708 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:19.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:19.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:19.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:19.987 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:20.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:20.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:20.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:20.260 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:20.362 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:20.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[107278]: 13.0 scrub starts 2024-09-13T15:04:20.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[107278]: 13.0 scrub ok 2024-09-13T15:04:20.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[107278]: 13.18 scrub starts 2024-09-13T15:04:20.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[107278]: 13.13 continuing backfill to osd.0 from (374'6347,394'12557] MIN to 394'12557 2024-09-13T15:04:20.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[107278]: 13.15 continuing backfill to osd.0 from (378'6832,394'12651] MIN to 394'12651 2024-09-13T15:04:20.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[107278]: 13.18 scrub ok 2024-09-13T15:04:20.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[104007]: 13.0 scrub starts 2024-09-13T15:04:20.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[104007]: 13.0 scrub ok 2024-09-13T15:04:20.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[104007]: 13.18 scrub starts 2024-09-13T15:04:20.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[104007]: 13.13 continuing backfill to osd.0 from (374'6347,394'12557] MIN to 394'12557 2024-09-13T15:04:20.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[104007]: 13.15 continuing backfill to osd.0 from (378'6832,394'12651] MIN to 394'12651 2024-09-13T15:04:20.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:20 smithi003 ceph-mon[104007]: 13.18 scrub ok 2024-09-13T15:04:20.436 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:20.481 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:04:20.481 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:04:20.502 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:20.534 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:20.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:20.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:20.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.677 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:20.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:04:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:04:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:20 smithi019 ceph-mon[91296]: 13.0 scrub starts 2024-09-13T15:04:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:20 smithi019 ceph-mon[91296]: 13.0 scrub ok 2024-09-13T15:04:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:20 smithi019 ceph-mon[91296]: 13.18 scrub starts 2024-09-13T15:04:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:20 smithi019 ceph-mon[91296]: 13.13 continuing backfill to osd.0 from (374'6347,394'12557] MIN to 394'12557 2024-09-13T15:04:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:20 smithi019 ceph-mon[91296]: 13.15 continuing backfill to osd.0 from (378'6832,394'12651] MIN to 394'12651 2024-09-13T15:04:20.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:20 smithi019 ceph-mon[91296]: 13.18 scrub ok 2024-09-13T15:04:20.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:20.797 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 5, 0, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:04:20.797 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:04:20.797 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_8 2024-09-13T15:04:20.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:04:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:20.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:20.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:20.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:21.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:21.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:21.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:21.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:21.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:21.378 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:21.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:21.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:04:21.479 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:21.513 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:21.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:21.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:21 smithi019 ceph-mon[91296]: pgmap v1429: 129 pgs: 1 peering, 2 remapped+peering, 2 unknown, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 121 active+clean; 2.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1/96894 objects degraded (0.001%); 2402/96894 objects misplaced (2.479%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T15:04:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1/96894 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:04:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:21 smithi003 ceph-mon[107278]: pgmap v1429: 129 pgs: 1 peering, 2 remapped+peering, 2 unknown, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 121 active+clean; 2.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1/96894 objects degraded (0.001%); 2402/96894 objects misplaced (2.479%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T15:04:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1/96894 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:04:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:21 smithi003 ceph-mon[104007]: pgmap v1429: 129 pgs: 1 peering, 2 remapped+peering, 2 unknown, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 121 active+clean; 2.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1/96894 objects degraded (0.001%); 2402/96894 objects misplaced (2.479%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T15:04:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1/96894 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:04:21.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:21.797 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:21.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:04:21.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:21.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:22.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:22.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:22.225 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:22.268 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:22.370 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:22.405 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:22.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:22 smithi019 ceph-mon[91296]: 13.4 scrub starts 2024-09-13T15:04:22.507 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:22.546 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:22.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:22.682 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:22.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:22 smithi003 ceph-mon[107278]: 13.4 scrub starts 2024-09-13T15:04:22.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:22 smithi003 ceph-mon[104007]: 13.4 scrub starts 2024-09-13T15:04:22.784 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:22.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:22.881 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:22.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:22.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:23.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:23.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:23.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:23.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:23.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:23.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:23 smithi019 ceph-mon[91296]: pgmap v1430: 129 pgs: 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 123 active+clean; 2.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1/98940 objects degraded (0.001%); 2402/98940 objects misplaced (2.428%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T15:04:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:23 smithi019 ceph-mon[91296]: 13.4 scrub ok 2024-09-13T15:04:23.465 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:23.505 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:23.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:23.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:23 smithi003 ceph-mon[107278]: pgmap v1430: 129 pgs: 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 123 active+clean; 2.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1/98940 objects degraded (0.001%); 2402/98940 objects misplaced (2.428%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T15:04:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:23 smithi003 ceph-mon[107278]: 13.4 scrub ok 2024-09-13T15:04:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:23 smithi003 ceph-mon[104007]: pgmap v1430: 129 pgs: 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 123 active+clean; 2.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1/98940 objects degraded (0.001%); 2402/98940 objects misplaced (2.428%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T15:04:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:23 smithi003 ceph-mon[104007]: 13.4 scrub ok 2024-09-13T15:04:23.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:23.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:23.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:23.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:24.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:24.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:24.177 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:24.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:24.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:24.364 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:04:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:04:24] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-09-13T15:04:24.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:24.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:24.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:24.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:24.693 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:24.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:24.767 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:24.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:24.869 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.921 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:24.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:25.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:25.062 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:25.164 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:25.203 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:25.305 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:25.344 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:25.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:25 smithi003 ceph-mon[107278]: 13.7 scrub starts 2024-09-13T15:04:25.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:25 smithi003 ceph-mon[107278]: pgmap v1431: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2073/87438 objects degraded (2.371%); 3609/87438 objects misplaced (4.127%); 5.7 MiB/s, 132 objects/s recovering 2024-09-13T15:04:25.424 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:25 smithi003 ceph-mon[104007]: 13.7 scrub starts 2024-09-13T15:04:25.424 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:25 smithi003 ceph-mon[104007]: pgmap v1431: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2073/87438 objects degraded (2.371%); 3609/87438 objects misplaced (4.127%); 5.7 MiB/s, 132 objects/s recovering 2024-09-13T15:04:25.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:25.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:25.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:25.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:25.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:25 smithi019 ceph-mon[91296]: 13.7 scrub starts 2024-09-13T15:04:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:25 smithi019 ceph-mon[91296]: pgmap v1431: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2073/87438 objects degraded (2.371%); 3609/87438 objects misplaced (4.127%); 5.7 MiB/s, 132 objects/s recovering 2024-09-13T15:04:25.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:25.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:25.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:26.009 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:26.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.052 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:26.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:26.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:26.195 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:26.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:26.324 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:04:26.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:26.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[107278]: 13.10 scrub starts 2024-09-13T15:04:26.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[107278]: 13.10 scrub ok 2024-09-13T15:04:26.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2230015417' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:26.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:26.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2073/87438 objects degraded (2.371%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:26.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[104007]: 13.10 scrub starts 2024-09-13T15:04:26.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[104007]: 13.10 scrub ok 2024-09-13T15:04:26.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2230015417' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:26.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:26.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2073/87438 objects degraded (2.371%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:26.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:26.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:26.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:26.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:26 smithi019 ceph-mon[91296]: 13.10 scrub starts 2024-09-13T15:04:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:26 smithi019 ceph-mon[91296]: 13.10 scrub ok 2024-09-13T15:04:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2230015417' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:26.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:26 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:26.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2073/87438 objects degraded (2.371%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:26.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:26.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:26.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:26.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.889 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:26.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:26.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:27.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:27.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:27.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:27.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:27.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:27.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:27.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:27.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:27.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:27.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:27.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[104007]: pgmap v1432: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2073/87438 objects degraded (2.371%); 3609/87438 objects misplaced (4.127%); 4.8 MiB/s, 113 objects/s recovering 2024-09-13T15:04:27.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:04:27.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:27.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[104007]: osdmap e399: 8 total, 8 up, 8 in 2024-09-13T15:04:27.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[104007]: 13.14 scrub starts 2024-09-13T15:04:27.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[104007]: 13.14 scrub ok 2024-09-13T15:04:27.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[107278]: pgmap v1432: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2073/87438 objects degraded (2.371%); 3609/87438 objects misplaced (4.127%); 4.8 MiB/s, 113 objects/s recovering 2024-09-13T15:04:27.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:04:27.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:27.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[107278]: osdmap e399: 8 total, 8 up, 8 in 2024-09-13T15:04:27.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[107278]: 13.14 scrub starts 2024-09-13T15:04:27.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:27 smithi003 ceph-mon[107278]: 13.14 scrub ok 2024-09-13T15:04:27.676 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:27 smithi019 ceph-mon[91296]: pgmap v1432: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2073/87438 objects degraded (2.371%); 3609/87438 objects misplaced (4.127%); 4.8 MiB/s, 113 objects/s recovering 2024-09-13T15:04:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:27 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:04:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:27 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:27 smithi019 ceph-mon[91296]: osdmap e399: 8 total, 8 up, 8 in 2024-09-13T15:04:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:27 smithi019 ceph-mon[91296]: 13.14 scrub starts 2024-09-13T15:04:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:27 smithi019 ceph-mon[91296]: 13.14 scrub ok 2024-09-13T15:04:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:27.781 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:27.816 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.853 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:27.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:27.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:27.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:28.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:28.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:28.142 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:04:28.144 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:04:28.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:28.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:28.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:28.413 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[107278]: 13.17 scrub starts 2024-09-13T15:04:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[107278]: 13.11 scrub starts 2024-09-13T15:04:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:28.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[107278]: 13.1 scrub starts 2024-09-13T15:04:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[104007]: 13.17 scrub starts 2024-09-13T15:04:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[104007]: 13.11 scrub starts 2024-09-13T15:04:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:28 smithi003 ceph-mon[104007]: 13.1 scrub starts 2024-09-13T15:04:28.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:28.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:28 smithi019 ceph-mon[91296]: 13.17 scrub starts 2024-09-13T15:04:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:28 smithi019 ceph-mon[91296]: 13.11 scrub starts 2024-09-13T15:04:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:28 smithi019 ceph-mon[91296]: 13.1 scrub starts 2024-09-13T15:04:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:28.805 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:28.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:28.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:28.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:29.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:29.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:29.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:29.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:29.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:29.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:29 smithi003 ceph-mon[107278]: from='client.37137 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:04:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:29 smithi003 ceph-mon[107278]: pgmap v1434: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 2073/81069 objects degraded (2.557%); 3198/81069 objects misplaced (3.945%); 1.7 MiB/s, 58 objects/s recovering 2024-09-13T15:04:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:29 smithi003 ceph-mon[104007]: from='client.37137 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:04:29.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:29 smithi003 ceph-mon[104007]: pgmap v1434: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 2073/81069 objects degraded (2.557%); 3198/81069 objects misplaced (3.945%); 1.7 MiB/s, 58 objects/s recovering 2024-09-13T15:04:29.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:29.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:04:29.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:29.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:29.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:29 smithi019 ceph-mon[91296]: from='client.37137 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:04:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:29 smithi019 ceph-mon[91296]: pgmap v1434: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 2073/81069 objects degraded (2.557%); 3198/81069 objects misplaced (3.945%); 1.7 MiB/s, 58 objects/s recovering 2024-09-13T15:04:29.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:29.852 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1433,"stamp":"2024-09-13T15:04:26.306022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1908905957,"num_objects":29146,"num_object_clones":0,"num_object_copies":87438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":3609,"num_objects_unfound":0,"num_objects_dirty":29146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171002,"num_write_kb":6201345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8601,"num_bytes_recovered":541852887,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171342,"ondisk_log_size":171342,"up":387,"acting":385,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":394,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16848272,"kb_used_data":6440144,"kb_used_omap":29,"kb_used_meta":10407778,"kb_avail":733112944,"statfs":{"total":767960285184,"available":750707654656,"internally_reserved":0,"allocated":6594707456,"data_stored":6575530144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":10657565198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-322633728,"num_objects":-4922,"num_object_clones":0,"num_object_copies":-14766,"num_objects_missing_on_primary":0,"num_objects_missing":-1107,"num_objects_degraded":-833,"num_objects_misplaced":-1064,"num_objects_unfound":0,"num_objects_dirty":-4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12313,"num_write_kb":-717856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":911,"num_bytes_recovered":40632320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.056383"},"pg_stats":[{"pgid":"13.11","version":"398'12829","reported_seq":12888,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.507413+0000","last_change":"2024-09-13T15:04:17.387975+0000","last_active":"2024-09-13T15:04:23.507413+0000","last_peered":"2024-09-13T15:04:23.507413+0000","last_clean":"2024-09-13T15:04:23.507413+0000","last_became_active":"2024-09-13T15:04:17.352314+0000","last_became_peered":"2024-09-13T15:04:17.352314+0000","last_unstale":"2024-09-13T15:04:23.507413+0000","last_undegraded":"2024-09-13T15:04:23.507413+0000","last_fullsized":"2024-09-13T15:04:23.507413+0000","mapping_epoch":395,"log_start":"374'6229","ondisk_log_start":"374'6229","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6600,"log_dups_size":0,"ondisk_log_size":6600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59670528,"num_objects":911,"num_object_clones":0,"num_object_copies":2733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6498,"num_write_kb":237056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1d","version":"79'100","reported_seq":36,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.258129+0000","last_change":"2024-09-13T15:03:48.428392+0000","last_active":"2024-09-13T15:04:19.258129+0000","last_peered":"2024-09-13T15:04:19.258129+0000","last_clean":"2024-09-13T15:04:19.258129+0000","last_became_active":"2024-09-13T15:03:46.867925+0000","last_became_peered":"2024-09-13T15:03:46.867925+0000","last_unstale":"2024-09-13T15:04:19.258129+0000","last_undegraded":"2024-09-13T15:04:19.258129+0000","last_fullsized":"2024-09-13T15:04:19.258129+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:48.428315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:48.428315+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:45.557387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":62,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257608+0000","last_change":"2024-09-13T15:02:19.429607+0000","last_active":"2024-09-13T15:04:19.257608+0000","last_peered":"2024-09-13T15:04:19.257608+0000","last_clean":"2024-09-13T15:04:19.257608+0000","last_became_active":"2024-09-13T15:02:19.418793+0000","last_became_peered":"2024-09-13T15:02:19.418793+0000","last_unstale":"2024-09-13T15:04:19.257608+0000","last_undegraded":"2024-09-13T15:04:19.257608+0000","last_fullsized":"2024-09-13T15:04:19.257608+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:03.520208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":370,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.255191+0000","last_change":"2024-09-13T14:48:29.121597+0000","last_active":"2024-09-13T15:04:18.255191+0000","last_peered":"2024-09-13T15:04:18.255191+0000","last_clean":"2024-09-13T15:04:18.255191+0000","last_became_active":"2024-09-13T14:48:29.121018+0000","last_became_peered":"2024-09-13T14:48:29.121018+0000","last_unstale":"2024-09-13T15:04:18.255191+0000","last_undegraded":"2024-09-13T15:04:18.255191+0000","last_fullsized":"2024-09-13T15:04:18.255191+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:47.572714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.10","version":"398'12950","reported_seq":13008,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.451197+0000","last_change":"2024-09-13T15:04:17.370525+0000","last_active":"2024-09-13T15:04:23.451197+0000","last_peered":"2024-09-13T15:04:23.451197+0000","last_clean":"2024-09-13T15:04:23.451197+0000","last_became_active":"2024-09-13T15:04:17.334691+0000","last_became_peered":"2024-09-13T15:04:17.334691+0000","last_unstale":"2024-09-13T15:04:23.451197+0000","last_undegraded":"2024-09-13T15:04:23.451197+0000","last_fullsized":"2024-09-13T15:04:23.451197+0000","mapping_epoch":395,"log_start":"378'6699","ondisk_log_start":"378'6699","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6251,"log_dups_size":0,"ondisk_log_size":6251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59604992,"num_objects":910,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":239200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":96,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.316047+0000","last_change":"2024-09-13T14:57:58.200463+0000","last_active":"2024-09-13T15:04:18.316047+0000","last_peered":"2024-09-13T15:04:18.316047+0000","last_clean":"2024-09-13T15:04:18.316047+0000","last_became_active":"2024-09-13T14:57:25.279086+0000","last_became_peered":"2024-09-13T14:57:25.279086+0000","last_unstale":"2024-09-13T15:04:18.316047+0000","last_undegraded":"2024-09-13T15:04:18.316047+0000","last_fullsized":"2024-09-13T15:04:18.316047+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:58.986716+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.258324+0000","last_change":"2024-09-13T15:03:46.871160+0000","last_active":"2024-09-13T15:04:19.258324+0000","last_peered":"2024-09-13T15:04:19.258324+0000","last_clean":"2024-09-13T15:04:19.258324+0000","last_became_active":"2024-09-13T15:03:46.870825+0000","last_became_peered":"2024-09-13T15:03:46.870825+0000","last_unstale":"2024-09-13T15:04:19.258324+0000","last_undegraded":"2024-09-13T15:04:19.258324+0000","last_fullsized":"2024-09-13T15:04:19.258324+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:54.702996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256875+0000","last_change":"2024-09-13T15:03:46.873412+0000","last_active":"2024-09-13T15:04:19.256875+0000","last_peered":"2024-09-13T15:04:19.256875+0000","last_clean":"2024-09-13T15:04:19.256875+0000","last_became_active":"2024-09-13T15:03:46.873094+0000","last_became_peered":"2024-09-13T15:03:46.873094+0000","last_unstale":"2024-09-13T15:04:19.256875+0000","last_undegraded":"2024-09-13T15:04:19.256875+0000","last_fullsized":"2024-09-13T15:04:19.256875+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:45.283467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.13","version":"398'12731","reported_seq":12818,"reported_epoch":398,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:04:22.839274+0000","last_change":"2024-09-13T15:04:19.287557+0000","last_active":"2024-09-13T15:04:22.839274+0000","last_peered":"2024-09-13T15:04:22.839274+0000","last_clean":"2024-09-13T15:02:18.361205+0000","last_became_active":"2024-09-13T15:04:19.260018+0000","last_became_peered":"2024-09-13T15:04:19.260018+0000","last_unstale":"2024-09-13T15:04:22.839274+0000","last_undegraded":"2024-09-13T15:04:22.839274+0000","last_fullsized":"2024-09-13T15:04:22.839274+0000","mapping_epoch":397,"log_start":"374'6347","ondisk_log_start":"374'6347","created":395,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6384,"log_dups_size":0,"ondisk_log_size":6384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61112320,"num_objects":933,"num_object_clones":0,"num_object_copies":2799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":933,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":236320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[3,4,5],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":933}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":101,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.257200+0000","last_change":"2024-09-13T14:58:28.426763+0000","last_active":"2024-09-13T15:04:18.257200+0000","last_peered":"2024-09-13T15:04:18.257200+0000","last_clean":"2024-09-13T15:04:18.257200+0000","last_became_active":"2024-09-13T14:58:28.426270+0000","last_became_peered":"2024-09-13T14:58:28.426270+0000","last_unstale":"2024-09-13T15:04:18.257200+0000","last_undegraded":"2024-09-13T15:04:18.257200+0000","last_fullsized":"2024-09-13T15:04:18.257200+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:08.991576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":417,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256162+0000","last_change":"2024-09-13T15:03:46.867138+0000","last_active":"2024-09-13T15:04:18.256162+0000","last_peered":"2024-09-13T15:04:18.256162+0000","last_clean":"2024-09-13T15:04:18.256162+0000","last_became_active":"2024-09-13T15:03:46.866870+0000","last_became_peered":"2024-09-13T15:03:46.866870+0000","last_unstale":"2024-09-13T15:04:18.256162+0000","last_undegraded":"2024-09-13T15:04:18.256162+0000","last_fullsized":"2024-09-13T15:04:18.256162+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:44.031064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"398'12834","reported_seq":12955,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:22.777580+0000","last_change":"2024-09-13T15:04:18.643135+0000","last_active":"2024-09-13T15:04:22.777580+0000","last_peered":"2024-09-13T15:04:22.777580+0000","last_clean":"2024-09-13T15:04:22.777580+0000","last_became_active":"2024-09-13T15:04:17.317738+0000","last_became_peered":"2024-09-13T15:04:17.317738+0000","last_unstale":"2024-09-13T15:04:22.777580+0000","last_undegraded":"2024-09-13T15:04:22.777580+0000","last_fullsized":"2024-09-13T15:04:22.777580+0000","mapping_epoch":395,"log_start":"377'6526","ondisk_log_start":"377'6526","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"396'12687","last_scrub_stamp":"2024-09-13T15:04:18.643071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:04:18.643071+0000","objects_scrubbed":1109,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:05.028397+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63045632,"num_objects":962,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":36,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.258217+0000","last_change":"2024-09-13T15:03:47.469254+0000","last_active":"2024-09-13T15:04:19.258217+0000","last_peered":"2024-09-13T15:04:19.258217+0000","last_clean":"2024-09-13T15:04:19.258217+0000","last_became_active":"2024-09-13T15:03:46.854242+0000","last_became_peered":"2024-09-13T15:03:46.854242+0000","last_unstale":"2024-09-13T15:04:19.258217+0000","last_undegraded":"2024-09-13T15:04:19.258217+0000","last_fullsized":"2024-09-13T15:04:19.258217+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:47.469211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:47.469211+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:59.563316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":101,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.255592+0000","last_change":"2024-09-13T14:58:28.429061+0000","last_active":"2024-09-13T15:04:18.255592+0000","last_peered":"2024-09-13T15:04:18.255592+0000","last_clean":"2024-09-13T15:04:18.255592+0000","last_became_active":"2024-09-13T14:58:28.428642+0000","last_became_peered":"2024-09-13T14:58:28.428642+0000","last_unstale":"2024-09-13T15:04:18.255592+0000","last_undegraded":"2024-09-13T15:04:18.255592+0000","last_fullsized":"2024-09-13T15:04:18.255592+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:03.197824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":427,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256245+0000","last_change":"2024-09-13T15:03:46.868016+0000","last_active":"2024-09-13T15:04:18.256245+0000","last_peered":"2024-09-13T15:04:18.256245+0000","last_clean":"2024-09-13T15:04:18.256245+0000","last_became_active":"2024-09-13T15:03:46.867643+0000","last_became_peered":"2024-09-13T15:03:46.867643+0000","last_unstale":"2024-09-13T15:04:18.256245+0000","last_undegraded":"2024-09-13T15:04:18.256245+0000","last_fullsized":"2024-09-13T15:04:18.256245+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:26.304605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"398'12845","reported_seq":12982,"reported_epoch":398,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:04:23.490108+0000","last_change":"2024-09-13T15:04:19.290294+0000","last_active":"2024-09-13T15:04:23.490108+0000","last_peered":"2024-09-13T15:04:23.490108+0000","last_clean":"2024-09-13T15:02:18.352546+0000","last_became_active":"2024-09-13T15:04:19.259583+0000","last_became_peered":"2024-09-13T15:04:19.259583+0000","last_unstale":"2024-09-13T15:04:23.490108+0000","last_undegraded":"2024-09-13T15:04:23.490108+0000","last_fullsized":"2024-09-13T15:04:23.490108+0000","mapping_epoch":397,"log_start":"378'6832","ondisk_log_start":"378'6832","created":395,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6013,"log_dups_size":0,"ondisk_log_size":6013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57835520,"num_objects":883,"num_object_clones":0,"num_object_copies":2649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":879,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6520,"num_write_kb":236864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,5,3],"avail_no_missing":["4","3","5"],"object_location_counts":[{"shards":"3,4,5","objects":883}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256967+0000","last_change":"2024-09-13T15:03:46.864596+0000","last_active":"2024-09-13T15:04:19.256967+0000","last_peered":"2024-09-13T15:04:19.256967+0000","last_clean":"2024-09-13T15:04:19.256967+0000","last_became_active":"2024-09-13T15:03:46.863107+0000","last_became_peered":"2024-09-13T15:03:46.863107+0000","last_unstale":"2024-09-13T15:04:19.256967+0000","last_undegraded":"2024-09-13T15:04:19.256967+0000","last_fullsized":"2024-09-13T15:04:19.256967+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:07.821841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":56,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.255289+0000","last_change":"2024-09-13T15:02:19.418023+0000","last_active":"2024-09-13T15:04:19.255289+0000","last_peered":"2024-09-13T15:04:19.255289+0000","last_clean":"2024-09-13T15:04:19.255289+0000","last_became_active":"2024-09-13T15:02:19.417503+0000","last_became_peered":"2024-09-13T15:02:19.417503+0000","last_unstale":"2024-09-13T15:04:19.255289+0000","last_undegraded":"2024-09-13T15:04:19.255289+0000","last_fullsized":"2024-09-13T15:04:19.255289+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:48.630767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":677,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.261401+0000","last_change":"2024-09-13T15:03:46.871823+0000","last_active":"2024-09-13T15:04:19.261401+0000","last_peered":"2024-09-13T15:04:19.261401+0000","last_clean":"2024-09-13T15:04:19.261401+0000","last_became_active":"2024-09-13T15:03:46.868753+0000","last_became_peered":"2024-09-13T15:03:46.868753+0000","last_unstale":"2024-09-13T15:04:19.261401+0000","last_undegraded":"2024-09-13T15:04:19.261401+0000","last_fullsized":"2024-09-13T15:04:19.261401+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:23.680514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.14","version":"398'13097","reported_seq":13153,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.517872+0000","last_change":"2024-09-13T15:04:17.355912+0000","last_active":"2024-09-13T15:04:23.517872+0000","last_peered":"2024-09-13T15:04:23.517872+0000","last_clean":"2024-09-13T15:04:23.517872+0000","last_became_active":"2024-09-13T15:04:17.310160+0000","last_became_peered":"2024-09-13T15:04:17.310160+0000","last_unstale":"2024-09-13T15:04:23.517872+0000","last_undegraded":"2024-09-13T15:04:23.517872+0000","last_fullsized":"2024-09-13T15:04:23.517872+0000","mapping_epoch":395,"log_start":"374'6476","ondisk_log_start":"374'6476","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6621,"log_dups_size":0,"ondisk_log_size":6621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58654720,"num_objects":895,"num_object_clones":0,"num_object_copies":2685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6639,"num_write_kb":241088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":93,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256847+0000","last_change":"2024-09-13T14:58:09.011981+0000","last_active":"2024-09-13T15:04:18.256847+0000","last_peered":"2024-09-13T15:04:18.256847+0000","last_clean":"2024-09-13T15:04:18.256847+0000","last_became_active":"2024-09-13T14:57:53.357674+0000","last_became_peered":"2024-09-13T14:57:53.357674+0000","last_unstale":"2024-09-13T15:04:18.256847+0000","last_undegraded":"2024-09-13T15:04:18.256847+0000","last_fullsized":"2024-09-13T15:04:18.256847+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.701912+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":379,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256240+0000","last_change":"2024-09-13T14:35:30.474113+0000","last_active":"2024-09-13T15:04:18.256240+0000","last_peered":"2024-09-13T15:04:18.256240+0000","last_clean":"2024-09-13T15:04:18.256240+0000","last_became_active":"2024-09-13T14:35:30.473677+0000","last_became_peered":"2024-09-13T14:35:30.473677+0000","last_unstale":"2024-09-13T15:04:18.256240+0000","last_undegraded":"2024-09-13T15:04:18.256240+0000","last_fullsized":"2024-09-13T15:04:18.256240+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:13.685048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.258026+0000","last_change":"2024-09-13T15:03:46.868133+0000","last_active":"2024-09-13T15:04:19.258026+0000","last_peered":"2024-09-13T15:04:19.258026+0000","last_clean":"2024-09-13T15:04:19.258026+0000","last_became_active":"2024-09-13T15:03:46.866551+0000","last_became_peered":"2024-09-13T15:03:46.866551+0000","last_unstale":"2024-09-13T15:04:19.258026+0000","last_undegraded":"2024-09-13T15:04:19.258026+0000","last_fullsized":"2024-09-13T15:04:19.258026+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:00.313559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.17","version":"398'13037","reported_seq":13091,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.503714+0000","last_change":"2024-09-13T15:04:17.336785+0000","last_active":"2024-09-13T15:04:23.503714+0000","last_peered":"2024-09-13T15:04:23.503714+0000","last_clean":"2024-09-13T15:04:23.503714+0000","last_became_active":"2024-09-13T15:04:17.305946+0000","last_became_peered":"2024-09-13T15:04:17.305946+0000","last_unstale":"2024-09-13T15:04:23.503714+0000","last_undegraded":"2024-09-13T15:04:23.503714+0000","last_fullsized":"2024-09-13T15:04:23.503714+0000","mapping_epoch":395,"log_start":"378'6858","ondisk_log_start":"378'6858","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60293120,"num_objects":920,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":240768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":109,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254526+0000","last_change":"2024-09-13T15:02:19.406391+0000","last_active":"2024-09-13T15:04:18.254526+0000","last_peered":"2024-09-13T15:04:18.254526+0000","last_clean":"2024-09-13T15:04:18.254526+0000","last_became_active":"2024-09-13T15:02:19.406040+0000","last_became_peered":"2024-09-13T15:02:19.406040+0000","last_unstale":"2024-09-13T15:04:18.254526+0000","last_undegraded":"2024-09-13T15:04:18.254526+0000","last_fullsized":"2024-09-13T15:04:18.254526+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:45.665420+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":334,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.264791+0000","last_change":"2024-09-13T15:03:46.870303+0000","last_active":"2024-09-13T15:04:19.264791+0000","last_peered":"2024-09-13T15:04:19.264791+0000","last_clean":"2024-09-13T15:04:19.264791+0000","last_became_active":"2024-09-13T15:03:46.869984+0000","last_became_peered":"2024-09-13T15:03:46.869984+0000","last_unstale":"2024-09-13T15:04:19.264791+0000","last_undegraded":"2024-09-13T15:04:19.264791+0000","last_fullsized":"2024-09-13T15:04:19.264791+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:38.720142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":71,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256566+0000","last_change":"2024-09-13T15:03:46.864512+0000","last_active":"2024-09-13T15:04:19.256566+0000","last_peered":"2024-09-13T15:04:19.256566+0000","last_clean":"2024-09-13T15:04:19.256566+0000","last_became_active":"2024-09-13T15:03:46.864120+0000","last_became_peered":"2024-09-13T15:03:46.864120+0000","last_unstale":"2024-09-13T15:04:19.256566+0000","last_undegraded":"2024-09-13T15:04:19.256566+0000","last_fullsized":"2024-09-13T15:04:19.256566+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:36.450217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.16","version":"398'12689","reported_seq":12765,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.529713+0000","last_change":"2024-09-13T15:04:17.349451+0000","last_active":"2024-09-13T15:04:23.529713+0000","last_peered":"2024-09-13T15:04:23.529713+0000","last_clean":"2024-09-13T15:04:23.529713+0000","last_became_active":"2024-09-13T15:04:17.316305+0000","last_became_peered":"2024-09-13T15:04:17.316305+0000","last_unstale":"2024-09-13T15:04:23.529713+0000","last_undegraded":"2024-09-13T15:04:23.529713+0000","last_fullsized":"2024-09-13T15:04:23.529713+0000","mapping_epoch":395,"log_start":"378'6664","ondisk_log_start":"378'6664","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60456960,"num_objects":923,"num_object_clones":0,"num_object_copies":2769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":235424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":96,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.255756+0000","last_change":"2024-09-13T14:58:05.212101+0000","last_active":"2024-09-13T15:04:18.255756+0000","last_peered":"2024-09-13T15:04:18.255756+0000","last_clean":"2024-09-13T15:04:18.255756+0000","last_became_active":"2024-09-13T14:57:25.275734+0000","last_became_peered":"2024-09-13T14:57:25.275734+0000","last_unstale":"2024-09-13T15:04:18.255756+0000","last_undegraded":"2024-09-13T15:04:18.255756+0000","last_fullsized":"2024-09-13T15:04:18.255756+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.212008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.212008+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:01.720211+0000","scrub_duration":0.0048969690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256367+0000","last_change":"2024-09-13T15:03:46.867411+0000","last_active":"2024-09-13T15:04:19.256367+0000","last_peered":"2024-09-13T15:04:19.256367+0000","last_clean":"2024-09-13T15:04:19.256367+0000","last_became_active":"2024-09-13T15:03:46.866890+0000","last_became_peered":"2024-09-13T15:03:46.866890+0000","last_unstale":"2024-09-13T15:04:19.256367+0000","last_undegraded":"2024-09-13T15:04:19.256367+0000","last_fullsized":"2024-09-13T15:04:19.256367+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:59.143319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":380,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254945+0000","last_change":"2024-09-13T14:57:53.364039+0000","last_active":"2024-09-13T15:04:18.254945+0000","last_peered":"2024-09-13T15:04:18.254945+0000","last_clean":"2024-09-13T15:04:18.254945+0000","last_became_active":"2024-09-13T14:57:53.363550+0000","last_became_peered":"2024-09-13T14:57:53.363550+0000","last_unstale":"2024-09-13T15:04:18.254945+0000","last_undegraded":"2024-09-13T15:04:18.254945+0000","last_fullsized":"2024-09-13T15:04:18.254945+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:36.171089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.19","version":"398'13072","reported_seq":17929,"reported_epoch":398,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T15:04:23.734540+0000","last_change":"2024-09-13T15:04:19.359918+0000","last_active":"2024-09-13T15:04:23.734540+0000","last_peered":"2024-09-13T15:04:23.734540+0000","last_clean":"2024-09-13T15:02:18.361788+0000","last_became_active":"2024-09-13T15:04:19.287507+0000","last_became_peered":"2024-09-13T15:04:19.287507+0000","last_unstale":"2024-09-13T15:04:23.734540+0000","last_undegraded":"2024-09-13T15:04:19.285438+0000","last_fullsized":"2024-09-13T15:04:19.255477+0000","mapping_epoch":397,"log_start":"373'5709","ondisk_log_start":"373'5709","created":395,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57376768,"num_objects":876,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6634,"num_write_kb":240288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1198,"num_bytes_recovered":78413824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,2],"avail_no_missing":["5","2","6"],"object_location_counts":[{"shards":"2,5","objects":297},{"shards":"2,5,6","objects":384}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":122,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.316294+0000","last_change":"2024-09-13T14:58:04.168950+0000","last_active":"2024-09-13T15:04:18.316294+0000","last_peered":"2024-09-13T15:04:18.316294+0000","last_clean":"2024-09-13T15:04:18.316294+0000","last_became_active":"2024-09-13T14:57:53.351301+0000","last_became_peered":"2024-09-13T14:57:53.351301+0000","last_unstale":"2024-09-13T15:04:18.316294+0000","last_undegraded":"2024-09-13T15:04:18.316294+0000","last_fullsized":"2024-09-13T15:04:18.316294+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:42.270600+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256562+0000","last_change":"2024-09-13T15:03:46.855014+0000","last_active":"2024-09-13T15:04:19.256562+0000","last_peered":"2024-09-13T15:04:19.256562+0000","last_clean":"2024-09-13T15:04:19.256562+0000","last_became_active":"2024-09-13T15:03:46.854333+0000","last_became_peered":"2024-09-13T15:03:46.854333+0000","last_unstale":"2024-09-13T15:04:19.256562+0000","last_undegraded":"2024-09-13T15:04:19.256562+0000","last_fullsized":"2024-09-13T15:04:19.256562+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:30.681891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257506+0000","last_change":"2024-09-13T15:03:46.871342+0000","last_active":"2024-09-13T15:04:19.257506+0000","last_peered":"2024-09-13T15:04:19.257506+0000","last_clean":"2024-09-13T15:04:19.257506+0000","last_became_active":"2024-09-13T15:03:46.863406+0000","last_became_peered":"2024-09-13T15:03:46.863406+0000","last_unstale":"2024-09-13T15:04:19.257506+0000","last_undegraded":"2024-09-13T15:04:19.257506+0000","last_fullsized":"2024-09-13T15:04:19.257506+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:53.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.18","version":"398'12710","reported_seq":12820,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:21.466171+0000","last_change":"2024-09-13T15:04:19.285866+0000","last_active":"2024-09-13T15:04:21.466171+0000","last_peered":"2024-09-13T15:04:21.466171+0000","last_clean":"2024-09-13T15:04:21.466171+0000","last_became_active":"2024-09-13T15:04:17.299560+0000","last_became_peered":"2024-09-13T15:04:17.299560+0000","last_unstale":"2024-09-13T15:04:21.466171+0000","last_undegraded":"2024-09-13T15:04:21.466171+0000","last_fullsized":"2024-09-13T15:04:21.466171+0000","mapping_epoch":395,"log_start":"371'4451","ondisk_log_start":"371'4451","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"396'12623","last_scrub_stamp":"2024-09-13T15:04:19.285650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:04:19.285650+0000","objects_scrubbed":1074,"log_size":8259,"log_dups_size":0,"ondisk_log_size":8259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:58.635302+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64684032,"num_objects":987,"num_object_clones":0,"num_object_copies":2961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256664+0000","last_change":"2024-09-13T15:03:46.859350+0000","last_active":"2024-09-13T15:04:19.256664+0000","last_peered":"2024-09-13T15:04:19.256664+0000","last_clean":"2024-09-13T15:04:19.256664+0000","last_became_active":"2024-09-13T15:03:46.853202+0000","last_became_peered":"2024-09-13T15:03:46.853202+0000","last_unstale":"2024-09-13T15:04:19.256664+0000","last_undegraded":"2024-09-13T15:04:19.256664+0000","last_fullsized":"2024-09-13T15:04:19.256664+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:42.594537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":79,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254369+0000","last_change":"2024-09-13T14:57:53.350622+0000","last_active":"2024-09-13T15:04:18.254369+0000","last_peered":"2024-09-13T15:04:18.254369+0000","last_clean":"2024-09-13T15:04:18.254369+0000","last_became_active":"2024-09-13T14:57:53.350292+0000","last_became_peered":"2024-09-13T14:57:53.350292+0000","last_unstale":"2024-09-13T15:04:18.254369+0000","last_undegraded":"2024-09-13T15:04:18.254369+0000","last_fullsized":"2024-09-13T15:04:18.254369+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:16.088338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":365,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.261308+0000","last_change":"2024-09-13T14:47:07.231179+0000","last_active":"2024-09-13T15:04:19.261308+0000","last_peered":"2024-09-13T15:04:19.261308+0000","last_clean":"2024-09-13T15:04:19.261308+0000","last_became_active":"2024-09-13T14:47:07.230740+0000","last_became_peered":"2024-09-13T14:47:07.230740+0000","last_unstale":"2024-09-13T15:04:19.261308+0000","last_undegraded":"2024-09-13T15:04:19.261308+0000","last_fullsized":"2024-09-13T15:04:19.261308+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:43:13.165444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":99,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256316+0000","last_change":"2024-09-13T15:02:19.406264+0000","last_active":"2024-09-13T15:04:18.256316+0000","last_peered":"2024-09-13T15:04:18.256316+0000","last_clean":"2024-09-13T15:04:18.256316+0000","last_became_active":"2024-09-13T15:02:19.405876+0000","last_became_peered":"2024-09-13T15:02:19.405876+0000","last_unstale":"2024-09-13T15:04:18.256316+0000","last_undegraded":"2024-09-13T15:04:18.256316+0000","last_fullsized":"2024-09-13T15:04:18.256316+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:19.208307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":193,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.316044+0000","last_change":"2024-09-13T15:03:46.877994+0000","last_active":"2024-09-13T15:04:18.316044+0000","last_peered":"2024-09-13T15:04:18.316044+0000","last_clean":"2024-09-13T15:04:18.316044+0000","last_became_active":"2024-09-13T15:03:46.859821+0000","last_became_peered":"2024-09-13T15:03:46.859821+0000","last_unstale":"2024-09-13T15:04:18.316044+0000","last_undegraded":"2024-09-13T15:04:18.316044+0000","last_fullsized":"2024-09-13T15:04:18.316044+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:44.527820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":299,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.253811+0000","last_change":"2024-09-13T14:47:40.126832+0000","last_active":"2024-09-13T15:04:18.253811+0000","last_peered":"2024-09-13T15:04:18.253811+0000","last_clean":"2024-09-13T15:04:18.253811+0000","last_became_active":"2024-09-13T14:47:40.126380+0000","last_became_peered":"2024-09-13T14:47:40.126380+0000","last_unstale":"2024-09-13T15:04:18.253811+0000","last_undegraded":"2024-09-13T15:04:18.253811+0000","last_fullsized":"2024-09-13T15:04:18.253811+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:10.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":212,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.299339+0000","last_change":"2024-09-13T15:03:46.857574+0000","last_active":"2024-09-13T15:04:19.299339+0000","last_peered":"2024-09-13T15:04:19.299339+0000","last_clean":"2024-09-13T15:04:19.299339+0000","last_became_active":"2024-09-13T15:03:46.857142+0000","last_became_peered":"2024-09-13T15:03:46.857142+0000","last_unstale":"2024-09-13T15:04:19.299339+0000","last_undegraded":"2024-09-13T15:04:19.299339+0000","last_fullsized":"2024-09-13T15:04:19.299339+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:12.884559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":367,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.255529+0000","last_change":"2024-09-13T14:21:52.615385+0000","last_active":"2024-09-13T15:04:18.255529+0000","last_peered":"2024-09-13T15:04:18.255529+0000","last_clean":"2024-09-13T15:04:18.255529+0000","last_became_active":"2024-09-13T14:21:52.615086+0000","last_became_peered":"2024-09-13T14:21:52.615086+0000","last_unstale":"2024-09-13T15:04:18.255529+0000","last_undegraded":"2024-09-13T15:04:18.255529+0000","last_fullsized":"2024-09-13T15:04:18.255529+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:51.066310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":360,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256318+0000","last_change":"2024-09-13T14:32:09.485348+0000","last_active":"2024-09-13T15:04:18.256318+0000","last_peered":"2024-09-13T15:04:18.256318+0000","last_clean":"2024-09-13T15:04:18.256318+0000","last_became_active":"2024-09-13T14:32:09.485017+0000","last_became_peered":"2024-09-13T14:32:09.485017+0000","last_unstale":"2024-09-13T15:04:18.256318+0000","last_undegraded":"2024-09-13T15:04:18.256318+0000","last_fullsized":"2024-09-13T15:04:18.256318+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.893315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"398'12715","reported_seq":12826,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:21.437931+0000","last_change":"2024-09-13T15:04:18.283465+0000","last_active":"2024-09-13T15:04:21.437931+0000","last_peered":"2024-09-13T15:04:21.437931+0000","last_clean":"2024-09-13T15:04:21.437931+0000","last_became_active":"2024-09-13T15:04:17.277138+0000","last_became_peered":"2024-09-13T15:04:17.277138+0000","last_unstale":"2024-09-13T15:04:21.437931+0000","last_undegraded":"2024-09-13T15:04:21.437931+0000","last_fullsized":"2024-09-13T15:04:21.437931+0000","mapping_epoch":362,"log_start":"371'4451","ondisk_log_start":"371'4451","created":362,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'12625","last_scrub_stamp":"2024-09-13T15:04:18.283375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:04:18.283375+0000","objects_scrubbed":1044,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:57.652240+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62521344,"num_objects":954,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":363,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.255322+0000","last_change":"2024-09-13T14:57:53.351374+0000","last_active":"2024-09-13T15:04:18.255322+0000","last_peered":"2024-09-13T15:04:18.255322+0000","last_clean":"2024-09-13T15:04:18.255322+0000","last_became_active":"2024-09-13T14:57:53.350803+0000","last_became_peered":"2024-09-13T14:57:53.350803+0000","last_unstale":"2024-09-13T15:04:18.255322+0000","last_undegraded":"2024-09-13T15:04:18.255322+0000","last_fullsized":"2024-09-13T15:04:18.255322+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:30.535837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":393,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254988+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T15:04:18.254988+0000","last_peered":"2024-09-13T15:04:18.254988+0000","last_clean":"2024-09-13T15:04:18.254988+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T15:04:18.254988+0000","last_undegraded":"2024-09-13T15:04:18.254988+0000","last_fullsized":"2024-09-13T15:04:18.254988+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":60,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256304+0000","last_change":"2024-09-13T15:02:19.432393+0000","last_active":"2024-09-13T15:04:19.256304+0000","last_peered":"2024-09-13T15:04:19.256304+0000","last_clean":"2024-09-13T15:04:19.256304+0000","last_became_active":"2024-09-13T15:02:19.419183+0000","last_became_peered":"2024-09-13T15:02:19.419183+0000","last_unstale":"2024-09-13T15:04:19.256304+0000","last_undegraded":"2024-09-13T15:04:19.256304+0000","last_fullsized":"2024-09-13T15:04:19.256304+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:51.143192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257652+0000","last_change":"2024-09-13T15:03:46.868597+0000","last_active":"2024-09-13T15:04:19.257652+0000","last_peered":"2024-09-13T15:04:19.257652+0000","last_clean":"2024-09-13T15:04:19.257652+0000","last_became_active":"2024-09-13T15:03:46.866939+0000","last_became_peered":"2024-09-13T15:03:46.866939+0000","last_unstale":"2024-09-13T15:04:19.257652+0000","last_undegraded":"2024-09-13T15:04:19.257652+0000","last_fullsized":"2024-09-13T15:04:19.257652+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.060483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"398'13056","reported_seq":19584,"reported_epoch":398,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T15:04:23.826431+0000","last_change":"2024-09-13T15:04:17.403017+0000","last_active":"2024-09-13T15:04:23.826431+0000","last_peered":"2024-09-13T15:04:23.826431+0000","last_clean":"2024-09-13T15:02:18.361788+0000","last_became_active":"2024-09-13T15:04:17.317295+0000","last_became_peered":"2024-09-13T15:04:17.317295+0000","last_unstale":"2024-09-13T15:04:23.826431+0000","last_undegraded":"2024-09-13T15:04:17.311547+0000","last_fullsized":"2024-09-13T15:04:17.251183+0000","mapping_epoch":376,"log_start":"373'5709","ondisk_log_start":"373'5709","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58425344,"num_objects":891,"num_object_clones":0,"num_object_copies":2673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6617,"num_write_kb":240288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":114458624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,2],"avail_no_missing":["5","2","6"],"object_location_counts":[{"shards":"2,5","objects":890}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256251+0000","last_change":"2024-09-13T15:03:46.865877+0000","last_active":"2024-09-13T15:04:19.256251+0000","last_peered":"2024-09-13T15:04:19.256251+0000","last_clean":"2024-09-13T15:04:19.256251+0000","last_became_active":"2024-09-13T15:03:46.865029+0000","last_became_peered":"2024-09-13T15:03:46.865029+0000","last_unstale":"2024-09-13T15:04:19.256251+0000","last_undegraded":"2024-09-13T15:04:19.256251+0000","last_fullsized":"2024-09-13T15:04:19.256251+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:28.871714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":234,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254612+0000","last_change":"2024-09-13T14:45:44.442684+0000","last_active":"2024-09-13T15:04:18.254612+0000","last_peered":"2024-09-13T15:04:18.254612+0000","last_clean":"2024-09-13T15:04:18.254612+0000","last_became_active":"2024-09-13T14:45:44.442355+0000","last_became_peered":"2024-09-13T14:45:44.442355+0000","last_unstale":"2024-09-13T15:04:18.254612+0000","last_undegraded":"2024-09-13T15:04:18.254612+0000","last_fullsized":"2024-09-13T15:04:18.254612+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:12.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":32,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.258531+0000","last_change":"2024-09-13T15:03:46.879528+0000","last_active":"2024-09-13T15:04:19.258531+0000","last_peered":"2024-09-13T15:04:19.258531+0000","last_clean":"2024-09-13T15:04:19.258531+0000","last_became_active":"2024-09-13T15:03:46.860753+0000","last_became_peered":"2024-09-13T15:03:46.860753+0000","last_unstale":"2024-09-13T15:04:19.258531+0000","last_undegraded":"2024-09-13T15:04:19.258531+0000","last_fullsized":"2024-09-13T15:04:19.258531+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:28.561501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":201,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.257161+0000","last_change":"2024-09-13T14:57:56.132062+0000","last_active":"2024-09-13T15:04:18.257161+0000","last_peered":"2024-09-13T15:04:18.257161+0000","last_clean":"2024-09-13T15:04:18.257161+0000","last_became_active":"2024-09-13T14:57:19.218181+0000","last_became_peered":"2024-09-13T14:57:19.218181+0000","last_unstale":"2024-09-13T15:04:18.257161+0000","last_undegraded":"2024-09-13T15:04:18.257161+0000","last_fullsized":"2024-09-13T15:04:18.257161+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:56.131976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:56.131976+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:31.359825+0000","scrub_duration":0.0061470450000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"398'12898","reported_seq":12984,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:21.513535+0000","last_change":"2024-09-13T15:00:55.873094+0000","last_active":"2024-09-13T15:04:21.513535+0000","last_peered":"2024-09-13T15:04:21.513535+0000","last_clean":"2024-09-13T15:04:21.513535+0000","last_became_active":"2024-09-13T15:00:55.872743+0000","last_became_peered":"2024-09-13T15:00:55.872743+0000","last_unstale":"2024-09-13T15:04:21.513535+0000","last_undegraded":"2024-09-13T15:04:21.513535+0000","last_fullsized":"2024-09-13T15:04:21.513535+0000","mapping_epoch":362,"log_start":"373'5660","ondisk_log_start":"373'5660","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:02.124501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130547712,"num_objects":1992,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12898,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":56,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.255391+0000","last_change":"2024-09-13T15:02:19.423088+0000","last_active":"2024-09-13T15:04:19.255391+0000","last_peered":"2024-09-13T15:04:19.255391+0000","last_clean":"2024-09-13T15:04:19.255391+0000","last_became_active":"2024-09-13T15:02:19.422675+0000","last_became_peered":"2024-09-13T15:02:19.422675+0000","last_unstale":"2024-09-13T15:04:19.255391+0000","last_undegraded":"2024-09-13T15:04:19.255391+0000","last_fullsized":"2024-09-13T15:04:19.255391+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:58.764834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":393,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.316326+0000","last_change":"2024-09-13T15:02:19.417107+0000","last_active":"2024-09-13T15:04:18.316326+0000","last_peered":"2024-09-13T15:04:18.316326+0000","last_clean":"2024-09-13T15:04:18.316326+0000","last_became_active":"2024-09-13T15:02:19.416741+0000","last_became_peered":"2024-09-13T15:02:19.416741+0000","last_unstale":"2024-09-13T15:04:18.316326+0000","last_undegraded":"2024-09-13T15:04:18.316326+0000","last_fullsized":"2024-09-13T15:04:18.316326+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:20.599091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":636,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.261336+0000","last_change":"2024-09-13T14:57:53.351358+0000","last_active":"2024-09-13T15:04:19.261336+0000","last_peered":"2024-09-13T15:04:19.261336+0000","last_clean":"2024-09-13T15:04:19.261336+0000","last_became_active":"2024-09-13T14:57:53.351021+0000","last_became_peered":"2024-09-13T14:57:53.351021+0000","last_unstale":"2024-09-13T15:04:19.261336+0000","last_undegraded":"2024-09-13T15:04:19.261336+0000","last_fullsized":"2024-09-13T15:04:19.261336+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.067202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":394,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254429+0000","last_change":"2024-09-13T15:03:46.869628+0000","last_active":"2024-09-13T15:04:18.254429+0000","last_peered":"2024-09-13T15:04:18.254429+0000","last_clean":"2024-09-13T15:04:18.254429+0000","last_became_active":"2024-09-13T15:03:46.869274+0000","last_became_peered":"2024-09-13T15:03:46.869274+0000","last_unstale":"2024-09-13T15:04:18.254429+0000","last_undegraded":"2024-09-13T15:04:18.254429+0000","last_fullsized":"2024-09-13T15:04:18.254429+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:21.999130+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"398'13105","reported_seq":23757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:22.816152+0000","last_change":"2024-09-13T15:04:16.517275+0000","last_active":"2024-09-13T15:04:22.816152+0000","last_peered":"2024-09-13T15:04:22.816152+0000","last_clean":"2024-09-13T15:04:22.816152+0000","last_became_active":"2024-09-13T15:03:46.901821+0000","last_became_peered":"2024-09-13T15:03:46.901821+0000","last_unstale":"2024-09-13T15:04:22.816152+0000","last_undegraded":"2024-09-13T15:04:22.816152+0000","last_fullsized":"2024-09-13T15:04:22.816152+0000","mapping_epoch":389,"log_start":"379'7434","ondisk_log_start":"379'7434","created":362,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:38.637521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115015680,"num_objects":1755,"num_object_clones":0,"num_object_copies":5265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13105,"num_write_kb":475520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5275,"num_bytes_recovered":345636864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":79,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.253564+0000","last_change":"2024-09-13T14:57:53.351087+0000","last_active":"2024-09-13T15:04:18.253564+0000","last_peered":"2024-09-13T15:04:18.253564+0000","last_clean":"2024-09-13T15:04:18.253564+0000","last_became_active":"2024-09-13T14:57:53.350744+0000","last_became_peered":"2024-09-13T14:57:53.350744+0000","last_unstale":"2024-09-13T15:04:18.253564+0000","last_undegraded":"2024-09-13T15:04:18.253564+0000","last_fullsized":"2024-09-13T15:04:18.253564+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:43.300957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":56,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257352+0000","last_change":"2024-09-13T15:02:19.410671+0000","last_active":"2024-09-13T15:04:19.257352+0000","last_peered":"2024-09-13T15:04:19.257352+0000","last_clean":"2024-09-13T15:04:19.257352+0000","last_became_active":"2024-09-13T15:02:19.410330+0000","last_became_peered":"2024-09-13T15:02:19.410330+0000","last_unstale":"2024-09-13T15:04:19.257352+0000","last_undegraded":"2024-09-13T15:04:19.257352+0000","last_fullsized":"2024-09-13T15:04:19.257352+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:30.403982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":99,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256565+0000","last_change":"2024-09-13T15:03:46.866800+0000","last_active":"2024-09-13T15:04:18.256565+0000","last_peered":"2024-09-13T15:04:18.256565+0000","last_clean":"2024-09-13T15:04:18.256565+0000","last_became_active":"2024-09-13T15:03:46.866557+0000","last_became_peered":"2024-09-13T15:03:46.866557+0000","last_unstale":"2024-09-13T15:04:18.256565+0000","last_undegraded":"2024-09-13T15:04:18.256565+0000","last_fullsized":"2024-09-13T15:04:18.256565+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:28.181919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":394,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254837+0000","last_change":"2024-09-13T15:03:46.851975+0000","last_active":"2024-09-13T15:04:18.254837+0000","last_peered":"2024-09-13T15:04:18.254837+0000","last_clean":"2024-09-13T15:04:18.254837+0000","last_became_active":"2024-09-13T15:03:46.851616+0000","last_became_peered":"2024-09-13T15:03:46.851616+0000","last_unstale":"2024-09-13T15:04:18.254837+0000","last_undegraded":"2024-09-13T15:04:18.254837+0000","last_fullsized":"2024-09-13T15:04:18.254837+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:01.819924+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"398'13253","reported_seq":13298,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.606845+0000","last_change":"2024-09-13T15:00:55.874092+0000","last_active":"2024-09-13T15:04:23.606845+0000","last_peered":"2024-09-13T15:04:23.606845+0000","last_clean":"2024-09-13T15:04:23.606845+0000","last_became_active":"2024-09-13T15:00:55.873709+0000","last_became_peered":"2024-09-13T15:00:55.873709+0000","last_unstale":"2024-09-13T15:04:23.606845+0000","last_undegraded":"2024-09-13T15:04:23.606845+0000","last_fullsized":"2024-09-13T15:04:23.606845+0000","mapping_epoch":362,"log_start":"380'7994","ondisk_log_start":"380'7994","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5259,"log_dups_size":0,"ondisk_log_size":5259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:20.813547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654104,"num_objects":1781,"num_object_clones":0,"num_object_copies":5343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13253,"num_write_kb":481025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":548,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.261439+0000","last_change":"2024-09-13T15:03:46.872633+0000","last_active":"2024-09-13T15:04:19.261439+0000","last_peered":"2024-09-13T15:04:19.261439+0000","last_clean":"2024-09-13T15:04:19.261439+0000","last_became_active":"2024-09-13T15:03:46.872123+0000","last_became_peered":"2024-09-13T15:03:46.872123+0000","last_unstale":"2024-09-13T15:04:19.261439+0000","last_undegraded":"2024-09-13T15:04:19.261439+0000","last_fullsized":"2024-09-13T15:04:19.261439+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:47.174643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":381,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256425+0000","last_change":"2024-09-13T15:02:19.406489+0000","last_active":"2024-09-13T15:04:18.256425+0000","last_peered":"2024-09-13T15:04:18.256425+0000","last_clean":"2024-09-13T15:04:18.256425+0000","last_became_active":"2024-09-13T15:02:19.405872+0000","last_became_peered":"2024-09-13T15:02:19.405872+0000","last_unstale":"2024-09-13T15:04:18.256425+0000","last_undegraded":"2024-09-13T15:04:18.256425+0000","last_fullsized":"2024-09-13T15:04:18.256425+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:15.343020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":168,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.302245+0000","last_change":"2024-09-13T14:58:28.428183+0000","last_active":"2024-09-13T15:04:19.302245+0000","last_peered":"2024-09-13T15:04:19.302245+0000","last_clean":"2024-09-13T15:04:19.302245+0000","last_became_active":"2024-09-13T14:58:28.427553+0000","last_became_peered":"2024-09-13T14:58:28.427553+0000","last_unstale":"2024-09-13T15:04:19.302245+0000","last_undegraded":"2024-09-13T15:04:19.302245+0000","last_fullsized":"2024-09-13T15:04:19.302245+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:59.108147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":56,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.254630+0000","last_change":"2024-09-13T15:02:19.420193+0000","last_active":"2024-09-13T15:04:19.254630+0000","last_peered":"2024-09-13T15:04:19.254630+0000","last_clean":"2024-09-13T15:04:19.254630+0000","last_became_active":"2024-09-13T15:02:19.419807+0000","last_became_peered":"2024-09-13T15:02:19.419807+0000","last_unstale":"2024-09-13T15:04:19.254630+0000","last_undegraded":"2024-09-13T15:04:19.254630+0000","last_fullsized":"2024-09-13T15:04:19.254630+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:13.929284+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"398'13070","reported_seq":13114,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.499382+0000","last_change":"2024-09-13T15:00:55.408720+0000","last_active":"2024-09-13T15:04:23.499382+0000","last_peered":"2024-09-13T15:04:23.499382+0000","last_clean":"2024-09-13T15:04:23.499382+0000","last_became_active":"2024-09-13T15:00:55.408379+0000","last_became_peered":"2024-09-13T15:00:55.408379+0000","last_unstale":"2024-09-13T15:04:23.499382+0000","last_undegraded":"2024-09-13T15:04:23.499382+0000","last_fullsized":"2024-09-13T15:04:23.499382+0000","mapping_epoch":362,"log_start":"379'7251","ondisk_log_start":"379'7251","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:24.296460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117833728,"num_objects":1798,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13070,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":389,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.299761+0000","last_change":"2024-09-13T15:02:19.420231+0000","last_active":"2024-09-13T15:04:19.299761+0000","last_peered":"2024-09-13T15:04:19.299761+0000","last_clean":"2024-09-13T15:04:19.299761+0000","last_became_active":"2024-09-13T15:02:19.419627+0000","last_became_peered":"2024-09-13T15:02:19.419627+0000","last_unstale":"2024-09-13T15:04:19.299761+0000","last_undegraded":"2024-09-13T15:04:19.299761+0000","last_fullsized":"2024-09-13T15:04:19.299761+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:21.403387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":475,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.261156+0000","last_change":"2024-09-13T14:45:44.444120+0000","last_active":"2024-09-13T15:04:19.261156+0000","last_peered":"2024-09-13T15:04:19.261156+0000","last_clean":"2024-09-13T15:04:19.261156+0000","last_became_active":"2024-09-13T14:45:44.443843+0000","last_became_peered":"2024-09-13T14:45:44.443843+0000","last_unstale":"2024-09-13T15:04:19.261156+0000","last_undegraded":"2024-09-13T15:04:19.261156+0000","last_fullsized":"2024-09-13T15:04:19.261156+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:15.401447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":693,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.264994+0000","last_change":"2024-09-13T15:03:46.871298+0000","last_active":"2024-09-13T15:04:19.264994+0000","last_peered":"2024-09-13T15:04:19.264994+0000","last_clean":"2024-09-13T15:04:19.264994+0000","last_became_active":"2024-09-13T15:03:46.870966+0000","last_became_peered":"2024-09-13T15:03:46.870966+0000","last_unstale":"2024-09-13T15:04:19.264994+0000","last_undegraded":"2024-09-13T15:04:19.264994+0000","last_fullsized":"2024-09-13T15:04:19.264994+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:22.463434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":175,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.315808+0000","last_change":"2024-09-13T15:02:19.406171+0000","last_active":"2024-09-13T15:04:18.315808+0000","last_peered":"2024-09-13T15:04:18.315808+0000","last_clean":"2024-09-13T15:04:18.315808+0000","last_became_active":"2024-09-13T15:02:19.405713+0000","last_became_peered":"2024-09-13T15:02:19.405713+0000","last_unstale":"2024-09-13T15:04:18.315808+0000","last_undegraded":"2024-09-13T15:04:18.315808+0000","last_fullsized":"2024-09-13T15:04:18.315808+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:13.657811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.f","version":"398'13227","reported_seq":13271,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.532715+0000","last_change":"2024-09-13T15:00:55.408205+0000","last_active":"2024-09-13T15:04:23.532715+0000","last_peered":"2024-09-13T15:04:23.532715+0000","last_clean":"2024-09-13T15:04:23.532715+0000","last_became_active":"2024-09-13T15:00:55.407801+0000","last_became_peered":"2024-09-13T15:00:55.407801+0000","last_unstale":"2024-09-13T15:04:23.532715+0000","last_undegraded":"2024-09-13T15:04:23.532715+0000","last_fullsized":"2024-09-13T15:04:23.532715+0000","mapping_epoch":362,"log_start":"379'7423","ondisk_log_start":"379'7423","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":5804,"log_dups_size":0,"ondisk_log_size":5804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:36.295309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117899264,"num_objects":1799,"num_object_clones":0,"num_object_copies":5397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13227,"num_write_kb":480832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":56,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256144+0000","last_change":"2024-09-13T15:02:19.419571+0000","last_active":"2024-09-13T15:04:19.256144+0000","last_peered":"2024-09-13T15:04:19.256144+0000","last_clean":"2024-09-13T15:04:19.256144+0000","last_became_active":"2024-09-13T15:02:19.419205+0000","last_became_peered":"2024-09-13T15:02:19.419205+0000","last_unstale":"2024-09-13T15:04:19.256144+0000","last_undegraded":"2024-09-13T15:04:19.256144+0000","last_fullsized":"2024-09-13T15:04:19.256144+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:04.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":70,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.254516+0000","last_change":"2024-09-13T15:03:46.866278+0000","last_active":"2024-09-13T15:04:19.254516+0000","last_peered":"2024-09-13T15:04:19.254516+0000","last_clean":"2024-09-13T15:04:19.254516+0000","last_became_active":"2024-09-13T15:03:46.865838+0000","last_became_peered":"2024-09-13T15:03:46.865838+0000","last_unstale":"2024-09-13T15:04:19.254516+0000","last_undegraded":"2024-09-13T15:04:19.254516+0000","last_fullsized":"2024-09-13T15:04:19.254516+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:02.073646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":430,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.315873+0000","last_change":"2024-09-13T15:02:19.408677+0000","last_active":"2024-09-13T15:04:18.315873+0000","last_peered":"2024-09-13T15:04:18.315873+0000","last_clean":"2024-09-13T15:04:18.315873+0000","last_became_active":"2024-09-13T15:02:19.408262+0000","last_became_peered":"2024-09-13T15:02:19.408262+0000","last_unstale":"2024-09-13T15:04:18.315873+0000","last_undegraded":"2024-09-13T15:04:18.315873+0000","last_fullsized":"2024-09-13T15:04:18.315873+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:47:22.461507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":334,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.253234+0000","last_change":"2024-09-13T14:47:59.730004+0000","last_active":"2024-09-13T15:04:18.253234+0000","last_peered":"2024-09-13T15:04:18.253234+0000","last_clean":"2024-09-13T15:04:18.253234+0000","last_became_active":"2024-09-13T14:47:07.240038+0000","last_became_peered":"2024-09-13T14:47:07.240038+0000","last_unstale":"2024-09-13T15:04:18.253234+0000","last_undegraded":"2024-09-13T15:04:18.253234+0000","last_fullsized":"2024-09-13T15:04:18.253234+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:22.466357+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.c","version":"398'13157","reported_seq":13201,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.810247+0000","last_change":"2024-09-13T15:00:55.873540+0000","last_active":"2024-09-13T15:04:23.810247+0000","last_peered":"2024-09-13T15:04:23.810247+0000","last_clean":"2024-09-13T15:04:23.810247+0000","last_became_active":"2024-09-13T15:00:55.872763+0000","last_became_peered":"2024-09-13T15:00:55.872763+0000","last_unstale":"2024-09-13T15:04:23.810247+0000","last_undegraded":"2024-09-13T15:04:23.810247+0000","last_fullsized":"2024-09-13T15:04:23.810247+0000","mapping_epoch":362,"log_start":"378'6819","ondisk_log_start":"378'6819","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6338,"log_dups_size":0,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:49.820345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116588544,"num_objects":1779,"num_object_clones":0,"num_object_copies":5337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13157,"num_write_kb":477952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":343,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254275+0000","last_change":"2024-09-13T14:21:53.699371+0000","last_active":"2024-09-13T15:04:18.254275+0000","last_peered":"2024-09-13T15:04:18.254275+0000","last_clean":"2024-09-13T15:04:18.254275+0000","last_became_active":"2024-09-13T14:21:53.699103+0000","last_became_peered":"2024-09-13T14:21:53.699103+0000","last_unstale":"2024-09-13T15:04:18.254275+0000","last_undegraded":"2024-09-13T15:04:18.254275+0000","last_fullsized":"2024-09-13T15:04:18.254275+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:44.175214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257210+0000","last_change":"2024-09-13T15:03:46.875410+0000","last_active":"2024-09-13T15:04:19.257210+0000","last_peered":"2024-09-13T15:04:19.257210+0000","last_clean":"2024-09-13T15:04:19.257210+0000","last_became_active":"2024-09-13T15:03:46.875103+0000","last_became_peered":"2024-09-13T15:03:46.875103+0000","last_unstale":"2024-09-13T15:04:19.257210+0000","last_undegraded":"2024-09-13T15:04:19.257210+0000","last_fullsized":"2024-09-13T15:04:19.257210+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:53.941854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":83,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254444+0000","last_change":"2024-09-13T14:57:53.360296+0000","last_active":"2024-09-13T15:04:18.254444+0000","last_peered":"2024-09-13T15:04:18.254444+0000","last_clean":"2024-09-13T15:04:18.254444+0000","last_became_active":"2024-09-13T14:57:53.347710+0000","last_became_peered":"2024-09-13T14:57:53.347710+0000","last_unstale":"2024-09-13T15:04:18.254444+0000","last_undegraded":"2024-09-13T15:04:18.254444+0000","last_fullsized":"2024-09-13T15:04:18.254444+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:43.099884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":175,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.315711+0000","last_change":"2024-09-13T15:02:19.407022+0000","last_active":"2024-09-13T15:04:18.315711+0000","last_peered":"2024-09-13T15:04:18.315711+0000","last_clean":"2024-09-13T15:04:18.315711+0000","last_became_active":"2024-09-13T15:02:19.406620+0000","last_became_peered":"2024-09-13T15:02:19.406620+0000","last_unstale":"2024-09-13T15:04:18.315711+0000","last_undegraded":"2024-09-13T15:04:18.315711+0000","last_fullsized":"2024-09-13T15:04:18.315711+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:37.100188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"398'12702","reported_seq":12779,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.577921+0000","last_change":"2024-09-13T15:04:17.316056+0000","last_active":"2024-09-13T15:04:23.577921+0000","last_peered":"2024-09-13T15:04:23.577921+0000","last_clean":"2024-09-13T15:04:23.577921+0000","last_became_active":"2024-09-13T15:04:17.282710+0000","last_became_peered":"2024-09-13T15:04:17.282710+0000","last_unstale":"2024-09-13T15:04:23.577921+0000","last_undegraded":"2024-09-13T15:04:23.577921+0000","last_fullsized":"2024-09-13T15:04:23.577921+0000","mapping_epoch":389,"log_start":"378'6664","ondisk_log_start":"378'6664","created":362,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59604992,"num_objects":909,"num_object_clones":0,"num_object_copies":2727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":235424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":335,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.261526+0000","last_change":"2024-09-13T15:03:46.856441+0000","last_active":"2024-09-13T15:04:19.261526+0000","last_peered":"2024-09-13T15:04:19.261526+0000","last_clean":"2024-09-13T15:04:19.261526+0000","last_became_active":"2024-09-13T15:03:46.856111+0000","last_became_peered":"2024-09-13T15:03:46.856111+0000","last_unstale":"2024-09-13T15:04:19.261526+0000","last_undegraded":"2024-09-13T15:04:19.261526+0000","last_fullsized":"2024-09-13T15:04:19.261526+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:44.991274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256469+0000","last_change":"2024-09-13T15:03:46.867861+0000","last_active":"2024-09-13T15:04:19.256469+0000","last_peered":"2024-09-13T15:04:19.256469+0000","last_clean":"2024-09-13T15:04:19.256469+0000","last_became_active":"2024-09-13T15:03:46.867422+0000","last_became_peered":"2024-09-13T15:03:46.867422+0000","last_unstale":"2024-09-13T15:04:19.256469+0000","last_undegraded":"2024-09-13T15:04:19.256469+0000","last_fullsized":"2024-09-13T15:04:19.256469+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:19.172678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":162,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.260696+0000","last_change":"2024-09-13T14:58:28.428608+0000","last_active":"2024-09-13T15:04:19.260696+0000","last_peered":"2024-09-13T15:04:19.260696+0000","last_clean":"2024-09-13T15:04:19.260696+0000","last_became_active":"2024-09-13T14:58:28.428282+0000","last_became_peered":"2024-09-13T14:58:28.428282+0000","last_unstale":"2024-09-13T15:04:19.260696+0000","last_undegraded":"2024-09-13T15:04:19.260696+0000","last_fullsized":"2024-09-13T15:04:19.260696+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.603606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.603606+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:10.695417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.7","version":"398'13053","reported_seq":13108,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.438500+0000","last_change":"2024-09-13T15:04:17.303434+0000","last_active":"2024-09-13T15:04:23.438500+0000","last_peered":"2024-09-13T15:04:23.438500+0000","last_clean":"2024-09-13T15:04:23.438500+0000","last_became_active":"2024-09-13T15:04:17.275877+0000","last_became_peered":"2024-09-13T15:04:17.275877+0000","last_unstale":"2024-09-13T15:04:23.438500+0000","last_undegraded":"2024-09-13T15:04:23.438500+0000","last_fullsized":"2024-09-13T15:04:23.438500+0000","mapping_epoch":362,"log_start":"378'6858","ondisk_log_start":"378'6858","created":362,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6195,"log_dups_size":0,"ondisk_log_size":6195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59244544,"num_objects":904,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6620,"num_write_kb":240768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":380,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256164+0000","last_change":"2024-09-13T14:47:40.131464+0000","last_active":"2024-09-13T15:04:18.256164+0000","last_peered":"2024-09-13T15:04:18.256164+0000","last_clean":"2024-09-13T15:04:18.256164+0000","last_became_active":"2024-09-13T14:47:40.131142+0000","last_became_peered":"2024-09-13T14:47:40.131142+0000","last_unstale":"2024-09-13T15:04:18.256164+0000","last_undegraded":"2024-09-13T15:04:18.256164+0000","last_fullsized":"2024-09-13T15:04:18.256164+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:13.318614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":361,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.255515+0000","last_change":"2024-09-13T14:35:30.478348+0000","last_active":"2024-09-13T15:04:18.255515+0000","last_peered":"2024-09-13T15:04:18.255515+0000","last_clean":"2024-09-13T15:04:18.255515+0000","last_became_active":"2024-09-13T14:35:30.477914+0000","last_became_peered":"2024-09-13T14:35:30.477914+0000","last_unstale":"2024-09-13T15:04:18.255515+0000","last_undegraded":"2024-09-13T15:04:18.255515+0000","last_fullsized":"2024-09-13T15:04:18.255515+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:13.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":73,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.255098+0000","last_change":"2024-09-13T15:02:36.128841+0000","last_active":"2024-09-13T15:04:19.255098+0000","last_peered":"2024-09-13T15:04:19.255098+0000","last_clean":"2024-09-13T15:04:19.255098+0000","last_became_active":"2024-09-13T15:02:19.422885+0000","last_became_peered":"2024-09-13T15:02:19.422885+0000","last_unstale":"2024-09-13T15:04:19.255098+0000","last_undegraded":"2024-09-13T15:04:19.255098+0000","last_fullsized":"2024-09-13T15:04:19.255098+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:02:36.128767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:02:36.128767+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:32.092793+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.4","version":"398'13099","reported_seq":13198,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.512634+0000","last_change":"2024-09-13T15:04:22.527133+0000","last_active":"2024-09-13T15:04:23.512634+0000","last_peered":"2024-09-13T15:04:23.512634+0000","last_clean":"2024-09-13T15:04:23.512634+0000","last_became_active":"2024-09-13T15:04:17.279773+0000","last_became_peered":"2024-09-13T15:04:17.279773+0000","last_unstale":"2024-09-13T15:04:23.512634+0000","last_undegraded":"2024-09-13T15:04:23.512634+0000","last_fullsized":"2024-09-13T15:04:23.512634+0000","mapping_epoch":362,"log_start":"374'6476","ondisk_log_start":"374'6476","created":362,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'13053","last_scrub_stamp":"2024-09-13T15:04:22.527048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:04:22.527048+0000","objects_scrubbed":896,"log_size":6623,"log_dups_size":0,"ondisk_log_size":6623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:30.566837+0000","scrub_duration":0.28104013,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55312384,"num_objects":844,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":377,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.257253+0000","last_change":"2024-09-13T14:35:30.475193+0000","last_active":"2024-09-13T15:04:18.257253+0000","last_peered":"2024-09-13T15:04:18.257253+0000","last_clean":"2024-09-13T15:04:18.257253+0000","last_became_active":"2024-09-13T14:35:30.474852+0000","last_became_peered":"2024-09-13T14:35:30.474852+0000","last_unstale":"2024-09-13T15:04:18.257253+0000","last_undegraded":"2024-09-13T15:04:18.257253+0000","last_fullsized":"2024-09-13T15:04:18.257253+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:32.509716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":381,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.253871+0000","last_change":"2024-09-13T14:47:40.129859+0000","last_active":"2024-09-13T15:04:18.253871+0000","last_peered":"2024-09-13T15:04:18.253871+0000","last_clean":"2024-09-13T15:04:18.253871+0000","last_became_active":"2024-09-13T14:47:40.128675+0000","last_became_peered":"2024-09-13T14:47:40.128675+0000","last_unstale":"2024-09-13T15:04:18.253871+0000","last_undegraded":"2024-09-13T15:04:18.253871+0000","last_fullsized":"2024-09-13T15:04:18.253871+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:18.840408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":231,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.261046+0000","last_change":"2024-09-13T15:03:46.869405+0000","last_active":"2024-09-13T15:04:19.261046+0000","last_peered":"2024-09-13T15:04:19.261046+0000","last_clean":"2024-09-13T15:04:19.261046+0000","last_became_active":"2024-09-13T15:03:46.868913+0000","last_became_peered":"2024-09-13T15:03:46.868913+0000","last_unstale":"2024-09-13T15:04:19.261046+0000","last_undegraded":"2024-09-13T15:04:19.261046+0000","last_fullsized":"2024-09-13T15:04:19.261046+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:02.098981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"398'12852","reported_seq":12989,"reported_epoch":398,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:04:23.550536+0000","last_change":"2024-09-13T15:04:17.313440+0000","last_active":"2024-09-13T15:04:23.550536+0000","last_peered":"2024-09-13T15:04:23.550536+0000","last_clean":"2024-09-13T15:02:18.352546+0000","last_became_active":"2024-09-13T15:04:17.279673+0000","last_became_peered":"2024-09-13T15:04:17.279673+0000","last_unstale":"2024-09-13T15:04:23.550536+0000","last_undegraded":"2024-09-13T15:04:23.550536+0000","last_fullsized":"2024-09-13T15:04:23.550536+0000","mapping_epoch":376,"log_start":"378'6832","ondisk_log_start":"378'6832","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6020,"log_dups_size":0,"ondisk_log_size":6020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57376768,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":871,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6526,"num_write_kb":236864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,5,3],"avail_no_missing":["4","3","5"],"object_location_counts":[{"shards":"3,4,5","objects":875}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":380,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.255342+0000","last_change":"2024-09-13T15:03:46.866312+0000","last_active":"2024-09-13T15:04:18.255342+0000","last_peered":"2024-09-13T15:04:18.255342+0000","last_clean":"2024-09-13T15:04:18.255342+0000","last_became_active":"2024-09-13T15:03:46.866038+0000","last_became_peered":"2024-09-13T15:03:46.866038+0000","last_unstale":"2024-09-13T15:04:18.255342+0000","last_undegraded":"2024-09-13T15:04:18.255342+0000","last_fullsized":"2024-09-13T15:04:18.255342+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:05.770254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":32,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257507+0000","last_change":"2024-09-13T15:03:46.881258+0000","last_active":"2024-09-13T15:04:19.257507+0000","last_peered":"2024-09-13T15:04:19.257507+0000","last_clean":"2024-09-13T15:04:19.257507+0000","last_became_active":"2024-09-13T15:03:46.866883+0000","last_became_peered":"2024-09-13T15:03:46.866883+0000","last_unstale":"2024-09-13T15:04:19.257507+0000","last_undegraded":"2024-09-13T15:04:19.257507+0000","last_fullsized":"2024-09-13T15:04:19.257507+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:12.364139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":221,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.315676+0000","last_change":"2024-09-13T15:03:46.863264+0000","last_active":"2024-09-13T15:04:18.315676+0000","last_peered":"2024-09-13T15:04:18.315676+0000","last_clean":"2024-09-13T15:04:18.315676+0000","last_became_active":"2024-09-13T15:03:46.862850+0000","last_became_peered":"2024-09-13T15:03:46.862850+0000","last_unstale":"2024-09-13T15:04:18.315676+0000","last_undegraded":"2024-09-13T15:04:18.315676+0000","last_fullsized":"2024-09-13T15:04:18.315676+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:57.707268+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2","version":"398'12841","reported_seq":12961,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:22.825171+0000","last_change":"2024-09-13T15:04:17.650546+0000","last_active":"2024-09-13T15:04:22.825171+0000","last_peered":"2024-09-13T15:04:22.825171+0000","last_clean":"2024-09-13T15:04:22.825171+0000","last_became_active":"2024-09-13T15:04:17.293048+0000","last_became_peered":"2024-09-13T15:04:17.293048+0000","last_unstale":"2024-09-13T15:04:22.825171+0000","last_undegraded":"2024-09-13T15:04:22.825171+0000","last_fullsized":"2024-09-13T15:04:22.825171+0000","mapping_epoch":390,"log_start":"377'6526","ondisk_log_start":"377'6526","created":362,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'12688","last_scrub_stamp":"2024-09-13T15:04:17.650389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:04:17.650389+0000","objects_scrubbed":1068,"log_size":6315,"log_dups_size":0,"ondisk_log_size":6315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:35.619423+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59965440,"num_objects":915,"num_object_clones":0,"num_object_copies":2745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":56,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.255196+0000","last_change":"2024-09-13T15:02:19.417296+0000","last_active":"2024-09-13T15:04:19.255196+0000","last_peered":"2024-09-13T15:04:19.255196+0000","last_clean":"2024-09-13T15:04:19.255196+0000","last_became_active":"2024-09-13T15:02:19.416930+0000","last_became_peered":"2024-09-13T15:02:19.416930+0000","last_unstale":"2024-09-13T15:04:19.255196+0000","last_undegraded":"2024-09-13T15:04:19.255196+0000","last_fullsized":"2024-09-13T15:04:19.255196+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:18.684193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257543+0000","last_change":"2024-09-13T15:03:46.869406+0000","last_active":"2024-09-13T15:04:19.257543+0000","last_peered":"2024-09-13T15:04:19.257543+0000","last_clean":"2024-09-13T15:04:19.257543+0000","last_became_active":"2024-09-13T15:03:46.866230+0000","last_became_peered":"2024-09-13T15:03:46.866230+0000","last_unstale":"2024-09-13T15:04:19.257543+0000","last_undegraded":"2024-09-13T15:04:19.257543+0000","last_fullsized":"2024-09-13T15:04:19.257543+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:40.914289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":221,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.255993+0000","last_change":"2024-09-13T15:03:46.855303+0000","last_active":"2024-09-13T15:04:18.255993+0000","last_peered":"2024-09-13T15:04:18.255993+0000","last_clean":"2024-09-13T15:04:18.255993+0000","last_became_active":"2024-09-13T15:03:46.854880+0000","last_became_peered":"2024-09-13T15:03:46.854880+0000","last_unstale":"2024-09-13T15:04:18.255993+0000","last_undegraded":"2024-09-13T15:04:18.255993+0000","last_fullsized":"2024-09-13T15:04:18.255993+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:00.146384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:00.146384+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:30.662335+0000","scrub_duration":0.0014649660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"398'12732","reported_seq":12839,"reported_epoch":398,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:04:22.777835+0000","last_change":"2024-09-13T15:04:17.354759+0000","last_active":"2024-09-13T15:04:22.777835+0000","last_peered":"2024-09-13T15:04:22.777835+0000","last_clean":"2024-09-13T15:02:18.361205+0000","last_became_active":"2024-09-13T15:04:17.293444+0000","last_became_peered":"2024-09-13T15:04:17.293444+0000","last_unstale":"2024-09-13T15:04:22.777835+0000","last_undegraded":"2024-09-13T15:04:22.777835+0000","last_fullsized":"2024-09-13T15:04:22.777835+0000","mapping_epoch":389,"log_start":"374'6347","ondisk_log_start":"374'6347","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6385,"log_dups_size":0,"ondisk_log_size":6385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046784,"num_objects":931,"num_object_clones":0,"num_object_copies":2793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":926,"num_objects_unfound":0,"num_objects_dirty":931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":236320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[3,4,5],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":931}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":383,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.253790+0000","last_change":"2024-09-13T14:57:35.357982+0000","last_active":"2024-09-13T15:04:18.253790+0000","last_peered":"2024-09-13T15:04:18.253790+0000","last_clean":"2024-09-13T15:04:18.253790+0000","last_became_active":"2024-09-13T14:57:35.357618+0000","last_became_peered":"2024-09-13T14:57:35.357618+0000","last_unstale":"2024-09-13T15:04:18.253790+0000","last_undegraded":"2024-09-13T15:04:18.253790+0000","last_fullsized":"2024-09-13T15:04:18.253790+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:50.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":402,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.316220+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:04:18.316220+0000","last_peered":"2024-09-13T15:04:18.316220+0000","last_clean":"2024-09-13T15:04:18.316220+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:04:18.316220+0000","last_undegraded":"2024-09-13T15:04:18.316220+0000","last_fullsized":"2024-09-13T15:04:18.316220+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":354,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254920+0000","last_change":"2024-09-13T14:57:53.363216+0000","last_active":"2024-09-13T15:04:18.254920+0000","last_peered":"2024-09-13T15:04:18.254920+0000","last_clean":"2024-09-13T15:04:18.254920+0000","last_became_active":"2024-09-13T14:57:53.362679+0000","last_became_peered":"2024-09-13T14:57:53.362679+0000","last_unstale":"2024-09-13T15:04:18.254920+0000","last_undegraded":"2024-09-13T15:04:18.254920+0000","last_fullsized":"2024-09-13T15:04:18.254920+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:48.885757+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"398'12946","reported_seq":13052,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.502985+0000","last_change":"2024-09-13T15:04:18.879969+0000","last_active":"2024-09-13T15:04:23.502985+0000","last_peered":"2024-09-13T15:04:23.502985+0000","last_clean":"2024-09-13T15:04:23.502985+0000","last_became_active":"2024-09-13T15:04:17.301843+0000","last_became_peered":"2024-09-13T15:04:17.301843+0000","last_unstale":"2024-09-13T15:04:23.502985+0000","last_undegraded":"2024-09-13T15:04:23.502985+0000","last_fullsized":"2024-09-13T15:04:23.502985+0000","mapping_epoch":362,"log_start":"378'6699","ondisk_log_start":"378'6699","created":362,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'12773","last_scrub_stamp":"2024-09-13T15:04:18.879902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:04:18.879902+0000","objects_scrubbed":1073,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:03.042547+0000","scrub_duration":0.109015899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58982400,"num_objects":900,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":173,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":407,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.255428+0000","last_change":"2024-09-13T15:03:46.866745+0000","last_active":"2024-09-13T15:04:18.255428+0000","last_peered":"2024-09-13T15:04:18.255428+0000","last_clean":"2024-09-13T15:04:18.255428+0000","last_became_active":"2024-09-13T15:03:46.866320+0000","last_became_peered":"2024-09-13T15:03:46.866320+0000","last_unstale":"2024-09-13T15:04:18.255428+0000","last_undegraded":"2024-09-13T15:04:18.255428+0000","last_fullsized":"2024-09-13T15:04:18.255428+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:36.155505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":364,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.316340+0000","last_change":"2024-09-13T14:32:09.482884+0000","last_active":"2024-09-13T15:04:18.316340+0000","last_peered":"2024-09-13T15:04:18.316340+0000","last_clean":"2024-09-13T15:04:18.316340+0000","last_became_active":"2024-09-13T14:32:09.482034+0000","last_became_peered":"2024-09-13T14:32:09.482034+0000","last_unstale":"2024-09-13T15:04:18.316340+0000","last_undegraded":"2024-09-13T15:04:18.316340+0000","last_fullsized":"2024-09-13T15:04:18.316340+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:30.197198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":32,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257820+0000","last_change":"2024-09-13T15:03:46.882026+0000","last_active":"2024-09-13T15:04:19.257820+0000","last_peered":"2024-09-13T15:04:19.257820+0000","last_clean":"2024-09-13T15:04:19.257820+0000","last_became_active":"2024-09-13T15:03:46.859557+0000","last_became_peered":"2024-09-13T15:03:46.859557+0000","last_unstale":"2024-09-13T15:04:19.257820+0000","last_undegraded":"2024-09-13T15:04:19.257820+0000","last_fullsized":"2024-09-13T15:04:19.257820+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:00.933755+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"398'12846","reported_seq":12902,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:23.494359+0000","last_change":"2024-09-13T15:04:19.252132+0000","last_active":"2024-09-13T15:04:23.494359+0000","last_peered":"2024-09-13T15:04:23.494359+0000","last_clean":"2024-09-13T15:04:23.494359+0000","last_became_active":"2024-09-13T15:04:17.318935+0000","last_became_peered":"2024-09-13T15:04:17.318935+0000","last_unstale":"2024-09-13T15:04:23.494359+0000","last_undegraded":"2024-09-13T15:04:23.494359+0000","last_fullsized":"2024-09-13T15:04:23.494359+0000","mapping_epoch":362,"log_start":"374'6229","ondisk_log_start":"374'6229","created":362,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:54.388834+0000","last_clean_scrub_stamp":"2024-09-13T15:00:54.388834+0000","objects_scrubbed":0,"log_size":6617,"log_dups_size":0,"ondisk_log_size":6617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58556416,"num_objects":893,"num_object_clones":0,"num_object_copies":2679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6514,"num_write_kb":237056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":79,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.256793+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:04:18.256793+0000","last_peered":"2024-09-13T15:04:18.256793+0000","last_clean":"2024-09-13T15:04:18.256793+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:04:18.256793+0000","last_undegraded":"2024-09-13T15:04:18.256793+0000","last_fullsized":"2024-09-13T15:04:18.256793+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":187,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.316369+0000","last_change":"2024-09-13T15:02:19.430869+0000","last_active":"2024-09-13T15:04:18.316369+0000","last_peered":"2024-09-13T15:04:18.316369+0000","last_clean":"2024-09-13T15:04:18.316369+0000","last_became_active":"2024-09-13T15:02:19.417063+0000","last_became_peered":"2024-09-13T15:02:19.417063+0000","last_unstale":"2024-09-13T15:04:18.316369+0000","last_undegraded":"2024-09-13T15:04:18.316369+0000","last_fullsized":"2024-09-13T15:04:18.316369+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:25.391998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":185,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.316025+0000","last_change":"2024-09-13T14:57:55.208230+0000","last_active":"2024-09-13T15:04:18.316025+0000","last_peered":"2024-09-13T15:04:18.316025+0000","last_clean":"2024-09-13T15:04:18.316025+0000","last_became_active":"2024-09-13T14:57:19.211671+0000","last_became_peered":"2024-09-13T14:57:19.211671+0000","last_unstale":"2024-09-13T15:04:18.316025+0000","last_undegraded":"2024-09-13T15:04:18.316025+0000","last_fullsized":"2024-09-13T15:04:18.316025+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.066938+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":489,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.254480+0000","last_change":"2024-09-13T15:02:19.426511+0000","last_active":"2024-09-13T15:04:18.254480+0000","last_peered":"2024-09-13T15:04:18.254480+0000","last_clean":"2024-09-13T15:04:18.254480+0000","last_became_active":"2024-09-13T15:02:19.391448+0000","last_became_peered":"2024-09-13T15:02:19.391448+0000","last_unstale":"2024-09-13T15:04:18.254480+0000","last_undegraded":"2024-09-13T15:04:18.254480+0000","last_fullsized":"2024-09-13T15:04:18.254480+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:40.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.258619+0000","last_change":"2024-09-13T15:03:46.870536+0000","last_active":"2024-09-13T15:04:19.258619+0000","last_peered":"2024-09-13T15:04:19.258619+0000","last_clean":"2024-09-13T15:04:19.258619+0000","last_became_active":"2024-09-13T15:03:46.869736+0000","last_became_peered":"2024-09-13T15:03:46.869736+0000","last_unstale":"2024-09-13T15:04:19.258619+0000","last_undegraded":"2024-09-13T15:04:19.258619+0000","last_fullsized":"2024-09-13T15:04:19.258619+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:45.432825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":79,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.253491+0000","last_change":"2024-09-13T14:57:53.356010+0000","last_active":"2024-09-13T15:04:18.253491+0000","last_peered":"2024-09-13T15:04:18.253491+0000","last_clean":"2024-09-13T15:04:18.253491+0000","last_became_active":"2024-09-13T14:57:53.354979+0000","last_became_peered":"2024-09-13T14:57:53.354979+0000","last_unstale":"2024-09-13T15:04:18.253491+0000","last_undegraded":"2024-09-13T15:04:18.253491+0000","last_fullsized":"2024-09-13T15:04:18.253491+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:38.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":365,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.315769+0000","last_change":"2024-09-13T14:57:35.358990+0000","last_active":"2024-09-13T15:04:18.315769+0000","last_peered":"2024-09-13T15:04:18.315769+0000","last_clean":"2024-09-13T15:04:18.315769+0000","last_became_active":"2024-09-13T14:57:35.358547+0000","last_became_peered":"2024-09-13T14:57:35.358547+0000","last_unstale":"2024-09-13T15:04:18.315769+0000","last_undegraded":"2024-09-13T15:04:18.315769+0000","last_fullsized":"2024-09-13T15:04:18.315769+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:45.386301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":70,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.256227+0000","last_change":"2024-09-13T15:03:46.864291+0000","last_active":"2024-09-13T15:04:19.256227+0000","last_peered":"2024-09-13T15:04:19.256227+0000","last_clean":"2024-09-13T15:04:19.256227+0000","last_became_active":"2024-09-13T15:03:46.863671+0000","last_became_peered":"2024-09-13T15:03:46.863671+0000","last_unstale":"2024-09-13T15:04:19.256227+0000","last_undegraded":"2024-09-13T15:04:19.256227+0000","last_fullsized":"2024-09-13T15:04:19.256227+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:28.672041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257408+0000","last_change":"2024-09-13T15:03:46.867845+0000","last_active":"2024-09-13T15:04:19.257408+0000","last_peered":"2024-09-13T15:04:19.257408+0000","last_clean":"2024-09-13T15:04:19.257408+0000","last_became_active":"2024-09-13T15:03:46.867336+0000","last_became_peered":"2024-09-13T15:03:46.867336+0000","last_unstale":"2024-09-13T15:04:19.257408+0000","last_undegraded":"2024-09-13T15:04:19.257408+0000","last_fullsized":"2024-09-13T15:04:19.257408+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:46.862346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":70,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.254531+0000","last_change":"2024-09-13T15:03:46.863848+0000","last_active":"2024-09-13T15:04:19.254531+0000","last_peered":"2024-09-13T15:04:19.254531+0000","last_clean":"2024-09-13T15:04:19.254531+0000","last_became_active":"2024-09-13T15:03:46.863523+0000","last_became_peered":"2024-09-13T15:03:46.863523+0000","last_unstale":"2024-09-13T15:04:19.254531+0000","last_undegraded":"2024-09-13T15:04:19.254531+0000","last_fullsized":"2024-09-13T15:04:19.254531+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:39.303427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":276,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.316106+0000","last_change":"2024-09-13T15:03:46.860138+0000","last_active":"2024-09-13T15:04:18.316106+0000","last_peered":"2024-09-13T15:04:18.316106+0000","last_clean":"2024-09-13T15:04:18.316106+0000","last_became_active":"2024-09-13T15:03:46.859819+0000","last_became_peered":"2024-09-13T15:03:46.859819+0000","last_unstale":"2024-09-13T15:04:18.316106+0000","last_undegraded":"2024-09-13T15:04:18.316106+0000","last_fullsized":"2024-09-13T15:04:18.316106+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:28.081805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":87,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.315763+0000","last_change":"2024-09-13T14:58:28.428896+0000","last_active":"2024-09-13T15:04:18.315763+0000","last_peered":"2024-09-13T15:04:18.315763+0000","last_clean":"2024-09-13T15:04:18.315763+0000","last_became_active":"2024-09-13T14:58:28.428522+0000","last_became_peered":"2024-09-13T14:58:28.428522+0000","last_unstale":"2024-09-13T15:04:18.315763+0000","last_undegraded":"2024-09-13T15:04:18.315763+0000","last_fullsized":"2024-09-13T15:04:18.315763+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:05.418586+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":380,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-13T15:04:18.315907+0000","last_change":"2024-09-13T14:48:29.177122+0000","last_active":"2024-09-13T15:04:18.315907+0000","last_peered":"2024-09-13T15:04:18.315907+0000","last_clean":"2024-09-13T15:04:18.315907+0000","last_became_active":"2024-09-13T14:48:29.129129+0000","last_became_peered":"2024-09-13T14:48:29.129129+0000","last_unstale":"2024-09-13T15:04:18.315907+0000","last_undegraded":"2024-09-13T15:04:18.315907+0000","last_fullsized":"2024-09-13T15:04:18.315907+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:06.917001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.257937+0000","last_change":"2024-09-13T15:03:46.871926+0000","last_active":"2024-09-13T15:04:19.257937+0000","last_peered":"2024-09-13T15:04:19.257937+0000","last_clean":"2024-09-13T15:04:19.257937+0000","last_became_active":"2024-09-13T15:03:46.868103+0000","last_became_peered":"2024-09-13T15:03:46.868103+0000","last_unstale":"2024-09-13T15:04:19.257937+0000","last_undegraded":"2024-09-13T15:04:19.257937+0000","last_fullsized":"2024-09-13T15:04:19.257937+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:43.957131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":70,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-13T15:04:19.254882+0000","last_change":"2024-09-13T15:03:46.864604+0000","last_active":"2024-09-13T15:04:19.254882+0000","last_peered":"2024-09-13T15:04:19.254882+0000","last_clean":"2024-09-13T15:04:19.254882+0000","last_became_active":"2024-09-13T15:03:46.864085+0000","last_became_peered":"2024-09-13T15:03:46.864085+0000","last_unstale":"2024-09-13T15:04:19.254882+0000","last_undegraded":"2024-09-13T15:04:19.254882+0000","last_fullsized":"2024-09-13T15:04:19.254882+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:32.524052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":26,"stat_sum":{"num_bytes":1908310040,"num_objects":29120,"num_object_clones":0,"num_object_copies":87360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":3609,"num_objects_unfound":0,"num_objects_dirty":29120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170938,"num_write_kb":6201313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8546,"num_bytes_recovered":540868608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6404714496,"data_stored":6404702280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":168183,"ondisk_log_size":168183,"up":78,"acting":76,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":1220,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365336,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2110772,"kb_used_data":601388,"kb_used_omap":1,"kb_used_meta":1509310,"kb_avail":91634380,"statfs":{"total":95995035648,"available":93833605120,"internally_reserved":0,"allocated":615821312,"data_stored":613408975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1545533784},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561566,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2503972,"kb_used_data":1003420,"kb_used_omap":1,"kb_used_meta":1500542,"kb_avail":91241180,"statfs":{"total":95995035648,"available":93430968320,"internally_reserved":0,"allocated":1027502080,"data_stored":1025105382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1536555352},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790498,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2556892,"kb_used_data":1062100,"kb_used_omap":1,"kb_used_meta":1494718,"kb_avail":91188260,"statfs":{"total":95995035648,"available":93376778240,"internally_reserved":0,"allocated":1087590400,"data_stored":1085198012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1530591576},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986726,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2802412,"kb_used_data":1308324,"kb_used_omap":1,"kb_used_meta":1493950,"kb_avail":90942740,"statfs":{"total":95995035648,"available":93125365760,"internally_reserved":0,"allocated":1339723776,"data_stored":1337319248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1529805144},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519881,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1619544,"kb_used_data":602192,"kb_used_omap":4,"kb_used_meta":1017339,"kb_avail":92125608,"statfs":{"total":95995035648,"available":94336622592,"internally_reserved":0,"allocated":616644608,"data_stored":614255782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":1041755584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304796,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2217440,"kb_used_data":1144600,"kb_used_omap":6,"kb_used_meta":1072825,"kb_avail":91527712,"statfs":{"total":95995035648,"available":93724377088,"internally_reserved":0,"allocated":1172070400,"data_stored":1169674440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":1098573255},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828100,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1856992,"kb_used_data":620952,"kb_used_omap":5,"kb_used_meta":1236026,"kb_avail":91888160,"statfs":{"total":95995035648,"available":94093475840,"internally_reserved":0,"allocated":635854848,"data_stored":633457825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1265690860},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482285965,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180248,"kb_used_data":97168,"kb_used_omap":5,"kb_used_meta":1083066,"kb_avail":92564904,"statfs":{"total":95995035648,"available":94786461696,"internally_reserved":0,"allocated":99500032,"data_stored":97110480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":1109059643},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93585408,"data_stored":93585408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":618074112,"data_stored":618070040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1152450560,"data_stored":1152450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":603783168,"data_stored":603783168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1295060992,"data_stored":1295056920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1035468800,"data_stored":1035468800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1014628352,"data_stored":1014628352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":591663104,"data_stored":591659032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:04:29.858 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T15:04:29.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_8 pg_num 36 2024-09-13T15:04:29.928 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:29.967 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:30.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:30.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:30.169 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:04:30.170 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:30.225 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:04:30.226 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:30.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:30.262 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:04:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:30.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:30.297 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:04:30.297 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:30.337 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:04:30.337 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:30.366 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:04:30.366 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:30.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:30.397 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:04:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:30.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:30 smithi003 ceph-mon[107278]: 13.7 deep-scrub starts 2024-09-13T15:04:30.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:30 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/188498082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:30 smithi003 ceph-mon[104007]: 13.7 deep-scrub starts 2024-09-13T15:04:30.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:30 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/188498082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:30.437 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:04:30.437 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:30.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:30.569 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:30.671 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:30 smithi019 ceph-mon[91296]: 13.7 deep-scrub starts 2024-09-13T15:04:30.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:30 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/188498082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:04:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:04:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:30.705 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:30.808 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.845 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:30.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:30.946 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:30.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:31.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:31.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:31.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:31.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:31.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:31.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:31.504 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:31.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:31.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:31.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:31 smithi003 ceph-mon[107278]: pgmap v1435: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2073/67314 objects degraded (3.080%); 2751/67314 objects misplaced (4.087%); 1.5 MiB/s, 103 objects/s recovering 2024-09-13T15:04:31.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2073/67314 objects degraded (3.080%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:31.615 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:31 smithi003 ceph-mon[107278]: 13.11 scrub starts 2024-09-13T15:04:31.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:31 smithi003 ceph-mon[104007]: pgmap v1435: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2073/67314 objects degraded (3.080%); 2751/67314 objects misplaced (4.087%); 1.5 MiB/s, 103 objects/s recovering 2024-09-13T15:04:31.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2073/67314 objects degraded (3.080%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:31.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:31 smithi003 ceph-mon[104007]: 13.11 scrub starts 2024-09-13T15:04:31.674 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:31 smithi019 ceph-mon[91296]: pgmap v1435: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2073/67314 objects degraded (3.080%); 2751/67314 objects misplaced (4.087%); 1.5 MiB/s, 103 objects/s recovering 2024-09-13T15:04:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2073/67314 objects degraded (3.080%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:31 smithi019 ceph-mon[91296]: 13.11 scrub starts 2024-09-13T15:04:31.714 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:31.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:31.816 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:31.945 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:32.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:32.081 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:32.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:32.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:32.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:32.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:32.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:32.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:32.473 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:32.610 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:32 smithi003 ceph-mon[107278]: 13.1 scrub starts 2024-09-13T15:04:32.611 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:32 smithi003 ceph-mon[104007]: 13.1 scrub starts 2024-09-13T15:04:32.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:32.662 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:32 smithi019 ceph-mon[91296]: 13.1 scrub starts 2024-09-13T15:04:32.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:32.800 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:32.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:32.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:33.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:33.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:33.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:33.215 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:33.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:33.329 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:04:33.329 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129107 12.a 12:51ee7cd8:::benchmark_data_smithi003_259359_object13481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.531378+0000", 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 383.79302469499999, 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3208663619999998, 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:04:33.330 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129107 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.531378+0000", 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.531378+0000", 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.531379+0000", 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6020000000000001e-06 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.531385+0000", 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3129999999999998e-06 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.531398+0000", 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2968e-05 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.531406+0000", 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0870000000000007e-06 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.531478+0000", 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2768999999999994e-05 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.531486+0000", 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2420000000000003e-06 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.500168+0000", 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96868247399999996 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.500181+0000", 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3016e-05 2024-09-13T15:04:33.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.771350+0000", 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2711686549999999 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.771504+0000", 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000154556 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:04:33.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.772088+0000", 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058377400000000001 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.812285+0000", 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040197322000000001 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.812465+0000", 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000179075 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.848000+0000", 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035535658999999997 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.848065+0000", 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4663000000000001e-05 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.852244+0000", 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004179187 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129110 12.a 12:57559ede:::benchmark_data_smithi003_259359_object13484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.532585+0000", 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 383.79181784799999, 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3243798290000002, 2024-09-13T15:04:33.338 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129110 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.532585+0000", 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.532585+0000", 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.532585+0000", 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6499999999999998e-07 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.532590+0000", 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5349999999999998e-06 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.532604+0000", 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4038e-05 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.532611+0000", 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0099999999999998e-06 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.532655+0000", 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3832000000000003e-05 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.532666+0000", 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1328000000000001e-05 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.500219+0000", 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96755267899999997 2024-09-13T15:04:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.500228+0000", 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7199999999999995e-06 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.790308+0000", 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2900804350000001 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.790449+0000", 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014102200000000001 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.790602+0000", 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015313499999999999 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.812333+0000", 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021730930999999998 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.812471+0000", 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013808700000000001 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.852309+0000", 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039837757000000001 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.852371+0000", 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1650999999999996e-05 2024-09-13T15:04:33.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.856964+0000", 2024-09-13T15:04:33.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045938039999999999 2024-09-13T15:04:33.346 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.346 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.346 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.347 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127130 13.b 13:d90b32dc:::benchmark_data_smithi003_293363_object7918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:04:33.347 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.241318+0000", 2024-09-13T15:04:33.347 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 105.083084824, 2024-09-13T15:04:33.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0674569620000001, 2024-09-13T15:04:33.348 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.348 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.348 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:33.348 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.348 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.348 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127130 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241318+0000", 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241318+0000", 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241320+0000", 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0640000000000001e-06 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241326+0000", 2024-09-13T15:04:33.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.754e-06 2024-09-13T15:04:33.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241340+0000", 2024-09-13T15:04:33.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3370000000000001e-05 2024-09-13T15:04:33.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241350+0000", 2024-09-13T15:04:33.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0139000000000001e-05 2024-09-13T15:04:33.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.242933+0000", 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015835319999999999 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.242944+0000", 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1032999999999999e-05 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.197291+0000", 2024-09-13T15:04:33.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.954346743 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.197302+0000", 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0978999999999999e-05 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.289014+0000", 2024-09-13T15:04:33.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.091712282 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.290488+0000", 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001473591 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4,5", 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.290633+0000", 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014513300000000001 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.293633+0000", 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00299957 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297084+0000", 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034512610000000002 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297110+0000", 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5732999999999999e-05 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.308591+0000", 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011481460000000001 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.308652+0000", 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1274999999999999e-05 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.308775+0000", 2024-09-13T15:04:33.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122043 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127131 13.b 13:d278da57:::benchmark_data_smithi003_293363_object7919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.243218+0000", 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 105.08118466499999, 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0678335479999999, 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.358 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127131 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243218+0000", 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243218+0000", 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243219+0000", 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.426e-06 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243234+0000", 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4453999999999999e-05 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243246+0000", 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1983e-05 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243254+0000", 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4740000000000005e-06 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243309+0000", 2024-09-13T15:04:33.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5216999999999997e-05 2024-09-13T15:04:33.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243334+0000", 2024-09-13T15:04:33.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4329999999999999e-05 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.197325+0000", 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.953991547 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.197333+0000", 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5920000000000003e-06 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.293680+0000", 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0963472320000001 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.293777+0000", 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7516000000000006e-05 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4,5", 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.293867+0000", 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9382000000000006e-05 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297117+0000", 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032505009999999998 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297305+0000", 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018791000000000001 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297361+0000", 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.613e-05 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.310877+0000", 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013515160999999999 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.310939+0000", 2024-09-13T15:04:33.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2268999999999996e-05 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.311051+0000", 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011242400000000001 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176522 13.b 13:dd5b9f0f:::benchmark_data_smithi003_293363_object57310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.828681+0000", 2024-09-13T15:04:33.367 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.495720945000002, 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0237406389999999, 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176522 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828681+0000", 2024-09-13T15:04:33.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828681+0000", 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828682+0000", 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9200000000000001e-07 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828684+0000", 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2639999999999998e-06 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828691+0000", 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.545e-06 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828696+0000", 2024-09-13T15:04:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9219999999999996e-06 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828744+0000", 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8066999999999999e-05 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828873+0000", 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000129395 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4,5", 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828994+0000", 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121124 2024-09-13T15:04:33.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.836675+0000", 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076807539999999997 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.840494+0000", 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038188470000000002 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.841971+0000", 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014773620000000001 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.869201+0000", 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027230061999999999 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.869211+0000", 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5140000000000007e-06 2024-09-13T15:04:33.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852408+0000", 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98319757399999996 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852422+0000", 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3716999999999999e-05 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176536 13.b 13:ddb01560:::benchmark_data_smithi003_293363_object57324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.855022+0000", 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.469380354000002, 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99743847200000002, 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176536 2024-09-13T15:04:33.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855022+0000", 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855022+0000", 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855023+0000", 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.252e-06 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855029+0000", 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3109999999999996e-06 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855042+0000", 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3067999999999999e-05 2024-09-13T15:04:33.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855050+0000", 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3010000000000007e-06 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855254+0000", 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020365800000000001 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:33.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855386+0000", 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000132547 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4,5", 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855517+0000", 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131326 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.871590+0000", 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016072521999999999 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.871619+0000", 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9026999999999998e-05 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852451+0000", 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.980831699 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852460+0000", 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7610000000000002e-06 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176559 13.b 13.8239baab (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.878857+0000", 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.445545815000003, 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97363370699999996, 2024-09-13T15:04:33.381 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176559 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878857+0000", 2024-09-13T15:04:33.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878857+0000", 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878858+0000", 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.359e-06 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878864+0000", 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6620000000000002e-06 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878873+0000", 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8849999999999994e-06 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878880+0000", 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3169999999999998e-06 2024-09-13T15:04:33.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878939+0000", 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9685000000000002e-05 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878947+0000", 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9980000000000003e-06 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852482+0000", 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97353494100000004 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852490+0000", 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8599999999999993e-06 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176618 13.b 13.834c39db (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.954483+0000", 2024-09-13T15:04:33.386 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.369919541000002, 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.898035413, 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176618 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954483+0000", 2024-09-13T15:04:33.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954483+0000", 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954484+0000", 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.64e-06 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954490+0000", 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4820000000000002e-06 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954507+0000", 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7000999999999999e-05 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954516+0000", 2024-09-13T15:04:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9570000000000008e-06 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954585+0000", 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8709999999999998e-05 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954595+0000", 2024-09-13T15:04:33.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9420000000000007e-06 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852510+0000", 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89791587799999995 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852518+0000", 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8029999999999997e-06 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176637 13.b 13.fc35e09b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.976426+0000", 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.347976451999997, 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87612334999999997, 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.392 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176637 2024-09-13T15:04:33.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.393 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976426+0000", 2024-09-13T15:04:33.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976426+0000", 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976428+0000", 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.762e-06 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976434+0000", 2024-09-13T15:04:33.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1709999999999999e-06 2024-09-13T15:04:33.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976448+0000", 2024-09-13T15:04:33.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4541e-05 2024-09-13T15:04:33.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976457+0000", 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3729999999999993e-06 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976534+0000", 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7105000000000001e-05 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976544+0000", 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0036e-05 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852541+0000", 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87599744099999999 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852549+0000", 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9209999999999992e-06 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176642 13.b 13.92db103b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.979747+0000", 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.344655002000003, 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87283178299999997, 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176642 2024-09-13T15:04:33.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979747+0000", 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979747+0000", 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979749+0000", 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.376e-06 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979754+0000", 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.558e-06 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979767+0000", 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.254e-05 2024-09-13T15:04:33.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979774+0000", 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2989999999999999e-06 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979836+0000", 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1471999999999997e-05 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979844+0000", 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3529999999999995e-06 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852571+0000", 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87272746199999995 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852579+0000", 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7230000000000004e-06 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.402 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176681 13.b 13.a6fc3bab (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.025138+0000", 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.299264315000002, 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.827471603, 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.403 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176681 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025138+0000", 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025138+0000", 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025139+0000", 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2130000000000001e-06 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025145+0000", 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2399999999999998e-06 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025155+0000", 2024-09-13T15:04:33.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0327e-05 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025162+0000", 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8360000000000003e-06 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025284+0000", 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121896 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025291+0000", 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7759999999999997e-06 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852602+0000", 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82731021599999999 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852610+0000", 2024-09-13T15:04:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0989999999999999e-06 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176684 13.b 13.cf4c3d6b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.026384+0000", 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.298018794000001, 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82625494399999999, 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:33.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176684 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026384+0000", 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026384+0000", 2024-09-13T15:04:33.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026384+0000", 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0999999999999997e-07 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026388+0000", 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.399e-06 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026397+0000", 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6149999999999997e-06 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026402+0000", 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7860000000000002e-06 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026465+0000", 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2617e-05 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026474+0000", 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.653e-06 2024-09-13T15:04:33.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852631+0000", 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82615713899999998 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852639+0000", 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8250000000000005e-06 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176690 13.b 13.a2ad84cb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.031425+0000", 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.292977733999997, 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82124366999999998, 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176690 2024-09-13T15:04:33.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031425+0000", 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031425+0000", 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031425+0000", 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1500000000000003e-07 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031430+0000", 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3059999999999999e-06 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031445+0000", 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5152e-05 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031453+0000", 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8290000000000008e-06 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031509+0000", 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6678999999999997e-05 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031519+0000", 2024-09-13T15:04:33.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0030999999999999e-05 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852661+0000", 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82114151800000001 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852668+0000", 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.34e-06 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176701 13.b 13.d16de8cb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.047337+0000", 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.277065534999998, 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.80578693899999998, 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176701 2024-09-13T15:04:33.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.047337+0000", 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.047337+0000", 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.047338+0000", 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4219999999999999e-06 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.047344+0000", 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3480000000000003e-06 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.047358+0000", 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3893000000000001e-05 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.047366+0000", 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2309999999999996e-06 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.047427+0000", 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0838999999999998e-05 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.047437+0000", 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0122000000000001e-05 2024-09-13T15:04:33.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853112+0000", 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.80567526899999997 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853124+0000", 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1815e-05 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176716 13.b 13.99aeec2b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.071107+0000", 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.253295553000001, 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.78204627400000004, 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176716 2024-09-13T15:04:33.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.071107+0000", 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.071107+0000", 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.071108+0000", 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.559e-06 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.071114+0000", 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7819999999999999e-06 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.071132+0000", 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7615999999999999e-05 2024-09-13T15:04:33.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.071139+0000", 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5789999999999998e-06 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.071197+0000", 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7614000000000001e-05 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.071206+0000", 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3270000000000007e-06 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853147+0000", 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.78194035699999997 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853153+0000", 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4400000000000002e-06 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176737 13.b 13.30391bbb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.098724+0000", 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.225678299000002, 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.75445228799999997, 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176737 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.098724+0000", 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.098724+0000", 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.098725+0000", 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.37e-07 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.098729+0000", 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8130000000000002e-06 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.098739+0000", 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0412e-05 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.098745+0000", 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1739999999999997e-06 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.098805+0000", 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9460999999999999e-05 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.098813+0000", 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0790000000000001e-06 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853170+0000", 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.75435746199999998 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853176+0000", 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1500000000000004e-06 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.435 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176756 13.b 13.a3064d9b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.129579+0000", 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.194823362999998, 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.72369041999999995, 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176756 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.129579+0000", 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.129579+0000", 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.129580+0000", 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1900000000000001e-07 2024-09-13T15:04:33.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.129584+0000", 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4050000000000002e-06 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.129593+0000", 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7960000000000001e-06 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.129600+0000", 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8820000000000003e-06 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.129670+0000", 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9506999999999997e-05 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.129678+0000", 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9390000000000003e-06 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853192+0000", 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.72351378899999996 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853269+0000", 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7182999999999996e-05 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176775 13.b 13.797682ab (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.160589+0000", 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.163813607999998, 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.69323565300000001, 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176775 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.441 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.160589+0000", 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.160589+0000", 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.160590+0000", 2024-09-13T15:04:33.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2999999999999999e-07 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.160594+0000", 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9439999999999998e-06 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.160604+0000", 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0015e-05 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.160610+0000", 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3609999999999997e-06 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.160670+0000", 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9867000000000001e-05 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.160677+0000", 2024-09-13T15:04:33.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2459999999999997e-06 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853814+0000", 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.69313638799999999 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853824+0000", 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0902e-05 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.445 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176784 13.b 13.c291179b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.179079+0000", 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.145323701999999, 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.67483935500000003, 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176784 2024-09-13T15:04:33.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.179079+0000", 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.179079+0000", 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.179080+0000", 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4699999999999999e-06 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.179085+0000", 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6959999999999996e-06 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.179098+0000", 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3559e-05 2024-09-13T15:04:33.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.179104+0000", 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.054e-06 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.179157+0000", 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2057e-05 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.179164+0000", 2024-09-13T15:04:33.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8140000000000001e-06 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853859+0000", 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.674694189 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.853918+0000", 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9515999999999997e-05 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:203844 13.8 13:1625f442:::benchmark_data_smithi003_293363_object84632:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:16.261053+0000", 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.063349455000001, 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.068389952, 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:04:33.451 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 203844 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261053+0000", 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261053+0000", 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261055+0000", 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1299999999999999e-06 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261063+0000", 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1499999999999999e-06 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:04:33.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261080+0000", 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6597999999999999e-05 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261090+0000", 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0103e-05 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.524007+0000", 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26291679200000001 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.524037+0000", 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9847000000000001e-05 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.291818+0000", 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.76778101099999996 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.291981+0000", 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016377500000000001 2024-09-13T15:04:33.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.292454+0000", 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000473144 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.297532+0000", 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005077285 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:04:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.297546+0000", 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3783e-05 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.329324+0000", 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.031778907000000002 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.329397+0000", 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2489000000000003e-05 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:04:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:04:33.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:04:33.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.329443+0000", 2024-09-13T15:04:33.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5938e-05 2024-09-13T15:04:33.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:04:33.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:04:33.458 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:04:33.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:04:33.511 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:33.541 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:33 smithi003 ceph-mon[107278]: 13.17 scrub starts 2024-09-13T15:04:33.542 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:33 smithi003 ceph-mon[107278]: pgmap v1436: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 2073/64914 objects degraded (3.193%); 2751/64914 objects misplaced (4.238%); 1.5 MiB/s, 103 objects/s recovering 2024-09-13T15:04:33.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:33 smithi003 ceph-mon[104007]: 13.17 scrub starts 2024-09-13T15:04:33.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:33 smithi003 ceph-mon[104007]: pgmap v1436: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 2073/64914 objects degraded (3.193%); 2751/64914 objects misplaced (4.238%); 1.5 MiB/s, 103 objects/s recovering 2024-09-13T15:04:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.551 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:33.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:33.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:33.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:33 smithi019 ceph-mon[91296]: 13.17 scrub starts 2024-09-13T15:04:33.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:33 smithi019 ceph-mon[91296]: pgmap v1436: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 2073/64914 objects degraded (3.193%); 2751/64914 objects misplaced (4.238%); 1.5 MiB/s, 103 objects/s recovering 2024-09-13T15:04:33.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:33.801 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.842 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:33.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:33.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:34.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:34.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:34.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:34.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:34.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:34.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:34 smithi003 ceph-mon[104007]: 13.7 scrub starts 2024-09-13T15:04:34.373 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:04:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:04:34] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T15:04:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:34.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:34.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:34.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:34.638 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:34.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:34.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:34 smithi019 ceph-mon[91296]: 13.7 scrub starts 2024-09-13T15:04:34.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:34 smithi019 ceph-mon[91296]: 13.11 scrub starts 2024-09-13T15:04:34.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:34 smithi003 ceph-mon[107278]: 13.7 scrub starts 2024-09-13T15:04:34.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:34 smithi003 ceph-mon[107278]: 13.11 scrub starts 2024-09-13T15:04:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:34 smithi003 ceph-mon[104007]: 13.11 scrub starts 2024-09-13T15:04:34.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:34.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:04:34.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:34.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:35.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:35.083 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:35.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:35.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:35.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:35.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:35.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:35.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:35.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:35 smithi003 ceph-mon[104007]: pgmap v1437: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1003 MiB data, 14 GiB used, 701 GiB / 715 GiB avail; 2073/48195 objects degraded (4.301%); 2001/48195 objects misplaced (4.152%); 0 B/s, 114 objects/s recovering 2024-09-13T15:04:35.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:35 smithi003 ceph-mon[104007]: osdmap e400: 8 total, 8 up, 8 in 2024-09-13T15:04:35.586 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:35 smithi003 ceph-mon[107278]: pgmap v1437: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1003 MiB data, 14 GiB used, 701 GiB / 715 GiB avail; 2073/48195 objects degraded (4.301%); 2001/48195 objects misplaced (4.152%); 0 B/s, 114 objects/s recovering 2024-09-13T15:04:35.586 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:35 smithi003 ceph-mon[107278]: osdmap e400: 8 total, 8 up, 8 in 2024-09-13T15:04:35.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:35 smithi019 ceph-mon[91296]: pgmap v1437: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1003 MiB data, 14 GiB used, 701 GiB / 715 GiB avail; 2073/48195 objects degraded (4.301%); 2001/48195 objects misplaced (4.152%); 0 B/s, 114 objects/s recovering 2024-09-13T15:04:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:35 smithi019 ceph-mon[91296]: osdmap e400: 8 total, 8 up, 8 in 2024-09-13T15:04:35.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:35.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:35.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:35.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:36.010 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.050 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:36.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:36.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:36.191 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:36.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:36.326 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:36.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:36.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:36.571 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:36.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:36 smithi019 ceph-mon[91296]: osdmap e401: 8 total, 8 up, 8 in 2024-09-13T15:04:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:36 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 2073/48195 objects degraded (4.301%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:36 smithi003 ceph-mon[107278]: osdmap e401: 8 total, 8 up, 8 in 2024-09-13T15:04:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:36 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 2073/48195 objects degraded (4.301%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:36 smithi003 ceph-mon[104007]: osdmap e401: 8 total, 8 up, 8 in 2024-09-13T15:04:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:36 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 2073/48195 objects degraded (4.301%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:36.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:36.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:36.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:36.884 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:36.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:37.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:37.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:37.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:37.253 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:37.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:37.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:37.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:37.465 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:37.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:37.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:37 smithi019 ceph-mon[91296]: 13.17 scrub starts 2024-09-13T15:04:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:37 smithi019 ceph-mon[91296]: pgmap v1440: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1003 MiB data, 14 GiB used, 701 GiB / 715 GiB avail; 2073/48195 objects degraded (4.301%); 2001/48195 objects misplaced (4.152%); 0 B/s, 143 objects/s recovering 2024-09-13T15:04:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:37 smithi003 ceph-mon[107278]: 13.17 scrub starts 2024-09-13T15:04:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:37 smithi003 ceph-mon[107278]: pgmap v1440: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1003 MiB data, 14 GiB used, 701 GiB / 715 GiB avail; 2073/48195 objects degraded (4.301%); 2001/48195 objects misplaced (4.152%); 0 B/s, 143 objects/s recovering 2024-09-13T15:04:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:37 smithi003 ceph-mon[104007]: 13.17 scrub starts 2024-09-13T15:04:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:37 smithi003 ceph-mon[104007]: pgmap v1440: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 1003 MiB data, 14 GiB used, 701 GiB / 715 GiB avail; 2073/48195 objects degraded (4.301%); 2001/48195 objects misplaced (4.152%); 0 B/s, 143 objects/s recovering 2024-09-13T15:04:37.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:37.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:37.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:37.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:37.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:38.009 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:38.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:38.147 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:38.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:38.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:38.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:38.416 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:04:38.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:38.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:38.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2588236226' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:38 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:38.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2588236226' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:38.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:38 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2588236226' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:38 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:38.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:38.721 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:38.823 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:38.861 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:38.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:38.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:38.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:39.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:39.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:39.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:39.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:39.372 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 13 pg_num to 36 2024-09-13T15:04:39.414 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:39.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:39.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:39.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:39 smithi019 ceph-mon[91296]: 13.7 scrub starts 2024-09-13T15:04:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:39 smithi019 ceph-mon[91296]: pgmap v1441: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 897 MiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2073/43092 objects degraded (4.811%); 1655/43092 objects misplaced (3.841%); 0 B/s, 78 objects/s recovering 2024-09-13T15:04:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:39 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:39 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:04:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:39 smithi019 ceph-mon[91296]: osdmap e402: 8 total, 8 up, 8 in 2024-09-13T15:04:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2471345920' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "36"} : dispatch 2024-09-13T15:04:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[107278]: 13.7 scrub starts 2024-09-13T15:04:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[107278]: pgmap v1441: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 897 MiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2073/43092 objects degraded (4.811%); 1655/43092 objects misplaced (3.841%); 0 B/s, 78 objects/s recovering 2024-09-13T15:04:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:04:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[107278]: osdmap e402: 8 total, 8 up, 8 in 2024-09-13T15:04:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2471345920' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "36"} : dispatch 2024-09-13T15:04:39.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[104007]: 13.7 scrub starts 2024-09-13T15:04:39.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[104007]: pgmap v1441: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 123 active+clean; 897 MiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2073/43092 objects degraded (4.811%); 1655/43092 objects misplaced (3.841%); 0 B/s, 78 objects/s recovering 2024-09-13T15:04:39.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:39.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:04:39.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[104007]: osdmap e402: 8 total, 8 up, 8 in 2024-09-13T15:04:39.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2471345920' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "36"} : dispatch 2024-09-13T15:04:39.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.692 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:39.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:39.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:39.830 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:39.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:39.972 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:40.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:40.108 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:40.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:40.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:40.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:40.383 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:40.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:40.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:40.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:04:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:04:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:40.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:40 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2471345920' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "36"}]': finished 2024-09-13T15:04:40.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:40 smithi019 ceph-mon[91296]: osdmap e403: 8 total, 8 up, 8 in 2024-09-13T15:04:40.583 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:40 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-13T15:04:40.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:40.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:40 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2471345920' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "36"}]': finished 2024-09-13T15:04:40.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:40 smithi003 ceph-mon[107278]: osdmap e403: 8 total, 8 up, 8 in 2024-09-13T15:04:40.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:40 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-13T15:04:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:40 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2471345920' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num", "val": "36"}]': finished 2024-09-13T15:04:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:40 smithi003 ceph-mon[104007]: osdmap e403: 8 total, 8 up, 8 in 2024-09-13T15:04:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:40 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-13T15:04:40.727 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:40.829 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:40.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:40.953 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:40.975 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:41.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:41.100 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:41.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:41.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:41.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:41.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:41.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:41.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:41.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:41 smithi003 ceph-mon[104007]: pgmap v1444: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 124 active+clean; 659 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 579/31695 objects degraded (1.827%); 1323/31695 objects misplaced (4.174%); 0 B/s, 94 objects/s recovering 2024-09-13T15:04:41.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:41 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-13T15:04:41.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:41 smithi003 ceph-mon[104007]: osdmap e404: 8 total, 8 up, 8 in 2024-09-13T15:04:41.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:41 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 579/31695 objects degraded (1.827%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:04:41.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:41 smithi003 ceph-mon[107278]: pgmap v1444: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 124 active+clean; 659 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 579/31695 objects degraded (1.827%); 1323/31695 objects misplaced (4.174%); 0 B/s, 94 objects/s recovering 2024-09-13T15:04:41.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:41 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-13T15:04:41.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:41 smithi003 ceph-mon[107278]: osdmap e404: 8 total, 8 up, 8 in 2024-09-13T15:04:41.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:41 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 579/31695 objects degraded (1.827%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:04:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:41.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:41 smithi019 ceph-mon[91296]: pgmap v1444: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 124 active+clean; 659 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 579/31695 objects degraded (1.827%); 1323/31695 objects misplaced (4.174%); 0 B/s, 94 objects/s recovering 2024-09-13T15:04:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:41 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-13T15:04:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:41 smithi019 ceph-mon[91296]: osdmap e404: 8 total, 8 up, 8 in 2024-09-13T15:04:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:41 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 579/31695 objects degraded (1.827%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:04:41.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:41.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:41.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:41.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:42.028 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:42.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:42.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:42.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:42.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:42.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:42.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:42.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:42.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:42.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:42 smithi003 ceph-mon[107278]: osdmap e405: 8 total, 8 up, 8 in 2024-09-13T15:04:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:42 smithi003 ceph-mon[107278]: 13.3 continuing backfill to osd.0 from (374'6347,403'13380] MIN to 403'13380 2024-09-13T15:04:42.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:42 smithi003 ceph-mon[107278]: 13.2 scrub starts 2024-09-13T15:04:42.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:42 smithi003 ceph-mon[104007]: osdmap e405: 8 total, 8 up, 8 in 2024-09-13T15:04:42.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:42 smithi003 ceph-mon[104007]: 13.3 continuing backfill to osd.0 from (374'6347,403'13380] MIN to 403'13380 2024-09-13T15:04:42.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:42 smithi003 ceph-mon[104007]: 13.2 scrub starts 2024-09-13T15:04:42.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:42.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:42.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:42.894 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:42.916 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:42 smithi019 ceph-mon[91296]: osdmap e405: 8 total, 8 up, 8 in 2024-09-13T15:04:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:42 smithi019 ceph-mon[91296]: 13.3 continuing backfill to osd.0 from (374'6347,403'13380] MIN to 403'13380 2024-09-13T15:04:42.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:42 smithi019 ceph-mon[91296]: 13.2 scrub starts 2024-09-13T15:04:43.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:43.041 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:43.142 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:43.166 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:43.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:43.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:43.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:43.414 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:43.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:43.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:43.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[107278]: pgmap v1447: 139 pgs: 9 unknown, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 125 active+clean; 620 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 579/29835 objects degraded (1.941%); 1323/29835 objects misplaced (4.434%); 0 B/s, 135 objects/s recovering 2024-09-13T15:04:43.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[107278]: 13.a scrub starts 2024-09-13T15:04:43.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[107278]: osdmap e406: 8 total, 8 up, 8 in 2024-09-13T15:04:43.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[107278]: 13.b scrub starts 2024-09-13T15:04:43.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:43.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:43.605 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[104007]: pgmap v1447: 139 pgs: 9 unknown, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 125 active+clean; 620 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 579/29835 objects degraded (1.941%); 1323/29835 objects misplaced (4.434%); 0 B/s, 135 objects/s recovering 2024-09-13T15:04:43.605 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[104007]: 13.a scrub starts 2024-09-13T15:04:43.605 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[104007]: osdmap e406: 8 total, 8 up, 8 in 2024-09-13T15:04:43.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[104007]: 13.b scrub starts 2024-09-13T15:04:43.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:43.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:43.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:43.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:43.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:43.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:43 smithi019 ceph-mon[91296]: pgmap v1447: 139 pgs: 9 unknown, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 125 active+clean; 620 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 579/29835 objects degraded (1.941%); 1323/29835 objects misplaced (4.434%); 0 B/s, 135 objects/s recovering 2024-09-13T15:04:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:43 smithi019 ceph-mon[91296]: 13.a scrub starts 2024-09-13T15:04:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:43 smithi019 ceph-mon[91296]: osdmap e406: 8 total, 8 up, 8 in 2024-09-13T15:04:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:43 smithi019 ceph-mon[91296]: 13.b scrub starts 2024-09-13T15:04:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:43.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:43.966 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:04:43.966 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:43.978 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:43.998 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:04:43.998 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:44.028 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:04:44.029 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:44.058 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:04:44.058 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:44.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:44.089 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:04:44.090 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:44.120 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:04:44.121 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:44.123 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:44.145 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:04:44.145 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:44.174 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:04:44.175 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:44.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:44.360 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:04:44] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T15:04:44.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:44.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:44.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:44.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:44.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.677 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:44.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[107278]: 13.1a scrub starts 2024-09-13T15:04:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[107278]: osdmap e407: 8 total, 8 up, 8 in 2024-09-13T15:04:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[107278]: 13.23 continuing backfill to osd.0 from (374'6347,403'13380] MIN to 403'13380 2024-09-13T15:04:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[107278]: 13.22 scrub starts 2024-09-13T15:04:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1463848064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[104007]: 13.1a scrub starts 2024-09-13T15:04:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[104007]: osdmap e407: 8 total, 8 up, 8 in 2024-09-13T15:04:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[104007]: 13.23 continuing backfill to osd.0 from (374'6347,403'13380] MIN to 403'13380 2024-09-13T15:04:44.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[104007]: 13.22 scrub starts 2024-09-13T15:04:44.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1463848064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:44.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:44.818 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:44.920 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:44 smithi019 ceph-mon[91296]: 13.1a scrub starts 2024-09-13T15:04:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:44 smithi019 ceph-mon[91296]: osdmap e407: 8 total, 8 up, 8 in 2024-09-13T15:04:44.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:44 smithi019 ceph-mon[91296]: 13.23 continuing backfill to osd.0 from (374'6347,403'13380] MIN to 403'13380 2024-09-13T15:04:44.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:44 smithi019 ceph-mon[91296]: 13.22 scrub starts 2024-09-13T15:04:44.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1463848064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:44.960 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:45.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:04:45.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:45.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:45.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:45.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:45.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:45.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:45.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:45.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:45.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.658 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:45.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:45.760 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:45.797 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:45.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:45 smithi019 ceph-mon[91296]: pgmap v1450: 139 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 132 active+clean; 432 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 579/20769 objects degraded (2.788%); 736/20769 objects misplaced (3.544%); 128 KiB/s, 251 objects/s recovering 2024-09-13T15:04:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:45 smithi019 ceph-mon[91296]: 13.1b scrub starts 2024-09-13T15:04:45.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:45 smithi003 ceph-mon[104007]: pgmap v1450: 139 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 132 active+clean; 432 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 579/20769 objects degraded (2.788%); 736/20769 objects misplaced (3.544%); 128 KiB/s, 251 objects/s recovering 2024-09-13T15:04:45.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:45 smithi003 ceph-mon[104007]: 13.1b scrub starts 2024-09-13T15:04:45.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:45 smithi003 ceph-mon[107278]: pgmap v1450: 139 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 132 active+clean; 432 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 579/20769 objects degraded (2.788%); 736/20769 objects misplaced (3.544%); 128 KiB/s, 251 objects/s recovering 2024-09-13T15:04:45.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:45 smithi003 ceph-mon[107278]: 13.1b scrub starts 2024-09-13T15:04:45.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:46.036 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.079 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:46.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:46.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:46.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:46.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:46.362 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:46.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:46.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:46.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:46.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:46.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:46.743 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 579/20769 objects degraded (2.788%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:04:46.744 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 579/20769 objects degraded (2.788%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:04:46.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:46.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:46.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 579/20769 objects degraded (2.788%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:04:46.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:47.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:47.082 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:47.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:47.202 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:04:47.202 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:04:47.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:47.221 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:47.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:47.357 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:47.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:47.494 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:47.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:47.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.633 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:47.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:47.692 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 119209 objects 2024-09-13T15:04:47.693 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :128.713 2024-09-13T15:04:47.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:47.785 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:47 smithi003 ceph-mon[107278]: pgmap v1451: 139 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 132 active+clean; 432 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 636 op/s; 579/20769 objects degraded (2.788%); 736/20769 objects misplaced (3.544%); 90 KiB/s, 176 objects/s recovering 2024-09-13T15:04:47.786 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:47 smithi003 ceph-mon[107278]: 13.2 scrub starts 2024-09-13T15:04:47.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:47 smithi003 ceph-mon[104007]: pgmap v1451: 139 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 132 active+clean; 432 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 636 op/s; 579/20769 objects degraded (2.788%); 736/20769 objects misplaced (3.544%); 90 KiB/s, 176 objects/s recovering 2024-09-13T15:04:47.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:47 smithi003 ceph-mon[104007]: 13.2 scrub starts 2024-09-13T15:04:47.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:47.881 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:47.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:47 smithi019 ceph-mon[91296]: pgmap v1451: 139 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 132 active+clean; 432 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 636 op/s; 579/20769 objects degraded (2.788%); 736/20769 objects misplaced (3.544%); 90 KiB/s, 176 objects/s recovering 2024-09-13T15:04:47.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:47 smithi019 ceph-mon[91296]: 13.2 scrub starts 2024-09-13T15:04:47.982 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:48.004 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:48.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.129 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:48.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:48.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:04:48.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:48.252 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 5, 0, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:04:48.252 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:04:48.252 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T15:04:48.252 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:04:48.253 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:04:48.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T15:04:48.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:48.355 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:48.377 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:48.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:48.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:48.514 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:04:48.551 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_8 2024-09-13T15:04:48.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_8 unique_pool_8 --yes-i-really-really-mean-it 2024-09-13T15:04:48.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:04:48.609 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:48.631 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:48.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:48.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:48.857 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:48.880 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:48.941 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:48.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:49.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:49.109 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:49.133 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:49.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:49.256 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:49.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:49.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:49.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:49.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:49.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:49.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:49.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:49.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:49.656 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:49 smithi019 ceph-mon[91296]: 13.a scrub starts 2024-09-13T15:04:49.656 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:49 smithi019 ceph-mon[91296]: pgmap v1452: 139 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 133 active+clean; 303 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 579/14595 objects degraded (3.967%); 395/14595 objects misplaced (2.706%); 76 KiB/s, 149 objects/s recovering 2024-09-13T15:04:49.656 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:49 smithi019 ceph-mon[91296]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T15:04:49.656 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:49 smithi019 ceph-mon[91296]: 13.b scrub starts 2024-09-13T15:04:49.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:49.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:49.862 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.885 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:49.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:49 smithi003 ceph-mon[107278]: 13.a scrub starts 2024-09-13T15:04:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:49 smithi003 ceph-mon[107278]: pgmap v1452: 139 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 133 active+clean; 303 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 579/14595 objects degraded (3.967%); 395/14595 objects misplaced (2.706%); 76 KiB/s, 149 objects/s recovering 2024-09-13T15:04:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:49 smithi003 ceph-mon[107278]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T15:04:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:49 smithi003 ceph-mon[107278]: 13.b scrub starts 2024-09-13T15:04:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:49 smithi003 ceph-mon[104007]: 13.a scrub starts 2024-09-13T15:04:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:49 smithi003 ceph-mon[104007]: pgmap v1452: 139 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 133 active+clean; 303 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 579/14595 objects degraded (3.967%); 395/14595 objects misplaced (2.706%); 76 KiB/s, 149 objects/s recovering 2024-09-13T15:04:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:49 smithi003 ceph-mon[104007]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T15:04:49.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:49 smithi003 ceph-mon[104007]: 13.b scrub starts 2024-09-13T15:04:49.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:50.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:50.061 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:50.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:50.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:50.191 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:04:50.191 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T15:04:50.192 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:04:50.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:50.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:50.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:50.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:04:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:04:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:50.485 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:50.512 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:50.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:50.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:50.636 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:50 smithi003 ceph-mon[107278]: 13.1a scrub starts 2024-09-13T15:04:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:50 smithi003 ceph-mon[107278]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T15:04:50.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:50 smithi003 ceph-mon[107278]: 13.22 scrub starts 2024-09-13T15:04:50.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:50 smithi003 ceph-mon[104007]: 13.1a scrub starts 2024-09-13T15:04:50.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:50 smithi003 ceph-mon[104007]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T15:04:50.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:50 smithi003 ceph-mon[104007]: 13.22 scrub starts 2024-09-13T15:04:50.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:50.839 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:50 smithi019 ceph-mon[91296]: 13.1a scrub starts 2024-09-13T15:04:50.839 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:50 smithi019 ceph-mon[91296]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T15:04:50.839 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:50 smithi019 ceph-mon[91296]: 13.22 scrub starts 2024-09-13T15:04:50.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:50.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:50.990 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:51.011 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:51.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:51.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:51.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:51.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:51.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:51.392 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:51.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:51.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:51.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:51.644 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:51 smithi003 ceph-mon[107278]: 13.23 scrub starts 2024-09-13T15:04:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:51 smithi003 ceph-mon[107278]: pgmap v1455: 139 pgs: 3 peering, 3 active+remapped+backfill_wait, 133 active+clean; 28 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 7/1398 objects misplaced (0.501%); 76 KiB/s, 236 objects/s recovering 2024-09-13T15:04:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:51 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 579/14595 objects degraded (3.967%), 1 pg degraded) 2024-09-13T15:04:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:51 smithi003 ceph-mon[107278]: 13.1b scrub starts 2024-09-13T15:04:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:51 smithi003 ceph-mon[104007]: 13.23 scrub starts 2024-09-13T15:04:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:51 smithi003 ceph-mon[104007]: pgmap v1455: 139 pgs: 3 peering, 3 active+remapped+backfill_wait, 133 active+clean; 28 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 7/1398 objects misplaced (0.501%); 76 KiB/s, 236 objects/s recovering 2024-09-13T15:04:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:51 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 579/14595 objects degraded (3.967%), 1 pg degraded) 2024-09-13T15:04:51.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:51 smithi003 ceph-mon[104007]: 13.1b scrub starts 2024-09-13T15:04:51.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:51.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:51.873 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:51.897 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:51.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:51 smithi019 ceph-mon[91296]: 13.23 scrub starts 2024-09-13T15:04:51.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:51 smithi019 ceph-mon[91296]: pgmap v1455: 139 pgs: 3 peering, 3 active+remapped+backfill_wait, 133 active+clean; 28 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 7/1398 objects misplaced (0.501%); 76 KiB/s, 236 objects/s recovering 2024-09-13T15:04:51.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:51 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 579/14595 objects degraded (3.967%), 1 pg degraded) 2024-09-13T15:04:51.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:51 smithi019 ceph-mon[91296]: 13.1b scrub starts 2024-09-13T15:04:51.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:52.024 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:52.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:52.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:52.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:52.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:52.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:52.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:52.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:52.537 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:52.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:52.664 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:52.758 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:52 smithi003 ceph-mon[107278]: 13.3 scrub starts 2024-09-13T15:04:52.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:52 smithi003 ceph-mon[104007]: 13.3 scrub starts 2024-09-13T15:04:52.765 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:52.790 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:52.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:52.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:52 smithi019 ceph-mon[91296]: 13.3 scrub starts 2024-09-13T15:04:53.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:53.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:53.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:53.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:53.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:53.298 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:53.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:53.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:53.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:53.555 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:53.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:53.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:53.726 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:53 smithi003 ceph-mon[107278]: 13.13 scrub starts 2024-09-13T15:04:53.726 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:53 smithi003 ceph-mon[107278]: pgmap v1456: 139 pgs: 3 peering, 3 active+remapped+backfill_wait, 133 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 7/717 objects misplaced (0.976%); 0 B/s, 72 objects/s recovering 2024-09-13T15:04:53.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:53 smithi003 ceph-mon[107278]: 13.2 scrub starts 2024-09-13T15:04:53.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:53 smithi003 ceph-mon[104007]: 13.13 scrub starts 2024-09-13T15:04:53.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:53 smithi003 ceph-mon[104007]: pgmap v1456: 139 pgs: 3 peering, 3 active+remapped+backfill_wait, 133 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 7/717 objects misplaced (0.976%); 0 B/s, 72 objects/s recovering 2024-09-13T15:04:53.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:53 smithi003 ceph-mon[104007]: 13.2 scrub starts 2024-09-13T15:04:53.768 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:04:53.768 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:04:53.769 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:04:53.769 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:53.769 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:53.769 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:53.776 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:04:53.776 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:53.782 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:53.791 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:04:53.792 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:53.807 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:04:53.808 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:53.809 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:53.825 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:04:53.825 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:53.839 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:04:53.840 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:53.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:53 smithi019 ceph-mon[91296]: 13.13 scrub starts 2024-09-13T15:04:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:53 smithi019 ceph-mon[91296]: pgmap v1456: 139 pgs: 3 peering, 3 active+remapped+backfill_wait, 133 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 7/717 objects misplaced (0.976%); 0 B/s, 72 objects/s recovering 2024-09-13T15:04:53.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:53 smithi019 ceph-mon[91296]: 13.2 scrub starts 2024-09-13T15:04:53.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:54.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:54.061 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:54.163 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:54.189 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:54.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:54.291 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:54.341 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:04:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:04:54] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T15:04:54.419 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:54.431 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:54.444 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:54.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:54.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:54.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:54 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3122148454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:54 smithi003 ceph-mon[107278]: 13.b scrub starts 2024-09-13T15:04:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T15:04:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:54 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3122148454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:54 smithi003 ceph-mon[104007]: 13.b scrub starts 2024-09-13T15:04:54.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T15:04:54.694 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:54.796 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:54.820 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:54.911 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:04:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:04:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:04:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:04:54.912 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:04:54.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:54 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3122148454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:54 smithi019 ceph-mon[91296]: 13.b scrub starts 2024-09-13T15:04:54.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T15:04:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:55.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:55.070 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:55.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:55.295 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:55.317 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:55.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:55.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:55.513 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:04:55.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:55.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:55.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:55.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:04:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:55 smithi019 ceph-mon[91296]: 13.a scrub starts 2024-09-13T15:04:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:55 smithi019 ceph-mon[91296]: pgmap v1457: 139 pgs: 4 active+clean+scrubbing, 135 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 0 B/s, 72 objects/s recovering 2024-09-13T15:04:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T15:04:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:55 smithi019 ceph-mon[91296]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T15:04:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1139697141' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:04:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:55 smithi019 ceph-mon[91296]: 13.1b scrub starts 2024-09-13T15:04:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:55.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[107278]: 13.a scrub starts 2024-09-13T15:04:55.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[107278]: pgmap v1457: 139 pgs: 4 active+clean+scrubbing, 135 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 0 B/s, 72 objects/s recovering 2024-09-13T15:04:55.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[107278]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1139697141' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[107278]: 13.1b scrub starts 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[104007]: 13.a scrub starts 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[104007]: pgmap v1457: 139 pgs: 4 active+clean+scrubbing, 135 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 0 B/s, 72 objects/s recovering 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[104007]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1139697141' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:55.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:04:55.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:55 smithi003 ceph-mon[104007]: 13.1b scrub starts 2024-09-13T15:04:55.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:55.814 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:55.864 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 5, 0, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:04:55.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:04:55.864 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-13T15:04:55.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-13T15:04:55.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:55.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:56.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:56.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:56.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:56.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:56.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:56.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:56.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:56.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:56.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:56.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:56.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:56.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[107278]: 13.1a scrub starts 2024-09-13T15:04:56.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:56.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1139697141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[107278]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[107278]: 13.b scrub starts 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[107278]: pgmap v1460: 139 pgs: 4 active+clean+scrubbing, 135 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 34 op/s 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[104007]: 13.1a scrub starts 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1139697141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[104007]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[104007]: 13.b scrub starts 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[104007]: pgmap v1460: 139 pgs: 4 active+clean+scrubbing, 135 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 34 op/s 2024-09-13T15:04:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:56 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T15:04:56.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:56.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:56.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:56 smithi019 ceph-mon[91296]: 13.1a scrub starts 2024-09-13T15:04:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:56 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1139697141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:56 smithi019 ceph-mon[91296]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T15:04:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:56 smithi019 ceph-mon[91296]: 13.b scrub starts 2024-09-13T15:04:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:56 smithi019 ceph-mon[91296]: pgmap v1460: 139 pgs: 4 active+clean+scrubbing, 135 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 34 op/s 2024-09-13T15:04:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:56 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T15:04:56.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:57.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:57.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:57.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:57.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:57.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:57.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:57.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:57.651 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:57.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:57.775 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:57.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.798 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:57.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:57.830 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:57 smithi019 ceph-mon[91296]: 13.13 scrub starts 2024-09-13T15:04:57.830 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:57 smithi019 ceph-mon[91296]: 13.a scrub starts 2024-09-13T15:04:57.830 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:57 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T15:04:57.830 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:57 smithi019 ceph-mon[91296]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T15:04:57.830 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:57 smithi019 ceph-mon[91296]: 13.10 starting backfill to osd.5 from (0'0,0'0] MAX to 407'13873 2024-09-13T15:04:57.830 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:57 smithi019 ceph-mon[91296]: 13.22 scrub starts 2024-09-13T15:04:57.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[107278]: 13.13 scrub starts 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[107278]: 13.a scrub starts 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[107278]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[107278]: 13.10 starting backfill to osd.5 from (0'0,0'0] MAX to 407'13873 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[107278]: 13.22 scrub starts 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[104007]: 13.13 scrub starts 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[104007]: 13.a scrub starts 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[104007]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[104007]: 13.10 starting backfill to osd.5 from (0'0,0'0] MAX to 407'13873 2024-09-13T15:04:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:57 smithi003 ceph-mon[104007]: 13.22 scrub starts 2024-09-13T15:04:57.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:57.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:57.948 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:58.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:58.046 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:58.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.172 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:58.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:58.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:58.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:58.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:58.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:58.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:58.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.672 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:58.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:58.677 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:58.773 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:58.795 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:58.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[107278]: 13.13 scrub starts 2024-09-13T15:04:58.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[107278]: 13.1a scrub starts 2024-09-13T15:04:58.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[107278]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T15:04:58.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[107278]: 13.1b scrub starts 2024-09-13T15:04:58.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:58.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:58.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[107278]: pgmap v1463: 139 pgs: 1 activating+remapped, 1 active+clean+scrubbing, 137 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:04:58.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[104007]: 13.13 scrub starts 2024-09-13T15:04:58.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[104007]: 13.1a scrub starts 2024-09-13T15:04:58.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[104007]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T15:04:58.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[104007]: 13.1b scrub starts 2024-09-13T15:04:58.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:58.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:58.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:58 smithi003 ceph-mon[104007]: pgmap v1463: 139 pgs: 1 activating+remapped, 1 active+clean+scrubbing, 137 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:04:58.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:58.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:58 smithi019 ceph-mon[91296]: 13.13 scrub starts 2024-09-13T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:58 smithi019 ceph-mon[91296]: 13.1a scrub starts 2024-09-13T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:58 smithi019 ceph-mon[91296]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:58 smithi019 ceph-mon[91296]: 13.1b scrub starts 2024-09-13T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:58 smithi019 ceph-mon[91296]: pgmap v1463: 139 pgs: 1 activating+remapped, 1 active+clean+scrubbing, 137 active+clean; 14 MiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:04:59.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:59.043 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:59.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:59.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:59.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:04:59.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:59.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:59.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:59.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:04:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:59.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:59.540 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_8' removed 2024-09-13T15:04:59.548 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:59.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:59.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[107278]: 13.3 scrub starts 2024-09-13T15:04:59.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[107278]: 13.a scrub starts 2024-09-13T15:04:59.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[107278]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T15:04:59.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[107278]: 13.11 starting backfill to osd.1 from (0'0,0'0] MAX to 407'13714 2024-09-13T15:04:59.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[107278]: 13.11 starting backfill to osd.5 from (0'0,0'0] MAX to 407'13714 2024-09-13T15:04:59.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[107278]: 13.2 scrub starts 2024-09-13T15:04:59.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1744783640' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[104007]: 13.3 scrub starts 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[104007]: 13.a scrub starts 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[104007]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[104007]: 13.11 starting backfill to osd.1 from (0'0,0'0] MAX to 407'13714 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[104007]: 13.11 starting backfill to osd.5 from (0'0,0'0] MAX to 407'13714 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[104007]: 13.2 scrub starts 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1744783640' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:59.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:04:59 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:59.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:59.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:59 smithi019 ceph-mon[91296]: 13.3 scrub starts 2024-09-13T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:59 smithi019 ceph-mon[91296]: 13.a scrub starts 2024-09-13T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:59 smithi019 ceph-mon[91296]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:59 smithi019 ceph-mon[91296]: 13.11 starting backfill to osd.1 from (0'0,0'0] MAX to 407'13714 2024-09-13T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:59 smithi019 ceph-mon[91296]: 13.11 starting backfill to osd.5 from (0'0,0'0] MAX to 407'13714 2024-09-13T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:59 smithi019 ceph-mon[91296]: 13.2 scrub starts 2024-09-13T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1744783640' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:04:59.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:59 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:59.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:04:59 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:04:59.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:59.956 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:04:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:00.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:00.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:00.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:00.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:00.306 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:00.330 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:00.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:00.453 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:00.544 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:05:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:05:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:00.555 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:00.579 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:00.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:00.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.705 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:00.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:00.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:00.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:00 smithi003 ceph-mon[107278]: 13.23 scrub starts 2024-09-13T15:05:00.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:00 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1744783640' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:05:00.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:00 smithi003 ceph-mon[107278]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T15:05:00.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:00 smithi003 ceph-mon[107278]: pgmap v1466: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:00.818 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:00 smithi003 ceph-mon[104007]: 13.23 scrub starts 2024-09-13T15:05:00.818 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:00 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1744783640' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:05:00.818 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:00 smithi003 ceph-mon[104007]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T15:05:00.818 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:00 smithi003 ceph-mon[104007]: pgmap v1466: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:00.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:00.845 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T15:05:00.846 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T15:05:00.846 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_9 2024-09-13T15:05:00.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_9 16 2024-09-13T15:05:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:00 smithi019 ceph-mon[91296]: 13.23 scrub starts 2024-09-13T15:05:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:00 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1744783640' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:05:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:00 smithi019 ceph-mon[91296]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T15:05:00.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:00 smithi019 ceph-mon[91296]: pgmap v1466: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:00.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:00.959 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:01.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:01.086 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:01.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:01.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:01.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:01.339 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:01.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:01.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:01.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:01.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:01.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:01.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:01.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:01.815 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:01.842 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:01.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:01.971 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:02.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:02.099 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:02.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:02.227 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:02.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:02.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:02.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:02.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:02.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:02.484 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:02.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:02.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:02.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:02.739 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:02.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:02.851 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-09-13T15:05:02.851 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-09-13T15:05:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-09-13T15:05:02.867 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:02.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:02.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:02.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:03.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:03.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:03.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:03.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:03.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:03.355 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:03.382 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:03.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:03.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:03.518 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:03 smithi003 ceph-mon[107278]: pgmap v1467: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:03.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:03 smithi003 ceph-mon[104007]: pgmap v1467: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:03.609 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:03.636 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:03.644 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:05:03.644 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T15:05:03.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T15:05:03.644 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:05:03.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.644 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129096 12.8 12:15f7dc4c:::benchmark_data_smithi003_259359_object13470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:05:03.644 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.516501+0000", 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 414.10044125799999, 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2720113149999999, 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129096 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516501+0000", 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516501+0000", 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516502+0000", 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8470000000000001e-06 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516508+0000", 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1420000000000001e-06 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516521+0000", 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2961e-05 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516530+0000", 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5359999999999998e-06 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516935+0000", 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00040524499999999999 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.516946+0000", 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0339999999999999e-05 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.504919+0000", 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98797316099999999 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.504934+0000", 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4715000000000001e-05 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.766100+0000", 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.261166786 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.766220+0000", 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119614 2024-09-13T15:05:03.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.766708+0000", 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00048800500000000001 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.782160+0000", 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015452097 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788271+0000", 2024-09-13T15:05:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061114000000000003 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788310+0000", 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8896999999999997e-05 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788347+0000", 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6692e-05 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788403+0000", 2024-09-13T15:05:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6465000000000002e-05 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788512+0000", 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108412 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46703.0:129111 12.8 12:149496a4:::benchmark_data_smithi003_259359_object13485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T14:58:09.535225+0000", 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 414.08171666599998, 2024-09-13T15:05:03.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2598992739999999, 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46703", 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 129111 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535225+0000", 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535225+0000", 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535226+0000", 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3209999999999999e-06 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535233+0000", 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4169999999999997e-06 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535247+0000", 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.393e-05 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535254+0000", 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0829999999999998e-06 2024-09-13T15:05:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535319+0000", 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5075999999999998e-05 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:09.535327+0000", 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0660000000000004e-06 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.506143+0000", 2024-09-13T15:05:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97081633899999997 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:10.506153+0000", 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3030000000000005e-06 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.775368+0000", 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2692155810000001 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.775502+0000", 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013353799999999999 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2,6", 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.775635+0000", 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133282 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.785399+0000", 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0097642519999999993 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788571+0000", 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031718509999999998 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.788631+0000", 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9475000000000002e-05 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.794958+0000", 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063276979999999997 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.795020+0000", 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1444000000000003e-05 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T14:58:11.795124+0000", 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010461799999999999 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127114 13.3 13:c1660045:::benchmark_data_smithi003_293363_object7902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T15:05:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.202037+0000", 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 135.41490449899999, 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.135729252, 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127114 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202037+0000", 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202037+0000", 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202039+0000", 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.875e-06 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202046+0000", 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4459999999999998e-06 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202060+0000", 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4323e-05 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202068+0000", 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3270000000000001e-06 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.203713+0000", 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001644413 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.203830+0000", 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116867 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.203956+0000", 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012679299999999999 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244242+0000", 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040285728 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244254+0000", 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1476e-05 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.328915+0000", 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0846615079999999 2024-09-13T15:05:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for missing object", 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329047+0000", 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013149400000000001 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.337707+0000", 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0086602120000000005 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.337767+0000", 2024-09-13T15:05:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9790000000000002e-05 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127123 13.3 13:cfeb70cd:::benchmark_data_smithi003_293363_object7911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.232129+0000", 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 135.384812772, 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1073596480000001, 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127123 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232129+0000", 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232129+0000", 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232131+0000", 2024-09-13T15:05:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.699e-06 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232137+0000", 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.224e-06 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232150+0000", 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3455000000000001e-05 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232158+0000", 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6029999999999999e-06 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244280+0000", 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012121709 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244287+0000", 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8340000000000001e-06 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329294+0000", 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0850076449999999 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329431+0000", 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013655399999999999 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329578+0000", 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014753499999999999 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336227+0000", 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066491049999999998 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336286+0000", 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9038e-05 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336352+0000", 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5582999999999999e-05 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339347+0000", 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029945530000000001 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339394+0000", 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7123000000000003e-05 2024-09-13T15:05:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339489+0000", 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4988000000000006e-05 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127141 13.3 13:c8a2a879:::benchmark_data_smithi003_293363_object7929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.256678+0000", 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 135.36026403899999, 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.082947579, 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127141 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256678+0000", 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256678+0000", 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256680+0000", 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9769999999999999e-06 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256687+0000", 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2980000000000003e-06 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256708+0000", 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1121999999999999e-05 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256716+0000", 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0460000000000006e-06 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256778+0000", 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2205999999999994e-05 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256787+0000", 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4100000000000008e-06 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.332032+0000", 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0752451359999999 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.332117+0000", 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4950000000000005e-05 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.332198+0000", 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0968000000000001e-05 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336368+0000", 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041701589999999997 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336418+0000", 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9521999999999998e-05 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336452+0000", 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4790999999999999e-05 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339529+0000", 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030764569999999999 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339561+0000", 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2246999999999998e-05 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339625+0000", 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4289999999999996e-05 2024-09-13T15:05:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176536 13.b 13:ddb01560:::benchmark_data_smithi003_293363_object57324:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.864930+0000", 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.752012097999994, 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.079589701, 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176536 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.864930+0000", 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.864930+0000", 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.864931+0000", 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3489999999999999e-06 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.864936+0000", 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0100000000000003e-06 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.864948+0000", 2024-09-13T15:05:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2231e-05 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.864958+0000", 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1980000000000007e-06 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865164+0000", 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000206856 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865173+0000", 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2579999999999995e-06 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.922672+0000", 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0574997500000001 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for missing object", 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.922742+0000", 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9612999999999999e-05 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944456+0000", 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021713896 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944519+0000", 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3540000000000005e-05 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176559 13.b 13:d55d9c41:::benchmark_data_smithi003_293363_object57347:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865021+0000", 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751920608999995, 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.070393551, 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176559 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865021+0000", 2024-09-13T15:05:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865021+0000", 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865022+0000", 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1399999999999997e-07 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865025+0000", 2024-09-13T15:05:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.96e-06 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865042+0000", 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7489000000000001e-05 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865049+0000", 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1899999999999998e-06 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865196+0000", 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000147133 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865203+0000", 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5080000000000002e-06 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.922787+0000", 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0575839920000001 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.922925+0000", 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013754199999999999 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.923401+0000", 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047600600000000001 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.932874+0000", 2024-09-13T15:05:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094736330000000004 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935227+0000", 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023525310000000002 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935276+0000", 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9221000000000002e-05 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935347+0000", 2024-09-13T15:05:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1007999999999995e-05 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935415+0000", 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7824000000000007e-05 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176618 13.b 13:db9c32c1:::benchmark_data_smithi003_293363_object57406:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865059+0000", 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751882664999997, 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.070536358, 2024-09-13T15:05:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176618 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865059+0000", 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865059+0000", 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865060+0000", 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7899999999999998e-07 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865062+0000", 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7030000000000002e-06 2024-09-13T15:05:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865069+0000", 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8680000000000001e-06 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865075+0000", 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4229999999999996e-06 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865224+0000", 2024-09-13T15:05:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014960700000000001 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865231+0000", 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3509999999999998e-06 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.930152+0000", 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0649212429999999 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.930240+0000", 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8545e-05 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.930321+0000", 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0183000000000001e-05 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935235+0000", 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049139819999999999 2024-09-13T15:05:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935464+0000", 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022982699999999999 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935496+0000", 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1396000000000002e-05 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935523+0000", 2024-09-13T15:05:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7032000000000001e-05 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935596+0000", 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2619000000000004e-05 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176637 13.b 13:d907ac3f:::benchmark_data_smithi003_293363_object57425:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865081+0000", 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751860335000003, 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.097294284, 2024-09-13T15:05:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176637 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865081+0000", 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865081+0000", 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865082+0000", 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8099999999999998e-07 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865084+0000", 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.182e-06 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865091+0000", 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5649999999999998e-06 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865096+0000", 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6420000000000004e-06 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865251+0000", 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015508899999999999 2024-09-13T15:05:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865258+0000", 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2219999999999999e-06 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935706+0000", 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0704488969999999 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935808+0000", 2024-09-13T15:05:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010106 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935897+0000", 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9606999999999998e-05 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944419+0000", 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0085219030000000008 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944552+0000", 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133387 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962201+0000", 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017648440000000001 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962260+0000", 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9416999999999997e-05 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962376+0000", 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115492 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176642 13.b 13:dc08db49:::benchmark_data_smithi003_293363_object57430:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751831964999994, 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.09749176, 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176642 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:05:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0200000000000003e-07 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865113+0000", 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.61e-06 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865120+0000", 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8870000000000003e-06 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865125+0000", 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4920000000000001e-06 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153487 2024-09-13T15:05:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865285+0000", 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3799999999999999e-06 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.938324+0000", 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0730388900000001 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.938395+0000", 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0896999999999996e-05 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.938507+0000", 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112172 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944565+0000", 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006058066 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944611+0000", 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6084999999999997e-05 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962436+0000", 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017824815000000001 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962496+0000", 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0374999999999997e-05 2024-09-13T15:05:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962602+0000", 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105202 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176681 13.b 13:d5dc3f65:::benchmark_data_smithi003_293363_object57469:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751809215999998, 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0977554540000001, 2024-09-13T15:05:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176681 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9999999999999998e-07 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865135+0000", 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2790000000000001e-06 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865142+0000", 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3849999999999999e-06 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865147+0000", 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2109999999999997e-06 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865307+0000", 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000159815 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865313+0000", 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1129999999999997e-06 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.941097+0000", 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0757844999999999 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.941212+0000", 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011456 2024-09-13T15:05:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.941347+0000", 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013543100000000001 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944571+0000", 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032240150000000002 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944667+0000", 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5983999999999994e-05 2024-09-13T15:05:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962729+0000", 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018061628999999999 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962782+0000", 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2652999999999999e-05 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962888+0000", 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010647899999999999 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176684 13.b 13:d6bc32f3:::benchmark_data_smithi003_293363_object57472:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751788132000001, 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.098160848, 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176684 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7899999999999999e-07 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865156+0000", 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2120000000000002e-06 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865163+0000", 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2979999999999997e-06 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865168+0000", 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.938e-06 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865339+0000", 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000171719 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865345+0000", 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.122e-06 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.726 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:03 smithi019 ceph-mon[91296]: pgmap v1467: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944711+0000", 2024-09-13T15:05:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0793655870000001 2024-09-13T15:05:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944815+0000", 2024-09-13T15:05:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010432199999999999 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.945061+0000", 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024528700000000001 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955429+0000", 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010368297 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955546+0000", 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117385 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963159+0000", 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076123060000000001 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963223+0000", 2024-09-13T15:05:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4424000000000005e-05 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963315+0000", 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1571999999999997e-05 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176690 13.b 13:d321b545:::benchmark_data_smithi003_293363_object57478:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751767951000005, 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.098341236, 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176690 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0400000000000002e-07 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865176+0000", 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.12e-06 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865191+0000", 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4345000000000001e-05 2024-09-13T15:05:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865196+0000", 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6949999999999997e-06 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865366+0000", 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016998100000000001 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865372+0000", 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9170000000000003e-06 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.948302+0000", 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0829294700000001 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.948432+0000", 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013043500000000001 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.948566+0000", 2024-09-13T15:05:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013371600000000001 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955473+0000", 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069065769999999997 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962132+0000", 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006659954 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963377+0000", 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012440940000000001 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963425+0000", 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8899000000000001e-05 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963515+0000", 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9629000000000005e-05 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176701 13.b 13:d317b68b:::benchmark_data_smithi003_293363_object57489:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751737953000003, 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0984964370000001, 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176701 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0100000000000005e-07 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865207+0000", 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4959999999999999e-06 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865213+0000", 2024-09-13T15:05:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6279999999999999e-06 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865219+0000", 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1059999999999999e-06 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865442+0000", 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022335700000000001 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865449+0000", 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1289999999999998e-06 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.952217+0000", 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.086767781 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.952348+0000", 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130698 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.952485+0000", 2024-09-13T15:05:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013729000000000001 2024-09-13T15:05:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955532+0000", 2024-09-13T15:05:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030474899999999999 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962140+0000", 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066075830000000002 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963560+0000", 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014204339999999999 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963608+0000", 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7970000000000003e-05 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963700+0000", 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1973999999999997e-05 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176716 13.b 13:d4377599:::benchmark_data_smithi003_293363_object57504:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751716735000002, 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0986561319999999, 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176716 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7599999999999998e-07 2024-09-13T15:05:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865228+0000", 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0619999999999999e-06 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865234+0000", 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3949999999999998e-06 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865239+0000", 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1699999999999996e-06 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865472+0000", 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023298500000000001 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865478+0000", 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9479999999999996e-06 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955574+0000", 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0900960959999999 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955659+0000", 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4608e-05 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955737+0000", 2024-09-13T15:05:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8046000000000005e-05 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962615+0000", 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068780509999999996 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962671+0000", 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6412999999999997e-05 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963738+0000", 2024-09-13T15:05:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010668470000000001 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963787+0000", 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8671000000000002e-05 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963881+0000", 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4463999999999995e-05 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176737 13.b 13:ddd89c0c:::benchmark_data_smithi003_293363_object57525:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865245+0000", 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751696554999995, 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.098807461, 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176737 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865245+0000", 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865245+0000", 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865246+0000", 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6300000000000001e-07 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865259+0000", 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3437e-05 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865266+0000", 2024-09-13T15:05:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0720000000000002e-06 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865272+0000", 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6500000000000001e-06 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865499+0000", 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022734299999999999 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865505+0000", 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1879999999999999e-06 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.958889+0000", 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.093383966 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.959023+0000", 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133664 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.959159+0000", 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013578400000000001 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962904+0000", 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037453019999999998 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963100+0000", 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000195565 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963924+0000", 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000824711 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963971+0000", 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6295000000000003e-05 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964053+0000", 2024-09-13T15:05:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2120999999999995e-05 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176756 13.b 13:d9b260c5:::benchmark_data_smithi003_293363_object57544:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751662366999994, 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1185605869999999, 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176756 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865280+0000", 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0800000000000005e-07 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865282+0000", 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2139999999999999e-06 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865288+0000", 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.297e-06 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865293+0000", 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9450000000000001e-06 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865527+0000", 2024-09-13T15:05:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023310799999999999 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865532+0000", 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9569999999999999e-06 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964086+0000", 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0985532360000001 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964196+0000", 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110356 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964312+0000", 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116267 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976732+0000", 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012419902999999999 2024-09-13T15:05:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976741+0000", 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8419999999999994e-06 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.983604+0000", 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006862511 2024-09-13T15:05:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.983720+0000", 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116086 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.983840+0000", 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120357 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176775 13.b 13:d5416e9e:::benchmark_data_smithi003_293363_object57563:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751642072999999, 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1439602769999999, 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176775 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.77e-07 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865302+0000", 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8989999999999999e-06 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865308+0000", 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3040000000000001e-06 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865313+0000", 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0599999999999998e-06 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866806+0000", 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001492581 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866814+0000", 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5619999999999998e-06 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.967407+0000", 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.100593173 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.967483+0000", 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6363000000000001e-05 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.967581+0000", 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7825000000000006e-05 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976749+0000", 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0091678430000000002 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976804+0000", 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5241000000000002e-05 2024-09-13T15:05:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009025+0000", 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032221078 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009088+0000", 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2738999999999997e-05 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009260+0000", 2024-09-13T15:05:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000172232 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176784 13.b 13:d9e88943:::benchmark_data_smithi003_293363_object57572:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751256257999998, 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.143711921, 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176784 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:05:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1600000000000001e-07 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865689+0000", 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7190000000000001e-06 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865697+0000", 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7959999999999996e-06 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865703+0000", 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7750000000000004e-06 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866835+0000", 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001132128 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866842+0000", 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6200000000000001e-06 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.970297+0000", 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1034550540000001 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.970390+0000", 2024-09-13T15:05:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3011999999999996e-05 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.970471+0000", 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1049000000000003e-05 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976847+0000", 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063753589999999997 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976866+0000", 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.897e-05 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009300+0000", 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032434642 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009336+0000", 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6041000000000003e-05 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009397+0000", 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1240000000000003e-05 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176811 13.b 13:d3189ffb:::benchmark_data_smithi003_293363_object57599:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865711+0000", 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.751230543000005, 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1460795859999999, 2024-09-13T15:05:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176811 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865711+0000", 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865711+0000", 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865712+0000", 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1100000000000001e-07 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865714+0000", 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0679999999999999e-06 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865720+0000", 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3029999999999996e-06 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865725+0000", 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.989e-06 2024-09-13T15:05:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866873+0000", 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011484360000000001 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866880+0000", 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5289999999999997e-06 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.972880+0000", 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.105999841 2024-09-13T15:05:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.973094+0000", 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000213888 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.973209+0000", 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011561300000000001 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976918+0000", 2024-09-13T15:05:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037089139999999998 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.000375+0000", 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023456290000000001 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.011630+0000", 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011255292 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:05:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.011734+0000", 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104122 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.011791+0000", 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6889999999999999e-05 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:03.789 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:05:03.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:03.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:03.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:04.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:04.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:04.163 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.193 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:04.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:04.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:04.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:04.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:04.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:05:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:05:04] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-13T15:05:04.449 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:04.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:04.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:04.675 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:04.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:05:04.702 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:04.805 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:04.828 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:04.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:04.952 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:04.968 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:05.054 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:05.072 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:05.072 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:05.072 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:05.076 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:05.108 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:05:05.108 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T15:05:05.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-13T15:05:05.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:05.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:05.302 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:05.325 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:05.357 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:05 smithi019 ceph-mon[91296]: pgmap v1468: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:05.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:05 smithi003 ceph-mon[107278]: pgmap v1468: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:05.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:05 smithi003 ceph-mon[104007]: pgmap v1468: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:05.427 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:05.449 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:05.551 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.574 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:05.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:05.676 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:05.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:05.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:05.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:05.947 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:06.049 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:06.073 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:06.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.198 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:06.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:06.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:06.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:06.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:06.445 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:06.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:06.630 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:05:06.630 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:05:06.631 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T15:05:06.631 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:05:06.661 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:06.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:06.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:06.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:06.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:06.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:06.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:07.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:07.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:07.173 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:07.199 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:07.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:07.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:07.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:07 smithi003 ceph-mon[107278]: pgmap v1469: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:07.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:07 smithi003 ceph-mon[104007]: pgmap v1469: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:07.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:07.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:07.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:07.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:07.620 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:07 smithi019 ceph-mon[91296]: pgmap v1469: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:07.652 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:05:07.676 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:07.718 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:07.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:07.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:07.917 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:07.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:07.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:07.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:08.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:08.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:08.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:08.218 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:08.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:08.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:08.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:08.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:08.571 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:08.593 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:08.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:08.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:08.820 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:08.844 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:08.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:08.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:09.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:09.093 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:09.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.217 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:09.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:09.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:09.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:09.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:09.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:09.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:09 smithi003 ceph-mon[107278]: pgmap v1470: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:09.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:09 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:09.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:09 smithi003 ceph-mon[104007]: pgmap v1470: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:09.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:09 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:09.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:09.587 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:05:09.588 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:05:09.588 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:05:09.588 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:05:09.588 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:05:09.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:09 smithi019 ceph-mon[91296]: pgmap v1470: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:09 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:09.696 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:09.718 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:09.820 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:09.842 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:09.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:09.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:10.068 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:10.091 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:10.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:10.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:10.221 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:05:10.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:10.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:10.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:10.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:10.460 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:05:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:05:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:10.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:10.471 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_9' created 2024-09-13T15:05:10.481 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:10.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:10.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:10.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3423847202' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-13T15:05:10.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:10 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-13T15:05:10.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3423847202' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-13T15:05:10.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:10 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-13T15:05:10.695 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:10.719 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:10.730 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3423847202' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-13T15:05:10.730 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:10 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-13T15:05:10.821 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:10.844 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:10.946 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:10.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:11.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.093 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:11.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:05:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:11.190 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:05:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:11.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:11.207 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:05:11.207 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:11.224 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:05:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:11.240 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:05:11.240 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:11.256 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:05:11.256 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:11.271 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:05:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:11.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:05:11.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:11.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:11.342 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:11.444 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:11.466 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:11.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:11.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:11.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:11 smithi019 ceph-mon[91296]: pgmap v1471: 103 pgs: 103 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:11 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-09-13T15:05:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:11 smithi019 ceph-mon[91296]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T15:05:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/442646763' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:11.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:11.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:11.818 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:11 smithi003 ceph-mon[107278]: pgmap v1471: 103 pgs: 103 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:11.818 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:11 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-09-13T15:05:11.819 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:11 smithi003 ceph-mon[107278]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T15:05:11.819 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/442646763' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:11.819 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:11 smithi003 ceph-mon[104007]: pgmap v1471: 103 pgs: 103 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:11.819 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:11 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-09-13T15:05:11.819 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:11 smithi003 ceph-mon[104007]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T15:05:11.819 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/442646763' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:11.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:11.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:11.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:11.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:11.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:12.067 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:12.093 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:12.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:12.217 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:12.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:12.339 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:12.339 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:12.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:12.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:12.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:12.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:12.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_9 rados --yes-i-really-mean-it || true 2024-09-13T15:05:12.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:12.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:12.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:12 smithi019 ceph-mon[91296]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T15:05:12.700 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:12.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.723 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:12.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:12.823 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:12.847 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:12.868 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:12 smithi003 ceph-mon[107278]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T15:05:12.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:12 smithi003 ceph-mon[104007]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T15:05:12.949 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:12.972 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:13.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:13.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:13.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:13.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:13.286 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:05:13.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:13.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:13.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:13.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:13.550 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:13.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:13.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:13.606 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[107278]: pgmap v1474: 119 pgs: 15 unknown, 104 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:13.606 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[107278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:05:13.606 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[107278]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T15:05:13.606 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:13.606 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:13.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[104007]: pgmap v1474: 119 pgs: 15 unknown, 104 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:13.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[104007]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:05:13.606 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[104007]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T15:05:13.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:13.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:13.673 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:13 smithi019 ceph-mon[91296]: pgmap v1474: 119 pgs: 15 unknown, 104 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:13.673 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:13 smithi019 ceph-mon[91296]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:05:13.674 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:13 smithi019 ceph-mon[91296]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T15:05:13.674 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:13.674 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:13.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:13.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:13.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:13.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:13.842 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:13.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:13.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:14.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:14.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-09-13T15:05:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-09-13T15:05:14.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:05:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:05:14] "GET /metrics HTTP/1.1" 200 36457 "" "Prometheus/2.51.0" 2024-09-13T15:05:14.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:14.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:14.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:14.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:14.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:14.565 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:14.590 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:14.692 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:14.716 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:14.818 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:14.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:14.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:14.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:15.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:15.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:15.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:15.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:15.287 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:15.287 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:15.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:15.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:05:15.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:15.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:15.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:15.467 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:15.491 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:05:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:05:15.562 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T15:05:15.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-13T15:05:15.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:15.593 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:15 smithi003 ceph-mon[107278]: pgmap v1476: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3525497775' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:05:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:15 smithi003 ceph-mon[104007]: pgmap v1476: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3525497775' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:05:15.695 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:15.717 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:15.740 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:15.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:15.841 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:15.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:15 smithi019 ceph-mon[91296]: pgmap v1476: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:15.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3525497775' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:05:15.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:15.964 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:16.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:05:16.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:16.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:16.191 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.214 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:16.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:16.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:16.321 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:16.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:16.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:16.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:16.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:16.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:16.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:16.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:16 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:16.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:16 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3525497775' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:05:16.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:16 smithi003 ceph-mon[104007]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T15:05:16.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:16 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:16.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:16 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3525497775' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:05:16.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:16 smithi003 ceph-mon[107278]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T15:05:16.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:16.702 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:16 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:16.702 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:16 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3525497775' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:05:16.702 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:16 smithi019 ceph-mon[91296]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T15:05:16.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:16.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:16.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:16.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:16.960 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:17.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:17.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:17.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:17.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:17.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:17.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:17.438 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:17.463 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:17.519 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:17.565 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:17.588 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:17.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:17.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:17.732 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:17 smithi019 ceph-mon[91296]: pgmap v1478: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:17.796 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:17 smithi003 ceph-mon[107278]: pgmap v1478: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:17.796 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:17 smithi003 ceph-mon[104007]: pgmap v1478: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:17.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.839 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:17.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:17.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:17.983 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:17.983 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:18.003 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:18.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:18.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:18.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:18.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:18.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:18.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:18.501 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:18.603 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:18.613 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:18.613 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:18.614 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:05:18.614 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:05:18.614 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:18.614 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:18 smithi019 ceph-mon[91296]: pgmap v1479: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:18.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:18.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:05:18.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:18.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:18.854 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:18.878 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:05:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:05:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:18.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[107278]: pgmap v1479: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:18.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:18.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:18.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:05:18.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:05:18.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:18.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:18 smithi003 ceph-mon[104007]: pgmap v1479: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:18.935 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:18.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:19.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.128 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:19.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:19.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:05:19.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:19.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:19.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:19.403 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:19.440 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:19.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:19.528 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:19.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:19.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:19.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:19.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:19.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:20.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:20.028 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:20.047 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:20.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:20.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:20.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:20.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:20.298 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:05:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:05:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:20.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:20.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:20.496 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:20.497 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:20.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:20.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:20.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:20.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:20.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:20.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:20.903 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:21.005 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.029 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:21.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:21.130 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:21.211 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:05:21.218 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_9' 2024-09-13T15:05:21.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:21.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:21.322 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:05:21.323 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:21.340 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:05:21.340 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:21.355 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:05:21.356 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:21.372 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:05:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:21.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:21.388 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:05:21.388 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:21.404 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:05:21.404 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:21.405 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:21.420 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:05:21.420 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:21.436 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:05:21.436 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:21.474 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:21.506 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:21.520 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:21 smithi003 ceph-mon[107278]: pgmap v1480: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:21.520 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1176111446' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:05:21.520 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:21 smithi003 ceph-mon[104007]: pgmap v1480: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:21.520 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1176111446' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:05:21.528 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:21.595 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:21 smithi019 ceph-mon[91296]: pgmap v1480: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:21.595 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1176111446' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:05:21.630 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:21.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:21.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:21.776 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:21.878 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:21.900 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:22.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.025 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:22.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:22.126 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:22.148 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:22.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:22.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1176111446' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:05:22.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:22 smithi003 ceph-mon[107278]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T15:05:22.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3315361277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:22.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1176111446' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:05:22.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:22 smithi003 ceph-mon[104007]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T15:05:22.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3315361277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:22.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:22.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:22.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:22.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:22.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:22.510 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1176111446' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:05:22.510 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:22 smithi019 ceph-mon[91296]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T15:05:22.510 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3315361277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:22.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:22.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:22.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:22.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:22.768 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:22.794 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:22.896 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:22.919 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:23.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:23.043 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:23.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:23.101 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:05:23.101 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:05:23.101 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T15:05:23.101 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:05:23.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:23.167 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:23.269 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:23.291 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:23.393 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:23.416 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:23.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:23 smithi019 ceph-mon[91296]: pgmap v1482: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:23 smithi019 ceph-mon[91296]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:05:23.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:23 smithi003 ceph-mon[107278]: pgmap v1482: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:23.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:23 smithi003 ceph-mon[107278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:05:23.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:23 smithi003 ceph-mon[104007]: pgmap v1482: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:23.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:23 smithi003 ceph-mon[104007]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:05:23.518 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:23.540 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:23.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:23.642 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:23.664 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:23.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:23.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:23.833 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:05:23.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:23.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:24.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-09-13T15:05:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-09-13T15:05:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:24.093 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:24.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:24.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:24.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:24.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:24.381 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:05:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:05:24] "GET /metrics HTTP/1.1" 200 36457 "" "Prometheus/2.51.0" 2024-09-13T15:05:24.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:24.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:24.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:24.519 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:24.541 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:24.566 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_9 bench 90 write' 2024-09-13T15:05:24.568 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T15:05:24.568 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T15:05:24.615 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-09-13T15:05:24.615 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T15:05:24.616 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_327099 2024-09-13T15:05:24.616 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:05:24.616 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-09-13T15:05:24.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:24.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:24.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:24.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:05:24.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:24.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:24.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:24.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:24.907 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:24.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:24.982 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:25.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:25.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:25.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:05:25.253 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T15:05:25.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-13T15:05:25.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:25.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:25.418 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:25.493 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:25 smithi019 ceph-mon[91296]: pgmap v1483: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:25.520 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:25.551 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:25.615 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3067 3051 190.655 190.688 0.00261121 0.0052244 2024-09-13T15:05:25.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:25.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:25.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:25 smithi003 ceph-mon[107278]: pgmap v1483: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:25.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:25 smithi003 ceph-mon[104007]: pgmap v1483: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:25.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:25.816 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:25.831 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:25.840 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:25.888 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:05:25.888 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:05:25.888 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:05:25.888 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:05:25.888 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:05:25.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:25.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:26.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.109 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:26.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:26.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:26.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:26.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:26.376 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:26.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:26.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:26.570 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:05:26.615 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 5844 5828 182.101 173.562 0.00463595 0.00547715 2024-09-13T15:05:26.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:26.652 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:26.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:26.787 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:26.890 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:26.906 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:05:26.907 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:26 smithi003 ceph-mon[107278]: pgmap v1484: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:26.907 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/245130954' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:26.907 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:26 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:26.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:26 smithi003 ceph-mon[104007]: pgmap v1484: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:26.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/245130954' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:26.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:26 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:26.925 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:26.968 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:27.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:27.061 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:27.137 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:26 smithi019 ceph-mon[91296]: pgmap v1484: 119 pgs: 119 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:05:27.137 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/245130954' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:27.137 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:26 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:27.164 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:27.195 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:27.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:27.330 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:27.432 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:27.471 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:27.573 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:27.601 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:27.615 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 8962 8946 186.353 194.875 0.00629413 0.00535449 2024-09-13T15:05:27.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:27.733 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:27.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:27.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:27.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:27.975 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:28.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:28.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:28.143 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:27 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:05:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:27 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:05:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:27 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:27 smithi019 ceph-mon[91296]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T15:05:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:28.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:27 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:05:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:05:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[107278]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T15:05:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:28.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:05:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:05:28.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:28.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[104007]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T15:05:28.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:28.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:27 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:28.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:28.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:28.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:28.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:28.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:28.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:28.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:28.616 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 11648 11632 181.731 167.875 0.00409807 0.00549561 2024-09-13T15:05:28.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:28.805 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:28.835 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:28.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:28.966 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:29.068 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:29.100 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:28 smithi019 ceph-mon[91296]: pgmap v1486: 119 pgs: 119 active+clean; 385 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 48 MiB/s wr, 768 op/s 2024-09-13T15:05:29.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:28 smithi003 ceph-mon[107278]: pgmap v1486: 119 pgs: 119 active+clean; 385 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 48 MiB/s wr, 768 op/s 2024-09-13T15:05:29.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:28 smithi003 ceph-mon[104007]: pgmap v1486: 119 pgs: 119 active+clean; 385 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 48 MiB/s wr, 768 op/s 2024-09-13T15:05:29.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:29.237 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:29.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:29.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:29.482 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:29.514 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:29.616 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 14332 14316 178.932 167.75 0.0129562 0.00557788 2024-09-13T15:05:29.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:29.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:29.770 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:29.801 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:29.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:29.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:30.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:30.072 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:30.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:30.206 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:30.209 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:174765 12.f 12:f0a86859:::benchmark_data_smithi003_259359_object59139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:09.410588+0000", 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 380.769956992, 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.279047816, 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 174765 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410588+0000", 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.211 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410588+0000", 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410587+0000", 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410592+0000", 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3739999999999997e-06 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.212 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410603+0000", 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0682999999999999e-05 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410634+0000", 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1080000000000001e-05 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410710+0000", 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5748999999999996e-05 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410833+0000", 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012261800000000001 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.410951+0000", 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000118779 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.416669+0000", 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057180470000000004 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.422507+0000", 2024-09-13T15:05:30.215 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0058372049999999998 2024-09-13T15:05:30.216 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.216 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.216 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.216 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689544+0000", 2024-09-13T15:05:30.216 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26703726700000002 2024-09-13T15:05:30.216 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.216 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.216 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.216 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689593+0000", 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9254e-05 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689636+0000", 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2843999999999998e-05 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.46703.0:174767 12.f 12:f1a47245:::benchmark_data_smithi003_259359_object59141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T15:05:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T14:59:09.412636+0000", 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 380.76790900899999, 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.27712137100000001, 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.46703", 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/1963956269", 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 174767 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412636+0000", 2024-09-13T15:05:30.218 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412636+0000", 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412635+0000", 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.219 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412641+0000", 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6579999999999999e-06 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412658+0000", 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7682000000000001e-05 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.412664+0000", 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.818e-06 2024-09-13T15:05:30.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.413722+0000", 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001057793 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.413846+0000", 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000123569 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:05:30.221 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.413969+0000", 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012367699999999999 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.416764+0000", 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027944110000000001 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.422532+0000", 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057681219999999997 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.222 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689683+0000", 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.26715064700000002 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689722+0000", 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9638000000000002e-05 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T14:59:09.689757+0000", 2024-09-13T15:05:30.223 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5339999999999997e-05 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.36909.0:77488 13.7 e372/362 13:eeaf4936:::benchmark_data_smithi003_293363_object77487:head v 372'4874, mlcod=372'4872)", 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:52.283562+0000", 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 217.89698313700001, 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25382753200000002, 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.224 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283562+0000", 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283562+0000", 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.225 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283561+0000", 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283587+0000", 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6443000000000001e-05 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283601+0000", 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3518e-05 2024-09-13T15:05:30.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283608+0000", 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8159999999999996e-06 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283652+0000", 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4401999999999999e-05 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.283672+0000", 2024-09-13T15:05:30.227 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9782000000000001e-05 2024-09-13T15:05:30.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.228 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.228 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.228 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537344+0000", 2024-09-13T15:05:30.228 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25367196800000003 2024-09-13T15:05:30.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.228 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.228 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.537390+0000", 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5686000000000003e-05 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.36909.0:77497 13.7 e372/362 13:e94a3da8:::benchmark_data_smithi003_293363_object77496:head v 372'4876, mlcod=372'4872)", 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:01:52.289980+0000", 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 217.890565272, 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25818336600000003, 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.229 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.230 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.230 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.230 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.230 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.289980+0000", 2024-09-13T15:05:30.230 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.230 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.230 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.230 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.230 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.289980+0000", 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.289979+0000", 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.290006+0000", 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6775999999999999e-05 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.231 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534627+0000", 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.244621595 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534647+0000", 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9950999999999999e-05 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534686+0000", 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9085999999999998e-05 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.534710+0000", 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.3839999999999999e-05 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.547839+0000", 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013128862999999999 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.233 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:01:52.548163+0000", 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000324075 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203840 13.4 13:2a6c846f:::benchmark_data_smithi003_293363_object84628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.245586+0000", 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 73.934959379999995, 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.36584723099999999, 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:05:30.234 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203840 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.245586+0000", 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.245586+0000", 2024-09-13T15:05:30.235 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.236 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.236 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.236 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.236 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.245585+0000", 2024-09-13T15:05:30.236 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:05:30.236 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.236 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.236 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.237 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.245588+0000", 2024-09-13T15:05:30.237 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7769999999999998e-06 2024-09-13T15:05:30.237 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.237 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.237 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.237 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.245595+0000", 2024-09-13T15:05:30.237 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.545e-06 2024-09-13T15:05:30.237 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.245598+0000", 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5700000000000001e-06 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.251359+0000", 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057609139999999998 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.240 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.251565+0000", 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00020614400000000001 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.251673+0000", 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000107488 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.241 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.611310+0000", 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.35963700999999998 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.611433+0000", 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012321200000000001 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.242 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203846 13.f 13:fdee251f:::benchmark_data_smithi003_293363_object84634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.253303+0000", 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 73.927242003999993, 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.33966548200000002, 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.243 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203846 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.253303+0000", 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.253303+0000", 2024-09-13T15:05:30.244 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.253302+0000", 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.253308+0000", 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9020000000000004e-06 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.253324+0000", 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6807999999999999e-05 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.253330+0000", 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.293e-06 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.246 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.256505+0000", 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0031754650000000001 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.256629+0000", 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012345499999999999 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.256740+0000", 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000111661 2024-09-13T15:05:30.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.263769+0000", 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0070288 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.587688+0000", 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.32391922200000001 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.248 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.592861+0000", 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0051726519999999998 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.592928+0000", 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7012000000000001e-05 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.249 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.592969+0000", 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0677999999999999e-05 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203839 13.11 13:8c49368b:::benchmark_data_smithi003_293363_object84627:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 73.919300860000007, 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.123564244, 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:05:30.250 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203839 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.251 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261243+0000", 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.252 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261250+0000", 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7020000000000001e-06 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261263+0000", 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2676e-05 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261269+0000", 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8490000000000002e-06 2024-09-13T15:05:30.253 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.502730+0000", 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.241461017 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.502806+0000", 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5870000000000004e-05 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.254 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.379709+0000", 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.876903561 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.379851+0000", 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014137500000000001 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.384603+0000", 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0047521660000000004 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.255 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.384809+0000", 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00020574999999999999 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203840 13.14 13:2a6c846f:::benchmark_data_smithi003_293363_object84628:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261277+0000", 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 73.919268535, 2024-09-13T15:05:30.256 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.089639778, 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203840 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261277+0000", 2024-09-13T15:05:30.257 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261277+0000", 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261276+0000", 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:05:30.258 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.259 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.259 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.259 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261279+0000", 2024-09-13T15:05:30.259 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7089999999999998e-06 2024-09-13T15:05:30.259 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.259 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.259 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261284+0000", 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3399999999999997e-06 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261288+0000", 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.45e-06 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612519+0000", 2024-09-13T15:05:30.260 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.35123135700000002 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612526+0000", 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3339999999999996e-06 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341783+0000", 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.72925640000000003 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.261 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341904+0000", 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120869 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350844+0000", 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0089401470000000007 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350916+0000", 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2562000000000001e-05 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.262 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203847 13.11 13:8898f2cd:::benchmark_data_smithi003_293363_object84635:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261293+0000", 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 73.919252426, 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1261630679999999, 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.263 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203847 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261293+0000", 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261293+0000", 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261292+0000", 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261295+0000", 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.193e-06 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.265 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261299+0000", 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6909999999999996e-06 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261302+0000", 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.19e-06 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.266 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.503068+0000", 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24176531500000001 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.503987+0000", 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00091926099999999999 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.379883+0000", 2024-09-13T15:05:30.267 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.87589635300000002 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.380025+0000", 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014113400000000001 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.387391+0000", 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0073666950000000004 2024-09-13T15:05:30.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.387456+0000", 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4535000000000005e-05 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203853 13.14 13:2dc48b45:::benchmark_data_smithi003_293363_object84641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.310940+0000", 2024-09-13T15:05:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 73.869605075999999, 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0401961900000001, 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203853 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310940+0000", 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310940+0000", 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310937+0000", 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999971 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.271 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310952+0000", 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4966e-05 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310990+0000", 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.756e-05 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.311000+0000", 2024-09-13T15:05:30.272 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0484e-05 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612544+0000", 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30154405099999998 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612549+0000", 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4390000000000002e-06 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.273 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341929+0000", 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.72938062400000003 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.342019+0000", 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9975000000000005e-05 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:05:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.342293+0000", 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00027399400000000001 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.345037+0000", 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002743255 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.347766+0000", 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027296099999999999 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.276 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.276 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350999+0000", 2024-09-13T15:05:30.276 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0032323930000000001 2024-09-13T15:05:30.277 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.277 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.351040+0000", 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1807999999999997e-05 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.351136+0000", 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5866000000000007e-05 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.278 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203861 13.1 13:85b724fb:::benchmark_data_smithi003_293363_object84649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.825629+0000", 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 73.354916075999995, 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.55450867100000001, 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.279 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203861 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.825629+0000", 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.280 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.825629+0000", 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.825628+0000", 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.825635+0000", 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3980000000000002e-06 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.281 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.825648+0000", 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2974e-05 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.825653+0000", 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9590000000000003e-06 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.825714+0000", 2024-09-13T15:05:30.282 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0918000000000002e-05 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.825728+0000", 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.362e-05 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.342632+0000", 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.51690438999999999 2024-09-13T15:05:30.283 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.342730+0000", 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8115000000000005e-05 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,4", 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.343115+0000", 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000385356 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.284 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350035+0000", 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0069199070000000003 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350703+0000", 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00066736300000000003 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.285 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.380063+0000", 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.029360654 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.380105+0000", 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1501000000000001e-05 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.380138+0000", 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2848999999999997e-05 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.286 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230557 13.4 13:2562e8b0:::benchmark_data_smithi003_293363_object111345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.432865+0000", 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 49.747679912000002, 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.25501063299999999, 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230557 2024-09-13T15:05:30.287 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.432865+0000", 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.432865+0000", 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.288 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.432864+0000", 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.432870+0000", 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6869999999999999e-06 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.432894+0000", 2024-09-13T15:05:30.289 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4017000000000001e-05 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.432900+0000", 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5139999999999998e-06 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.432947+0000", 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7095000000000002e-05 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.290 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.433055+0000", 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010802999999999999 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.433462+0000", 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00040637799999999999 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.436580+0000", 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0031183280000000001 2024-09-13T15:05:30.291 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.631407+0000", 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19482653899999999 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.687784+0000", 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.056377113 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.292 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.687834+0000", 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0308e-05 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.687876+0000", 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1829000000000002e-05 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.293 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230539 13.f 13:f205da92:::benchmark_data_smithi003_293363_object111327:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521183+0000", 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 49.659361658999998, 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96228324200000004, 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230539 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.294 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521183+0000", 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521183+0000", 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521182+0000", 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:05:30.295 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521192+0000", 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0701e-05 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521207+0000", 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4569000000000001e-05 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.296 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521213+0000", 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4409999999999998e-06 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561286+0000", 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040072137000000001 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561291+0000", 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9390000000000002e-06 2024-09-13T15:05:30.297 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483382+0000", 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92209029899999995 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483467+0000", 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4948000000000001e-05 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.298 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230542 13.1f 13:fe3a49e0:::benchmark_data_smithi003_293363_object111330:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 49.659322781999997, 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99456133199999996, 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.299 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230542 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.300 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521225+0000", 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7439999999999999e-06 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.301 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521230+0000", 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.558e-06 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521233+0000", 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3809999999999999e-06 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561254+0000", 2024-09-13T15:05:30.302 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040020864000000003 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561262+0000", 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3529999999999997e-06 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515697+0000", 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95443503600000001 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.303 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515784+0000", 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6948999999999997e-05 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230548 13.f 13:f2b282bc:::benchmark_data_smithi003_293363_object111336:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521238+0000", 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 49.65930762, 2024-09-13T15:05:30.304 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96231032900000002, 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230548 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521238+0000", 2024-09-13T15:05:30.305 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521238+0000", 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521237+0000", 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.306 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521239+0000", 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9539999999999998e-06 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521244+0000", 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8770000000000001e-06 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.307 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521247+0000", 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1329999999999999e-06 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.563020+0000", 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.041772769000000001 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.563051+0000", 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1293e-05 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483490+0000", 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92043902 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483548+0000", 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7589e-05 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230549 13.1 13:828281d2:::benchmark_data_smithi003_293363_object111337:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521251+0000", 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 49.659294584999998, 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93879572200000005, 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230549 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.310 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521251+0000", 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521251+0000", 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.311 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521250+0000", 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521252+0000", 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.001e-06 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521257+0000", 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8670000000000002e-06 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521260+0000", 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.004e-06 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523807+0000", 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002547218 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.313 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523836+0000", 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8328000000000001e-05 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.459966+0000", 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93613046200000005 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460046+0000", 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0149999999999997e-05 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230559 13.1f 13:feac4672:::benchmark_data_smithi003_293363_object111347:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 49.659280201999998, 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.000411124, 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230559 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:05:30.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521266+0000", 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8840000000000001e-06 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521271+0000", 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6990000000000002e-06 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521274+0000", 2024-09-13T15:05:30.317 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1939999999999998e-06 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561310+0000", 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040035846 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561316+0000", 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0349999999999998e-06 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.318 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515819+0000", 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95450267600000005 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515931+0000", 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000111594 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.516267+0000", 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000336163 2024-09-13T15:05:30.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.519875+0000", 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003608385 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.519881+0000", 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3010000000000003e-06 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.320 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.521593+0000", 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0017120709999999999 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.521641+0000", 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7216000000000003e-05 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.521676+0000", 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5380999999999999e-05 2024-09-13T15:05:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230561 13.1 13:826700a8:::benchmark_data_smithi003_293363_object111349:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521278+0000", 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 49.659267380999999, 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95642722800000002, 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.322 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230561 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521278+0000", 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521278+0000", 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521277+0000", 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521279+0000", 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9659999999999999e-06 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521284+0000", 2024-09-13T15:05:30.324 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5759999999999999e-06 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521287+0000", 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.96e-06 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523964+0000", 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026767449999999999 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.325 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523988+0000", 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4579e-05 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460069+0000", 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93608031000000003 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460199+0000", 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013029900000000001 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,4", 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460537+0000", 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00033807200000000003 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477550+0000", 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.017013495999999999 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.327 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477558+0000", 2024-09-13T15:05:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2479999999999999e-06 2024-09-13T15:05:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477616+0000", 2024-09-13T15:05:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7852000000000001e-05 2024-09-13T15:05:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477669+0000", 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3695e-05 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477705+0000", 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5771e-05 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230577 13.f 13:f354a541:::benchmark_data_smithi003_293363_object111365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.775742+0000", 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 49.404803057999999, 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.74209197199999999, 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230577 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.775742+0000", 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.775742+0000", 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.775740+0000", 2024-09-13T15:05:30.331 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999976 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.775752+0000", 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1888999999999999e-05 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.775776+0000", 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4250999999999999e-05 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.332 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.775785+0000", 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7299999999999994e-06 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.775955+0000", 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000170147 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.775983+0000", 2024-09-13T15:05:30.333 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8543000000000001e-05 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483570+0000", 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.70758660699999998 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483684+0000", 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000114385 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:05:30.334 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.484078+0000", 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000393867 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515683+0000", 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.031605217999999997 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.517717+0000", 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002033974 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.335 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.517756+0000", 2024-09-13T15:05:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8791000000000002e-05 2024-09-13T15:05:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.517799+0000", 2024-09-13T15:05:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2870000000000001e-05 2024-09-13T15:05:30.336 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.517834+0000", 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4961e-05 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:3799 14.4 14:2b0f9b88:::benchmark_data_smithi003_327099_object3798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:05:25.919258+0000", 2024-09-13T15:05:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 4.2612867899999998, 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.024197659, 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 3799 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.338 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.919258+0000", 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.919258+0000", 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.919257+0000", 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:05:30.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.919333+0000", 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5495999999999997e-05 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.919345+0000", 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2235000000000001e-05 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.919350+0000", 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0429999999999998e-06 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.919398+0000", 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7933999999999999e-05 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.919536+0000", 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000138135 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.919695+0000", 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015879499999999999 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.922908+0000", 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003212861 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.923265+0000", 2024-09-13T15:05:30.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00035691899999999998 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.943369+0000", 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.020104770000000001 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.943424+0000", 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4308999999999998e-05 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:05:30.343 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:05:25.943456+0000", 2024-09-13T15:05:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2357000000000002e-05 2024-09-13T15:05:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:05:30.344 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:05:30.344 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:05:30.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:30.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:30.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:05:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:05:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:30.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:30.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:30.616 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 16520 16504 171.9 136.75 0.0123577 0.00580957 2024-09-13T15:05:30.626 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:30.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:30.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:30.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:30.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:05:31.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:31.066 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:31.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:05:31.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:31.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:31.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:31.359 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:31.387 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:31 smithi003 ceph-mon[107278]: pgmap v1487: 119 pgs: 119 active+clean; 662 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T15:05:31.388 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:31 smithi003 ceph-mon[104007]: pgmap v1487: 119 pgs: 119 active+clean; 662 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T15:05:31.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:31 smithi019 ceph-mon[91296]: pgmap v1487: 119 pgs: 119 active+clean; 662 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T15:05:31.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:31.464 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-09-13T15:05:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-09-13T15:05:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-09-13T15:05:31.494 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:05:31.494 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:05:31.495 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:05:31.495 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:05:31.495 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:31.495 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:31.495 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:31.495 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:31.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:31.519 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:05:31.520 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:31.555 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:05:31.555 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:31.591 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:05:31.591 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:31.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:31.616 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 18444 18428 164.515 120.25 0.00407817 0.00607245 2024-09-13T15:05:31.624 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:05:31.624 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:31.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:31.741 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:31.771 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:31.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:31.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:32.014 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:32.046 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:32.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:32.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:32.291 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:32.322 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:32.424 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:32.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2040897851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:32.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2040897851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:32.457 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:05:32.535 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T15:05:32.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-13T15:05:32.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:32.616 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 20875 20859 162.938 151.938 0.00964684 0.00613197 2024-09-13T15:05:32.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:32.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2040897851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:32.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:32.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:32.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:32.923 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:33.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:33.072 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:33.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:33.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:33.180 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:33.205 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:33.212 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:33.314 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:33.342 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:33 smithi019 ceph-mon[91296]: pgmap v1488: 119 pgs: 119 active+clean; 662 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T15:05:33.353 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:33.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:33.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:33 smithi003 ceph-mon[107278]: pgmap v1488: 119 pgs: 119 active+clean; 662 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T15:05:33.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:33 smithi003 ceph-mon[104007]: pgmap v1488: 119 pgs: 119 active+clean; 662 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T15:05:33.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:33.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:33.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:33.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:33.617 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 22881 22865 158.758 125.375 0.00649053 0.00629256 2024-09-13T15:05:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:33.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:33.657 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:33.657 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:33.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:33.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:33.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:33.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:33.994 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:34.016 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:34.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:34.145 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:34.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:34.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:34.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:34.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:05:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:05:34] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T15:05:34.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:34.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:34.617 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 24643 24627 153.894 110.125 0.00549675 0.00649227 2024-09-13T15:05:34.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:34.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:34.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:34.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:34.926 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:34.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:35.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:05:35.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:35.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:35.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:35.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:35.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:35.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:35.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:35.570 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:35.617 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 26278 26262 149.193 102.188 0.0045378 0.00669655 2024-09-13T15:05:35.671 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:35 smithi019 ceph-mon[91296]: pgmap v1489: 119 pgs: 119 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T15:05:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:35 smithi003 ceph-mon[107278]: pgmap v1489: 119 pgs: 119 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T15:05:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:35 smithi003 ceph-mon[104007]: pgmap v1489: 119 pgs: 119 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T15:05:35.705 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:35.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:35.950 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:35.982 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:35.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:36.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:36.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:36.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:05:36.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:36.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:36.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:36.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:36.495 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:36.524 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:36.541 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:36.617 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 28058 28042 146.03 111.25 0.00310321 0.00684107 2024-09-13T15:05:36.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:36.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:36.766 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:36.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:36.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:36.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:36.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:37.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:37.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:37.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:37.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:37.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:37.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:37.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:37.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:37.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:37.617 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 29682 29666 142.604 101.5 0.0137846 0.00700634 2024-09-13T15:05:37.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:37 smithi019 ceph-mon[91296]: pgmap v1490: 119 pgs: 119 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T15:05:37.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:37 smithi003 ceph-mon[104007]: pgmap v1490: 119 pgs: 119 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T15:05:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:37 smithi003 ceph-mon[107278]: pgmap v1490: 119 pgs: 119 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T15:05:37.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:37.757 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:37.860 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:37.883 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:37.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:38.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:38.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:38.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:38.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:38.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:38.485 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:38.485 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:38.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:38.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:38.617 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 31250 31234 139.417 98 0.00345594 0.00716703 2024-09-13T15:05:38.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:38.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:38.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:38.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:39.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:39.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:39.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:39.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:05:39.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:39.287 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:39.327 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:39.430 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:39.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:39 smithi019 ceph-mon[91296]: pgmap v1491: 119 pgs: 119 active+clean; 1.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 151 MiB/s wr, 2.41k op/s 2024-09-13T15:05:39.465 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:39.541 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:05:39.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:39.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:39.617 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 33049 33033 137.618 112.438 0.00453983 0.00726039 2024-09-13T15:05:39.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:39 smithi003 ceph-mon[107278]: pgmap v1491: 119 pgs: 119 active+clean; 1.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 151 MiB/s wr, 2.41k op/s 2024-09-13T15:05:39.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:39 smithi003 ceph-mon[104007]: pgmap v1491: 119 pgs: 119 active+clean; 1.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 151 MiB/s wr, 2.41k op/s 2024-09-13T15:05:39.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:39.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:39.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:39.996 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:40.018 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:40.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:40.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:40.152 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:40.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:40.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:40.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:40.346 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:05:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:05:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:40.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:40.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:40.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:40.568 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:40.617 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 34783 34767 135.79 108.375 0.0184881 0.00735699 2024-09-13T15:05:40.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:40.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:40.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:40.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:40.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:40.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:40.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:41.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:41.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:41.236 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:41.267 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:41.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:41.378 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:41.395 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:41.487 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:41 smithi003 ceph-mon[104007]: pgmap v1492: 119 pgs: 119 active+clean; 1.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 127 MiB/s wr, 2.04k op/s 2024-09-13T15:05:41.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:41 smithi003 ceph-mon[107278]: pgmap v1492: 119 pgs: 119 active+clean; 1.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 127 MiB/s wr, 2.04k op/s 2024-09-13T15:05:41.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:41.530 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:41.596 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:41 smithi019 ceph-mon[91296]: pgmap v1492: 119 pgs: 119 active+clean; 1.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 127 MiB/s wr, 2.04k op/s 2024-09-13T15:05:41.617 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 36511 36495 134.155 108 0.0205422 0.00744724 2024-09-13T15:05:41.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:41.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:41.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:41.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:41.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:41.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:41.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:42.043 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:42.043 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:42.067 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-09-13T15:05:42.067 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-09-13T15:05:42.067 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-09-13T15:05:42.067 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-09-13T15:05:42.067 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-09-13T15:05:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-09-13T15:05:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-09-13T15:05:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-09-13T15:05:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-09-13T15:05:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-09-13T15:05:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-09-13T15:05:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-09-13T15:05:42.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:42.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:42.211 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:42.243 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:42.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:42.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:42.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:42.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:42.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:42.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 38104 38088 132.232 99.5625 0.00564189 0.00755675 2024-09-13T15:05:42.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:42.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:42.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:42.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:42.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T15:05:42.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:42.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:42.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:43.053 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:43.079 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:43.181 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:43.219 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:43.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:43.262 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:05:43.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:43 smithi019 ceph-mon[91296]: pgmap v1493: 119 pgs: 119 active+clean; 1.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-13T15:05:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/273302155' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:05:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:43 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:05:43.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:43.464 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:43.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:43.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:43.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 39636 39620 130.312 95.75 0.00586383 0.00766843 2024-09-13T15:05:43.629 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:43 smithi003 ceph-mon[107278]: pgmap v1493: 119 pgs: 119 active+clean; 1.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-13T15:05:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/273302155' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:05:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:43 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:05:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:43 smithi003 ceph-mon[104007]: pgmap v1493: 119 pgs: 119 active+clean; 1.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-13T15:05:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/273302155' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:05:43.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:43 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:05:43.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:43.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:43.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:43.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:44.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:44.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:44.155 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:44.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:05:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:05:44] "GET /metrics HTTP/1.1" 200 36520 "" "Prometheus/2.51.0" 2024-09-13T15:05:44.187 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:44.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:44.429 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:44.463 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:44.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:44.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:44.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T15:05:44.617563+0000 min lat: 0.00114407 max lat: 0.172613 avg lat: 0.00777715 2024-09-13T15:05:44.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:05:44.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 15 41016 41001 128.112 86.3125 0.127452 0.00777715 2024-09-13T15:05:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:44 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:44 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:05:44.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:44 smithi019 ceph-mon[91296]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T15:05:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:44 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:44 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:05:44.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:44 smithi003 ceph-mon[104007]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T15:05:44.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:44 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:44.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:44 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:05:44.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:44 smithi003 ceph-mon[107278]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T15:05:44.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:44.713 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:44.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:44.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:44.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:44.986 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:45.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:45.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:45.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:45.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:45.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:45.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:45.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:45.522 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:45.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 42297 42281 125.82 80 0.00169863 0.00793975 2024-09-13T15:05:45.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:45.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:45 smithi019 ceph-mon[91296]: pgmap v1495: 119 pgs: 119 active+clean; 2.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T15:05:45.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:45 smithi003 ceph-mon[104007]: pgmap v1495: 119 pgs: 119 active+clean; 2.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T15:05:45.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:45 smithi003 ceph-mon[107278]: pgmap v1495: 119 pgs: 119 active+clean; 2.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T15:05:45.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:45.789 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:45.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:45.913 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:46.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:46.042 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:46.144 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:46.174 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:46.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:46.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:46.306 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:46.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:46.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:46.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:46.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:46.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 43852 43836 124.518 97.1875 0.00383128 0.00802559 2024-09-13T15:05:46.683 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:46.711 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:05:46.800 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T15:05:46.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-13T15:05:46.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:46.838 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:46.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:46.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:46.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:47.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:47.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:47.246 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:47.268 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:47.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:47.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:47 smithi003 ceph-mon[107278]: pgmap v1496: 119 pgs: 119 active+clean; 2.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T15:05:47.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:47 smithi003 ceph-mon[104007]: pgmap v1496: 119 pgs: 119 active+clean; 2.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T15:05:47.495 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:47.517 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:47.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 44887 44871 121.917 64.6875 0.0106501 0.00819716 2024-09-13T15:05:47.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:47.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:47 smithi019 ceph-mon[91296]: pgmap v1496: 119 pgs: 119 active+clean; 2.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T15:05:47.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:47.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:47.901 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:47.930 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:48.032 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:48.060 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:48.162 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:48.190 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:48.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:48.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:48.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:48.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 46546 46530 121.157 103.688 0.0114274 0.00824736 2024-09-13T15:05:48.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:48.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:48.747 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:48.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:48.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:48.912 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:49.014 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:49.043 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:49.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:49.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:49.185 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:49.287 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:49.317 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:49.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:49.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:49 smithi003 ceph-mon[107278]: pgmap v1497: 119 pgs: 119 active+clean; 2.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-13T15:05:49.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:49 smithi003 ceph-mon[104007]: pgmap v1497: 119 pgs: 119 active+clean; 2.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-13T15:05:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:49.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:49.566 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:49.594 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:49.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 48043 48027 120.053 93.5625 0.0149352 0.00832456 2024-09-13T15:05:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:49 smithi019 ceph-mon[91296]: pgmap v1497: 119 pgs: 119 active+clean; 2.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-13T15:05:49.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:49.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:49.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:49.839 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:49.866 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:49.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:50.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:50.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:50.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:50.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:50.299 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:50.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:50.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:50.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:50.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:50.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:50.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 49522 49506 118.99 92.4375 0.00266921 0.00838834 2024-09-13T15:05:50.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:05:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:05:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:05:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:50.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:50.866 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:50.997 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:51.027 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:51.129 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:51.156 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:51.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:51.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:51.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:51.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:51.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:51.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:51.619 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 50534 50518 116.925 63.25 0.00223465 0.00846969 2024-09-13T15:05:51.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:51.671 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:51 smithi019 ceph-mon[91296]: pgmap v1498: 119 pgs: 119 active+clean; 2.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T15:05:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:51 smithi003 ceph-mon[107278]: pgmap v1498: 119 pgs: 119 active+clean; 2.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T15:05:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:51 smithi003 ceph-mon[104007]: pgmap v1498: 119 pgs: 119 active+clean; 2.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T15:05:51.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:51.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:51.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:52.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:52.059 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:52.161 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:52.194 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:52.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:52.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:52.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:52.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:52.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:52.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:52.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:52.618 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 51710 51694 115.374 73.5 0.00339247 0.00864901 2024-09-13T15:05:52.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:52.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:52.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:52.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:52.858 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:52.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:52.959 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:53.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:53.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:53.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:53.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:53.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:53.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:53.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:53.532 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:53.546 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:53 smithi019 ceph-mon[91296]: pgmap v1499: 119 pgs: 119 active+clean; 2.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T15:05:53.621 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 52649 52633 113.411 58.6875 0.00340342 0.00881156 2024-09-13T15:05:53.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:53.666 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:53.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:53 smithi003 ceph-mon[104007]: pgmap v1499: 119 pgs: 119 active+clean; 2.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T15:05:53.668 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:53 smithi003 ceph-mon[107278]: pgmap v1499: 119 pgs: 119 active+clean; 2.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T15:05:53.767 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:53.796 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:53.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:53.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:54.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:54.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:54.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:05:54.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:05:54.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:05:54.092 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:05:54.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:54.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:54.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:54.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:54.092 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:05:54.092 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:05:54.093 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:05:54.093 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:54.093 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:54.093 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:54.101 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:05:54.101 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:54.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:54.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:54.306 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:05:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:05:54] "GET /metrics HTTP/1.1" 200 36520 "" "Prometheus/2.51.0" 2024-09-13T15:05:54.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:54.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:54.452 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:54.480 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:54.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:54.621 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 54037 54021 112.522 86.75 0.00278546 0.00888132 2024-09-13T15:05:54.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:54 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2684904605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:54.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:54 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2684904605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:54.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:54 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2684904605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:54.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:54.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:54.870 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:54.899 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:55.000 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:55.031 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:55.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:55.160 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:55.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.295 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:55.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:55.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:55.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:55.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:55.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:55.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:55 smithi003 ceph-mon[107278]: pgmap v1500: 119 pgs: 119 active+clean; 3.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-13T15:05:55.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:55 smithi003 ceph-mon[104007]: pgmap v1500: 119 pgs: 119 active+clean; 3.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-13T15:05:55.621 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 55476 55460 111.793 89.9375 0.00653753 0.00893849 2024-09-13T15:05:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:55 smithi019 ceph-mon[91296]: pgmap v1500: 119 pgs: 119 active+clean; 3.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-13T15:05:55.706 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:55.732 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:55.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:55.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:55.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:56.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:56.087 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:05:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:05:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:56027 14.b 14:dd5c96e6:::benchmark_data_smithi003_327099_object56026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-13T15:05:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.043194+0000", 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.012181468000000001, 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012213577, 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56027 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.043194+0000", 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.043194+0000", 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.043195+0000", 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4000000000000002e-07 2024-09-13T15:05:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.043214+0000", 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9775999999999998e-05 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.043222+0000", 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6729999999999992e-06 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.043228+0000", 2024-09-13T15:05:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9560000000000002e-06 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.046697+0000", 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034693559999999998 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.047768+0000", 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010709389999999999 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:05:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.048081+0000", 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031290199999999999 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.053973+0000", 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058920750000000001 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.054273+0000", 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029985500000000002 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:56031 14.3 14:c0ec31d1:::benchmark_data_smithi003_327099_object56030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.045775+0000", 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0096003600000000005, 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00971203, 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56031 2024-09-13T15:05:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.045775+0000", 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.045775+0000", 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.045777+0000", 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5340000000000001e-06 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.045941+0000", 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016437900000000001 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.045954+0000", 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3067999999999999e-05 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.045962+0000", 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5959999999999998e-06 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.049444+0000", 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034820110000000001 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.049622+0000", 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017861999999999999 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.049752+0000", 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000129395 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.054915+0000", 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051636980000000004 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:56008 14.3 14:c4c64216:::benchmark_data_smithi003_327099_object56007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.029763+0000", 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.025612599, 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.02579211, 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56008 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.029763+0000", 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.029763+0000", 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.029764+0000", 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.012e-06 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.029918+0000", 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015376100000000001 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.029935+0000", 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7535999999999999e-05 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.029943+0000", 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0790000000000001e-06 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.032399+0000", 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024559819999999998 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.032538+0000", 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000138412 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.032674+0000", 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013608000000000001 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.035390+0000", 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027162380000000002 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:56036 14.7 e422/416 14:e8867cd9:::benchmark_data_smithi003_327099_object56035:head v 422'3497, mlcod=422'3495)", 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.055032+0000", 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00034343299999999998, 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058625300000000001, 2024-09-13T15:05:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055032+0000", 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055032+0000", 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055033+0000", 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0500000000000002e-07 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055142+0000", 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010912300000000001 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055167+0000", 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4706999999999999e-05 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055174+0000", 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3209999999999999e-06 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055295+0000", 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120721 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055314+0000", 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9487000000000002e-05 2024-09-13T15:05:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.47273.0:56027 14.b e422/416)", 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.055466+0000", 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9999094, 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019703900000000001, 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055466+0000", 2024-09-13T15:05:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055466+0000", 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055466+0000", 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.72e-07 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055469+0000", 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.571e-06 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055477+0000", 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9149999999999996e-06 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055482+0000", 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3739999999999997e-06 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.47273.0:56031 14.3 e422/416)", 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.055495+0000", 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9998803, 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020928099999999999, 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055495+0000", 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055495+0000", 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055495+0000", 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3599999999999999e-07 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055498+0000", 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5299999999999999e-06 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055505+0000", 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8920000000000003e-06 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055510+0000", 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7809999999999996e-06 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:56037 14.c 14:3e6cfe6b:::benchmark_data_smithi003_327099_object56036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.055575+0000", 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9998002, 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000171986, 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:05:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56037 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055575+0000", 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055575+0000", 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055577+0000", 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.48e-06 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055601+0000", 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3981e-05 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055612+0000", 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0743e-05 2024-09-13T15:05:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055618+0000", 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2709999999999998e-06 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055716+0000", 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8121999999999993e-05 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:56038 14.c 14.5bc4819c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.055645+0000", 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9997306, 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015310700000000001, 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56038 2024-09-13T15:05:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055645+0000", 2024-09-13T15:05:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055645+0000", 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055646+0000", 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0299999999999999e-07 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055667+0000", 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1488000000000002e-05 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055676+0000", 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4800000000000001e-06 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055682+0000", 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.049e-06 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:56039 14.b 14.32dc0c3b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-13T15:05:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.055713+0000", 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9996624, 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012762099999999999, 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56039 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055713+0000", 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055713+0000", 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055714+0000", 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0999999999999998e-07 2024-09-13T15:05:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055765+0000", 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1768000000000003e-05 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055774+0000", 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3499999999999997e-06 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055789+0000", 2024-09-13T15:05:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5634e-05 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.47273.0:56008 14.3 e422/416)", 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:05:56.055818+0000", 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9995575, 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000124843, 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:05:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055818+0000", 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055818+0000", 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055819+0000", 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9500000000000003e-07 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055822+0000", 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.038e-06 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055830+0000", 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9920000000000007e-06 2024-09-13T15:05:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:05:56.055835+0000", 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6269999999999997e-06 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 10 2024-09-13T15:05:56.127 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:05:56.128 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:56.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:56.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:56.361 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:56.399 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:56.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:56.621 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 56782 56766 110.85 81.625 0.00704614 0.0090111 2024-09-13T15:05:56.682 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:05:56 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a[77474]: logger=cleanup t=2024-09-13T15:05:56.419223097Z level=info msg="Completed cleanup jobs" duration=13.113408ms 2024-09-13T15:05:56.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:56.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:56.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:56.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:57.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:57.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:57.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:57.252 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:57.274 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:57.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:57.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:05:57.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:57.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:57.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:57.621 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 58011 57995 109.819 76.8125 0.0142864 0.0091003 2024-09-13T15:05:57.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:57 smithi019 ceph-mon[91296]: pgmap v1501: 119 pgs: 119 active+clean; 3.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:05:57.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:57 smithi003 ceph-mon[107278]: pgmap v1501: 119 pgs: 119 active+clean; 3.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:05:57.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:57 smithi003 ceph-mon[104007]: pgmap v1501: 119 pgs: 119 active+clean; 3.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:05:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:57.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:57.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:57.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:57.884 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-09-13T15:05:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-09-13T15:05:57.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:58.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:58.102 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:58.132 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:58.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:58.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:58.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:58.391 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:58.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:58.519 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:58.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:58.622 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 59315 59299 108.986 81.5 0.0118813 0.00916979 2024-09-13T15:05:58.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:58.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:58.790 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:58.892 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:58.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:59.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:59.057 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:59.085 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:59.085 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:59.085 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:05:58 smithi003 ceph-mon[107278]: pgmap v1502: 119 pgs: 119 active+clean; 3.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T15:05:59.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:59.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:59.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:05:58 smithi003 ceph-mon[104007]: pgmap v1502: 119 pgs: 119 active+clean; 3.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T15:05:59.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:05:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:05:58 smithi019 ceph-mon[91296]: pgmap v1502: 119 pgs: 119 active+clean; 3.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T15:05:59.189 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:59.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:59.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:59.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T15:05:59.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:59.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:59.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:59.624 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 60616 60600 108.195 81.3125 0.0137034 0.00923592 2024-09-13T15:05:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:59.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:59.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:59.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:05:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:59.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:05:59.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:00.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:00.105 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:06:00.195 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T15:06:00.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-13T15:06:00.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:00.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:00.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:00.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:00.414 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:00.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:00.542 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:00.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:00.622 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 61616 61600 106.925 62.5 0.0100676 0.00934751 2024-09-13T15:06:00.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:00.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:06:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:06:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:00.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:00.800 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:00.902 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:00.930 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:01.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:01.074 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:01.151 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:01.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:01.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:01.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:01.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:01.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:01.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:01.499 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:01.601 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:01.621 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 62791 62775 106.02 73.4375 0.012913 0.0094272 2024-09-13T15:06:01.627 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:01 smithi019 ceph-mon[91296]: pgmap v1503: 119 pgs: 119 active+clean; 3.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-13T15:06:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:01 smithi003 ceph-mon[104007]: pgmap v1503: 119 pgs: 119 active+clean; 3.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-13T15:06:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:01 smithi003 ceph-mon[107278]: pgmap v1503: 119 pgs: 119 active+clean; 3.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-13T15:06:01.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:01.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:01.799 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:01.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.916 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:01.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:02.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:02.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:02.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:02.186 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:02.288 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:02.317 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:02.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:02.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:02.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:02.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:02.622 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 63995 63979 105.21 75.25 0.00781309 0.0094986 2024-09-13T15:06:02.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:02.734 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:02.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:02.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:02.867 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:02.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:03.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:03.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:03.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:03.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:03.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:03 smithi003 ceph-mon[104007]: pgmap v1504: 119 pgs: 119 active+clean; 3.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T15:06:03.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:03 smithi003 ceph-mon[107278]: pgmap v1504: 119 pgs: 119 active+clean; 3.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T15:06:03.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:03.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:03.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:03.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:03.623 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 65344 65328 104.67 84.3125 0.00825144 0.00954866 2024-09-13T15:06:03.670 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:03.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:03 smithi019 ceph-mon[91296]: pgmap v1504: 119 pgs: 119 active+clean; 3.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T15:06:03.698 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:03.750 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:03.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:03.828 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:03.930 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:03.958 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:04.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:04.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:04.207 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:04.238 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:04.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:04.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:04.421 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:06:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:06:04] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-09-13T15:06:04.479 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:04.508 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:04.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:04.624 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T15:06:04.623391+0000 min lat: 0.00114407 max lat: 0.340087 avg lat: 0.00961035 2024-09-13T15:06:04.624 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:06:04.624 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 66580 66564 103.984 77.25 0.0162111 0.00961035 2024-09-13T15:06:04.639 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:04.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:04.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:04.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:04.926 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:05.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:05.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:05.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:05.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:05.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:05.358 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:05 smithi003 ceph-mon[104007]: pgmap v1505: 119 pgs: 119 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T15:06:05.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:05 smithi003 ceph-mon[107278]: pgmap v1505: 119 pgs: 119 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T15:06:05.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:05.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:05.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:05.625 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 67731 67715 103.199 71.9375 0.0126789 0.00968452 2024-09-13T15:06:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:05 smithi019 ceph-mon[91296]: pgmap v1505: 119 pgs: 119 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T15:06:05.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:05.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:05.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:05.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:05.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:06.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:06.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:06.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:06.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:06.260 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:06:06.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:06.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:06.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:06.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:06.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:06.626 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 68926 68910 102.52 74.6875 0.0150625 0.00974777 2024-09-13T15:06:06.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:06.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:06 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1139235733' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:06 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:06 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1139235733' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:06 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:06 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1139235733' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:06.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:06 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:06.743 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:06.768 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:06.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:06.870 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:06.896 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:06.998 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.027 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:07.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:07.129 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:07.160 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:07.261 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:07.290 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:07.393 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:07.419 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:07.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:07.508 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:07 smithi003 ceph-mon[104007]: pgmap v1506: 119 pgs: 119 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T15:06:07.508 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:07 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:07.508 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:07 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:07.508 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:07 smithi003 ceph-mon[104007]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T15:06:07.508 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:07 smithi003 ceph-mon[107278]: pgmap v1506: 119 pgs: 119 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T15:06:07.508 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:07 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:07.508 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:07 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:07.508 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:07 smithi003 ceph-mon[107278]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T15:06:07.522 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:07.549 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:07.625 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 70103 70087 101.847 73.5625 0.00449521 0.00981311 2024-09-13T15:06:07.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:07 smithi019 ceph-mon[91296]: pgmap v1506: 119 pgs: 119 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T15:06:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:07 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:07 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:07 smithi019 ceph-mon[91296]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T15:06:07.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:07.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:07.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:07.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:07.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:08.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:08.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:08.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:08.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:08.353 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:08.386 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:08.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:08.488 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:08.515 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:08.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:08.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:08.626 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 71349 71333 101.301 77.875 0.0167492 0.00986598 2024-09-13T15:06:08.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:08.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:08.789 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:08.848 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:06:08.849 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:06:08.849 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:06:08.849 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:06:08.849 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:06:08.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:08.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:09.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:09.052 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:09.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:09.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:09.284 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:09.311 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:09.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:09.546 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:09.580 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:09.626 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 72551 72535 100.719 75.125 0.0108385 0.00992274 2024-09-13T15:06:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:09 smithi019 ceph-mon[91296]: pgmap v1508: 119 pgs: 119 active+clean; 4.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-13T15:06:09.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:09 smithi003 ceph-mon[104007]: pgmap v1508: 119 pgs: 119 active+clean; 4.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-13T15:06:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:09 smithi003 ceph-mon[107278]: pgmap v1508: 119 pgs: 119 active+clean; 4.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-13T15:06:09.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:09.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:09.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:09.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:09.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:10.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:10.121 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:10.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:10.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:06:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:10.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:10.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:10.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:10.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:06:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:06:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:10.626 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 73674 73658 100.055 70.1875 0.0164692 0.00998942 2024-09-13T15:06:10.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:10.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:10.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:10.808 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.837 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:10.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:10.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.972 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:10.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:11.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:11.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:11.146 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:06:11.146 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:11.186 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:06:11.186 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:11.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:11.224 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:06:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:11.231 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:11.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:11.257 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:06:11.257 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:11.290 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:06:11.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:11.323 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:06:11.323 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:11.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:11.356 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:06:11.356 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:11.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:11.386 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:06:11.386 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:11.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:11.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:11 smithi003 ceph-mon[104007]: pgmap v1509: 119 pgs: 119 active+clean; 4.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T15:06:11.487 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1979553705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:11.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:11 smithi003 ceph-mon[107278]: pgmap v1509: 119 pgs: 119 active+clean; 4.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T15:06:11.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1979553705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:11.514 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:11 smithi019 ceph-mon[91296]: pgmap v1509: 119 pgs: 119 active+clean; 4.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T15:06:11.514 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1979553705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:11.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:11.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:11.627 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 74768 74752 99.3806 68.375 0.0059214 0.010057 2024-09-13T15:06:11.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:11.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:11.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:11.909 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:11.936 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-09-13T15:06:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-09-13T15:06:11.997 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-09-13T15:06:11.997 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-09-13T15:06:12.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:12.070 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:12.172 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:12.201 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:12.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:12.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:12.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:12.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:12.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:12.624 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:12.626 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 75908 75892 98.7945 71.25 0.00904784 0.0101162 2024-09-13T15:06:12.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:12.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:12.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:06:12.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:12.891 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:12.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:13.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:13.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:13.160 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:13.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:06:13.271 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T15:06:13.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-13T15:06:13.286 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:13.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:13.311 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:13.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:13.414 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:13.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:13.503 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:13 smithi003 ceph-mon[107278]: pgmap v1510: 119 pgs: 119 active+clean; 4.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T15:06:13.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:13.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:13 smithi003 ceph-mon[104007]: pgmap v1510: 119 pgs: 119 active+clean; 4.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T15:06:13.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:13.517 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:13.546 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:13.627 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 76930 76914 98.0819 63.875 0.0266103 0.0101894 2024-09-13T15:06:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:13.647 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:13.672 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:13 smithi019 ceph-mon[91296]: pgmap v1510: 119 pgs: 119 active+clean; 4.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T15:06:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:13.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:13.800 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:13.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:13.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:13.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:14.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:14.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:14.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:14.220 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:14.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:14.368 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:06:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:06:14] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-13T15:06:14.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:14.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:06:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:14.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:14.627 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 78005 77989 97.4636 67.1875 0.0151884 0.0102557 2024-09-13T15:06:14.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:14.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:14.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:14.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:14.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:14.974 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:15.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:15.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:15.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:15.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:15.362 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:15.394 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:15.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.524 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:15.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:15.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:15.627 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 79032 79016 96.8111 64.1875 0.0114377 0.0103234 2024-09-13T15:06:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.658 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:15.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:15 smithi019 ceph-mon[91296]: pgmap v1511: 119 pgs: 119 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T15:06:15.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:15 smithi003 ceph-mon[107278]: pgmap v1511: 119 pgs: 119 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T15:06:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:15 smithi003 ceph-mon[104007]: pgmap v1511: 119 pgs: 119 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T15:06:15.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:15.785 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:15.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:15.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:15.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:15.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:16.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:16.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:16.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:16.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:16.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:16.239 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:16.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:16.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:16.482 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:16.512 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:16.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:16.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:16.628 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 80109 80093 96.2437 67.3125 0.0120639 0.0103837 2024-09-13T15:06:16.642 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:16.744 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:16.772 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:16.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:16.902 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:17.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:17.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:17.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:17.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:17.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:17.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:17.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:17.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:17.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:17.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:17.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:17.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:17.627 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 81159 81143 95.6661 65.625 0.0133949 0.0104466 2024-09-13T15:06:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:17 smithi019 ceph-mon[91296]: pgmap v1512: 119 pgs: 119 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T15:06:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:17 smithi003 ceph-mon[104007]: pgmap v1512: 119 pgs: 119 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T15:06:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:17 smithi003 ceph-mon[107278]: pgmap v1512: 119 pgs: 119 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T15:06:17.695 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:17.726 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:17.828 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:17.852 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:17.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:17.984 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:18.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:18.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:18.229 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:18.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:18.359 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:18.387 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:18.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:18.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:18.577 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:18 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:06:18.577 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:18 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:06:18.629 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 82201 82185 95.1005 65.125 0.00796484 0.0105098 2024-09-13T15:06:18.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:18 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:06:18.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:18.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:18.893 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:06:18.893 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:06:18.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:18.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:18.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:19.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.109 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:19.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:19.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:19.240 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:19.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:19.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.374 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:19.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:19.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:19 smithi003 ceph-mon[104007]: pgmap v1513: 119 pgs: 119 active+clean; 4.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:06:19.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:19 smithi003 ceph-mon[107278]: pgmap v1513: 119 pgs: 119 active+clean; 4.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:06:19.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:19.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:19.621 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:19.627 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 83247 83231 94.56 65.375 0.00441084 0.0105694 2024-09-13T15:06:19.651 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:19 smithi019 ceph-mon[91296]: pgmap v1513: 119 pgs: 119 active+clean; 4.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:06:19.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:19.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:19.871 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:19.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:19.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:19.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:20.048 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:20.082 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:20.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:20.214 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:20.299 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:06:20.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:20.347 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:20.449 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:20.480 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:20.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:20.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:20.616 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:20.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:20.627 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 84288 84272 94.0332 65.0625 0.0041817 0.0106293 2024-09-13T15:06:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:06:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:06:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:20 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/249953356' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:20.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:20 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/249953356' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:20 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/249953356' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:20.717 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:20.746 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:20.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T15:06:20.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:20.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:21.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:21.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:21.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:21.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:21.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:21.361 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:21.389 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:21 smithi019 ceph-mon[91296]: pgmap v1514: 119 pgs: 119 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-13T15:06:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:21 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/249953356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:06:21.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:21 smithi019 ceph-mon[91296]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T15:06:21.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:21.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:21.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:21.627 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 85316 85300 93.5106 64.25 0.0245234 0.0106882 2024-09-13T15:06:21.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:21.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:21 smithi003 ceph-mon[107278]: pgmap v1514: 119 pgs: 119 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-13T15:06:21.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:21 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:21.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/249953356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:06:21.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:21 smithi003 ceph-mon[107278]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T15:06:21.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:21 smithi003 ceph-mon[104007]: pgmap v1514: 119 pgs: 119 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-13T15:06:21.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:21 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:21.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/249953356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:06:21.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:21 smithi003 ceph-mon[104007]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T15:06:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.676 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:21.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:21.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:21.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:21.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:21.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:22.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:22.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:22.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:22.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:22.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:22.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:22.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:22.541 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:22.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:22.628 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 86338 86322 92.9996 63.875 0.0151867 0.0107473 2024-09-13T15:06:22.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:22.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:22.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:22.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:22.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:22.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:22.976 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:23.078 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:23.107 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:23.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:23.209 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:23.237 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:23.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:23.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:23.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:06:23.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:23 smithi019 ceph-mon[91296]: pgmap v1516: 119 pgs: 119 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:06:23.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:23 smithi003 ceph-mon[104007]: pgmap v1516: 119 pgs: 119 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:06:23.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:23 smithi003 ceph-mon[107278]: pgmap v1516: 119 pgs: 119 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T15:06:23.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:23.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:23.628 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 87337 87321 92.4816 62.4375 0.0208425 0.0108068 2024-09-13T15:06:23.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:23.672 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:23.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:23.807 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:23.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:23.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:23.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:24.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:24.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:24.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:24.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:24.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:06:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:06:24] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-13T15:06:24.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:24.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:24.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:24.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:24.628 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T15:06:24.628040+0000 min lat: 0.00114407 max lat: 0.340087 avg lat: 0.0108689 2024-09-13T15:06:24.628 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:06:24.628 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 88307 88291 91.9498 60.625 0.0158889 0.0108689 2024-09-13T15:06:24.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:24.763 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:24.794 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:24.896 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:24.926 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:25.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:25.057 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:25.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:24 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:25.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:24 smithi019 ceph-mon[91296]: pgmap v1517: 119 pgs: 119 active+clean; 5.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:06:25.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:25.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:24 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:25.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:24 smithi003 ceph-mon[107278]: pgmap v1517: 119 pgs: 119 active+clean; 5.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:06:25.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:25.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:24 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:25.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:24 smithi003 ceph-mon[104007]: pgmap v1517: 119 pgs: 119 active+clean; 5.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:06:25.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:25.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:25.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:25.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:25.471 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:25.573 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:25.628 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 89373 89357 91.5346 66.625 0.0171523 0.0109191 2024-09-13T15:06:25.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:25.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:25.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:25.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:25.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:26.007 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:26.109 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:26.144 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:26.246 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:26.378 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:26.414 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:26.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:26.538 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:26.629 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 90303 90287 90.9956 58.125 0.00335256 0.0109601 2024-09-13T15:06:26.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.663 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:26.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:26.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:26.789 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:26.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:26.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:27.034 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:27.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:27.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:27.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:27.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:27.456 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:27.483 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:27.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:27 smithi003 ceph-mon[107278]: pgmap v1518: 119 pgs: 119 active+clean; 5.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:06:27.502 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:27 smithi003 ceph-mon[104007]: pgmap v1518: 119 pgs: 119 active+clean; 5.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:06:27.519 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:27.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:27.614 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-09-13T15:06:27.614 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-09-13T15:06:27.614 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-09-13T15:06:27.614 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-09-13T15:06:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-09-13T15:06:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-09-13T15:06:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-09-13T15:06:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-09-13T15:06:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-09-13T15:06:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-09-13T15:06:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-09-13T15:06:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-09-13T15:06:27.629 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 91125 91109 90.3667 51.375 0.0301904 0.0110597 2024-09-13T15:06:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:27.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:27 smithi019 ceph-mon[91296]: pgmap v1518: 119 pgs: 119 active+clean; 5.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:06:27.739 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:27.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:27.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:28.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:28.045 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:28.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:28.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:28.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:28.298 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:06:28.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:28.340 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:06:28.340 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:28.381 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:06:28.381 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:28.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:28.382 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:28.417 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:06:28.417 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:28.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:28.447 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:28.449 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:06:28.449 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:28.479 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:06:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:28.512 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:06:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:06:28.545 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:28.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:28.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:28.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:28.629 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 92061 92045 89.8687 58.5 0.0267754 0.0111216 2024-09-13T15:06:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:06:28.694 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T15:06:28.694 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:28.724 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:28.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:28.967 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:28.997 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:28.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:29.099 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:29.133 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:29.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:29.267 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:29.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:29 smithi003 ceph-mon[107278]: pgmap v1519: 119 pgs: 119 active+clean; 5.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T15:06:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:29 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/763337193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:29.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:29 smithi003 ceph-mon[104007]: pgmap v1519: 119 pgs: 119 active+clean; 5.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T15:06:29.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:29 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/763337193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:29.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:29.539 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:29.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:29.629 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 93056 93040 89.4428 62.1875 0.0189917 0.0111749 2024-09-13T15:06:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:29 smithi019 ceph-mon[91296]: pgmap v1519: 119 pgs: 119 active+clean; 5.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T15:06:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:29 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/763337193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:29.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:29.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:29.881 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:29.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:30.009 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:30.110 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:30.138 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:30.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:30.266 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:30.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:30.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:30.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:30.527 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:30.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:30.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:30.630 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 93985 93969 88.9673 58.0625 0.0296363 0.0112343 2024-09-13T15:06:30.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:30.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:06:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:06:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:30.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:30.807 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:30.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:30.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:31.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:31.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:31.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:31.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:31.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:31.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:31.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:31.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:06:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:31.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:31.571 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:31 smithi003 ceph-mon[107278]: pgmap v1520: 119 pgs: 119 active+clean; 5.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 62 MiB/s wr, 988 op/s 2024-09-13T15:06:31.571 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:31 smithi003 ceph-mon[104007]: pgmap v1520: 119 pgs: 119 active+clean; 5.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 62 MiB/s wr, 988 op/s 2024-09-13T15:06:31.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:31.630 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 94840 94824 88.437 53.4375 0.033601 0.0113019 2024-09-13T15:06:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:31.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:31 smithi019 ceph-mon[91296]: pgmap v1520: 119 pgs: 119 active+clean; 5.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 62 MiB/s wr, 988 op/s 2024-09-13T15:06:31.775 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:31.800 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:31.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:31.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:32.057 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:32.082 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:32.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:32.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:32.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:32.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:32.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:32.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:32.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:32.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:32.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:32.630 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 95509 95493 87.7501 41.8125 0.0330662 0.0113905 2024-09-13T15:06:32.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:32.743 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:32.770 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:32.872 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:32.898 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:33.000 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:33.029 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:33.131 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:33.157 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:33.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:33.289 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:33.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:33.415 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:33.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:33 smithi003 ceph-mon[104007]: pgmap v1521: 119 pgs: 119 active+clean; 5.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 52 MiB/s wr, 832 op/s 2024-09-13T15:06:33.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:33 smithi003 ceph-mon[107278]: pgmap v1521: 119 pgs: 119 active+clean; 5.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 52 MiB/s wr, 832 op/s 2024-09-13T15:06:33.518 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:33.545 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:33.630 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 96494 96478 87.3705 61.5625 0.00584125 0.0114387 2024-09-13T15:06:33.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.679 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:33.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:33.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:33 smithi019 ceph-mon[91296]: pgmap v1521: 119 pgs: 119 active+clean; 5.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 52 MiB/s wr, 832 op/s 2024-09-13T15:06:33.695 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 5, 0, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:06:33.695 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:06:33.695 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [1, 2, 4, 6, 7, 5, 0, 3] 2024-09-13T15:06:33.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-13T15:06:33.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:33.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:33.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:33.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:33.951 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:34.053 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:34.080 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:34.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:34.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:34.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:06:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:06:34] "GET /metrics HTTP/1.1" 200 36516 "" "Prometheus/2.51.0" 2024-09-13T15:06:34.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:34.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:34.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:34.632 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 97403 97387 86.9322 56.8125 0.00477801 0.0114982 2024-09-13T15:06:34.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:34.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:34.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:34.851 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:34.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:35.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:35.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:35.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:35.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:35.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:35.287 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:35.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:35.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:35.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:35 smithi003 ceph-mon[104007]: pgmap v1522: 119 pgs: 119 active+clean; 5.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-13T15:06:35.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:35 smithi003 ceph-mon[107278]: pgmap v1522: 119 pgs: 119 active+clean; 5.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-13T15:06:35.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:35.560 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:35.632 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 98292 98276 86.4904 55.5625 0.0270951 0.0115568 2024-09-13T15:06:35.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:35 smithi019 ceph-mon[91296]: pgmap v1522: 119 pgs: 119 active+clean; 5.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-13T15:06:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:35.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:35.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:35.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:35.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:36.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:36.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:36.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:36.127 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:06:36.127 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:06:36.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.127 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:97936 14.2 e424/416 14:48310f4e:::benchmark_data_smithi003_327099_object97935:head v 424'6127, mlcod=424'6125)", 2024-09-13T15:06:36.127 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:06:35.244804+0000", 2024-09-13T15:06:36.127 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.003794319, 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003820468, 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.244804+0000", 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:06:36.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.244804+0000", 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.244804+0000", 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8100000000000002e-07 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.244846+0000", 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2123999999999999e-05 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.244864+0000", 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7971000000000001e-05 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.244871+0000", 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6900000000000003e-06 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.244976+0000", 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010520500000000001 2024-09-13T15:06:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.244996+0000", 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9259e-05 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:97941 14.2 e424/416 14:4e26037d:::benchmark_data_smithi003_327099_object97940:head v 424'6128, mlcod=424'6125)", 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:06:35.246235+0000", 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0023633, 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024433469999999998, 2024-09-13T15:06:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.246235+0000", 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.246235+0000", 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.246235+0000", 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999995e-07 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.246322+0000", 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6982000000000002e-05 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.246341+0000", 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8578999999999999e-05 2024-09-13T15:06:36.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.246349+0000", 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6469999999999998e-06 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.248618+0000", 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022697070000000001 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:35.248667+0000", 2024-09-13T15:06:36.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8826999999999999e-05 2024-09-13T15:06:36.135 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:06:36.135 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:06:36.135 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:06:36.135 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:06:36.135 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-13T15:06:36.135 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 2 2024-09-13T15:06:36.135 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:06:36.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:36.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:36.288 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:36.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:36.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:36.522 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:36.549 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:36.632 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 99184 99168 86.0634 55.75 0.0207156 0.0116133 2024-09-13T15:06:36.651 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:36.676 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:36.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:36.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:36.805 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:36.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:36.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:06:36.935 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:36.974 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:36.977 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:37.036 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:37.065 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:37.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:37.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:37.296 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:06:37.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:37.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:37.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:37 smithi003 ceph-mon[104007]: pgmap v1523: 119 pgs: 119 active+clean; 5.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 48 MiB/s wr, 767 op/s 2024-09-13T15:06:37.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2679389858' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:37.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:37 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:37.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:37 smithi003 ceph-mon[107278]: pgmap v1523: 119 pgs: 119 active+clean; 5.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 48 MiB/s wr, 767 op/s 2024-09-13T15:06:37.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2679389858' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:37.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:37 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:37.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:37.481 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:37.583 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:37.608 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:37.632 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 100026 100010 85.6054 52.625 0.0197504 0.0116761 2024-09-13T15:06:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:37 smithi019 ceph-mon[91296]: pgmap v1523: 119 pgs: 119 active+clean; 5.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 48 MiB/s wr, 767 op/s 2024-09-13T15:06:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2679389858' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:37 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:37.709 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:37.743 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:37.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:37.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:38.017 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:38.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:38.144 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:38.246 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:38.277 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:38.379 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.408 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:38.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:38.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:38.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:38.632 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 100979 100963 85.2534 59.5625 0.00559636 0.011724 2024-09-13T15:06:38.643 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:38 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:06:38.643 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:38 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:06:38.643 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:38 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:38.643 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:38 smithi003 ceph-mon[104007]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T15:06:38.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:38 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:06:38.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:38 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:06:38.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:38 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:38.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:38 smithi003 ceph-mon[107278]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T15:06:38.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:38 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:06:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:38 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:06:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:38 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:38 smithi019 ceph-mon[91296]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T15:06:38.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:38.804 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:38.833 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:38.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:38.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:38.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:39.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:39.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:39.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:39.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:39.359 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:39.388 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:39.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:39 smithi003 ceph-mon[104007]: pgmap v1525: 119 pgs: 119 active+clean; 5.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 57 MiB/s wr, 910 op/s 2024-09-13T15:06:39.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:39 smithi003 ceph-mon[107278]: pgmap v1525: 119 pgs: 119 active+clean; 5.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 57 MiB/s wr, 910 op/s 2024-09-13T15:06:39.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:39.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:39.632 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 101920 101904 84.9008 58.8125 0.00725939 0.0117732 2024-09-13T15:06:39.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:39 smithi019 ceph-mon[91296]: pgmap v1525: 119 pgs: 119 active+clean; 5.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 57 MiB/s wr, 910 op/s 2024-09-13T15:06:39.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:39.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:39.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:39.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:40.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:40.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:40.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:40.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:40.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:40.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:40.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:40.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:40.582 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:06:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:06:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:40.632 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 102833 102817 84.5345 57.0625 0.00918208 0.0118238 2024-09-13T15:06:40.652 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:40.662 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:06:40.662 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:40.685 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:40.701 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:06:40.701 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:40.732 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:06:40.733 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:40.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:40.790 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:06:40.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:40.822 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:06:40.823 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:40.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:40.857 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:06:40.857 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:40.887 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:06:40.888 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:40.926 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:06:40.926 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:40.931 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:40.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:41.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:41.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:41.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:41.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:41.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:41.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:41.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:41.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:41.485 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:41.516 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:41.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:41.632 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 103642 103626 84.0933 50.5625 0.0033731 0.0118728 2024-09-13T15:06:41.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:41 smithi019 ceph-mon[91296]: pgmap v1526: 119 pgs: 119 active+clean; 6.1 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s 2024-09-13T15:06:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:41 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4139013808' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:41.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:41 smithi003 ceph-mon[104007]: pgmap v1526: 119 pgs: 119 active+clean; 6.1 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s 2024-09-13T15:06:41.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:41 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4139013808' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:41.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:41 smithi003 ceph-mon[107278]: pgmap v1526: 119 pgs: 119 active+clean; 6.1 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s 2024-09-13T15:06:41.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:41 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4139013808' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:41.743 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.768 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:41.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:41.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:41.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:41.993 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:42.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:06:42.018 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:42.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:42.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:42.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:42.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:42.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:42.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:42.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:42.600 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:42.633 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 104404 104388 83.6248 47.625 0.0240191 0.0119529 2024-09-13T15:06:42.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.732 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:42.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:42.833 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.862 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:42.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:42.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:43.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:43.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:43.140 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:43.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:43.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:43.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:43.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:43.566 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:43 smithi003 ceph-mon[107278]: pgmap v1527: 119 pgs: 119 active+clean; 6.1 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s 2024-09-13T15:06:43.566 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:43.566 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:43.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:43 smithi003 ceph-mon[104007]: pgmap v1527: 119 pgs: 119 active+clean; 6.1 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s 2024-09-13T15:06:43.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:43.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:43.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:43.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:43.634 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 105279 105263 83.2579 54.6875 0.0057366 0.0120057 2024-09-13T15:06:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:43 smithi019 ceph-mon[91296]: pgmap v1527: 119 pgs: 119 active+clean; 6.1 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s 2024-09-13T15:06:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:43.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:43.734 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:43.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:43.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:43.863 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:43.965 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:43.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:44.100 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:44.129 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:44.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:44.261 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:44.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:44.395 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:44.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:06:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:06:44] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-09-13T15:06:44.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:44.521 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:44.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:44.634 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-09-13T15:06:44.633854+0000 min lat: 0.00114407 max lat: 0.340087 avg lat: 0.0120505 2024-09-13T15:06:44.634 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:06:44.634 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 106211 106195 82.9453 58.25 0.00944092 0.0120505 2024-09-13T15:06:44.651 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:44.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:44.779 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:44.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:44.917 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:44.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:45.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:45.095 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:45.197 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:45.225 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:45.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:45.333 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.4. 2024-09-13T15:06:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:45.474 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:45.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:45.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:45.620 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:45.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:45.634 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 106811 106795 82.3843 37.5 0.00388679 0.012095 2024-09-13T15:06:45.668 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:45 smithi003 ceph-mon[107278]: pgmap v1528: 119 pgs: 119 active+clean; 6.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-13T15:06:45.668 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:45 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2882425344' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T15:06:45.668 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:45 smithi003 ceph-mon[107278]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T15:06:45.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:45 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:06:45.239+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e425 definitely_dead 0 2024-09-13T15:06:45.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:45 smithi003 ceph-mon[104007]: pgmap v1528: 119 pgs: 119 active+clean; 6.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-13T15:06:45.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:45 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2882425344' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T15:06:45.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:45 smithi003 ceph-mon[104007]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T15:06:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:45 smithi019 ceph-mon[91296]: pgmap v1528: 119 pgs: 119 active+clean; 6.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-13T15:06:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:45 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2882425344' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T15:06:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:45 smithi019 ceph-mon[91296]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T15:06:45.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:45.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:45.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:45.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:45.994 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:46.095 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:46.118 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:46.219 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:46.241 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:46.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:46.366 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:46.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:46.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:46.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:46.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:46.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:46 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2882425344' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T15:06:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:46 smithi003 ceph-mon[104007]: osdmap e426: 8 total, 8 up, 7 in 2024-09-13T15:06:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:46 smithi003 ceph-mon[104007]: osdmap e427: 8 total, 8 up, 7 in 2024-09-13T15:06:46.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:46 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2882425344' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T15:06:46.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:46 smithi003 ceph-mon[107278]: osdmap e426: 8 total, 8 up, 7 in 2024-09-13T15:06:46.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:46 smithi003 ceph-mon[107278]: osdmap e427: 8 total, 8 up, 7 in 2024-09-13T15:06:46.634 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 106811 106795 81.3797 0 - 0.012095 2024-09-13T15:06:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:46 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2882425344' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T15:06:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:46 smithi019 ceph-mon[91296]: osdmap e426: 8 total, 8 up, 7 in 2024-09-13T15:06:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:46 smithi019 ceph-mon[91296]: osdmap e427: 8 total, 8 up, 7 in 2024-09-13T15:06:46.718 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:46.743 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:46.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:46.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:46.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:06:46.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:46.995 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:47.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:47.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:47.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:47.307 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:47.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:47.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:47.583 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:47.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:47.634 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 107067 107051 80.5921 8 0.0126979 0.0124025 2024-09-13T15:06:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:47 smithi019 ceph-mon[91296]: 1.e scrub starts 2024-09-13T15:06:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:47 smithi019 ceph-mon[91296]: 1.e scrub ok 2024-09-13T15:06:47.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:47 smithi019 ceph-mon[91296]: pgmap v1531: 119 pgs: 119 active+clean; 6.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 959 op/s 2024-09-13T15:06:47.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:47 smithi019 ceph-mon[91296]: 1.5 scrub starts 2024-09-13T15:06:47.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:47 smithi019 ceph-mon[91296]: 1.5 scrub ok 2024-09-13T15:06:47.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:47 smithi019 ceph-mon[91296]: osdmap e428: 8 total, 8 up, 7 in 2024-09-13T15:06:47.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:47 smithi019 ceph-mon[91296]: 14.9 starting backfill to osd.0 from (0'0,0'0] MAX to 425'6711 2024-09-13T15:06:47.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[104007]: 1.e scrub starts 2024-09-13T15:06:47.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[104007]: 1.e scrub ok 2024-09-13T15:06:47.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[104007]: pgmap v1531: 119 pgs: 119 active+clean; 6.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 959 op/s 2024-09-13T15:06:47.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[104007]: 1.5 scrub starts 2024-09-13T15:06:47.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[104007]: 1.5 scrub ok 2024-09-13T15:06:47.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[104007]: osdmap e428: 8 total, 8 up, 7 in 2024-09-13T15:06:47.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[104007]: 14.9 starting backfill to osd.0 from (0'0,0'0] MAX to 425'6711 2024-09-13T15:06:47.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[107278]: 1.e scrub starts 2024-09-13T15:06:47.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[107278]: 1.e scrub ok 2024-09-13T15:06:47.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[107278]: pgmap v1531: 119 pgs: 119 active+clean; 6.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 959 op/s 2024-09-13T15:06:47.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[107278]: 1.5 scrub starts 2024-09-13T15:06:47.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[107278]: 1.5 scrub ok 2024-09-13T15:06:47.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[107278]: osdmap e428: 8 total, 8 up, 7 in 2024-09-13T15:06:47.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:47 smithi003 ceph-mon[107278]: 14.9 starting backfill to osd.0 from (0'0,0'0] MAX to 425'6711 2024-09-13T15:06:47.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:47.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.724 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:47.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:47.825 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:47.854 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:47.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:47.984 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:48.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:48.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:48.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:48.212 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.241 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:48.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:48.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:48.370 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:48 smithi003 ceph-mon[107278]: 14.e starting backfill to osd.1 from (0'0,0'0] MAX to 425'6768 2024-09-13T15:06:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:48 smithi003 ceph-mon[107278]: 1.1a scrub starts 2024-09-13T15:06:48.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:48 smithi003 ceph-mon[107278]: 1.1a scrub ok 2024-09-13T15:06:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:48 smithi003 ceph-mon[104007]: 14.e starting backfill to osd.1 from (0'0,0'0] MAX to 425'6768 2024-09-13T15:06:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:48 smithi003 ceph-mon[104007]: 1.1a scrub starts 2024-09-13T15:06:48.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:48 smithi003 ceph-mon[104007]: 1.1a scrub ok 2024-09-13T15:06:48.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:48.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:48.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:48.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:48.635 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 107758 107742 80.1467 43.1875 0.0403106 0.0124719 2024-09-13T15:06:48.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:48 smithi019 ceph-mon[91296]: 14.e starting backfill to osd.1 from (0'0,0'0] MAX to 425'6768 2024-09-13T15:06:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:48 smithi019 ceph-mon[91296]: 1.1a scrub starts 2024-09-13T15:06:48.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:48 smithi019 ceph-mon[91296]: 1.1a scrub ok 2024-09-13T15:06:48.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:48.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:48.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.919 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:48.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:49.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:49.052 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:49.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:49.182 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:49.284 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:49.311 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:49.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:49.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:49.478 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:49 smithi003 ceph-mon[107278]: pgmap v1533: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 6.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 53 MiB/s wr, 852 op/s; 13420/317454 objects misplaced (4.227%); 1.2 MiB/s, 20 objects/s recovering 2024-09-13T15:06:49.478 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:49 smithi003 ceph-mon[104007]: pgmap v1533: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 6.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 53 MiB/s wr, 852 op/s; 13420/317454 objects misplaced (4.227%); 1.2 MiB/s, 20 objects/s recovering 2024-09-13T15:06:49.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:49.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:49.635 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 108426 108410 79.6949 41.75 0.035901 0.0125419 2024-09-13T15:06:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:49 smithi019 ceph-mon[91296]: pgmap v1533: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 6.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 53 MiB/s wr, 852 op/s; 13420/317454 objects misplaced (4.227%); 1.2 MiB/s, 20 objects/s recovering 2024-09-13T15:06:49.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:49.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:49.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:49.892 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:49.994 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:50.025 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:50.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:50.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:50.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:50.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:50.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:50.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:50.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:50.635 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 109118 109102 79.2709 43.25 0.0331471 0.0126088 2024-09-13T15:06:50.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:50.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:50 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 13275/322680 objects degraded (4.114%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:50.664 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:50 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 13275/322680 objects degraded (4.114%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:50.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:50 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 13275/322680 objects degraded (4.114%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:06:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:06:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:50.727 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:06:50.727 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:06:50.745 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:50.775 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:50.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:50.907 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:51.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:51.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:51.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:51.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:51.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:51.319 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:51.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:51.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:51.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:51.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:51.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:06:51.636 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 15 109813 109798 78.8588 43.5 0.00732982 0.0126717 2024-09-13T15:06:51.657 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 6, 7, 5, 0, 3] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:06:51.657 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:06:51.657 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-13T15:06:51.657 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:06:51.657 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:06:51.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-13T15:06:51.677 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:51 smithi019 ceph-mon[91296]: pgmap v1534: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 26 MiB/s wr, 419 op/s; 13275/322680 objects degraded (4.114%); 26702/322680 objects misplaced (8.275%); 1.9 MiB/s, 1 keys/s, 32 objects/s recovering 2024-09-13T15:06:51.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:51 smithi003 ceph-mon[107278]: pgmap v1534: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 26 MiB/s wr, 419 op/s; 13275/322680 objects degraded (4.114%); 26702/322680 objects misplaced (8.275%); 1.9 MiB/s, 1 keys/s, 32 objects/s recovering 2024-09-13T15:06:51.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:51 smithi003 ceph-mon[104007]: pgmap v1534: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 26 MiB/s wr, 419 op/s; 13275/322680 objects degraded (4.114%); 26702/322680 objects misplaced (8.275%); 1.9 MiB/s, 1 keys/s, 32 objects/s recovering 2024-09-13T15:06:51.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:51.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:51.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:51.884 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:51.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:52.010 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:52.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:52.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:52.176 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:52.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:52.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:52.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:52.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:52.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:52.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:52.636 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:52.636 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 110428 110412 78.3987 38.375 0.0994152 0.0127475 2024-09-13T15:06:52.642 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:52.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:52.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:52.895 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:52.927 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:53.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:53.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:53.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[107278]: pgmap v1535: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 23 MiB/s wr, 367 op/s; 13275/322680 objects degraded (4.114%); 26702/322680 objects misplaced (8.275%); 1.7 MiB/s, 1 keys/s, 28 objects/s recovering 2024-09-13T15:06:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:06:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:06:53.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:53.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:53.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:53.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[104007]: pgmap v1535: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 23 MiB/s wr, 367 op/s; 13275/322680 objects degraded (4.114%); 26702/322680 objects misplaced (8.275%); 1.7 MiB/s, 1 keys/s, 28 objects/s recovering 2024-09-13T15:06:53.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:06:53.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:06:53.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:53.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:53.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:53.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:53.388 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:53.388 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:53.388 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:53 smithi019 ceph-mon[91296]: pgmap v1535: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 23 MiB/s wr, 367 op/s; 13275/322680 objects degraded (4.114%); 26702/322680 objects misplaced (8.275%); 1.7 MiB/s, 1 keys/s, 28 objects/s recovering 2024-09-13T15:06:53.388 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:06:53.389 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:06:53.389 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:53.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:53.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:53.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:53.639 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 111074 111058 77.9693 40.375 0.0113046 0.0128184 2024-09-13T15:06:53.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:53.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:53.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:53.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:53.935 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:53.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:54.038 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:06:54.038 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T15:06:54.039 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:06:54.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:54.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:54.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:54.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:54.317 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:06:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:06:54] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-09-13T15:06:54.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:54.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:54.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:54.534 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:54.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:54.639 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 90 1 111723 111722 77.5641 41.5 0.0280502 0.0128871 2024-09-13T15:06:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:54.761 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:54.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:54.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:54.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:54.908 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:55.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:55.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:55.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:55.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:55.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:55.266 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:55 smithi003 ceph-mon[104007]: pgmap v1536: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 44 MiB/s wr, 704 op/s; 13275/332667 objects degraded (3.990%); 26436/332667 objects misplaced (7.947%); 6.3 MiB/s, 1 keys/s, 101 objects/s recovering 2024-09-13T15:06:55.266 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1502663939' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:55.267 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:55 smithi003 ceph-mon[107278]: pgmap v1536: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 44 MiB/s wr, 704 op/s; 13275/332667 objects degraded (3.990%); 26436/332667 objects misplaced (7.947%); 6.3 MiB/s, 1 keys/s, 101 objects/s recovering 2024-09-13T15:06:55.267 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1502663939' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:55.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:55.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:55.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:55 smithi019 ceph-mon[91296]: pgmap v1536: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 44 MiB/s wr, 704 op/s; 13275/332667 objects degraded (3.990%); 26436/332667 objects misplaced (7.947%); 6.3 MiB/s, 1 keys/s, 101 objects/s recovering 2024-09-13T15:06:55.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1502663939' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:55.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:55.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:55.551 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:06:55.639 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0284 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 111723 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 77.5609 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 35.084 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 194.875 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1240 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 561.343 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3118 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-09-13T15:06:55.640 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0128872 2024-09-13T15:06:55.641 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.025744 2024-09-13T15:06:55.641 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.05909 2024-09-13T15:06:55.641 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00114407 2024-09-13T15:06:55.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:55.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:55.676 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T15:06:55.771 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:55.801 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:55.903 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:55.933 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:56.035 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:56.164 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:56.194 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:56.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:56.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:56.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:56.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:56.565 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:56.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:56 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:56.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1502663939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:06:56.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:56 smithi003 ceph-mon[107278]: osdmap e429: 8 total, 8 up, 7 in 2024-09-13T15:06:56.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 13275/332667 objects degraded (3.990%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:56.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:56 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:56.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1502663939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:06:56.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:56 smithi003 ceph-mon[104007]: osdmap e429: 8 total, 8 up, 7 in 2024-09-13T15:06:56.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 13275/332667 objects degraded (3.990%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:56.594 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:56 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1502663939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:06:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:56 smithi019 ceph-mon[91296]: osdmap e429: 8 total, 8 up, 7 in 2024-09-13T15:06:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 13275/332667 objects degraded (3.990%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:56.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:56.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:56.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:56.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:57.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:57.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:57.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:57.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:57 smithi003 ceph-mon[107278]: pgmap v1538: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 39 MiB/s wr, 629 op/s; 13275/332667 objects degraded (3.990%); 26436/332667 objects misplaced (7.947%); 5.6 MiB/s, 1 keys/s, 90 objects/s recovering 2024-09-13T15:06:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:57 smithi003 ceph-mon[104007]: pgmap v1538: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 39 MiB/s wr, 629 op/s; 13275/332667 objects degraded (3.990%); 26436/332667 objects misplaced (7.947%); 5.6 MiB/s, 1 keys/s, 90 objects/s recovering 2024-09-13T15:06:57.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.469 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:57.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:57.494 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:06:57.494 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:06:57.570 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:57.596 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:57 smithi019 ceph-mon[91296]: pgmap v1538: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 39 MiB/s wr, 629 op/s; 13275/332667 objects degraded (3.990%); 26436/332667 objects misplaced (7.947%); 5.6 MiB/s, 1 keys/s, 90 objects/s recovering 2024-09-13T15:06:57.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:57.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:57.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:57.870 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:57.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:58.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:58.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:58.146 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:58.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:58.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:58.276 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:58.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:58.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:58.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:06:58.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:58.554 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:58.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:58.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:58.811 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:58.838 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:58.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:06:58.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:59.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:59.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:59.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:59.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:58 smithi003 ceph-mon[107278]: pgmap v1539: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 113 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 34 MiB/s wr, 596 op/s; 13275/332085 objects degraded (3.997%); 25436/332085 objects misplaced (7.659%); 10 MiB/s, 0 keys/s, 167 objects/s recovering 2024-09-13T15:06:59.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:59.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:59.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:59.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:58 smithi003 ceph-mon[104007]: pgmap v1539: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 113 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 34 MiB/s wr, 596 op/s; 13275/332085 objects degraded (3.997%); 25436/332085 objects misplaced (7.659%); 10 MiB/s, 0 keys/s, 167 objects/s recovering 2024-09-13T15:06:59.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:59.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:58 smithi019 ceph-mon[91296]: pgmap v1539: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 113 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 34 MiB/s wr, 596 op/s; 13275/332085 objects degraded (3.997%); 25436/332085 objects misplaced (7.659%); 10 MiB/s, 0 keys/s, 167 objects/s recovering 2024-09-13T15:06:59.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:06:59.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:59.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:59.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:59.331 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:06:59.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:59.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:59.357 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:59.372 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:06:59.373 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:59.412 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:06:59.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:59.419 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:59.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:59.463 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:06:59.467 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:59.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:59.508 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:06:59.508 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:59.541 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:06:59.541 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:59.571 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:06:59.571 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:59.603 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:06:59.603 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:59.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:59.639 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:06:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:59.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:59.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:59.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:59.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:59.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:06:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4045141360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:59.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:06:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4045141360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:59.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:06:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:00.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:00.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:00.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:06:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4045141360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:00.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:00.229 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:00.331 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:00.359 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:00.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:00.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:00.603 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:00.628 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:07:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:07:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:00.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:00.759 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:00.766 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 6, 7, 5, 0, 3] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:07:00.766 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:07:00.766 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [1, 2, 6, 7, 5, 0, 3] 2024-09-13T15:07:00.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd out 1 2024-09-13T15:07:00.860 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:00.890 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:00 smithi003 ceph-mon[104007]: pgmap v1540: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 618 op/s; 13275/329250 objects degraded (4.032%); 25153/329250 objects misplaced (7.639%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:07:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:00 smithi003 ceph-mon[107278]: pgmap v1540: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 618 op/s; 13275/329250 objects degraded (4.032%); 25153/329250 objects misplaced (7.639%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:07:00.992 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:01.021 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:01.122 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:01.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:01.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:00 smithi019 ceph-mon[91296]: pgmap v1540: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 618 op/s; 13275/329250 objects degraded (4.032%); 25153/329250 objects misplaced (7.639%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:07:01.252 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:01.276 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:01.378 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:01.400 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:01.502 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:01.524 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:01.626 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:01.648 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:01.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:01.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:01.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:01.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:01.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:01 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 13275/329250 objects degraded (4.032%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:01.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:01 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 13275/329250 objects degraded (4.032%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:02.025 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:02.055 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:02.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:02.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:01 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 13275/329250 objects degraded (4.032%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:02.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:02.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:02.325 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:02.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:02.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:02.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:02.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:02.727 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:02.757 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:02.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:02.889 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:02.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:03.020 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:03.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:02 smithi003 ceph-mon[104007]: pgmap v1541: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 618 op/s; 13275/329250 objects degraded (4.032%); 25153/329250 objects misplaced (7.639%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:07:03.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:02 smithi003 ceph-mon[107278]: pgmap v1541: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 618 op/s; 13275/329250 objects degraded (4.032%); 25153/329250 objects misplaced (7.639%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:07:03.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:03.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:03.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:02 smithi019 ceph-mon[91296]: pgmap v1541: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 618 op/s; 13275/329250 objects degraded (4.032%); 25153/329250 objects misplaced (7.639%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:07:03.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:03.303 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:03.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:03.431 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:03.475 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:07:03.475 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:07:03.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.475 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:116733 14.e 14:7eab3038:::benchmark_data_smithi003_327099_object5007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-13T15:07:03.475 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:03.456484+0000", 2024-09-13T15:07:03.475 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.016673100999999999, 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0167022, 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 116733 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.456484+0000", 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.456484+0000", 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.456485+0000", 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5370000000000001e-06 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.456492+0000", 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.562e-06 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.456512+0000", 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0789999999999999e-05 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:03.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.456522+0000", 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2310000000000002e-06 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.457963+0000", 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00144141 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.458107+0000", 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014404399999999999 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.458232+0000", 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000124447 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.464909+0000", 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066774620000000003 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.464947+0000", 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8189999999999999e-05 2024-09-13T15:07:03.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.465549+0000", 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00060163399999999998 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:116735 14.1 e429/416 14:8d0c1cfa:::benchmark_data_smithi003_327099_object5009:head v 429'7286, mlcod=429'7284)", 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:03.470559+0000", 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.002598038, 2024-09-13T15:07:03.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027130259999999999, 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.470559+0000", 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.470559+0000", 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:03.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.470559+0000", 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3900000000000004e-07 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.470565+0000", 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4099999999999999e-06 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.470580+0000", 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5068e-05 2024-09-13T15:07:03.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.470587+0000", 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6179999999999997e-06 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.470699+0000", 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111351 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:07:03.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.470729+0000", 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0026e-05 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:116727 14.b e429/416 14:dbfbd1c9:::benchmark_data_smithi003_327099_object5001:head v 429'7108, mlcod=429'7104)", 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:03.471208+0000", 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0019482270000000001, 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021126650000000001, 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:07:03.485 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.471208+0000", 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.471208+0000", 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:03.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.471209+0000", 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4199999999999995e-07 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.471213+0000", 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8279999999999996e-06 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.471221+0000", 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0299999999999994e-06 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.471226+0000", 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5659999999999997e-06 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.471284+0000", 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7844999999999999e-05 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:03.471306+0000", 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1373e-05 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:03.488 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:03.489 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:03.489 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:03.489 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-13T15:07:03.489 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 3 2024-09-13T15:07:03.489 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:07:03.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:03.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:03.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:03.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:03.864 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:03.883 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:07:03.966 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:03.996 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:04.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:07:04.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:04.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:03 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/751004094' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:04.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:03 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:04.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:03 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/751004094' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:04.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:03 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:04.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:07:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:07:04] "GET /metrics HTTP/1.1" 200 36554 "" "Prometheus/2.51.0" 2024-09-13T15:07:04.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:03 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/751004094' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:04.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:03 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:04.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:04.286 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:04.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:04.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:04.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:04.571 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:04.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:04.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:04.814 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:04.840 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:04.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:04.974 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:05.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:05.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:04 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:04 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:05.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:04 smithi019 ceph-mon[91296]: osdmap e430: 8 total, 8 up, 7 in 2024-09-13T15:07:05.183 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:04 smithi019 ceph-mon[91296]: pgmap v1543: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 114 B/s rd, 6.0 MiB/s wr, 648 op/s; 13293/320442 objects degraded (4.148%); 23976/320442 objects misplaced (7.482%); 14 MiB/s, 230 objects/s recovering 2024-09-13T15:07:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:04 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:04 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:04 smithi003 ceph-mon[107278]: osdmap e430: 8 total, 8 up, 7 in 2024-09-13T15:07:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:04 smithi003 ceph-mon[107278]: pgmap v1543: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 114 B/s rd, 6.0 MiB/s wr, 648 op/s; 13293/320442 objects degraded (4.148%); 23976/320442 objects misplaced (7.482%); 14 MiB/s, 230 objects/s recovering 2024-09-13T15:07:05.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:04 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:05.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:04 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:05.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:04 smithi003 ceph-mon[104007]: osdmap e430: 8 total, 8 up, 7 in 2024-09-13T15:07:05.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:04 smithi003 ceph-mon[104007]: pgmap v1543: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 114 B/s rd, 6.0 MiB/s wr, 648 op/s; 13293/320442 objects degraded (4.148%); 23976/320442 objects misplaced (7.482%); 14 MiB/s, 230 objects/s recovering 2024-09-13T15:07:05.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:05.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:05.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:05.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:05.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:05.508 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:05.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:05.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:05.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:05.794 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:05.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:05.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:06.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:06.076 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:06.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:06.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:05 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 13293/320442 objects degraded (4.148%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:06.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:05 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 13293/320442 objects degraded (4.148%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:06.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:05 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 13293/320442 objects degraded (4.148%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:06.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:06.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:06.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:06.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:06.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:06.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:06.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:06.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:06.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:06.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:06.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:06.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:06.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:06.901 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:07.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:07.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:07.144 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:07.167 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:07.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:06 smithi003 ceph-mon[107278]: pgmap v1544: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 5.4 MiB/s wr, 579 op/s; 13293/320442 objects degraded (4.148%); 23976/320442 objects misplaced (7.482%); 13 MiB/s, 206 objects/s recovering 2024-09-13T15:07:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:06 smithi003 ceph-mon[104007]: pgmap v1544: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 5.4 MiB/s wr, 579 op/s; 13293/320442 objects degraded (4.148%); 23976/320442 objects misplaced (7.482%); 13 MiB/s, 206 objects/s recovering 2024-09-13T15:07:07.268 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:07.299 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:07.401 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:07.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:06 smithi019 ceph-mon[91296]: pgmap v1544: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 5.4 MiB/s wr, 579 op/s; 13293/320442 objects degraded (4.148%); 23976/320442 objects misplaced (7.482%); 13 MiB/s, 206 objects/s recovering 2024-09-13T15:07:07.531 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:07.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:07:07.559 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:07.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:07.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:07.769 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:07.820 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:07.852 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:07.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:07.976 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:08.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:08.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:08.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:08.230 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:08.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:08.362 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:08.424 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:08.464 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:08.492 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:08.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:08.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:08.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:08.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:08.788 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:08.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:09.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:09.064 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:09.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:09.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:09.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:09.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:09.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:09.556 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:09 smithi003 ceph-mon[107278]: pgmap v1545: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 113 active+clean; 6.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 3.2 MiB/s wr, 586 op/s; 13293/317559 objects degraded (4.186%); 23118/317559 objects misplaced (7.280%); 10 MiB/s, 163 objects/s recovering 2024-09-13T15:07:09.556 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:09 smithi003 ceph-mon[104007]: pgmap v1545: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 113 active+clean; 6.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 3.2 MiB/s wr, 586 op/s; 13293/317559 objects degraded (4.186%); 23118/317559 objects misplaced (7.280%); 10 MiB/s, 163 objects/s recovering 2024-09-13T15:07:09.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:09.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:09 smithi019 ceph-mon[91296]: pgmap v1545: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 113 active+clean; 6.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 3.2 MiB/s wr, 586 op/s; 13293/317559 objects degraded (4.186%); 23118/317559 objects misplaced (7.280%); 10 MiB/s, 163 objects/s recovering 2024-09-13T15:07:09.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:09.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:09.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:09.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:10.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:10.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:10.157 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:10.185 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:10.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:10.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:10.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:10.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:10.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:07:10.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:10.580 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:10.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:07:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:07:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.608 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:10.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:10.709 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:10.738 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:10.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:10.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:11.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:11.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:11.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:11.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:11.323 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:11.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:11.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:11 smithi003 ceph-mon[104007]: pgmap v1546: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 616 op/s; 13392/310734 objects degraded (4.310%); 22781/310734 objects misplaced (7.331%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T15:07:11.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 13392/310734 objects degraded (4.310%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:11.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:11 smithi003 ceph-mon[107278]: pgmap v1546: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 616 op/s; 13392/310734 objects degraded (4.310%); 22781/310734 objects misplaced (7.331%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T15:07:11.362 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 13392/310734 objects degraded (4.310%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:11.425 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:11 smithi019 ceph-mon[91296]: pgmap v1546: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 113 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 616 op/s; 13392/310734 objects degraded (4.310%); 22781/310734 objects misplaced (7.331%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T15:07:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 13392/310734 objects degraded (4.310%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:11.455 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:11.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:11.584 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:11.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:11.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:11.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:11.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:11.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:11.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:11.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:12.140 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:12.175 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:12.277 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:12.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:12.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:12.447 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:12.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:12.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:12.580 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:12.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:12.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:12 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:07:12.366+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e430 definitely_dead 0 2024-09-13T15:07:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:12.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:12.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:12.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:12.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:12.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:13.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:13.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:13.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:13.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:13.267 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.1. 2024-09-13T15:07:13.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.286 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:13.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:13.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:13.411 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[104007]: pgmap v1547: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 113 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 673 op/s; 13392/309042 objects degraded (4.333%); 22598/309042 objects misplaced (7.312%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T15:07:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/328144485' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T15:07:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T15:07:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[104007]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T15:07:13.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[107278]: pgmap v1547: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 113 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 673 op/s; 13392/309042 objects degraded (4.333%); 22598/309042 objects misplaced (7.312%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T15:07:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/328144485' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T15:07:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T15:07:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[107278]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T15:07:13.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:13.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:13.537 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:13.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:13.662 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:13 smithi019 ceph-mon[91296]: pgmap v1547: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 113 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 673 op/s; 13392/309042 objects degraded (4.333%); 22598/309042 objects misplaced (7.312%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T15:07:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:13 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/328144485' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T15:07:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:13 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T15:07:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:13 smithi019 ceph-mon[91296]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T15:07:13.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:13.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.789 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:13.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:13.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:14.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:14.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:14.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:14.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:14.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:14.290 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:14.392 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:14.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:14 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T15:07:14.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:14 smithi003 ceph-mon[107278]: osdmap e431: 8 total, 8 up, 6 in 2024-09-13T15:07:14.393 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:07:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:07:14] "GET /metrics HTTP/1.1" 200 36560 "" "Prometheus/2.51.0" 2024-09-13T15:07:14.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:14 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T15:07:14.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:14 smithi003 ceph-mon[104007]: osdmap e431: 8 total, 8 up, 6 in 2024-09-13T15:07:14.416 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:14.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:14.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:14.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:14.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:14 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T15:07:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:14 smithi019 ceph-mon[91296]: osdmap e431: 8 total, 8 up, 6 in 2024-09-13T15:07:14.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:14.789 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:14.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:14.915 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:15.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:15.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:15.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:15.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:15.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.291 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:15.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:15.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:15.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:15.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:15.547 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:15.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:15 smithi019 ceph-mon[91296]: pgmap v1549: 119 pgs: 1 active+recovering+undersized+remapped, 38 peering, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 652 op/s; 13516/300849 objects degraded (4.493%); 21620/300849 objects misplaced (7.186%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T15:07:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:15 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:07:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:15 smithi019 ceph-mon[91296]: osdmap e432: 8 total, 8 up, 6 in 2024-09-13T15:07:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:15 smithi019 ceph-mon[91296]: 14.e starting backfill to osd.0 from (0'0,0'0] MAX to 430'7794 2024-09-13T15:07:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:15 smithi003 ceph-mon[104007]: pgmap v1549: 119 pgs: 1 active+recovering+undersized+remapped, 38 peering, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 652 op/s; 13516/300849 objects degraded (4.493%); 21620/300849 objects misplaced (7.186%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T15:07:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:15 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:07:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:15 smithi003 ceph-mon[104007]: osdmap e432: 8 total, 8 up, 6 in 2024-09-13T15:07:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:15 smithi003 ceph-mon[104007]: 14.e starting backfill to osd.0 from (0'0,0'0] MAX to 430'7794 2024-09-13T15:07:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:15 smithi003 ceph-mon[107278]: pgmap v1549: 119 pgs: 1 active+recovering+undersized+remapped, 38 peering, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 652 op/s; 13516/300849 objects degraded (4.493%); 21620/300849 objects misplaced (7.186%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T15:07:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:15 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:07:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:15 smithi003 ceph-mon[107278]: osdmap e432: 8 total, 8 up, 6 in 2024-09-13T15:07:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:15 smithi003 ceph-mon[107278]: 14.e starting backfill to osd.0 from (0'0,0'0] MAX to 430'7794 2024-09-13T15:07:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:15.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:15.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:15.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:15.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:16.060 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.097 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:16.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:16.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:16.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:16.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:16.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: osdmap e433: 8 total, 8 up, 6 in 2024-09-13T15:07:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: 14.a starting backfill to osd.6 from (0'0,0'0] MAX to 430'7745 2024-09-13T15:07:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: 14.2 starting backfill to osd.5 from (0'0,0'0] MAX to 430'7715 2024-09-13T15:07:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: 14.1 starting backfill to osd.6 from (0'0,0'0] MAX to 430'7703 2024-09-13T15:07:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: 14.8 starting backfill to osd.3 from (0'0,0'0] MAX to 430'7792 2024-09-13T15:07:16.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: 14.6 starting backfill to osd.3 from (0'0,0'0] MAX to 430'7765 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: 14.3 starting backfill to osd.5 from (0'0,0'0] MAX to 430'7730 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: 14.f starting backfill to osd.5 from (0'0,0'0] MAX to 430'7656 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: 14.b starting backfill to osd.6 from (0'0,0'0] MAX to 430'7517 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 13516/300849 objects degraded (4.493%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: osdmap e433: 8 total, 8 up, 6 in 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: 14.a starting backfill to osd.6 from (0'0,0'0] MAX to 430'7745 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: 14.2 starting backfill to osd.5 from (0'0,0'0] MAX to 430'7715 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: 14.1 starting backfill to osd.6 from (0'0,0'0] MAX to 430'7703 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: 14.8 starting backfill to osd.3 from (0'0,0'0] MAX to 430'7792 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: 14.6 starting backfill to osd.3 from (0'0,0'0] MAX to 430'7765 2024-09-13T15:07:16.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: 14.3 starting backfill to osd.5 from (0'0,0'0] MAX to 430'7730 2024-09-13T15:07:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: 14.f starting backfill to osd.5 from (0'0,0'0] MAX to 430'7656 2024-09-13T15:07:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: 14.b starting backfill to osd.6 from (0'0,0'0] MAX to 430'7517 2024-09-13T15:07:16.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 13516/300849 objects degraded (4.493%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:16.451 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:07:16.451 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:07:16.451 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:07:16.451 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:07:16.451 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:07:16.452 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:16.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:16.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:16.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:16.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: osdmap e433: 8 total, 8 up, 6 in 2024-09-13T15:07:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: 14.a starting backfill to osd.6 from (0'0,0'0] MAX to 430'7745 2024-09-13T15:07:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: 14.2 starting backfill to osd.5 from (0'0,0'0] MAX to 430'7715 2024-09-13T15:07:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: 14.1 starting backfill to osd.6 from (0'0,0'0] MAX to 430'7703 2024-09-13T15:07:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: 14.8 starting backfill to osd.3 from (0'0,0'0] MAX to 430'7792 2024-09-13T15:07:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: 14.6 starting backfill to osd.3 from (0'0,0'0] MAX to 430'7765 2024-09-13T15:07:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: 14.3 starting backfill to osd.5 from (0'0,0'0] MAX to 430'7730 2024-09-13T15:07:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: 14.f starting backfill to osd.5 from (0'0,0'0] MAX to 430'7656 2024-09-13T15:07:16.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: 14.b starting backfill to osd.6 from (0'0,0'0] MAX to 430'7517 2024-09-13T15:07:16.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 13516/300849 objects degraded (4.493%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:16.768 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:16.801 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:16.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:16.903 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:16.941 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:17.043 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:17.070 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:17.172 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:17.203 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:17.305 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:17.333 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:17 smithi003 ceph-mon[107278]: pgmap v1552: 119 pgs: 1 active+recovering+undersized+remapped, 38 peering, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 548 op/s; 13516/300849 objects degraded (4.493%); 21620/300849 objects misplaced (7.186%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T15:07:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:17 smithi003 ceph-mon[104007]: pgmap v1552: 119 pgs: 1 active+recovering+undersized+remapped, 38 peering, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 548 op/s; 13516/300849 objects degraded (4.493%); 21620/300849 objects misplaced (7.186%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T15:07:17.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:17.467 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:17.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:07:17.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:17.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:17 smithi019 ceph-mon[91296]: pgmap v1552: 119 pgs: 1 active+recovering+undersized+remapped, 38 peering, 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 548 op/s; 13516/300849 objects degraded (4.493%); 21620/300849 objects misplaced (7.186%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T15:07:17.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:17.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:17.821 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:07:17.821 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:07:17.860 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:07:17.860 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:07:17.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:17.894 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:07:17.894 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:07:17.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:17.934 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:07:17.935 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:07:17.967 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:07:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:07:18.000 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:07:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:07:18.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:18.033 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:07:18.033 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:07:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:18.077 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:07:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:07:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:18.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:18.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:18.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:18.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:18.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:18.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:18.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:18.626 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:18.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3271451211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:18.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3271451211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3271451211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:18.728 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:18.758 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:18.860 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:18.886 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:18.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:19.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:19.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:19.125 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:19.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:19.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:19.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:19.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:19.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:19.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:19.567 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:19.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:19 smithi019 ceph-mon[91296]: pgmap v1553: 119 pgs: 1 active+recovering+undersized+remapped, 38 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 73 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 562 op/s; 13516/298914 objects degraded (4.522%); 36654/298914 objects misplaced (12.262%); 7.6 MiB/s, 121 objects/s recovering 2024-09-13T15:07:19.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:19 smithi003 ceph-mon[107278]: pgmap v1553: 119 pgs: 1 active+recovering+undersized+remapped, 38 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 73 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 562 op/s; 13516/298914 objects degraded (4.522%); 36654/298914 objects misplaced (12.262%); 7.6 MiB/s, 121 objects/s recovering 2024-09-13T15:07:19.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:19 smithi003 ceph-mon[104007]: pgmap v1553: 119 pgs: 1 active+recovering+undersized+remapped, 38 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 73 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 562 op/s; 13516/298914 objects degraded (4.522%); 36654/298914 objects misplaced (12.262%); 7.6 MiB/s, 121 objects/s recovering 2024-09-13T15:07:19.700 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:19.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:19.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:19.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:19.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:20.093 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:20.126 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:20.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:20.266 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:20.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:20.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:20.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:20.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:20.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:20.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:20.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:20.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:20 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-13T15:07:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:07:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:07:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:20.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:20 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-13T15:07:20.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:20 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-13T15:07:20.701 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:20.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:20.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:20.950 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:20.981 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:21.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:21.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:21.222 INFO:tasks.thrashosds.thrasher:in_osds: [2, 6, 7, 5, 0, 3] out_osds: [4, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:07:21.222 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:07:21.222 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-13T15:07:21.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-13T15:07:21.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:21.258 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:21.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.399 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:21.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:21.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:21.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:21 smithi003 ceph-mon[104007]: pgmap v1554: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 659 op/s; 13515/294072 objects degraded (4.596%); 72896/294072 objects misplaced (24.788%); 5.4 MiB/s, 87 objects/s recovering 2024-09-13T15:07:21.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 13515/294072 objects degraded (4.596%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:21.615 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:21 smithi003 ceph-mon[107278]: pgmap v1554: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 659 op/s; 13515/294072 objects degraded (4.596%); 72896/294072 objects misplaced (24.788%); 5.4 MiB/s, 87 objects/s recovering 2024-09-13T15:07:21.615 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 13515/294072 objects degraded (4.596%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:21.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:21 smithi019 ceph-mon[91296]: pgmap v1554: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 659 op/s; 13515/294072 objects degraded (4.596%); 72896/294072 objects misplaced (24.788%); 5.4 MiB/s, 87 objects/s recovering 2024-09-13T15:07:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 13515/294072 objects degraded (4.596%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:21.686 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:21.788 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:21.817 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:21.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:21.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:07:21.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:22.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:22.091 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:22.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:22.219 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:22.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:22.351 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:22.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:22.482 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:22.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:22.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:22.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:22.773 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:22.774 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:22.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:22.818 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:07:22.819 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:07:22.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:22.907 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:23.009 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:23.038 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:23.084 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:23.140 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:23.168 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:23.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:23.301 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:23.375 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:07:23.403 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.435 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:23.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:23.455 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:23 smithi003 ceph-mon[104007]: pgmap v1555: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 369 op/s; 13515/291987 objects degraded (4.629%); 72201/291987 objects misplaced (24.727%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T15:07:23.455 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:23 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/750465237' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:23.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:23 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:23.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:23 smithi003 ceph-mon[107278]: pgmap v1555: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 369 op/s; 13515/291987 objects degraded (4.629%); 72201/291987 objects misplaced (24.727%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T15:07:23.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:23 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/750465237' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:23.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:23 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:23.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:23.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:23.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:23.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:23 smithi019 ceph-mon[91296]: pgmap v1555: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 369 op/s; 13515/291987 objects degraded (4.629%); 72201/291987 objects misplaced (24.727%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T15:07:23.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:23 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/750465237' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:23.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:23 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:23.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:23.847 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:23.949 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:23.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:24.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:24.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:24.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:07:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:07:24] "GET /metrics HTTP/1.1" 200 36560 "" "Prometheus/2.51.0" 2024-09-13T15:07:24.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:24.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:24.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:24.383 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:24.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:24.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:24.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:24.667 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:24 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:24 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:07:24.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:24 smithi019 ceph-mon[91296]: osdmap e434: 8 total, 8 up, 6 in 2024-09-13T15:07:24.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:24 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:24.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:24 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:07:24.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:24 smithi003 ceph-mon[107278]: osdmap e434: 8 total, 8 up, 6 in 2024-09-13T15:07:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:24 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:24 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:07:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:24 smithi003 ceph-mon[104007]: osdmap e434: 8 total, 8 up, 6 in 2024-09-13T15:07:24.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:24.796 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:24.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:24.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:25.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:25.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:25.217 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:25.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:25.350 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:25.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:25.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:25.480 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:25.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:25.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:25 smithi019 ceph-mon[91296]: pgmap v1557: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 669 op/s; 13665/282927 objects degraded (4.830%); 70158/282927 objects misplaced (24.797%); 4.1 MiB/s, 67 objects/s recovering 2024-09-13T15:07:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:25 smithi003 ceph-mon[104007]: pgmap v1557: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 669 op/s; 13665/282927 objects degraded (4.830%); 70158/282927 objects misplaced (24.797%); 4.1 MiB/s, 67 objects/s recovering 2024-09-13T15:07:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:25 smithi003 ceph-mon[107278]: pgmap v1557: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 669 op/s; 13665/282927 objects degraded (4.830%); 70158/282927 objects misplaced (24.797%); 4.1 MiB/s, 67 objects/s recovering 2024-09-13T15:07:25.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:25.769 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:25.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:26.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:26.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:26.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:26.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:26.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:26.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:26.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:26.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:26.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:26.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 13665/282927 objects degraded (4.830%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:26.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 13665/282927 objects degraded (4.830%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:26.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 13665/282927 objects degraded (4.830%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:26.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:26.757 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:26.859 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:26.889 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:26.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:27.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:27.126 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:27.157 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:27.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:27.295 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:27.397 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:27.424 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:27.526 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:27.562 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:27.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:27.664 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:27 smithi019 ceph-mon[91296]: pgmap v1558: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 597 op/s; 13665/282927 objects degraded (4.830%); 70158/282927 objects misplaced (24.797%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T15:07:27.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:27 smithi003 ceph-mon[104007]: pgmap v1558: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 597 op/s; 13665/282927 objects degraded (4.830%); 70158/282927 objects misplaced (24.797%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T15:07:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:27 smithi003 ceph-mon[107278]: pgmap v1558: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 597 op/s; 13665/282927 objects degraded (4.830%); 70158/282927 objects misplaced (24.797%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T15:07:27.698 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:27.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:27.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:27.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:27.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:28.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:28.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:28.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:28.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:28.240 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:28.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:28.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:28.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:28.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:28.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:28.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:28.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:28.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:28.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:28.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:28.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:28.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:28.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:28.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:29.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:29.050 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:29.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:29.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:29.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:29.310 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:29.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:29.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:29.551 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:29.579 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:29.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:29.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:29 smithi019 ceph-mon[91296]: pgmap v1559: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 13665/279174 objects degraded (4.895%); 69144/279174 objects misplaced (24.767%); 3.6 MiB/s, 58 objects/s recovering 2024-09-13T15:07:29.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:29 smithi003 ceph-mon[107278]: pgmap v1559: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 13665/279174 objects degraded (4.895%); 69144/279174 objects misplaced (24.767%); 3.6 MiB/s, 58 objects/s recovering 2024-09-13T15:07:29.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:29 smithi003 ceph-mon[104007]: pgmap v1559: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 13665/279174 objects degraded (4.895%); 69144/279174 objects misplaced (24.767%); 3.6 MiB/s, 58 objects/s recovering 2024-09-13T15:07:29.709 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:29.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:29.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:29.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:29.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:30.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:30.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:30.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:30.264 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:30.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:07:30.365 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:30.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:30.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.525 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:30.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:30.626 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:30.659 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:07:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:07:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:30.761 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:30.789 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:30.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:30.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:30.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:31.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:31.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:31.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:31.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:31.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:31.506 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:31.537 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:31.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:07:31.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:31.664 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:31 smithi003 ceph-mon[104007]: pgmap v1560: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 767 op/s; 13916/271023 objects degraded (5.135%); 67144/271023 objects misplaced (24.774%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T15:07:31.664 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 13916/271023 objects degraded (5.135%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:31.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:31 smithi003 ceph-mon[107278]: pgmap v1560: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 767 op/s; 13916/271023 objects degraded (5.135%); 67144/271023 objects misplaced (24.774%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T15:07:31.664 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 13916/271023 objects degraded (5.135%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:31 smithi019 ceph-mon[91296]: pgmap v1560: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 767 op/s; 13916/271023 objects degraded (5.135%); 67144/271023 objects misplaced (24.774%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T15:07:31.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 13916/271023 objects degraded (5.135%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:31.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:31.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:31.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:31.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:32.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:32.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:32.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:32.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:32.323 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:07:32.323 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:07:32.324 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:07:32.324 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:07:32.343 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:32.354 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:07:32.354 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:07:32.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:32.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:32.402 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:07:32.402 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:07:32.429 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:07:32.429 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:07:32.446 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:07:32.447 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:07:32.476 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:07:32.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4215016048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:32.477 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4215016048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:32.477 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:07:32.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:32.509 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:07:32.509 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:07:32.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:32.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:32.665 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:32.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:32.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:32.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:32.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4215016048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:32.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:07:32.652+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e434 definitely_dead 0 2024-09-13T15:07:32.936 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:33.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:33.060 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:33.162 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:33.185 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:33.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:33.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:33.427 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:33.461 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:33.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:33.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:33.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:33.629 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.4. 2024-09-13T15:07:33.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:33.718 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:33.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:33.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:33 smithi019 ceph-mon[91296]: pgmap v1561: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 768 op/s; 13916/268896 objects degraded (5.175%); 66435/268896 objects misplaced (24.707%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T15:07:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:33 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1162942423' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T15:07:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:33 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T15:07:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:33 smithi003 ceph-mon[107278]: pgmap v1561: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 768 op/s; 13916/268896 objects degraded (5.175%); 66435/268896 objects misplaced (24.707%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T15:07:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:33 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1162942423' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T15:07:33.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:33 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T15:07:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:33 smithi003 ceph-mon[104007]: pgmap v1561: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 105 active+clean; 5.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 768 op/s; 13916/268896 objects degraded (5.175%); 66435/268896 objects misplaced (24.707%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T15:07:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:33 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1162942423' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T15:07:33.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:33 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T15:07:33.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:34.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:34.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:34.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:34.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:34.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:34.330 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:34.355 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:07:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:07:34] "GET /metrics HTTP/1.1" 200 36555 "" "Prometheus/2.51.0" 2024-09-13T15:07:34.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:34.497 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:34.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:34.573 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-13T15:07:34.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:34.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:34.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:34.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:34.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:34 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T15:07:34.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:34 smithi003 ceph-mon[107278]: osdmap e435: 8 total, 8 up, 7 in 2024-09-13T15:07:34.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:34 smithi003 ceph-mon[107278]: osdmap e436: 8 total, 8 up, 7 in 2024-09-13T15:07:34.805 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:34 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T15:07:34.805 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:34 smithi003 ceph-mon[104007]: osdmap e435: 8 total, 8 up, 7 in 2024-09-13T15:07:34.805 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:34 smithi003 ceph-mon[104007]: osdmap e436: 8 total, 8 up, 7 in 2024-09-13T15:07:34.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:34.893 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:34 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T15:07:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:34 smithi019 ceph-mon[91296]: osdmap e435: 8 total, 8 up, 7 in 2024-09-13T15:07:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:34 smithi019 ceph-mon[91296]: osdmap e436: 8 total, 8 up, 7 in 2024-09-13T15:07:34.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:35.021 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:35.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:35.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:35.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:35.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:35.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:35.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:35.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:35.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:35.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:35.668 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:35 smithi003 ceph-mon[104007]: pgmap v1563: 119 pgs: 2 active+remapped+backfilling, 5 remapped+peering, 38 peering, 7 active+remapped+backfill_wait, 67 active+clean; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 728 op/s; 48965/260943 objects misplaced (18.765%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T15:07:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:35 smithi003 ceph-mon[104007]: 14.a starting backfill to osd.4 from (0'0,0'0] MAX to 434'8600 2024-09-13T15:07:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:35 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13916/268896 objects degraded (5.175%), 4 pgs degraded) 2024-09-13T15:07:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:35 smithi003 ceph-mon[104007]: 14.3 starting backfill to osd.4 from (0'0,0'0] MAX to 434'8577 2024-09-13T15:07:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:35 smithi003 ceph-mon[107278]: pgmap v1563: 119 pgs: 2 active+remapped+backfilling, 5 remapped+peering, 38 peering, 7 active+remapped+backfill_wait, 67 active+clean; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 728 op/s; 48965/260943 objects misplaced (18.765%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T15:07:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:35 smithi003 ceph-mon[107278]: 14.a starting backfill to osd.4 from (0'0,0'0] MAX to 434'8600 2024-09-13T15:07:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:35 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13916/268896 objects degraded (5.175%), 4 pgs degraded) 2024-09-13T15:07:35.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:35 smithi003 ceph-mon[107278]: 14.3 starting backfill to osd.4 from (0'0,0'0] MAX to 434'8577 2024-09-13T15:07:35.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:35.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:35.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:35 smithi019 ceph-mon[91296]: pgmap v1563: 119 pgs: 2 active+remapped+backfilling, 5 remapped+peering, 38 peering, 7 active+remapped+backfill_wait, 67 active+clean; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 728 op/s; 48965/260943 objects misplaced (18.765%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T15:07:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:35 smithi019 ceph-mon[91296]: 14.a starting backfill to osd.4 from (0'0,0'0] MAX to 434'8600 2024-09-13T15:07:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:35 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13916/268896 objects degraded (5.175%), 4 pgs degraded) 2024-09-13T15:07:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:35 smithi019 ceph-mon[91296]: 14.3 starting backfill to osd.4 from (0'0,0'0] MAX to 434'8577 2024-09-13T15:07:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:36.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:36.092 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:36.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:36.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:36.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:36.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:36.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:36.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:36.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:36.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:36.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:36 smithi003 ceph-mon[107278]: osdmap e437: 8 total, 8 up, 7 in 2024-09-13T15:07:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:36 smithi003 ceph-mon[104007]: osdmap e437: 8 total, 8 up, 7 in 2024-09-13T15:07:36.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:36.808 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:36.910 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:36.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:36 smithi019 ceph-mon[91296]: osdmap e437: 8 total, 8 up, 7 in 2024-09-13T15:07:36.943 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:37.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:37.077 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:37.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:37.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:37.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:37.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.362 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:37.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:37.463 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:37.491 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:37.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:37.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:37.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:37 smithi003 ceph-mon[107278]: pgmap v1566: 119 pgs: 2 active+remapped+backfilling, 5 remapped+peering, 38 peering, 7 active+remapped+backfill_wait, 67 active+clean; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 48965/260943 objects misplaced (18.765%); 3.3 MiB/s, 51 objects/s recovering 2024-09-13T15:07:37.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:37 smithi003 ceph-mon[104007]: pgmap v1566: 119 pgs: 2 active+remapped+backfilling, 5 remapped+peering, 38 peering, 7 active+remapped+backfill_wait, 67 active+clean; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 48965/260943 objects misplaced (18.765%); 3.3 MiB/s, 51 objects/s recovering 2024-09-13T15:07:37.733 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:37.772 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:37.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:07:37.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.905 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:37.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:37.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:37 smithi019 ceph-mon[91296]: pgmap v1566: 119 pgs: 2 active+remapped+backfilling, 5 remapped+peering, 38 peering, 7 active+remapped+backfill_wait, 67 active+clean; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 48965/260943 objects misplaced (18.765%); 3.3 MiB/s, 51 objects/s recovering 2024-09-13T15:07:38.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:38.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:38.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:38.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:38.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.319 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:38.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:38.420 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:38.459 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:38.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.597 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:38.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:38.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:38.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:38.823 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:38.857 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:38.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:38.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:39.099 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:39.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.141 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:39.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:39.242 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:39.281 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:39.383 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:39.425 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:39.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:39.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.566 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:39.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:39.574 INFO:tasks.thrashosds.thrasher:in_osds: [2, 6, 7, 5, 0, 3, 4] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:07:39.574 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:07:39.575 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T15:07:39.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-09-13T15:07:39.667 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:39.700 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:39.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:39.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:39.929 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:39 smithi019 ceph-mon[91296]: pgmap v1567: 119 pgs: 3 active+remapped+backfilling, 5 remapped+peering, 38 peering, 5 active+remapped+backfill_wait, 68 active+clean; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 511 op/s; 42050/259581 objects misplaced (16.199%); 15 MiB/s, 246 objects/s recovering 2024-09-13T15:07:39.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:39 smithi003 ceph-mon[104007]: pgmap v1567: 119 pgs: 3 active+remapped+backfilling, 5 remapped+peering, 38 peering, 5 active+remapped+backfill_wait, 68 active+clean; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 511 op/s; 42050/259581 objects misplaced (16.199%); 15 MiB/s, 246 objects/s recovering 2024-09-13T15:07:39.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:39 smithi003 ceph-mon[107278]: pgmap v1567: 119 pgs: 3 active+remapped+backfilling, 5 remapped+peering, 38 peering, 5 active+remapped+backfill_wait, 68 active+clean; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 511 op/s; 42050/259581 objects misplaced (16.199%); 15 MiB/s, 246 objects/s recovering 2024-09-13T15:07:39.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:40.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:40.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:40.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:40.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:40.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:40.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:40.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:40.501 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:40.533 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:07:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:07:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:40.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:40.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:40.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.775 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:40.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:40.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:40.912 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:40.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:40 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 5597/252240 objects degraded (2.219%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:40 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 5597/252240 objects degraded (2.219%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:40 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 5597/252240 objects degraded (2.219%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:41.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.050 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:41.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:41.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:41.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:41.292 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:07:41.292 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:07:41.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.292 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:141788 14.a 14:5f46fc13:::benchmark_data_smithi003_327099_object30062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:07:41.292 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:41.276058+0000", 2024-09-13T15:07:41.292 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0093783010000000003, 2024-09-13T15:07:41.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094081000000000008, 2024-09-13T15:07:41.292 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:41.292 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141788 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.276058+0000", 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.276058+0000", 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.276060+0000", 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2410000000000001e-06 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.276064+0000", 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7840000000000003e-06 2024-09-13T15:07:41.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.276077+0000", 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.218e-05 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.276084+0000", 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7600000000000002e-06 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.276419+0000", 2024-09-13T15:07:41.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000334297 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.276564+0000", 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014547599999999999 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4,5", 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.276705+0000", 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014113 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280330+0000", 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036249609999999999 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:141795 14.a 14:5dd1998f:::benchmark_data_smithi003_327099_object30069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:41.277874+0000", 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0075632169999999997, 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076676720000000004, 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:41.297 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141795 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.277874+0000", 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.277874+0000", 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.277875+0000", 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.421e-06 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.277880+0000", 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1220000000000003e-06 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.277892+0000", 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2299e-05 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.277900+0000", 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4819999999999997e-06 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.283569+0000", 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056690839999999996 2024-09-13T15:07:41.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.283708+0000", 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013871099999999999 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4,5", 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.283846+0000", 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013829700000000001 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:141791 14.3 e437/435 14:c2fbfd33:::benchmark_data_smithi003_327099_object30065:head v 437'8911, mlcod=437'8908)", 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:41.280171+0000", 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0052659289999999999, 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054320590000000004, 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280171+0000", 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280171+0000", 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280172+0000", 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6499999999999998e-07 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280178+0000", 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.942e-06 2024-09-13T15:07:41.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280190+0000", 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2198000000000001e-05 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280196+0000", 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5599999999999999e-06 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:07:41.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280292+0000", 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6136000000000002e-05 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280428+0000", 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013587699999999999 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.47273.0:141788 14.a e437/435)", 2024-09-13T15:07:41.305 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:41.280658+0000", 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0047786649999999996, 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049898629999999998, 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280658+0000", 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:41.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280658+0000", 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280659+0000", 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6599999999999996e-07 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280663+0000", 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1300000000000003e-06 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280673+0000", 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0530000000000001e-05 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.280681+0000", 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0729999999999999e-06 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.47273.0:141788 14.a e437/435)", 2024-09-13T15:07:41.308 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:41.281913+0000", 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0035233899999999999, 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037722049999999998, 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.281913+0000", 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.281913+0000", 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.281914+0000", 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5000000000000003e-07 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.281918+0000", 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9999999999999998e-06 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.281928+0000", 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0513000000000001e-05 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.281935+0000", 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9179999999999997e-06 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.47273.0:141788 14.a e437/435)", 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:41.283454+0000", 2024-09-13T15:07:41.311 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0019827209999999998, 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002267519, 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.283454+0000", 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:41.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.283454+0000", 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.283455+0000", 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5600000000000005e-07 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.283459+0000", 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.267e-06 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.283469+0000", 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0067e-05 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.283476+0000", 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6449999999999999e-06 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:141800 14.6 e437/432 14:66720c21:::benchmark_data_smithi003_327099_object30074:head v 437'8932, mlcod=437'8930)", 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:41.284823+0000", 2024-09-13T15:07:41.314 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00061334400000000004, 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00093740800000000003, 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284823+0000", 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284823+0000", 2024-09-13T15:07:41.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284825+0000", 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1030000000000001e-06 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284832+0000", 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2259999999999999e-06 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:41.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284843+0000", 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1299e-05 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284856+0000", 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3332999999999999e-05 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284975+0000", 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011848700000000001 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:07:41.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284995+0000", 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9882e-05 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:141792 14.3 e437/435)", 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:41.284877+0000", 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00056009500000000004, 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092827999999999997, 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-09-13T15:07:41.318 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284877+0000", 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284877+0000", 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:07:41.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284877+0000", 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1e-07 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284881+0000", 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.557e-06 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284887+0000", 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6449999999999999e-06 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:07:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:07:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:41.284892+0000", 2024-09-13T15:07:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5800000000000002e-06 2024-09-13T15:07:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:07:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:07:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-13T15:07:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 8 2024-09-13T15:07:41.321 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:07:41.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:41.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:41.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:41.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:41.615 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:41.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:41.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:41 smithi003 ceph-mon[107278]: pgmap v1568: 119 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 5.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 5597/252240 objects degraded (2.219%); 38944/252240 objects misplaced (15.439%); 34 MiB/s, 1 keys/s, 551 objects/s recovering 2024-09-13T15:07:41.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:41 smithi003 ceph-mon[104007]: pgmap v1568: 119 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 5.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 5597/252240 objects degraded (2.219%); 38944/252240 objects misplaced (15.439%); 34 MiB/s, 1 keys/s, 551 objects/s recovering 2024-09-13T15:07:41.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:41.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:41.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:41.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:41 smithi019 ceph-mon[91296]: pgmap v1568: 119 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 5.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 5597/252240 objects degraded (2.219%); 38944/252240 objects misplaced (15.439%); 34 MiB/s, 1 keys/s, 551 objects/s recovering 2024-09-13T15:07:42.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:42.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:42.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:42.216 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:42.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:42.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:42.500 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:42.602 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:42.640 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:42.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:42.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:42.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:42.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:07:42.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:43.026 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:43.064 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:43.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:43.195 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:43.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:43.448 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:43.451 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:43.456 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:43.485 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:43.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:43.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:43 smithi003 ceph-mon[107278]: pgmap v1569: 119 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 5.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 2.5 KiB/s wr, 464 op/s; 5597/249903 objects degraded (2.240%); 37787/249903 objects misplaced (15.121%); 32 MiB/s, 1 keys/s, 520 objects/s recovering 2024-09-13T15:07:43.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:43 smithi003 ceph-mon[104007]: pgmap v1569: 119 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 5.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 2.5 KiB/s wr, 464 op/s; 5597/249903 objects degraded (2.240%); 37787/249903 objects misplaced (15.121%); 32 MiB/s, 1 keys/s, 520 objects/s recovering 2024-09-13T15:07:43.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:43.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:43.763 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:43.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:43 smithi019 ceph-mon[91296]: pgmap v1569: 119 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 5.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 2.5 KiB/s wr, 464 op/s; 5597/249903 objects degraded (2.240%); 37787/249903 objects misplaced (15.121%); 32 MiB/s, 1 keys/s, 520 objects/s recovering 2024-09-13T15:07:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:44.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:44.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:44.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:44.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:44.301 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:44.339 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:44.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:07:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:07:44] "GET /metrics HTTP/1.1" 200 36560 "" "Prometheus/2.51.0" 2024-09-13T15:07:44.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:44.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:44.574 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:07:44.594 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.633 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:44.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:44.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:44.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:44.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:44.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:44 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3491348788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:44 smithi019 ceph-mon[91296]: pgmap v1570: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s wr, 733 op/s; 5597/239508 objects degraded (2.337%); 35095/239508 objects misplaced (14.653%); 34 MiB/s, 1 keys/s, 582 objects/s recovering 2024-09-13T15:07:44.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:44 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3491348788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:44.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:44 smithi003 ceph-mon[104007]: pgmap v1570: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s wr, 733 op/s; 5597/239508 objects degraded (2.337%); 35095/239508 objects misplaced (14.653%); 34 MiB/s, 1 keys/s, 582 objects/s recovering 2024-09-13T15:07:44.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:44 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3491348788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:44.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:44 smithi003 ceph-mon[107278]: pgmap v1570: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s wr, 733 op/s; 5597/239508 objects degraded (2.337%); 35095/239508 objects misplaced (14.653%); 34 MiB/s, 1 keys/s, 582 objects/s recovering 2024-09-13T15:07:45.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:45.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:45.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.195 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:45.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:45.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.338 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:45.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:45.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:45.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:45.613 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:45.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:45.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:45.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:45 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:07:45.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:45 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3491348788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:45.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:45 smithi003 ceph-mon[107278]: osdmap e438: 8 total, 8 up, 7 in 2024-09-13T15:07:45.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:45 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:07:45.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:45 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3491348788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:45.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:45 smithi003 ceph-mon[104007]: osdmap e438: 8 total, 8 up, 7 in 2024-09-13T15:07:45.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:45.889 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:45 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:07:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:45 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3491348788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:45.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:45 smithi019 ceph-mon[91296]: osdmap e438: 8 total, 8 up, 7 in 2024-09-13T15:07:45.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:46.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:46.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:46.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:46.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:46.315 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:46.418 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:46.455 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:46.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:46.567 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:46.593 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:46.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:46.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:46.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:46.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:46.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:46 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5597/239508 objects degraded (2.337%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:46 smithi019 ceph-mon[91296]: pgmap v1572: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s wr, 717 op/s; 5597/239508 objects degraded (2.337%); 35095/239508 objects misplaced (14.653%); 33 MiB/s, 0 keys/s, 569 objects/s recovering 2024-09-13T15:07:46.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:46 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5597/239508 objects degraded (2.337%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:46.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:46 smithi003 ceph-mon[107278]: pgmap v1572: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s wr, 717 op/s; 5597/239508 objects degraded (2.337%); 35095/239508 objects misplaced (14.653%); 33 MiB/s, 0 keys/s, 569 objects/s recovering 2024-09-13T15:07:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:46 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5597/239508 objects degraded (2.337%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:46 smithi003 ceph-mon[104007]: pgmap v1572: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s wr, 717 op/s; 5597/239508 objects degraded (2.337%); 35095/239508 objects misplaced (14.653%); 33 MiB/s, 0 keys/s, 569 objects/s recovering 2024-09-13T15:07:47.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:47.045 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:47.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:47.148 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:47.188 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:47.290 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:47.331 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:47.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:47.433 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:47.472 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:47.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:47.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:47.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:47.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:47.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:47.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:07:47.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:47.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:48.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:48.095 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:48.197 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:48.233 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:48.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:48.365 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:48.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.503 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:48.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:48.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:48.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:48.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:48.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:48.816 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:07:48.816 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:07:48.864 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:07:48.864 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:07:48.873 INFO:teuthology.orchestra.run.smithi003.stdout:moved 4 / 357 (1.12045%) 2024-09-13T15:07:48.873 INFO:teuthology.orchestra.run.smithi003.stdout:avg 51 2024-09-13T15:07:48.873 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.61168 -> 6.1412 (expected baseline 6.61168) 2024-09-13T15:07:48.873 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 39 -> 40 pgs (0.764706 -> 0.784314 * mean) 2024-09-13T15:07:48.873 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 61 -> 62 pgs (1.19608 -> 1.21569 * mean) 2024-09-13T15:07:48.873 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:07:48.873 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-09-13T15:07:48.873 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 0.05 2024-09-13T15:07:48.874 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T15:07:48.874 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0352 2024-09-13T15:07:48.874 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0387 2024-09-13T15:07:48.874 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 weight 1.0000 -> 0.9500 2024-09-13T15:07:48.874 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 weight 1.0000 -> 0.9500 2024-09-13T15:07:48.874 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T15:07:48.904 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:07:48.905 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:07:48.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:48.938 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:48.957 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:07:48.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:07:48.996 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:07:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:07:49.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:49.041 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:07:49.041 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:07:49.076 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:07:49.077 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:07:49.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:49.112 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:07:49.112 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:07:49.132 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:07:48 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T15:07:48.869+0000 7f0f7ea9d640 -1 mgr.server reply reply (2) No such file or directory no change 2024-09-13T15:07:49.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:49.219 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:49.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.360 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:49.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:49.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:49 smithi003 ceph-mon[104007]: pgmap v1573: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s wr, 758 op/s; 5597/236913 objects degraded (2.362%); 34045/236913 objects misplaced (14.370%); 30 MiB/s, 0 keys/s, 513 objects/s recovering 2024-09-13T15:07:49.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4028054150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:49.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:49 smithi003 ceph-mon[107278]: pgmap v1573: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s wr, 758 op/s; 5597/236913 objects degraded (2.362%); 34045/236913 objects misplaced (14.370%); 30 MiB/s, 0 keys/s, 513 objects/s recovering 2024-09-13T15:07:49.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4028054150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:49.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:49.498 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:49.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:49 smithi019 ceph-mon[91296]: pgmap v1573: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s wr, 758 op/s; 5597/236913 objects degraded (2.362%); 34045/236913 objects misplaced (14.370%); 30 MiB/s, 0 keys/s, 513 objects/s recovering 2024-09-13T15:07:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4028054150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:49.746 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:49.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:49.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:50.026 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:50.060 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:50.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:50.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:50.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:50.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:50.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:50.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:50 smithi003 ceph-mon[104007]: from='client.57022 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:07:50.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:50 smithi003 ceph-mon[107278]: from='client.57022 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:07:50.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:50.485 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:50.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.623 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:50.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:50.644 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T15:07:50.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-13T15:07:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:07:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:07:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:50.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:50 smithi019 ceph-mon[91296]: from='client.57022 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:07:50.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:50.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:50.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:50.895 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:50.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:50.997 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:51.036 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:51.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:51.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:51.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:51.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:51.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:51.464 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:51.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:51.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:51.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:51 smithi003 ceph-mon[104007]: pgmap v1574: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 5597/225558 objects degraded (2.481%); 31374/225558 objects misplaced (13.910%); 23 MiB/s, 434 objects/s recovering 2024-09-13T15:07:51.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:51 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5597/225558 objects degraded (2.481%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:51.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:51 smithi003 ceph-mon[107278]: pgmap v1574: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 5597/225558 objects degraded (2.481%); 31374/225558 objects misplaced (13.910%); 23 MiB/s, 434 objects/s recovering 2024-09-13T15:07:51.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:51 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5597/225558 objects degraded (2.481%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:51 smithi019 ceph-mon[91296]: pgmap v1574: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 5597/225558 objects degraded (2.481%); 31374/225558 objects misplaced (13.910%); 23 MiB/s, 434 objects/s recovering 2024-09-13T15:07:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:51 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5597/225558 objects degraded (2.481%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:51.714 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:51.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:51.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:51.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:52.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:52.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:52.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:52.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:52.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:52.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:52.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:52.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:52.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:52.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:52.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:52.754 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:52.770 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:52.774 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:52.815 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:07:52.816 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:07:52.856 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:52.894 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:52.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:53.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:53.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:53.289 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:53.332 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:53.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:53.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:53.563 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:53.599 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:53 smithi019 ceph-mon[91296]: pgmap v1575: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 901 op/s; 5597/222840 objects degraded (2.512%); 30402/222840 objects misplaced (13.643%); 21 MiB/s, 412 objects/s recovering 2024-09-13T15:07:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:53 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:07:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:53 smithi003 ceph-mon[107278]: pgmap v1575: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 901 op/s; 5597/222840 objects degraded (2.512%); 30402/222840 objects misplaced (13.643%); 21 MiB/s, 412 objects/s recovering 2024-09-13T15:07:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:53 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:07:53.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:53 smithi003 ceph-mon[104007]: pgmap v1575: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 901 op/s; 5597/222840 objects degraded (2.512%); 30402/222840 objects misplaced (13.643%); 21 MiB/s, 412 objects/s recovering 2024-09-13T15:07:53.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:53 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:07:53.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:53.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:53.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:53.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:53.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:54.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:54.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:54.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:54.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:07:54.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:54.294 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:54.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.429 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:54.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:07:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:07:54] "GET /metrics HTTP/1.1" 200 36560 "" "Prometheus/2.51.0" 2024-09-13T15:07:54.529 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:54.557 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:54.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:54.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:54.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:54.784 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:54.812 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:54.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:07:54.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:07:54.867 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:54 smithi019 ceph-mon[91296]: pgmap v1576: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 5597/211515 objects degraded (2.646%); 27834/211515 objects misplaced (13.159%); 21 MiB/s, 412 objects/s recovering 2024-09-13T15:07:54.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:07:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:07:54.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:54 smithi003 ceph-mon[104007]: pgmap v1576: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 5597/211515 objects degraded (2.646%); 27834/211515 objects misplaced (13.159%); 21 MiB/s, 412 objects/s recovering 2024-09-13T15:07:54.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:07:54.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:07:54.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:54 smithi003 ceph-mon[107278]: pgmap v1576: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 5597/211515 objects degraded (2.646%); 27834/211515 objects misplaced (13.159%); 21 MiB/s, 412 objects/s recovering 2024-09-13T15:07:54.951 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:55.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:55.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:55.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:55.222 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:55.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:55.349 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:55.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:55.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:55.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:55.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:55.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:55.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:55.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:55.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:55.872 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:55.905 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:55 smithi019 ceph-mon[91296]: osdmap e439: 8 total, 8 up, 7 in 2024-09-13T15:07:55.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:55 smithi003 ceph-mon[104007]: osdmap e439: 8 total, 8 up, 7 in 2024-09-13T15:07:55.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:55 smithi003 ceph-mon[107278]: osdmap e439: 8 total, 8 up, 7 in 2024-09-13T15:07:55.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:56.002 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:56.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:07:56.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:56.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:56.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:56.282 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:56.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:56.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:56.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:56.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:56.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:56.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:56.787 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:56.819 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:56.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:56 smithi019 ceph-mon[91296]: osdmap e440: 8 total, 8 up, 7 in 2024-09-13T15:07:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:56 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 5597/211515 objects degraded (2.646%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:56.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:56 smithi019 ceph-mon[91296]: pgmap v1579: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 5597/211515 objects degraded (2.646%); 27834/211515 objects misplaced (13.159%); 21 MiB/s, 438 objects/s recovering 2024-09-13T15:07:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:56 smithi003 ceph-mon[104007]: osdmap e440: 8 total, 8 up, 7 in 2024-09-13T15:07:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:56 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 5597/211515 objects degraded (2.646%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:56 smithi003 ceph-mon[104007]: pgmap v1579: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 5597/211515 objects degraded (2.646%); 27834/211515 objects misplaced (13.159%); 21 MiB/s, 438 objects/s recovering 2024-09-13T15:07:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:56 smithi003 ceph-mon[107278]: osdmap e440: 8 total, 8 up, 7 in 2024-09-13T15:07:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:56 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 5597/211515 objects degraded (2.646%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:07:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:56 smithi003 ceph-mon[107278]: pgmap v1579: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 5597/211515 objects degraded (2.646%); 27834/211515 objects misplaced (13.159%); 21 MiB/s, 438 objects/s recovering 2024-09-13T15:07:56.956 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:57.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:57.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:57.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:57.225 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:57.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:57.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:57.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:57.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:57.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:57.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:57.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:57.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:57.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:57.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:58.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:58.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:58.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:58.182 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:58.283 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:58.312 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:58.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:58.441 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:58.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:58.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:58.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:58.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:58.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.845 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:58.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:58 smithi019 ceph-mon[91296]: osdmap e441: 8 total, 8 up, 7 in 2024-09-13T15:07:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:07:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:58 smithi019 ceph-mon[91296]: pgmap v1581: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 753 op/s; 5597/209280 objects degraded (2.674%); 26768/209280 objects misplaced (12.791%); 20 MiB/s, 378 objects/s recovering 2024-09-13T15:07:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:58 smithi003 ceph-mon[107278]: osdmap e441: 8 total, 8 up, 7 in 2024-09-13T15:07:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:07:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:58.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:58 smithi003 ceph-mon[107278]: pgmap v1581: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 753 op/s; 5597/209280 objects degraded (2.674%); 26768/209280 objects misplaced (12.791%); 20 MiB/s, 378 objects/s recovering 2024-09-13T15:07:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:58 smithi003 ceph-mon[104007]: osdmap e441: 8 total, 8 up, 7 in 2024-09-13T15:07:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:07:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:58.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:58 smithi003 ceph-mon[104007]: pgmap v1581: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 4.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 753 op/s; 5597/209280 objects degraded (2.674%); 26768/209280 objects misplaced (12.791%); 20 MiB/s, 378 objects/s recovering 2024-09-13T15:07:58.946 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:58.991 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:59.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:59.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:59.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:59.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:59.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:59.404 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:59.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:59.538 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:59.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:59.676 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:59.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:59.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:59.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:07:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:59.919 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:59.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:07:59 smithi003 ceph-mon[104007]: osdmap e442: 8 total, 8 up, 7 in 2024-09-13T15:07:59.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:07:59 smithi003 ceph-mon[107278]: osdmap e442: 8 total, 8 up, 7 in 2024-09-13T15:07:59.934 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:07:59 smithi019 ceph-mon[91296]: osdmap e442: 8 total, 8 up, 7 in 2024-09-13T15:07:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.964 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:07:59.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:00.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.102 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:00.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:00.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:00.235 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:00.337 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:00.379 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:00.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:00.523 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:00.617 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:08:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:08:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:00.624 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:00.651 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:00.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:00.786 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:00.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:00.926 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:01.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:01.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:01.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:01.201 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:01.303 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:01.337 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:01.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:01.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:01.589 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:01.618 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:01 smithi019 ceph-mon[91296]: pgmap v1583: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 595 op/s; 3716/201504 objects degraded (1.844%); 24428/201504 objects misplaced (12.123%); 27 MiB/s, 513 objects/s recovering 2024-09-13T15:08:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:01 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:01 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3716/201504 objects degraded (1.844%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:01 smithi003 ceph-mon[104007]: pgmap v1583: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 595 op/s; 3716/201504 objects degraded (1.844%); 24428/201504 objects misplaced (12.123%); 27 MiB/s, 513 objects/s recovering 2024-09-13T15:08:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:01 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:01.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:01 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3716/201504 objects degraded (1.844%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:01 smithi003 ceph-mon[107278]: pgmap v1583: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 595 op/s; 3716/201504 objects degraded (1.844%); 24428/201504 objects misplaced (12.123%); 27 MiB/s, 513 objects/s recovering 2024-09-13T15:08:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:01 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:01.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:01 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3716/201504 objects degraded (1.844%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:01.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:01.752 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:01.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:01.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:01.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:01.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:02.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:02.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:02.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:02.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:02.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:02.373 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:02.384 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:02.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:02.560 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:02.662 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:02.691 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:02.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:02.824 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:02.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:02.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:03.099 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:03.131 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:03.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:03.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:03.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:03.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:03.542 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:03.572 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:03.616 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:03 smithi003 ceph-mon[107278]: pgmap v1584: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 4.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 3716/199248 objects degraded (1.865%); 23274/199248 objects misplaced (11.681%); 31 MiB/s, 565 objects/s recovering 2024-09-13T15:08:03.617 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:03 smithi003 ceph-mon[104007]: pgmap v1584: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 4.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 3716/199248 objects degraded (1.865%); 23274/199248 objects misplaced (11.681%); 31 MiB/s, 565 objects/s recovering 2024-09-13T15:08:03.673 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:03.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:03 smithi019 ceph-mon[91296]: pgmap v1584: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 4.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 3716/199248 objects degraded (1.865%); 23274/199248 objects misplaced (11.681%); 31 MiB/s, 565 objects/s recovering 2024-09-13T15:08:03.701 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:03.803 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:03.829 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:03.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:04.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:04.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:04.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:04.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:08:03 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T15:08:03.916+0000 7f0f7ea9d640 -1 mgr.server reply reply (2) No such file or directory no change 2024-09-13T15:08:04.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:08:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:08:04] "GET /metrics HTTP/1.1" 200 36554 "" "Prometheus/2.51.0" 2024-09-13T15:08:04.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:04.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:04.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:04.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:04.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.468 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:04.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:04.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:04.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:04.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:04 smithi003 ceph-mon[107278]: osdmap e443: 8 total, 8 up, 7 in 2024-09-13T15:08:04.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:04 smithi003 ceph-mon[104007]: osdmap e443: 8 total, 8 up, 7 in 2024-09-13T15:08:04.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:04 smithi019 ceph-mon[91296]: osdmap e443: 8 total, 8 up, 7 in 2024-09-13T15:08:04.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.753 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:04.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:04.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:04.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:04.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:05.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:05.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:05.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:05.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:05.315 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:05.334 INFO:teuthology.orchestra.run.smithi003.stdout:moved 12 / 357 (3.36134%) 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:avg 51 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.61168 -> 6.04743 (expected baseline 6.61168) 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 39 -> 40 pgs (0.764706 -> 0.784314 * mean) 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 61 -> 62 pgs (1.19608 -> 1.21569 * mean) 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 3 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0322 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0354 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 weight 1.0000 -> 0.9007 2024-09-13T15:08:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 weight 1.0000 -> 0.8955 2024-09-13T15:08:05.336 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T15:08:05.417 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:05.432 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:08:05.451 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:05.553 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:05.584 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:05 smithi019 ceph-mon[91296]: from='client.37344 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:05 smithi019 ceph-mon[91296]: pgmap v1586: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 3.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 3716/188910 objects degraded (1.967%); 20638/188910 objects misplaced (10.925%); 38 MiB/s, 699 objects/s recovering 2024-09-13T15:08:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:05 smithi019 ceph-mon[91296]: osdmap e444: 8 total, 8 up, 7 in 2024-09-13T15:08:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/367080246' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:05 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:05.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[104007]: from='client.37344 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:05.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[104007]: pgmap v1586: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 3.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 3716/188910 objects degraded (1.967%); 20638/188910 objects misplaced (10.925%); 38 MiB/s, 699 objects/s recovering 2024-09-13T15:08:05.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[104007]: osdmap e444: 8 total, 8 up, 7 in 2024-09-13T15:08:05.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/367080246' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:05.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:05.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[107278]: from='client.37344 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:05.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[107278]: pgmap v1586: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 3.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 3716/188910 objects degraded (1.967%); 20638/188910 objects misplaced (10.925%); 38 MiB/s, 699 objects/s recovering 2024-09-13T15:08:05.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[107278]: osdmap e444: 8 total, 8 up, 7 in 2024-09-13T15:08:05.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/367080246' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:05.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:05 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:05.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:05.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:05.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:05.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:05.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:06.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:06.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:06.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.320 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:06.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:06.422 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:06.448 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T15:08:06.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-09-13T15:08:06.457 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:06.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:06.593 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:06.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:06 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:06 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:08:06.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:06 smithi019 ceph-mon[91296]: osdmap e445: 8 total, 8 up, 7 in 2024-09-13T15:08:06.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:06 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3716/188910 objects degraded (1.967%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:06.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:06.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:08:06.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[104007]: osdmap e445: 8 total, 8 up, 7 in 2024-09-13T15:08:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3716/188910 objects degraded (1.967%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:06.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:08:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[107278]: osdmap e445: 8 total, 8 up, 7 in 2024-09-13T15:08:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3716/188910 objects degraded (1.967%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:06.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:06.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:06.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:06.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:06.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:06.977 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:07.012 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:07.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:07.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.159 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:07.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:07.260 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:07.292 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:07.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:07.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:07.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:07.572 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:07.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:07 smithi019 ceph-mon[91296]: pgmap v1589: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 3.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 699 op/s; 3716/188910 objects degraded (1.967%); 20638/188910 objects misplaced (10.925%); 24 MiB/s, 411 objects/s recovering 2024-09-13T15:08:07.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:07 smithi003 ceph-mon[104007]: pgmap v1589: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 3.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 699 op/s; 3716/188910 objects degraded (1.967%); 20638/188910 objects misplaced (10.925%); 24 MiB/s, 411 objects/s recovering 2024-09-13T15:08:07.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:07 smithi003 ceph-mon[107278]: pgmap v1589: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 109 active+clean; 3.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 699 op/s; 3716/188910 objects degraded (1.967%); 20638/188910 objects misplaced (10.925%); 24 MiB/s, 411 objects/s recovering 2024-09-13T15:08:07.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:07.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:07.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:08.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:08.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:08.249 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.290 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:08.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:08.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.432 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:08.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:08.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:08.561 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:08.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:08.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:08.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:08.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:08.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:08.863 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:08.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:08.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:08.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:08.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:09.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:09.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:09.219 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:09.242 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:09.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:08:09.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:09.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:09.506 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:09.534 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:09.635 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:09.657 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:09 smithi003 ceph-mon[104007]: pgmap v1590: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 110 active+clean; 3.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 3716/186765 objects degraded (1.990%); 20273/186765 objects misplaced (10.855%); 16 MiB/s, 290 objects/s recovering 2024-09-13T15:08:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:09 smithi003 ceph-mon[104007]: osdmap e446: 8 total, 8 up, 7 in 2024-09-13T15:08:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:09 smithi003 ceph-mon[107278]: pgmap v1590: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 110 active+clean; 3.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 3716/186765 objects degraded (1.990%); 20273/186765 objects misplaced (10.855%); 16 MiB/s, 290 objects/s recovering 2024-09-13T15:08:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:09 smithi003 ceph-mon[107278]: osdmap e446: 8 total, 8 up, 7 in 2024-09-13T15:08:09.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.792 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:09.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:09.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:09.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:09 smithi019 ceph-mon[91296]: pgmap v1590: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 110 active+clean; 3.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 3716/186765 objects degraded (1.990%); 20273/186765 objects misplaced (10.855%); 16 MiB/s, 290 objects/s recovering 2024-09-13T15:08:09.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:09 smithi019 ceph-mon[91296]: osdmap e446: 8 total, 8 up, 7 in 2024-09-13T15:08:10.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:10.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:10.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:10.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:10.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:10.473 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:08:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:08:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:10.484 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:10.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:10.604 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:08:10.604 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:08:10.604 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:08:10.604 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:08:10.604 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:08:10.604 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:08:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:08:10.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.624 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:10.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:10.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:10.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:10.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:10.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:10.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:10.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:10 smithi003 ceph-mon[104007]: osdmap e447: 8 total, 8 up, 7 in 2024-09-13T15:08:10.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4231075973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:10.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:10 smithi003 ceph-mon[107278]: osdmap e447: 8 total, 8 up, 7 in 2024-09-13T15:08:10.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4231075973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:10 smithi019 ceph-mon[91296]: osdmap e447: 8 total, 8 up, 7 in 2024-09-13T15:08:10.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4231075973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:11.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:11.033 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:11.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:11.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:11.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:11.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:11.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:11.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:11.672 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:11 smithi003 ceph-mon[107278]: pgmap v1593: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 110 active+clean; 3.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 698 op/s; 3716/176694 objects degraded (2.103%); 18005/176694 objects misplaced (10.190%); 14 MiB/s, 318 objects/s recovering 2024-09-13T15:08:11.672 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:11 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3716/176694 objects degraded (2.103%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:11.672 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:11 smithi003 ceph-mon[104007]: pgmap v1593: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 110 active+clean; 3.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 698 op/s; 3716/176694 objects degraded (2.103%); 18005/176694 objects misplaced (10.190%); 14 MiB/s, 318 objects/s recovering 2024-09-13T15:08:11.672 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:11 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3716/176694 objects degraded (2.103%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:11.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:11 smithi019 ceph-mon[91296]: pgmap v1593: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 110 active+clean; 3.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 698 op/s; 3716/176694 objects degraded (2.103%); 18005/176694 objects misplaced (10.190%); 14 MiB/s, 318 objects/s recovering 2024-09-13T15:08:11.684 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:11 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3716/176694 objects degraded (2.103%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:11.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:11.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:11.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:11.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:11.996 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:12.039 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:12.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:12.173 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:12.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:12.317 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:12.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:12.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:12.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:12.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:12.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.758 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:12.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:12.783 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:08:12.783 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T15:08:12.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T15:08:12.783 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:08:12.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.783 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127114 13.3 13:c1660045:::benchmark_data_smithi003_293363_object7902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.202037+0000", 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 324.46089408799997, 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.135729252, 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127114 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202037+0000", 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202037+0000", 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202039+0000", 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.875e-06 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202046+0000", 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4459999999999998e-06 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202060+0000", 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4323e-05 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202068+0000", 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3270000000000001e-06 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.203713+0000", 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001644413 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.203830+0000", 2024-09-13T15:08:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116867 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.203956+0000", 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012679299999999999 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244242+0000", 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040285728 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244254+0000", 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1476e-05 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.328915+0000", 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0846615079999999 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for missing object", 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329047+0000", 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013149400000000001 2024-09-13T15:08:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.337707+0000", 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0086602120000000005 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.337767+0000", 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9790000000000002e-05 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127123 13.3 13:cfeb70cd:::benchmark_data_smithi003_293363_object7911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.232129+0000", 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 324.43080236100002, 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1073596480000001, 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127123 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232129+0000", 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232129+0000", 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232131+0000", 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.699e-06 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232137+0000", 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.224e-06 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232150+0000", 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3455000000000001e-05 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232158+0000", 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6029999999999999e-06 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244280+0000", 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012121709 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244287+0000", 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8340000000000001e-06 2024-09-13T15:08:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329294+0000", 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0850076449999999 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329431+0000", 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013655399999999999 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329578+0000", 2024-09-13T15:08:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014753499999999999 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336227+0000", 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066491049999999998 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336286+0000", 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9038e-05 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336352+0000", 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5582999999999999e-05 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339347+0000", 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029945530000000001 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339394+0000", 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7123000000000003e-05 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339489+0000", 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4988000000000006e-05 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127141 13.3 13:c8a2a879:::benchmark_data_smithi003_293363_object7929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.256678+0000", 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 324.406253628, 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.082947579, 2024-09-13T15:08:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127141 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256678+0000", 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256678+0000", 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256680+0000", 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9769999999999999e-06 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256687+0000", 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2980000000000003e-06 2024-09-13T15:08:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256708+0000", 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1121999999999999e-05 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256716+0000", 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0460000000000006e-06 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256778+0000", 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2205999999999994e-05 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256787+0000", 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4100000000000008e-06 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.332032+0000", 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0752451359999999 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.332117+0000", 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4950000000000005e-05 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.332198+0000", 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0968000000000001e-05 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336368+0000", 2024-09-13T15:08:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041701589999999997 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336418+0000", 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9521999999999998e-05 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336452+0000", 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4790999999999999e-05 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339529+0000", 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030764569999999999 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339561+0000", 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2246999999999998e-05 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339625+0000", 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4289999999999996e-05 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176637 13.b 13:d907ac3f:::benchmark_data_smithi003_293363_object57425:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865081+0000", 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79784992399999, 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.097294284, 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176637 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865081+0000", 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865081+0000", 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865082+0000", 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8099999999999998e-07 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865084+0000", 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.182e-06 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865091+0000", 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5649999999999998e-06 2024-09-13T15:08:12.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865096+0000", 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6420000000000004e-06 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865251+0000", 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015508899999999999 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865258+0000", 2024-09-13T15:08:12.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2219999999999999e-06 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935706+0000", 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0704488969999999 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935808+0000", 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010106 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935897+0000", 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9606999999999998e-05 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944419+0000", 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0085219030000000008 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944552+0000", 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133387 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962201+0000", 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017648440000000001 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962260+0000", 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9416999999999997e-05 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962376+0000", 2024-09-13T15:08:12.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115492 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176642 13.b 13:dc08db49:::benchmark_data_smithi003_293363_object57430:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.797821554, 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.09749176, 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.813 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176642 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:08:12.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0200000000000003e-07 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865113+0000", 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.61e-06 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865120+0000", 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8870000000000003e-06 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865125+0000", 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4920000000000001e-06 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153487 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865285+0000", 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3799999999999999e-06 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.938324+0000", 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0730388900000001 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.938395+0000", 2024-09-13T15:08:12.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0896999999999996e-05 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.938507+0000", 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112172 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944565+0000", 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006058066 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944611+0000", 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6084999999999997e-05 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962436+0000", 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017824815000000001 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962496+0000", 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0374999999999997e-05 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962602+0000", 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105202 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176681 13.b 13:d5dc3f65:::benchmark_data_smithi003_293363_object57469:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79779880500001, 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0977554540000001, 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.820 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176681 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9999999999999998e-07 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865135+0000", 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2790000000000001e-06 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865142+0000", 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3849999999999999e-06 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865147+0000", 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2109999999999997e-06 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865307+0000", 2024-09-13T15:08:12.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000159815 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865313+0000", 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1129999999999997e-06 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.941097+0000", 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0757844999999999 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.941212+0000", 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011456 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.941347+0000", 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013543100000000001 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944571+0000", 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032240150000000002 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944667+0000", 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5983999999999994e-05 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962729+0000", 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018061628999999999 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962782+0000", 2024-09-13T15:08:12.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2652999999999999e-05 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962888+0000", 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010647899999999999 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176684 13.b 13:d6bc32f3:::benchmark_data_smithi003_293363_object57472:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79777772099999, 2024-09-13T15:08:12.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.098160848, 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176684 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:08:12.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7899999999999999e-07 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865156+0000", 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2120000000000002e-06 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865163+0000", 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2979999999999997e-06 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865168+0000", 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.938e-06 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865339+0000", 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000171719 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865345+0000", 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.122e-06 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944711+0000", 2024-09-13T15:08:12.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0793655870000001 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944815+0000", 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010432199999999999 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.945061+0000", 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024528700000000001 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955429+0000", 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010368297 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955546+0000", 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117385 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963159+0000", 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076123060000000001 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963223+0000", 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4424000000000005e-05 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963315+0000", 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1571999999999997e-05 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.834 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176690 13.b 13:d321b545:::benchmark_data_smithi003_293363_object57478:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79775754000002, 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.098341236, 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176690 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0400000000000002e-07 2024-09-13T15:08:12.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865176+0000", 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.12e-06 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865191+0000", 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4345000000000001e-05 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865196+0000", 2024-09-13T15:08:12.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6949999999999997e-06 2024-09-13T15:08:12.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865366+0000", 2024-09-13T15:08:12.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016998100000000001 2024-09-13T15:08:12.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865372+0000", 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9170000000000003e-06 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.948302+0000", 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0829294700000001 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.948432+0000", 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013043500000000001 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.948566+0000", 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013371600000000001 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955473+0000", 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069065769999999997 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962132+0000", 2024-09-13T15:08:12.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006659954 2024-09-13T15:08:12.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963377+0000", 2024-09-13T15:08:12.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012440940000000001 2024-09-13T15:08:12.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963425+0000", 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8899000000000001e-05 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963515+0000", 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9629000000000005e-05 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176701 13.b 13:d317b68b:::benchmark_data_smithi003_293363_object57489:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79772754200002, 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0984964370000001, 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176701 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0100000000000005e-07 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865207+0000", 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4959999999999999e-06 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865213+0000", 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6279999999999999e-06 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865219+0000", 2024-09-13T15:08:12.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1059999999999999e-06 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865442+0000", 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022335700000000001 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865449+0000", 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1289999999999998e-06 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.952217+0000", 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.086767781 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.952348+0000", 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130698 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.952485+0000", 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013729000000000001 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955532+0000", 2024-09-13T15:08:12.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030474899999999999 2024-09-13T15:08:12.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962140+0000", 2024-09-13T15:08:12.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066075830000000002 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963560+0000", 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014204339999999999 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963608+0000", 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7970000000000003e-05 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963700+0000", 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1973999999999997e-05 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176716 13.b 13:d4377599:::benchmark_data_smithi003_293363_object57504:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79770632399999, 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0986561319999999, 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.850 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176716 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7599999999999998e-07 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865228+0000", 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0619999999999999e-06 2024-09-13T15:08:12.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865234+0000", 2024-09-13T15:08:12.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3949999999999998e-06 2024-09-13T15:08:12.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865239+0000", 2024-09-13T15:08:12.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1699999999999996e-06 2024-09-13T15:08:12.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865472+0000", 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023298500000000001 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865478+0000", 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9479999999999996e-06 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955574+0000", 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0900960959999999 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955659+0000", 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4608e-05 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955737+0000", 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8046000000000005e-05 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962615+0000", 2024-09-13T15:08:12.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068780509999999996 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962671+0000", 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6412999999999997e-05 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963738+0000", 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010668470000000001 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963787+0000", 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8671000000000002e-05 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963881+0000", 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4463999999999995e-05 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.858 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176737 13.b 13:ddd89c0c:::benchmark_data_smithi003_293363_object57525:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865245+0000", 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79768614400001, 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.098807461, 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176737 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865245+0000", 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865245+0000", 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865246+0000", 2024-09-13T15:08:12.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6300000000000001e-07 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865259+0000", 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3437e-05 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865266+0000", 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0720000000000002e-06 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865272+0000", 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6500000000000001e-06 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865499+0000", 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022734299999999999 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865505+0000", 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1879999999999999e-06 2024-09-13T15:08:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.958889+0000", 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.093383966 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.959023+0000", 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133664 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.959159+0000", 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013578400000000001 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962904+0000", 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037453019999999998 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963100+0000", 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000195565 2024-09-13T15:08:12.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963924+0000", 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000824711 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963971+0000", 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6295000000000003e-05 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964053+0000", 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2120999999999995e-05 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176756 13.b 13:d9b260c5:::benchmark_data_smithi003_293363_object57544:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79765195599998, 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1185605869999999, 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.866 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176756 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865280+0000", 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0800000000000005e-07 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865282+0000", 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2139999999999999e-06 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865288+0000", 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.297e-06 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865293+0000", 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9450000000000001e-06 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865527+0000", 2024-09-13T15:08:12.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023310799999999999 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865532+0000", 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9569999999999999e-06 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964086+0000", 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0985532360000001 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964196+0000", 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110356 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964312+0000", 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116267 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976732+0000", 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012419902999999999 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976741+0000", 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8419999999999994e-06 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.983604+0000", 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006862511 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.983720+0000", 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116086 2024-09-13T15:08:12.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.983840+0000", 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120357 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176775 13.b 13:d5416e9e:::benchmark_data_smithi003_293363_object57563:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79763166200001, 2024-09-13T15:08:12.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1439602769999999, 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176775 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:08:12.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.77e-07 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.875 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865302+0000", 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8989999999999999e-06 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865308+0000", 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3040000000000001e-06 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865313+0000", 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0599999999999998e-06 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866806+0000", 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001492581 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866814+0000", 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5619999999999998e-06 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.967407+0000", 2024-09-13T15:08:12.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.100593173 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.967483+0000", 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6363000000000001e-05 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.967581+0000", 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7825000000000006e-05 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976749+0000", 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0091678430000000002 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976804+0000", 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5241000000000002e-05 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009025+0000", 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032221078 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009088+0000", 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2738999999999997e-05 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009260+0000", 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000172232 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176784 13.b 13:d9e88943:::benchmark_data_smithi003_293363_object57572:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.797245847, 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.143711921, 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176784 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.881 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:08:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1600000000000001e-07 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865689+0000", 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7190000000000001e-06 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865697+0000", 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7959999999999996e-06 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.883 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865703+0000", 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7750000000000004e-06 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866835+0000", 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001132128 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866842+0000", 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6200000000000001e-06 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.970297+0000", 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1034550540000001 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.970390+0000", 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3011999999999996e-05 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.970471+0000", 2024-09-13T15:08:12.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1049000000000003e-05 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976847+0000", 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063753589999999997 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976866+0000", 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.897e-05 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009300+0000", 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032434642 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009336+0000", 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6041000000000003e-05 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009397+0000", 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1240000000000003e-05 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.887 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176811 13.b 13:d3189ffb:::benchmark_data_smithi003_293363_object57599:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865711+0000", 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.79722013200001, 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1460795859999999, 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:12.888 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176811 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865711+0000", 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865711+0000", 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865712+0000", 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1100000000000001e-07 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865714+0000", 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0679999999999999e-06 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865720+0000", 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3029999999999996e-06 2024-09-13T15:08:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865725+0000", 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.989e-06 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866873+0000", 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011484360000000001 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866880+0000", 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5289999999999997e-06 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.972880+0000", 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.105999841 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.973094+0000", 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000213888 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.973209+0000", 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011561300000000001 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976918+0000", 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037089139999999998 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.000375+0000", 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023456290000000001 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.011630+0000", 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011255292 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.011734+0000", 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104122 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.011791+0000", 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6889999999999999e-05 2024-09-13T15:08:12.894 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:123386 14.1 14:880e9125:::benchmark_data_smithi003_327099_object11660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:13.271860+0000", 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 59.391070953000003, 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1641921220000002, 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:08:12.895 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123386 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271860+0000", 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271860+0000", 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271862+0000", 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.686e-06 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271869+0000", 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2949999999999996e-06 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271887+0000", 2024-09-13T15:08:12.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7829000000000002e-05 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271896+0000", 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3950000000000001e-06 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.272239+0000", 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034359100000000002 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.272371+0000", 2024-09-13T15:08:12.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013179500000000001 2024-09-13T15:08:12.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:08:12.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.272500+0000", 2024-09-13T15:08:12.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:12.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012896399999999999 2024-09-13T15:08:12.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.279137+0000", 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066367229999999998 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.291636+0000", 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012499148999999999 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.312212+0000", 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020576522 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.312229+0000", 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6231000000000001e-05 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.315906+0000", 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0036772819999999 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.315921+0000", 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5559000000000001e-05 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.435973+0000", 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1200512170000001 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.436053+0000", 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9883999999999996e-05 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.903 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:123398 14.1 14:89cc6598:::benchmark_data_smithi003_327099_object11672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:13.319706+0000", 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 59.343225285000003, 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1364959680000002, 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:08:12.904 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123398 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319706+0000", 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319706+0000", 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319708+0000", 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8670000000000001e-06 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319716+0000", 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0139999999999998e-06 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319736+0000", 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0196000000000001e-05 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319749+0000", 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2745000000000001e-05 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319869+0000", 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120412 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319882+0000", 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2543e-05 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.315954+0000", 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99607232400000001 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.315963+0000", 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9730000000000003e-06 2024-09-13T15:08:12.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.436320+0000", 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.120356753 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.436436+0000", 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011609599999999999 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:08:12.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.438424+0000", 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019879099999999999 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.450917+0000", 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012492771999999999 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.451974+0000", 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010571580000000001 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.452675+0000", 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070099099999999998 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.456061+0000", 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033863970000000002 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.456107+0000", 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5420000000000002e-05 2024-09-13T15:08:12.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.456202+0000", 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5396999999999995e-05 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:136678 14.9 14:9c1f4eb4:::benchmark_data_smithi003_327099_object24952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:33.401566+0000", 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 39.261365751, 2024-09-13T15:08:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0827195519999999, 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136678 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401566+0000", 2024-09-13T15:08:12.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401566+0000", 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401567+0000", 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6360000000000001e-06 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401573+0000", 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5879999999999997e-06 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401586+0000", 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3542e-05 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401594+0000", 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8849999999999999e-06 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401653+0000", 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8292999999999999e-05 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401761+0000", 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010793000000000001 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401885+0000", 2024-09-13T15:08:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012484400000000001 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.404245+0000", 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002359407 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.405002+0000", 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000757312 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.405090+0000", 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7656000000000002e-05 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.448173+0000", 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.043083370000000003 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.448187+0000", 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3782e-05 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:34.484215+0000", 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0360284209999999 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:34.484285+0000", 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9886000000000001e-05 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:165890 14.1 14:80f7ed4e:::benchmark_data_smithi003_327099_object54164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-13T15:08:12.919 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:08:10.462456+0000", 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.200475033, 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036014286, 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 165890 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462456+0000", 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462456+0000", 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462458+0000", 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0449999999999999e-06 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462465+0000", 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6019999999999996e-06 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462479+0000", 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3838e-05 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462488+0000", 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3419999999999997e-06 2024-09-13T15:08:12.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.468537+0000", 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060492410000000003 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.468645+0000", 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010728 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.468789+0000", 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014477500000000001 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.472393+0000", 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036035170000000001 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.486618+0000", 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014224785 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.498354+0000", 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011736411 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.498429+0000", 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4887000000000003e-05 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.498471+0000", 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1563000000000002e-05 2024-09-13T15:08:12.925 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:165895 14.9 14:95eed53f:::benchmark_data_smithi003_327099_object54169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:08:10.462645+0000", 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.2002861070000002, 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035697244000000003, 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:08:12.926 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 165895 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462645+0000", 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462645+0000", 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:08:12.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462646+0000", 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2600000000000002e-07 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462649+0000", 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2009999999999999e-06 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:08:12.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462657+0000", 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2409999999999995e-06 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.462663+0000", 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0569999999999998e-06 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.465439+0000", 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027758890000000001 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.465547+0000", 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010791299999999999 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.465651+0000", 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103967 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.471890+0000", 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062383880000000001 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.472213+0000", 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032370499999999998 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.498191+0000", 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025977916 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.498298+0000", 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106986 2024-09-13T15:08:12.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:10.498343+0000", 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4354999999999998e-05 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:08:12.932 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:08:12.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:13.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:13.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.238 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:13.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:13.339 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:13.367 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:13.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:08:13.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:13.501 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:13.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:13.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:13.670 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:13 smithi003 ceph-mon[107278]: pgmap v1594: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 110 active+clean; 3.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 712 op/s; 3716/174333 objects degraded (2.132%); 17743/174333 objects misplaced (10.178%); 12 MiB/s, 272 objects/s recovering 2024-09-13T15:08:13.670 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:13.670 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:13.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:13 smithi003 ceph-mon[104007]: pgmap v1594: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 110 active+clean; 3.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 712 op/s; 3716/174333 objects degraded (2.132%); 17743/174333 objects misplaced (10.178%); 12 MiB/s, 272 objects/s recovering 2024-09-13T15:08:13.671 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:13.671 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:13.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:13.805 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:13.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:13.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:13 smithi019 ceph-mon[91296]: pgmap v1594: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 110 active+clean; 3.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 712 op/s; 3716/174333 objects degraded (2.132%); 17743/174333 objects misplaced (10.178%); 12 MiB/s, 272 objects/s recovering 2024-09-13T15:08:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:14.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:14.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:14.188 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:14.219 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:14.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.352 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:14.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:14.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:08:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:08:14] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T15:08:14.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:14.496 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:14.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:14.638 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:14.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:14.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:14.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:14.883 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.905 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:14.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:15.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:15.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:15.156 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:15.184 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:15.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:15.423 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:15.462 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:15.564 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:15.598 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:15 smithi003 ceph-mon[107278]: pgmap v1595: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 3716/164157 objects degraded (2.264%); 16746/164157 objects misplaced (10.201%); 11 MiB/s, 316 objects/s recovering 2024-09-13T15:08:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:15 smithi003 ceph-mon[104007]: pgmap v1595: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 3716/164157 objects degraded (2.264%); 16746/164157 objects misplaced (10.201%); 11 MiB/s, 316 objects/s recovering 2024-09-13T15:08:15.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:15.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:15.782 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 357 (0%) 2024-09-13T15:08:15.782 INFO:teuthology.orchestra.run.smithi003.stdout:avg 51 2024-09-13T15:08:15.782 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.61168 -> 6.61168 (expected baseline 6.61168) 2024-09-13T15:08:15.783 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 39 -> 39 pgs (0.764706 -> 0.764706 * mean) 2024-09-13T15:08:15.783 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 61 -> 61 pgs (1.19608 -> 1.19608 * mean) 2024-09-13T15:08:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:08:15.783 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-09-13T15:08:15.783 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 0.05 2024-09-13T15:08:15.783 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T15:08:15.783 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0296 2024-09-13T15:08:15.783 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0326 2024-09-13T15:08:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T15:08:15.834 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:15.866 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:15.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:15 smithi019 ceph-mon[91296]: pgmap v1595: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 3716/164157 objects degraded (2.264%); 16746/164157 objects misplaced (10.201%); 11 MiB/s, 316 objects/s recovering 2024-09-13T15:08:15.968 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:16.001 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:16.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:16.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:16.229 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:16.252 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:16.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:16.376 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:16.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:16.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:16.606 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:16.629 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:16.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:16.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:16.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:16.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:16 smithi019 ceph-mon[91296]: from='client.57046 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:16 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 3716/164157 objects degraded (2.264%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:16 smithi019 ceph-mon[91296]: osdmap e448: 8 total, 8 up, 7 in 2024-09-13T15:08:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:16 smithi003 ceph-mon[104007]: from='client.57046 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:16 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 3716/164157 objects degraded (2.264%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:16.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:16 smithi003 ceph-mon[104007]: osdmap e448: 8 total, 8 up, 7 in 2024-09-13T15:08:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:16 smithi003 ceph-mon[107278]: from='client.57046 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:16 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 3716/164157 objects degraded (2.264%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:16.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:16 smithi003 ceph-mon[107278]: osdmap e448: 8 total, 8 up, 7 in 2024-09-13T15:08:16.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:17.014 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:17.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:17.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:17.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:17.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:17.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:17.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:17.500 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:17.538 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:17.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.681 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:17.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:17.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:17.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:17.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:17.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:17 smithi019 ceph-mon[91296]: pgmap v1597: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 934 op/s; 3716/164157 objects degraded (2.264%); 16746/164157 objects misplaced (10.201%); 9.8 MiB/s, 308 objects/s recovering 2024-09-13T15:08:17.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:17 smithi019 ceph-mon[91296]: osdmap e449: 8 total, 8 up, 7 in 2024-09-13T15:08:17.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:17 smithi003 ceph-mon[104007]: pgmap v1597: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 934 op/s; 3716/164157 objects degraded (2.264%); 16746/164157 objects misplaced (10.201%); 9.8 MiB/s, 308 objects/s recovering 2024-09-13T15:08:17.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:17 smithi003 ceph-mon[104007]: osdmap e449: 8 total, 8 up, 7 in 2024-09-13T15:08:17.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:17 smithi003 ceph-mon[107278]: pgmap v1597: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 934 op/s; 3716/164157 objects degraded (2.264%); 16746/164157 objects misplaced (10.201%); 9.8 MiB/s, 308 objects/s recovering 2024-09-13T15:08:17.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:17 smithi003 ceph-mon[107278]: osdmap e449: 8 total, 8 up, 7 in 2024-09-13T15:08:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:18.067 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:18.095 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:18.197 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.231 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:18.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:18.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:18.367 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:18.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:18.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:18.520 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:18.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:18.647 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:18.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:18.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:18.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:18.907 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:19.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:19.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:19.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:19.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:19.183 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:19.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:19.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:19.329 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:19.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:19.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:19.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.613 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:19.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:19.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:19.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:19.855 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:19.890 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:19.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:19 smithi019 ceph-mon[91296]: pgmap v1599: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 617 op/s; 3716/161868 objects degraded (2.296%); 16460/161868 objects misplaced (10.169%); 0 B/s, 84 objects/s recovering 2024-09-13T15:08:19.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:19 smithi003 ceph-mon[107278]: pgmap v1599: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 617 op/s; 3716/161868 objects degraded (2.296%); 16460/161868 objects misplaced (10.169%); 0 B/s, 84 objects/s recovering 2024-09-13T15:08:19.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:19 smithi003 ceph-mon[104007]: pgmap v1599: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 617 op/s; 3716/161868 objects degraded (2.296%); 16460/161868 objects misplaced (10.169%); 0 B/s, 84 objects/s recovering 2024-09-13T15:08:19.992 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:20.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.034 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:20.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:20.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:20.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:20.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:20.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:20.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:20.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:20.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:20.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:20.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:08:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:08:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:20.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:20.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:20.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:20.977 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:21.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:21.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:21.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:21.237 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:21.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:21.339 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:21.373 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:21.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:21.512 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:21.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:21.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:21.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:21.722 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:08:21.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:21.756 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:21 smithi019 ceph-mon[91296]: pgmap v1600: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 111 active+clean; 3.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 1828/153066 objects degraded (1.194%); 15635/153066 objects misplaced (10.215%); 2.6 MiB/s, 180 objects/s recovering 2024-09-13T15:08:21.756 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:21 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1828/153066 objects degraded (1.194%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:21.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:21.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:21.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:21 smithi003 ceph-mon[107278]: pgmap v1600: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 111 active+clean; 3.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 1828/153066 objects degraded (1.194%); 15635/153066 objects misplaced (10.215%); 2.6 MiB/s, 180 objects/s recovering 2024-09-13T15:08:21.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:21 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1828/153066 objects degraded (1.194%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:21.801 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:21 smithi003 ceph-mon[104007]: pgmap v1600: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 111 active+clean; 3.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 1828/153066 objects degraded (1.194%); 15635/153066 objects misplaced (10.215%); 2.6 MiB/s, 180 objects/s recovering 2024-09-13T15:08:21.801 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:21 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1828/153066 objects degraded (1.194%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:21.896 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:21.930 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:21.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:22.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:22.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:22.159 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:22.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:22.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:22.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:22.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:22.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:22.505 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:22.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:22.565 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T15:08:22.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-13T15:08:22.607 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:22.637 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:22.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:22.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:22.794 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:22.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:22.895 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.931 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:22.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:23.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:23.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:23.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:23.216 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:23.318 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:23.349 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:23.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:23.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:23.484 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:23.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:23.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:23.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:23 smithi003 ceph-mon[104007]: pgmap v1601: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 111 active+clean; 3.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 558 op/s; 1828/150747 objects degraded (1.213%); 15388/150747 objects misplaced (10.208%); 2.6 MiB/s, 95 objects/s recovering 2024-09-13T15:08:23.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:23 smithi003 ceph-mon[107278]: pgmap v1601: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 111 active+clean; 3.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 558 op/s; 1828/150747 objects degraded (1.213%); 15388/150747 objects misplaced (10.208%); 2.6 MiB/s, 95 objects/s recovering 2024-09-13T15:08:23.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:23.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:23.813 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:23.813 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:23.824 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:23 smithi019 ceph-mon[91296]: pgmap v1601: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 111 active+clean; 3.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 558 op/s; 1828/150747 objects degraded (1.213%); 15388/150747 objects misplaced (10.208%); 2.6 MiB/s, 95 objects/s recovering 2024-09-13T15:08:23.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:23.909 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:24.011 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:24.059 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:24.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:24.275 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:08:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:08:24] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T15:08:24.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:24.321 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:08:24.322 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:08:24.322 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T15:08:24.322 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:08:24.357 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:24.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:24.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:24.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:24.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:24.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:24.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:24.784 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:24.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:24.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:25.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:25.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:25.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:25.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:25.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:25.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:25.477 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:25.526 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:08:25.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:25.718 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:25.752 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:25.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:08:25.854 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:25.859 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:25 smithi019 ceph-mon[91296]: pgmap v1602: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 1828/138768 objects degraded (1.317%); 14210/138768 objects misplaced (10.240%); 2.5 MiB/s, 156 objects/s recovering 2024-09-13T15:08:25.859 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:25 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3790163729' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:25.859 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:25 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:25.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:25 smithi003 ceph-mon[104007]: pgmap v1602: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 1828/138768 objects degraded (1.317%); 14210/138768 objects misplaced (10.240%); 2.5 MiB/s, 156 objects/s recovering 2024-09-13T15:08:25.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:25 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3790163729' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:25.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:25 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:25.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:25 smithi003 ceph-mon[107278]: pgmap v1602: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 1828/138768 objects degraded (1.317%); 14210/138768 objects misplaced (10.240%); 2.5 MiB/s, 156 objects/s recovering 2024-09-13T15:08:25.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:25 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3790163729' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:25.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:25 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:26.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:26.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:26.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:26.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:26.253 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:26.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:26.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:26.429 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.475 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:26.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:26.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:26.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:26 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:26 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:08:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:26 smithi019 ceph-mon[91296]: osdmap e450: 8 total, 8 up, 7 in 2024-09-13T15:08:26.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:26 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1828/138768 objects degraded (1.317%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:26.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:26.747 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:26.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:26.880 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:26 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:26 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:08:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:26 smithi003 ceph-mon[104007]: osdmap e450: 8 total, 8 up, 7 in 2024-09-13T15:08:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:26 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1828/138768 objects degraded (1.317%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:26 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:26.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:26 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:08:26.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:26 smithi003 ceph-mon[107278]: osdmap e450: 8 total, 8 up, 7 in 2024-09-13T15:08:26.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:26 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1828/138768 objects degraded (1.317%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:26.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:27.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:27.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:27.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:27.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:27.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:27.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:27.430 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:27.471 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:27.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:27.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:27.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:27.727 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:27.761 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:27.863 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:27.902 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:27 smithi019 ceph-mon[91296]: pgmap v1604: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 908 op/s; 1828/138768 objects degraded (1.317%); 14210/138768 objects misplaced (10.240%); 2.2 MiB/s, 139 objects/s recovering 2024-09-13T15:08:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:27 smithi003 ceph-mon[107278]: pgmap v1604: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 908 op/s; 1828/138768 objects degraded (1.317%); 14210/138768 objects misplaced (10.240%); 2.2 MiB/s, 139 objects/s recovering 2024-09-13T15:08:27.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:27 smithi003 ceph-mon[104007]: pgmap v1604: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 908 op/s; 1828/138768 objects degraded (1.317%); 14210/138768 objects misplaced (10.240%); 2.2 MiB/s, 139 objects/s recovering 2024-09-13T15:08:28.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:28.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:28.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:28.181 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:28.237 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 357 (0%) 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout:avg 51 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.61168 -> 6.61168 (expected baseline 6.61168) 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 39 -> 39 pgs (0.764706 -> 0.764706 * mean) 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 61 -> 61 pgs (1.19608 -> 1.19608 * mean) 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 0.05 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0273 2024-09-13T15:08:28.238 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.2733 2024-09-13T15:08:28.239 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T15:08:28.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:28.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:28.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:28.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:28.577 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:28.602 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:28.621 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:08:28.621 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:08:28.622 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:08:28.622 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:08:28.622 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:08:28.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:28.735 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:28.820 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:28.821 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:28.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:28.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:28.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.025 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:29.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:29.126 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:29.160 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:29.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:29.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:29.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:29.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:08:29.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:29.451 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:29.553 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:29.575 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:29 smithi019 ceph-mon[91296]: from='client.37368 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:29.575 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:29 smithi019 ceph-mon[91296]: pgmap v1605: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 1828/135954 objects degraded (1.345%); 13907/135954 objects misplaced (10.229%); 2.1 MiB/s, 130 objects/s recovering 2024-09-13T15:08:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.598 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:29.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:29.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:29.724 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:29.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:29.844 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:29.867 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:29.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:29 smithi003 ceph-mon[107278]: from='client.37368 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:29.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:29 smithi003 ceph-mon[107278]: pgmap v1605: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 1828/135954 objects degraded (1.345%); 13907/135954 objects misplaced (10.229%); 2.1 MiB/s, 130 objects/s recovering 2024-09-13T15:08:29.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:29 smithi003 ceph-mon[104007]: from='client.37368 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:29.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:29 smithi003 ceph-mon[104007]: pgmap v1605: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 1828/135954 objects degraded (1.345%); 13907/135954 objects misplaced (10.229%); 2.1 MiB/s, 130 objects/s recovering 2024-09-13T15:08:29.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:29.991 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:30.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:30.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:30.157 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-13T15:08:30.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-13T15:08:30.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:30.239 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:30.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:30.363 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:30.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:08:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:08:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:30.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:30.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:30.591 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:30.615 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:30.636 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:08:30.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:08:30.637 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:30 smithi003 ceph-mon[107278]: osdmap e451: 8 total, 8 up, 7 in 2024-09-13T15:08:30.637 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:30 smithi003 ceph-mon[107278]: pgmap v1607: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 1828/123663 objects degraded (1.478%); 12633/123663 objects misplaced (10.216%); 0 B/s, 152 objects/s recovering 2024-09-13T15:08:30.638 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:30 smithi003 ceph-mon[104007]: osdmap e451: 8 total, 8 up, 7 in 2024-09-13T15:08:30.638 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:30 smithi003 ceph-mon[104007]: pgmap v1607: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 1828/123663 objects degraded (1.478%); 12633/123663 objects misplaced (10.216%); 0 B/s, 152 objects/s recovering 2024-09-13T15:08:30.669 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:08:30.669 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:08:30.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:30.727 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:08:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:08:30.742 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:30.763 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:08:30.764 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:08:30.792 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:08:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:08:30.824 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:08:30.824 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:08:30.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:30.857 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:08:30.857 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:08:30.887 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:08:30.887 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:08:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:30.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:30 smithi019 ceph-mon[91296]: osdmap e451: 8 total, 8 up, 7 in 2024-09-13T15:08:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:30 smithi019 ceph-mon[91296]: pgmap v1607: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 1828/123663 objects degraded (1.478%); 12633/123663 objects misplaced (10.216%); 0 B/s, 152 objects/s recovering 2024-09-13T15:08:30.990 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:31.019 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:31.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.152 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:31.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:31.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:08:31.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:31.281 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:31.382 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:31.407 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:31.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:31.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:31.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:31.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:31.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:31.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:31.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:31 smithi019 ceph-mon[91296]: osdmap e452: 8 total, 8 up, 7 in 2024-09-13T15:08:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4205728753' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:31 smithi019 ceph-mon[91296]: Health check update: Degraded data redundancy: 1828/123663 objects degraded (1.478%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:31 smithi003 ceph-mon[107278]: osdmap e452: 8 total, 8 up, 7 in 2024-09-13T15:08:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4205728753' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:31 smithi003 ceph-mon[107278]: Health check update: Degraded data redundancy: 1828/123663 objects degraded (1.478%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:31 smithi003 ceph-mon[104007]: osdmap e452: 8 total, 8 up, 7 in 2024-09-13T15:08:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4205728753' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:31 smithi003 ceph-mon[104007]: Health check update: Degraded data redundancy: 1828/123663 objects degraded (1.478%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:31.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:31.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:31.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:32.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:32.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:32.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:32.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:32.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:32.393 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:32.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:32.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:32.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:32.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:32 smithi003 ceph-mon[107278]: pgmap v1609: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 1828/121461 objects degraded (1.505%); 12365/121461 objects misplaced (10.180%); 0 B/s, 100 objects/s recovering 2024-09-13T15:08:32.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:32 smithi003 ceph-mon[104007]: pgmap v1609: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 1828/121461 objects degraded (1.505%); 12365/121461 objects misplaced (10.180%); 0 B/s, 100 objects/s recovering 2024-09-13T15:08:32.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:32.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:32.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:32 smithi019 ceph-mon[91296]: pgmap v1609: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+remapped+backfill_wait, 111 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 1828/121461 objects degraded (1.505%); 12365/121461 objects misplaced (10.180%); 0 B/s, 100 objects/s recovering 2024-09-13T15:08:32.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:33.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:33.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:33.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:33.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:33.280 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:33.382 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:33.412 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:33.514 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:33.549 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:33.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:33.683 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:33.788 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.825 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:33.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:33.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:33.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:34.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:34.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:34.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:34.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:34.348 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:34.349 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:08:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:08:34] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T15:08:34.377 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:34.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:34.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:34.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:34.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:34 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1828/121461 objects degraded (1.505%), 1 pg degraded) 2024-09-13T15:08:34.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:34 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1828/121461 objects degraded (1.505%), 1 pg degraded) 2024-09-13T15:08:34.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:34 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1828/121461 objects degraded (1.505%), 1 pg degraded) 2024-09-13T15:08:34.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:34.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:34.901 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:34.924 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:34.926 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:35.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:35.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:35.164 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:35.189 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:35.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:35.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:35.429 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:35.461 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:35.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:35.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:35 smithi019 ceph-mon[91296]: pgmap v1610: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 11434/111951 objects misplaced (10.213%); 2.3 MiB/s, 141 objects/s recovering 2024-09-13T15:08:35.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:35 smithi003 ceph-mon[104007]: pgmap v1610: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 11434/111951 objects misplaced (10.213%); 2.3 MiB/s, 141 objects/s recovering 2024-09-13T15:08:35.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:35 smithi003 ceph-mon[107278]: pgmap v1610: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 11434/111951 objects misplaced (10.213%); 2.3 MiB/s, 141 objects/s recovering 2024-09-13T15:08:35.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:35.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:35.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:35.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:35.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:36.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:36.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:36.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:36.252 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:36.265 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:08:36.265 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:08:36.288 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:36.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:36.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:36.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:36.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:36.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:36.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:36 smithi019 ceph-mon[91296]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T15:08:36.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:36 smithi003 ceph-mon[107278]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T15:08:36.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:36 smithi003 ceph-mon[104007]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T15:08:36.683 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:36.708 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:36.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:36.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:36.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:36.989 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:36.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:08:37.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:37.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:37.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:37.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:37.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:37.412 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:37 smithi019 ceph-mon[91296]: pgmap v1611: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 999 op/s; 11434/111951 objects misplaced (10.213%); 2.3 MiB/s, 141 objects/s recovering 2024-09-13T15:08:37.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.421 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:37.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:37.480 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:37.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:37.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:37.581 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:37 smithi003 ceph-mon[104007]: pgmap v1611: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 999 op/s; 11434/111951 objects misplaced (10.213%); 2.3 MiB/s, 141 objects/s recovering 2024-09-13T15:08:37.585 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:37 smithi003 ceph-mon[107278]: pgmap v1611: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 999 op/s; 11434/111951 objects misplaced (10.213%); 2.3 MiB/s, 141 objects/s recovering 2024-09-13T15:08:37.661 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:37.693 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:37.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:37.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:37.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:38.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:38.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.150 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:38.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:38.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:38.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:38.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:38.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:38.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:38.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:38.688 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:38.717 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:38.734 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:38.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:38.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:38.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:38 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:38 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:38.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:38 smithi019 ceph-mon[91296]: pgmap v1612: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 551 op/s; 11128/109308 objects misplaced (10.180%); 2.2 MiB/s, 52 objects/s recovering 2024-09-13T15:08:38.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:39.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:39.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:38 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:38 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:39.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:38 smithi003 ceph-mon[107278]: pgmap v1612: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 551 op/s; 11128/109308 objects misplaced (10.180%); 2.2 MiB/s, 52 objects/s recovering 2024-09-13T15:08:39.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:38 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:39.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:38 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:39.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:38 smithi003 ceph-mon[104007]: pgmap v1612: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 551 op/s; 11128/109308 objects misplaced (10.180%); 2.2 MiB/s, 52 objects/s recovering 2024-09-13T15:08:39.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:39.305 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:39.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:08:39.409 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:39.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:39.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:39.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:39.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:39.733 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:39.777 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:08:39.777 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:08:39.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:39.862 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:39.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:40.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:40.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:40.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:40.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:40.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:40.321 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:40.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:40.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:08:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:08:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:40.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:40.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:08:40.566 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:40.594 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:40.632 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 357 (0%) 2024-09-13T15:08:40.632 INFO:teuthology.orchestra.run.smithi003.stdout:avg 51 2024-09-13T15:08:40.632 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.61168 -> 6.61168 (expected baseline 6.61168) 2024-09-13T15:08:40.632 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 39 -> 39 pgs (0.764706 -> 0.764706 * mean) 2024-09-13T15:08:40.632 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 61 -> 61 pgs (1.19608 -> 1.19608 * mean) 2024-09-13T15:08:40.633 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:08:40.633 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-09-13T15:08:40.633 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 0.05 2024-09-13T15:08:40.633 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-09-13T15:08:40.633 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0237 2024-09-13T15:08:40.633 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.2367 2024-09-13T15:08:40.633 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-09-13T15:08:40.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:40.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:40.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:40.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:40.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:40.896 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:40 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:40 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:40.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:40 smithi003 ceph-mon[107278]: pgmap v1613: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 9992/97812 objects misplaced (10.216%); 1.9 MiB/s, 81 objects/s recovering 2024-09-13T15:08:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:40 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:40.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:40 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:40.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:40 smithi003 ceph-mon[104007]: pgmap v1613: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 9992/97812 objects misplaced (10.216%); 1.9 MiB/s, 81 objects/s recovering 2024-09-13T15:08:40.988 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.026 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:41.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:41.030 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:40 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:41.030 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:40 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:41.030 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:40 smithi019 ceph-mon[91296]: pgmap v1613: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 2.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 9992/97812 objects misplaced (10.216%); 1.9 MiB/s, 81 objects/s recovering 2024-09-13T15:08:41.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:41.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:41.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:41.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:41.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:41.443 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:41.545 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:41.569 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:41.573 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:41.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:41.818 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:41.854 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:41.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:42.021 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:41 smithi003 ceph-mon[104007]: from='client.47429 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:42.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:41 smithi003 ceph-mon[107278]: from='client.47429 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:42.080 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:41 smithi019 ceph-mon[91296]: from='client.47429 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:08:42.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:42.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:42.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:42.284 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:42.386 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:42.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:42.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:42.569 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:42.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:42.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:42.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:42.860 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:42.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:42 smithi003 ceph-mon[107278]: pgmap v1614: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 9707/95118 objects misplaced (10.205%); 1.6 MiB/s, 70 objects/s recovering 2024-09-13T15:08:42.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:42 smithi003 ceph-mon[104007]: pgmap v1614: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 9707/95118 objects misplaced (10.205%); 1.6 MiB/s, 70 objects/s recovering 2024-09-13T15:08:42.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:43.003 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:42 smithi019 ceph-mon[91296]: pgmap v1614: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 9707/95118 objects misplaced (10.205%); 1.6 MiB/s, 70 objects/s recovering 2024-09-13T15:08:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:43.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:43.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:43.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:43.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:43.283 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:43.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:43.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:43.481 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:08:43.482 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:08:43.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:43.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:43.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:43.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:43.768 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:08:43.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:43.843 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:43.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:43.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3959073988' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:08:43.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:43 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:08:43.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:43.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3959073988' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:08:43.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:43 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:08:43.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:43.982 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:44.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:44.154 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:44.154 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3959073988' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:08:44.154 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:43 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:08:44.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:08:44.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:44.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.293 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:44.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:44.347 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:08:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:08:44] "GET /metrics HTTP/1.1" 200 36546 "" "Prometheus/2.51.0" 2024-09-13T15:08:44.394 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:44.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.439 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:44.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:44.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:44.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:44.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:44.641 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:44.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:44.730 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:44.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:44.870 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:44 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:44 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:08:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:44 smithi019 ceph-mon[91296]: osdmap e453: 8 total, 8 up, 7 in 2024-09-13T15:08:44.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:44 smithi019 ceph-mon[91296]: pgmap v1616: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 8565/83427 objects misplaced (10.266%); 0 B/s, 71 objects/s recovering 2024-09-13T15:08:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:44 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:44 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:08:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:44 smithi003 ceph-mon[107278]: osdmap e453: 8 total, 8 up, 7 in 2024-09-13T15:08:44.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:44 smithi003 ceph-mon[107278]: pgmap v1616: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 8565/83427 objects misplaced (10.266%); 0 B/s, 71 objects/s recovering 2024-09-13T15:08:44.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:44 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:44.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:44 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:08:44.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:44 smithi003 ceph-mon[104007]: osdmap e453: 8 total, 8 up, 7 in 2024-09-13T15:08:44.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:44 smithi003 ceph-mon[104007]: pgmap v1616: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 8565/83427 objects misplaced (10.266%); 0 B/s, 71 objects/s recovering 2024-09-13T15:08:44.972 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:45.010 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:45.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:45.149 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:45.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:45.286 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:45.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:45.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:45.531 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:45.561 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:45.663 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:45.695 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:45.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:45.830 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:45.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:45.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:46.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:46.115 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:46.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:46.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.253 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:46.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:46.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.387 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:46.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:46.488 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:46.517 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:46.618 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:46.650 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:46.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:46.699 INFO:teuthology.orchestra.run.smithi019.stdout:ERROR: (22) Invalid argument 2024-09-13T15:08:46.700 INFO:teuthology.orchestra.run.smithi019.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:08:46.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:46.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:46.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:47.023 INFO:tasks.thrashosds.thrasher:in_osds: [2, 6, 7, 5, 0, 3, 4] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:08:47.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T15:08:47.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:47.046 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:08:47.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:08:47.092 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:08:47.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:08:47.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:47.140 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:08:47.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:08:47.185 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:08:47.185 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:08:47.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:47.223 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:08:47.224 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:08:47.243 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:47.265 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:08:47.265 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:08:47.302 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:08:47.303 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:08:47.340 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:08:47.340 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:08:47.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:47.371 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:47 smithi019 ceph-mon[91296]: pgmap v1617: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 8565/83427 objects misplaced (10.266%); 0 B/s, 71 objects/s recovering 2024-09-13T15:08:47.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1502755014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:47.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:47.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:47.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:47.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:47.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:47 smithi003 ceph-mon[104007]: pgmap v1617: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 8565/83427 objects misplaced (10.266%); 0 B/s, 71 objects/s recovering 2024-09-13T15:08:47.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1502755014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:47.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:47 smithi003 ceph-mon[107278]: pgmap v1617: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 8565/83427 objects misplaced (10.266%); 0 B/s, 71 objects/s recovering 2024-09-13T15:08:47.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1502755014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:47.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:47.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:08:47.822 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:47.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:47.962 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:48.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:08:48.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:48.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:48.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:48.250 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:48.266 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:48.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:48.386 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:48.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:48.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:48.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:48.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:48.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:48.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:48.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:49.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:49.105 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:49.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:49.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:49.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:49.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:49.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:49.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:49 smithi019 ceph-mon[91296]: pgmap v1618: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 8240/80538 objects misplaced (10.231%); 0 B/s, 71 objects/s recovering 2024-09-13T15:08:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:49 smithi003 ceph-mon[104007]: pgmap v1618: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 8240/80538 objects misplaced (10.231%); 0 B/s, 71 objects/s recovering 2024-09-13T15:08:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:49 smithi003 ceph-mon[107278]: pgmap v1618: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 8240/80538 objects misplaced (10.231%); 0 B/s, 71 objects/s recovering 2024-09-13T15:08:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:49.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:49.804 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:49.841 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:49.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:49.985 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:49.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:50.087 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:50.120 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:50.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:08:50.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:50.261 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:50.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:50.363 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:50.395 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:50.432 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:08:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:08:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:50.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:50.531 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:50.554 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:50.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:50.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:08:50.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:08:50.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:50.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:50.648 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:08:50.648 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:08:50.648 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.648 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.47273.0:202711 14.b e453/432 14:d10ff028:::benchmark_data_smithi003_327099_object90985:head v 453'12394, mlcod=453'12392)", 2024-09-13T15:08:50.648 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:50.591879+0000", 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.0041208959999999998, 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0041522829999999997, 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "started", 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.591879+0000", 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.591879+0000", 2024-09-13T15:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.591878+0000", 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.591890+0000", 2024-09-13T15:08:50.650 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2544999999999999e-05 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.591912+0000", 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1226e-05 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.591918+0000", 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7849999999999997e-06 2024-09-13T15:08:50.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.592092+0000", 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000174198 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.592110+0000", 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8315e-05 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.652 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_repop(client.47273.0:202716 14.b e453/432 14:d0ccc2e5:::benchmark_data_smithi003_327099_object90990:head v 453'12395, mlcod=453'12392)", 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:50.593268+0000", 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.0027319950000000001, 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028187170000000001, 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "started", 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.593268+0000", 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:50.653 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.593268+0000", 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.593267+0000", 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.593273+0000", 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0299999999999999e-06 2024-09-13T15:08:50.654 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.593301+0000", 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8447000000000001e-05 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.593307+0000", 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5559999999999998e-06 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.594421+0000", 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0011140779999999999 2024-09-13T15:08:50.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.594456+0000", 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5123999999999998e-05 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:202715 14.2 14:43f81cd7:::benchmark_data_smithi003_327099_object90989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:50.590928+0000", 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.0050714009999999997, 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0051948469999999998, 2024-09-13T15:08:50.656 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 202715 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.590928+0000", 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.657 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.590928+0000", 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.590927+0000", 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:50.658 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.590935+0000", 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2770000000000005e-06 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.590953+0000", 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7666999999999999e-05 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.590959+0000", 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1399999999999997e-06 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:50.659 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.591021+0000", 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2325999999999999e-05 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.591150+0000", 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012918700000000001 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.591272+0000", 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000121777 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.594318+0000", 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003046296 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:50.594744+0000", 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00042563900000000001 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-09-13T15:08:50.661 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 3 2024-09-13T15:08:50.662 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:08:50.671 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:50.773 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:50.802 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:50.904 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:08:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:08:50.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:50.934 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:08:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:08:50.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:50.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:51.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:51.069 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:51.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:51.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:51.318 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:51.354 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:51.429 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:08:51.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:51.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:51.603 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:51.633 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:51 smithi019 ceph-mon[91296]: pgmap v1619: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 962 op/s; 7046/68916 objects misplaced (10.224%); 0 B/s, 76 objects/s recovering 2024-09-13T15:08:51.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:51.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:51.837 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:51.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.917 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:51.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:51.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:51 smithi003 ceph-mon[107278]: pgmap v1619: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 962 op/s; 7046/68916 objects misplaced (10.224%); 0 B/s, 76 objects/s recovering 2024-09-13T15:08:51.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:51 smithi003 ceph-mon[104007]: pgmap v1619: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 962 op/s; 7046/68916 objects misplaced (10.224%); 0 B/s, 76 objects/s recovering 2024-09-13T15:08:52.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:52.061 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:52.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:52.188 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:52.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:52.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:52.374 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:52.476 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:52.503 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:52.598 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T15:08:52.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:52.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:52.756 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:52.876 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:52 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3558792470' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:08:52.876 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:52 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:08:52.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:52.910 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:52 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3558792470' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:08:52.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:52 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:08:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:52 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3558792470' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:08:52.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:52 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:08:53.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:53.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:53.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:53.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:53.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:53.295 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:53.397 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:53.424 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:53.525 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:53.550 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:53.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:53 smithi019 ceph-mon[91296]: pgmap v1620: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 966 op/s; 6732/66108 objects misplaced (10.183%); 0 B/s, 76 objects/s recovering 2024-09-13T15:08:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:53 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:08:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:53 smithi019 ceph-mon[91296]: osdmap e454: 8 total, 8 up, 7 in 2024-09-13T15:08:53.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:53.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:53.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:53.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:53 smithi003 ceph-mon[104007]: pgmap v1620: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 966 op/s; 6732/66108 objects misplaced (10.183%); 0 B/s, 76 objects/s recovering 2024-09-13T15:08:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:53 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:08:53.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:53 smithi003 ceph-mon[104007]: osdmap e454: 8 total, 8 up, 7 in 2024-09-13T15:08:53.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:53 smithi003 ceph-mon[107278]: pgmap v1620: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 966 op/s; 6732/66108 objects misplaced (10.183%); 0 B/s, 76 objects/s recovering 2024-09-13T15:08:53.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:53 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:08:53.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:53 smithi003 ceph-mon[107278]: osdmap e454: 8 total, 8 up, 7 in 2024-09-13T15:08:53.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:53.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:54.055 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:08:54.055 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:08:54.055 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:08:54.055 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:08:54.055 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:08:54.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:54.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:54.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:54.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:54.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:54.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:08:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:08:54] "GET /metrics HTTP/1.1" 200 36546 "" "Prometheus/2.51.0" 2024-09-13T15:08:54.478 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:54.511 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:54.613 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:54 smithi019 ceph-mon[91296]: osdmap e455: 8 total, 8 up, 7 in 2024-09-13T15:08:54.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:54 smithi019 ceph-mon[91296]: pgmap v1623: 119 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s; 5209/55605 objects misplaced (9.368%); 4.0 MiB/s, 159 objects/s recovering 2024-09-13T15:08:54.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:54.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:08:54.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:54.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:54 smithi003 ceph-mon[104007]: osdmap e455: 8 total, 8 up, 7 in 2024-09-13T15:08:54.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:54 smithi003 ceph-mon[104007]: pgmap v1623: 119 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s; 5209/55605 objects misplaced (9.368%); 4.0 MiB/s, 159 objects/s recovering 2024-09-13T15:08:54.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:54 smithi003 ceph-mon[107278]: osdmap e455: 8 total, 8 up, 7 in 2024-09-13T15:08:54.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:54 smithi003 ceph-mon[107278]: pgmap v1623: 119 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s; 5209/55605 objects misplaced (9.368%); 4.0 MiB/s, 159 objects/s recovering 2024-09-13T15:08:54.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:55.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:55.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:55.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:55.220 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:55.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:55.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:55.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:55.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:55.513 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:55.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:55.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:55.756 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:55.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:55.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:55.896 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:55 smithi019 ceph-mon[91296]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T15:08:55.921 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:55 smithi003 ceph-mon[107278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T15:08:55.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:55 smithi003 ceph-mon[104007]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T15:08:55.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:56.033 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.071 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:56.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:56.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T15:08:56.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:56.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:56.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:56.400 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:56.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:56.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:56.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:56.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:56.820 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:56.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:56.904 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:56 smithi019 ceph-mon[91296]: pgmap v1624: 119 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 5209/55605 objects misplaced (9.368%); 4.0 MiB/s, 159 objects/s recovering 2024-09-13T15:08:56.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:56 smithi003 ceph-mon[107278]: pgmap v1624: 119 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 5209/55605 objects misplaced (9.368%); 4.0 MiB/s, 159 objects/s recovering 2024-09-13T15:08:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:56 smithi003 ceph-mon[104007]: pgmap v1624: 119 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 5209/55605 objects misplaced (9.368%); 4.0 MiB/s, 159 objects/s recovering 2024-09-13T15:08:56.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:56.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:57.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:57.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:57.241 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:57.271 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:57.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:57.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:57.506 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:57.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:57.549 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:57.637 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:08:57.637 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:08:57.637 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:08:57.637 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:08:57.637 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203839 13.11 13:8c49368b:::benchmark_data_smithi003_293363_object84627:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 281.37127829399998, 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.123564244, 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203839 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.638 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261243+0000", 2024-09-13T15:08:57.639 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:08:57.640 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.640 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.640 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.640 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261250+0000", 2024-09-13T15:08:57.640 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7020000000000001e-06 2024-09-13T15:08:57.640 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.640 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.640 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.640 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261263+0000", 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2676e-05 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261269+0000", 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8490000000000002e-06 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.502730+0000", 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.241461017 2024-09-13T15:08:57.641 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.502806+0000", 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5870000000000004e-05 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.379709+0000", 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.876903561 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:08:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.379851+0000", 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014137500000000001 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.384603+0000", 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0047521660000000004 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.384809+0000", 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00020574999999999999 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.643 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203840 13.14 13:2a6c846f:::benchmark_data_smithi003_293363_object84628:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261277+0000", 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 281.37124596899997, 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.089639778, 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203840 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261277+0000", 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261277+0000", 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.645 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261276+0000", 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261279+0000", 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7089999999999998e-06 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261284+0000", 2024-09-13T15:08:57.646 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3399999999999997e-06 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261288+0000", 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.45e-06 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612519+0000", 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.35123135700000002 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.647 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612526+0000", 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3339999999999996e-06 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341783+0000", 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.72925640000000003 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341904+0000", 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120869 2024-09-13T15:08:57.648 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350844+0000", 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0089401470000000007 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350916+0000", 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2562000000000001e-05 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.649 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203847 13.11 13:8898f2cd:::benchmark_data_smithi003_293363_object84635:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261293+0000", 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 281.37122986000003, 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1261630679999999, 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203847 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.650 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261293+0000", 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261293+0000", 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261292+0000", 2024-09-13T15:08:57.651 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:08:57.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.652 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.652 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.652 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261295+0000", 2024-09-13T15:08:57.652 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.193e-06 2024-09-13T15:08:57.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.652 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.652 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.652 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261299+0000", 2024-09-13T15:08:57.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:57.653 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6909999999999996e-06 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261302+0000", 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.19e-06 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.503068+0000", 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24176531500000001 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.503987+0000", 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00091926099999999999 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.379883+0000", 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.87589635300000002 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.655 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.380025+0000", 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014113400000000001 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.387391+0000", 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0073666950000000004 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.387456+0000", 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4535000000000005e-05 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.656 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203853 13.14 13:2dc48b45:::benchmark_data_smithi003_293363_object84641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.310940+0000", 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 281.32158250999998, 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0401961900000001, 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203853 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.657 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310940+0000", 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310940+0000", 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310937+0000", 2024-09-13T15:08:57.658 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999971 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310952+0000", 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4966e-05 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310990+0000", 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.756e-05 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.311000+0000", 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0484e-05 2024-09-13T15:08:57.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612544+0000", 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30154405099999998 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612549+0000", 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4390000000000002e-06 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341929+0000", 2024-09-13T15:08:57.660 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.72938062400000003 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.342019+0000", 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9975000000000005e-05 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.342293+0000", 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00027399400000000001 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.661 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.345037+0000", 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002743255 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.347766+0000", 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027296099999999999 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350999+0000", 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0032323930000000001 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.662 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.351040+0000", 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1807999999999997e-05 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.351136+0000", 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5866000000000007e-05 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230539 13.f 13:f205da92:::benchmark_data_smithi003_293363_object111327:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:08:57.663 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521183+0000", 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 257.11133909300003, 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96228324200000004, 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230539 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521183+0000", 2024-09-13T15:08:57.664 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521183+0000", 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521182+0000", 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521192+0000", 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0701e-05 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521207+0000", 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4569000000000001e-05 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521213+0000", 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4409999999999998e-06 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.666 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561286+0000", 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040072137000000001 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561291+0000", 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9390000000000002e-06 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483382+0000", 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92209029899999995 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483467+0000", 2024-09-13T15:08:57.667 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4948000000000001e-05 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230542 13.1f 13:fe3a49e0:::benchmark_data_smithi003_293363_object111330:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 257.11130021600002, 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99456133199999996, 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:57.668 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230542 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.669 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521225+0000", 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7439999999999999e-06 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521230+0000", 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.558e-06 2024-09-13T15:08:57.670 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521233+0000", 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3809999999999999e-06 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561254+0000", 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040020864000000003 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561262+0000", 2024-09-13T15:08:57.671 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3529999999999997e-06 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515697+0000", 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95443503600000001 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515784+0000", 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6948999999999997e-05 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.672 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230548 13.f 13:f2b282bc:::benchmark_data_smithi003_293363_object111336:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521238+0000", 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 257.11128505400001, 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96231032900000002, 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230548 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.673 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521238+0000", 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521238+0000", 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521237+0000", 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.674 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521239+0000", 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9539999999999998e-06 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521244+0000", 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8770000000000001e-06 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521247+0000", 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1329999999999999e-06 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.563020+0000", 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.041772769000000001 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.563051+0000", 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1293e-05 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483490+0000", 2024-09-13T15:08:57.676 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92043902 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483548+0000", 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7589e-05 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230559 13.1f 13:feac4672:::benchmark_data_smithi003_293363_object111347:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 257.111257636, 2024-09-13T15:08:57.677 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.000411124, 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230559 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.678 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521266+0000", 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8840000000000001e-06 2024-09-13T15:08:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521271+0000", 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6990000000000002e-06 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521274+0000", 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1939999999999998e-06 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561310+0000", 2024-09-13T15:08:57.680 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040035846 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561316+0000", 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0349999999999998e-06 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515819+0000", 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95450267600000005 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.681 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515931+0000", 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000111594 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.516267+0000", 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000336163 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.519875+0000", 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003608385 2024-09-13T15:08:57.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.519881+0000", 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3010000000000003e-06 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.521593+0000", 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0017120709999999999 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.521641+0000", 2024-09-13T15:08:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7216000000000003e-05 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.521676+0000", 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5380999999999999e-05 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230561 13.1 13:826700a8:::benchmark_data_smithi003_293363_object111349:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:08:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521278+0000", 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 257.11124481500002, 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95642722800000002, 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230561 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521278+0000", 2024-09-13T15:08:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521278+0000", 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521277+0000", 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521279+0000", 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9659999999999999e-06 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521284+0000", 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5759999999999999e-06 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521287+0000", 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.96e-06 2024-09-13T15:08:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523964+0000", 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026767449999999999 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523988+0000", 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4579e-05 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460069+0000", 2024-09-13T15:08:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93608031000000003 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460199+0000", 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013029900000000001 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,4", 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460537+0000", 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00033807200000000003 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477550+0000", 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.017013495999999999 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477558+0000", 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2479999999999999e-06 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477616+0000", 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7852000000000001e-05 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477669+0000", 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3695e-05 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477705+0000", 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5771e-05 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:123397 14.2 14:450a99e6:::benchmark_data_smithi003_327099_object11671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:13.318784+0000", 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 104.31373848299999, 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1419773559999999, 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123397 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318784+0000", 2024-09-13T15:08:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318784+0000", 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318783+0000", 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318789+0000", 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7660000000000001e-06 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318802+0000", 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2238e-05 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318807+0000", 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.366e-06 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318865+0000", 2024-09-13T15:08:57.696 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:57 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3574876533' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:57.696 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:57 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3574876533' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:57.696 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7877000000000002e-05 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318876+0000", 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1472e-05 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.283892+0000", 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96501579199999998 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.697 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.283906+0000", 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.428e-05 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.438947+0000", 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1550406930000001 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.439080+0000", 2024-09-13T15:08:57.698 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013328999999999999 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.439655+0000", 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00057495500000000004 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.451726+0000", 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012070292999999999 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.454406+0000", 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026805399999999999 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.456100+0000", 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0016936970000000001 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460596+0000", 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.004495995 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.700 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460694+0000", 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7740999999999998e-05 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460761+0000", 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7727999999999999e-05 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.701 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:123404 14.2 14:476c8c71:::benchmark_data_smithi003_327099_object11678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:13.351072+0000", 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 104.28145016800001, 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1098517659999998, 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123404 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.702 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351072+0000", 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351072+0000", 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351071+0000", 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:08:57.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351079+0000", 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1149999999999994e-06 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351091+0000", 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1989e-05 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.704 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351096+0000", 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4099999999999999e-06 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351159+0000", 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2504999999999999e-05 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351169+0000", 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0475999999999999e-05 2024-09-13T15:08:57.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.283932+0000", 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93276217100000003 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.283941+0000", 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6940000000000007e-06 2024-09-13T15:08:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.447343+0000", 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.163402015 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.447471+0000", 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012805600000000001 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.447606+0000", 2024-09-13T15:08:57.707 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013456399999999999 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.454100+0000", 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.006493993 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.454434+0000", 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00033445099999999999 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.708 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.456132+0000", 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001697282 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460787+0000", 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0046557869999999998 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460815+0000", 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7246e-05 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.709 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460924+0000", 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010948899999999999 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:136685 14.d 14:b60d3feb:::benchmark_data_smithi003_327099_object24959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:33.424329+0000", 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 84.208193456999993, 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1967100070000001, 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:08:57.710 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 136685 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424329+0000", 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.711 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424329+0000", 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424328+0000", 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424336+0000", 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4840000000000004e-06 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.712 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424350+0000", 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4416999999999999e-05 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424356+0000", 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6749999999999999e-06 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.433661+0000", 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0093052819999999998 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.713 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.433790+0000", 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012886899999999999 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.433958+0000", 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000167768 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.462602+0000", 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.028644284999999998 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.714 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.462611+0000", 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1900000000000001e-06 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.471863+0000", 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0092517629999999997 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.471873+0000", 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8719999999999997e-06 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.715 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.610659+0000", 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.138785876 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.610801+0000", 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014245999999999999 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.620973+0000", 2024-09-13T15:08:57.716 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010171556 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.621039+0000", 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6094999999999997e-05 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:136694 14.d 14:b9a67a18:::benchmark_data_smithi003_327099_object24968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:33.597900+0000", 2024-09-13T15:08:57.717 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 84.034622541000005, 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.022468412, 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 136694 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597900+0000", 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.718 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597900+0000", 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597899+0000", 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597906+0000", 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.046e-06 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.719 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597919+0000", 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3482999999999999e-05 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597926+0000", 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5289999999999997e-06 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597980+0000", 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4045000000000002e-05 2024-09-13T15:08:57.720 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597988+0000", 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0819999999999999e-06 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.610835+0000", 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.012847584 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.611001+0000", 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000165401 2024-09-13T15:08:57.721 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.611107+0000", 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106289 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.614125+0000", 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003018195 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.614161+0000", 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6185e-05 2024-09-13T15:08:57.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.619810+0000", 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0056483030000000004 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.619849+0000", 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8782999999999998e-05 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.620368+0000", 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00051989600000000005 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.723 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183555 14.d 14:b5c49246:::benchmark_data_smithi003_327099_object71829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.585663+0000", 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 28.046859788999999, 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.061434201, 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183555 2024-09-13T15:08:57.724 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585663+0000", 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585663+0000", 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.725 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585661+0000", 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585668+0000", 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6279999999999999e-06 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585680+0000", 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2451e-05 2024-09-13T15:08:57.726 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585687+0000", 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7159999999999997e-06 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.590533+0000", 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.004846342 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.590542+0000", 2024-09-13T15:08:57.727 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2770000000000005e-06 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.609871+0000", 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.019329073 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.609991+0000", 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120322 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.728 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.610071+0000", 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9628000000000004e-05 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621753+0000", 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011681904 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.646936+0000", 2024-09-13T15:08:57.729 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.025183245 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647004+0000", 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8289999999999998e-05 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647058+0000", 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3816000000000001e-05 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.730 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647097+0000", 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9104999999999999e-05 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183556 14.d 14:b3561416:::benchmark_data_smithi003_327099_object71830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.587770+0000", 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 28.044752304999999, 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.059504521, 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.731 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183556 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587770+0000", 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.732 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587770+0000", 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587769+0000", 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587778+0000", 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9849999999999995e-06 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.733 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587806+0000", 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8422999999999999e-05 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587814+0000", 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4819999999999997e-06 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.590563+0000", 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027496199999999999 2024-09-13T15:08:57.734 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.590569+0000", 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.699e-06 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.612536+0000", 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0219669389999999 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.735 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.612636+0000", 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.9932000000000005e-05 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.612714+0000", 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8164999999999994e-05 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621757+0000", 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0090431339999999995 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.736 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647147+0000", 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.025390341 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647190+0000", 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2422999999999997e-05 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.737 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647237+0000", 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7064999999999997e-05 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647275+0000", 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7935000000000002e-05 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183564 14.d 14:baefa26f:::benchmark_data_smithi003_327099_object71838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.593574+0000", 2024-09-13T15:08:57.738 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 28.03894816, 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0538798110000001, 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183564 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593574+0000", 2024-09-13T15:08:57.739 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593574+0000", 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593574+0000", 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.740 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593580+0000", 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8769999999999998e-06 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593594+0000", 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4511e-05 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593600+0000", 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1229999999999997e-06 2024-09-13T15:08:57.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.594261+0000", 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00066062200000000003 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.594269+0000", 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4109999999999996e-06 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.742 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.615210+0000", 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.02094094 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.615293+0000", 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3270999999999996e-05 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.615370+0000", 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6464999999999994e-05 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.743 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.626464+0000", 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011094405999999999 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647299+0000", 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.020834386999999999 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647399+0000", 2024-09-13T15:08:57.744 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000100677 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647429+0000", 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0159e-05 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647454+0000", 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4732999999999999e-05 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.745 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183565 14.d 14:bc68576d:::benchmark_data_smithi003_327099_object71839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.593626+0000", 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 28.038896215000001, 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.053956822, 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183565 2024-09-13T15:08:57.746 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593626+0000", 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593626+0000", 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.747 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593626+0000", 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593630+0000", 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8120000000000001e-06 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593637+0000", 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8029999999999997e-06 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.748 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593642+0000", 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2150000000000003e-06 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.594288+0000", 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000646669 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.749 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.594294+0000", 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3040000000000004e-06 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.617719+0000", 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0234252319999999 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.750 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.617800+0000", 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1549000000000001e-05 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.617875+0000", 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4802000000000007e-05 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.626471+0000", 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0085958059999999992 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.751 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647501+0000", 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.021029835 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647528+0000", 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7654e-05 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.752 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647557+0000", 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8319000000000001e-05 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647583+0000", 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.641e-05 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.753 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183571 14.d 14:b36892d7:::benchmark_data_smithi003_327099_object71845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:08:57.755 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.598281+0000", 2024-09-13T15:08:57.755 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 28.034241608999999, 2024-09-13T15:08:57.755 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0494433430000001, 2024-09-13T15:08:57.755 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.755 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.755 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.755 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.755 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183571 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598281+0000", 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598281+0000", 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598280+0000", 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598285+0000", 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0209999999999999e-06 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.757 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598296+0000", 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0852999999999999e-05 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598302+0000", 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9370000000000001e-06 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598384+0000", 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1879000000000007e-05 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.758 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598392+0000", 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5199999999999997e-06 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.619787+0000", 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.021394374 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.619860+0000", 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3506999999999999e-05 2024-09-13T15:08:57.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.619962+0000", 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101642 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.630766+0000", 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010804309 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.760 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647628+0000", 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016861830000000001 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647654+0000", 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5999999999999998e-05 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.761 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647700+0000", 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5797000000000002e-05 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647724+0000", 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4471e-05 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183614 14.d 14:b95de5de:::benchmark_data_smithi003_327099_object71888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.659145+0000", 2024-09-13T15:08:57.762 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 27.973377329000002, 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98874907499999998, 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183614 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659145+0000", 2024-09-13T15:08:57.763 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659145+0000", 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659144+0000", 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.764 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659151+0000", 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9630000000000001e-06 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659166+0000", 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4878000000000001e-05 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659172+0000", 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.852e-06 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.765 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659230+0000", 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8239000000000003e-05 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659243+0000", 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2908000000000001e-05 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621788+0000", 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.962545659 2024-09-13T15:08:57.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621858+0000", 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9383999999999998e-05 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621956+0000", 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7905999999999994e-05 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.630772+0000", 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0088158490000000006 2024-09-13T15:08:57.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647765+0000", 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016993726000000001 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647842+0000", 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6320000000000001e-05 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647864+0000", 2024-09-13T15:08:57.768 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2785e-05 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647894+0000", 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9887999999999999e-05 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183619 14.d 14:bd455a57:::benchmark_data_smithi003_327099_object71893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:08:57.769 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.663599+0000", 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 27.968923452999999, 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98442653800000002, 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183619 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:08:57.770 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663599+0000", 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663599+0000", 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663597+0000", 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.771 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663605+0000", 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6720000000000004e-06 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663621+0000", 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6157000000000001e-05 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663629+0000", 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0849999999999997e-06 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663689+0000", 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9589000000000002e-05 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663700+0000", 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1355000000000001e-05 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.623787+0000", 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.960086471 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.623907+0000", 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012057600000000001 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.624030+0000", 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012242700000000001 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.632872+0000", 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0088422350000000004 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.774 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647794+0000", 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014921719 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647944+0000", 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015033399999999999 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647988+0000", 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4246999999999998e-05 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:08:57.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:08:57.776 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:08:57.776 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.648026+0000", 2024-09-13T15:08:57.776 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7292000000000001e-05 2024-09-13T15:08:57.776 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.776 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.776 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.776 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:08:57.776 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:08:57.776 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:08:57.777 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:57 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3574876533' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:57.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:57.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:57.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:57.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:58.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:58.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:58.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:58.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:58.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:58.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:58.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:58.485 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:58.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:58.623 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:58.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:58.757 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:58.859 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:58.893 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:58 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:08:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:58 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:08:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:58 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3574876533' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:08:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:58 smithi019 ceph-mon[91296]: osdmap e456: 8 total, 8 up, 7 in 2024-09-13T15:08:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:58.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:08:58 smithi019 ceph-mon[91296]: pgmap v1626: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 4232/53319 objects misplaced (7.937%); 14 MiB/s, 283 objects/s recovering 2024-09-13T15:08:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:08:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:08:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3574876533' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:08:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[104007]: osdmap e456: 8 total, 8 up, 7 in 2024-09-13T15:08:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:58.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[104007]: pgmap v1626: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 4232/53319 objects misplaced (7.937%); 14 MiB/s, 283 objects/s recovering 2024-09-13T15:08:58.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:08:58.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:08:58.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3574876533' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:08:58.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[107278]: osdmap e456: 8 total, 8 up, 7 in 2024-09-13T15:08:58.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:08:58.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:58.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:08:58 smithi003 ceph-mon[107278]: pgmap v1626: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 4232/53319 objects misplaced (7.937%); 14 MiB/s, 283 objects/s recovering 2024-09-13T15:08:58.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:08:58.996 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:59.054 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:59.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:59.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:59.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:59.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:59.345 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:59.428 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:08:59.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:59.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:59.571 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:59.604 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:59.706 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:59.744 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:08:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:59.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:08:59.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:59.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:00.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:00.136 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:00.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.261 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:00.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:00.362 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:00.385 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:00.487 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:00.509 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:00.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:00.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:00.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:00.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:09:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:09:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:09:00.737 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:00.759 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:00.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:00.883 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:00.964 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:00 smithi019 ceph-mon[91296]: osdmap e457: 8 total, 8 up, 7 in 2024-09-13T15:09:00.964 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:00 smithi019 ceph-mon[91296]: 1.e deep-scrub starts 2024-09-13T15:09:00.964 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:00 smithi019 ceph-mon[91296]: 1.e deep-scrub ok 2024-09-13T15:09:00.964 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:00 smithi019 ceph-mon[91296]: pgmap v1628: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 866 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s; 2807/41613 objects misplaced (6.745%); 17 MiB/s, 322 objects/s recovering 2024-09-13T15:09:00.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:01.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:01.123 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:01.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:00 smithi003 ceph-mon[104007]: osdmap e457: 8 total, 8 up, 7 in 2024-09-13T15:09:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:00 smithi003 ceph-mon[104007]: 1.e deep-scrub starts 2024-09-13T15:09:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:00 smithi003 ceph-mon[104007]: 1.e deep-scrub ok 2024-09-13T15:09:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:00 smithi003 ceph-mon[104007]: pgmap v1628: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 866 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s; 2807/41613 objects misplaced (6.745%); 17 MiB/s, 322 objects/s recovering 2024-09-13T15:09:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:00 smithi003 ceph-mon[107278]: osdmap e457: 8 total, 8 up, 7 in 2024-09-13T15:09:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:00 smithi003 ceph-mon[107278]: 1.e deep-scrub starts 2024-09-13T15:09:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:00 smithi003 ceph-mon[107278]: 1.e deep-scrub ok 2024-09-13T15:09:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:00 smithi003 ceph-mon[107278]: pgmap v1628: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 866 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s; 2807/41613 objects misplaced (6.745%); 17 MiB/s, 322 objects/s recovering 2024-09-13T15:09:01.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:01.273 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:211338 14.f 14:f57f5426:::benchmark_data_smithi003_327099_object99612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:01.268585+0000", 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.0031408830000000001, 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003173276, 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211338 2024-09-13T15:09:01.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.268585+0000", 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.268585+0000", 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:01.275 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.268584+0000", 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.268590+0000", 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3910000000000003e-06 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.268605+0000", 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4769e-05 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.276 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.268611+0000", 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9359999999999996e-06 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.268813+0000", 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00020197300000000001 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.268923+0000", 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000109936 2024-09-13T15:09:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.269025+0000", 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101881 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 1 2024-09-13T15:09:01.278 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:09:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:01.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:01.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:01.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:01.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:01.705 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:01.733 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:01.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:01.867 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:01.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:01 smithi019 ceph-mon[91296]: osdmap e458: 8 total, 8 up, 7 in 2024-09-13T15:09:01.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:01 smithi003 ceph-mon[104007]: osdmap e458: 8 total, 8 up, 7 in 2024-09-13T15:09:01.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:01 smithi003 ceph-mon[107278]: osdmap e458: 8 total, 8 up, 7 in 2024-09-13T15:09:01.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:02.001 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:02.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:09:02.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:02.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:02.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:02.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:02.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:02.416 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:09:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:02.540 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:02.578 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:02.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:02.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:02.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:02.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:02.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:02.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:02.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:03.095 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:03.122 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:03.157 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:02 smithi003 ceph-mon[107278]: pgmap v1630: 119 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 115 active+clean; 817 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 906 op/s; 2544/39282 objects misplaced (6.476%); 16 MiB/s, 253 objects/s recovering 2024-09-13T15:09:03.158 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:02 smithi003 ceph-mon[104007]: pgmap v1630: 119 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 115 active+clean; 817 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 906 op/s; 2544/39282 objects misplaced (6.476%); 16 MiB/s, 253 objects/s recovering 2024-09-13T15:09:03.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:02 smithi019 ceph-mon[91296]: pgmap v1630: 119 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 115 active+clean; 817 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 906 op/s; 2544/39282 objects misplaced (6.476%); 16 MiB/s, 253 objects/s recovering 2024-09-13T15:09:03.244 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:03.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.270 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:03.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:03.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:03.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:03.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:03.521 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:03.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:03.646 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:03.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:03.773 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:03.875 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:03.900 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:04.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:04.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:04.099 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:03 smithi019 ceph-mon[91296]: osdmap e459: 8 total, 8 up, 7 in 2024-09-13T15:09:04.135 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:04.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:04.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:03 smithi003 ceph-mon[107278]: osdmap e459: 8 total, 8 up, 7 in 2024-09-13T15:09:04.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:09:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:09:04] "GET /metrics HTTP/1.1" 200 36546 "" "Prometheus/2.51.0" 2024-09-13T15:09:04.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:03 smithi003 ceph-mon[104007]: osdmap e459: 8 total, 8 up, 7 in 2024-09-13T15:09:04.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:04.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:04.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:04.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:04.543 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:04.574 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:04.676 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:04.711 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:04.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:04.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:04.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:04.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:05.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:05.092 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:04 smithi019 ceph-mon[91296]: osdmap e460: 8 total, 8 up, 7 in 2024-09-13T15:09:05.092 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:04 smithi019 ceph-mon[91296]: pgmap v1633: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 661 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 830/31761 objects misplaced (2.613%); 21 MiB/s, 335 objects/s recovering 2024-09-13T15:09:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:05.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:04 smithi003 ceph-mon[107278]: osdmap e460: 8 total, 8 up, 7 in 2024-09-13T15:09:05.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:04 smithi003 ceph-mon[107278]: pgmap v1633: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 661 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 830/31761 objects misplaced (2.613%); 21 MiB/s, 335 objects/s recovering 2024-09-13T15:09:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:04 smithi003 ceph-mon[104007]: osdmap e460: 8 total, 8 up, 7 in 2024-09-13T15:09:05.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:04 smithi003 ceph-mon[104007]: pgmap v1633: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 661 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 830/31761 objects misplaced (2.613%); 21 MiB/s, 335 objects/s recovering 2024-09-13T15:09:05.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:05.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:05.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:05.411 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:05.417 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:09:05.418 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:09:05.418 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:09:05.418 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:09:05.418 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:09:05.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:05.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:05.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:05.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:05.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:05.819 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:05.920 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:05.952 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:06.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:06.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:06.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:06.209 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:06.311 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:06.335 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:06.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:06.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:06.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:06.580 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:09:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.586 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:06.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:06.687 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:06.711 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:06.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:06.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:06.836 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:06.856 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:09:06.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:06.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:07.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:07.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:07.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:07.151 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:06 smithi019 ceph-mon[91296]: osdmap e461: 8 total, 8 up, 7 in 2024-09-13T15:09:07.151 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:06 smithi019 ceph-mon[91296]: pgmap v1635: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 661 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 605 op/s; 830/31761 objects misplaced (2.613%); 17 MiB/s, 274 objects/s recovering 2024-09-13T15:09:07.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:06 smithi003 ceph-mon[107278]: osdmap e461: 8 total, 8 up, 7 in 2024-09-13T15:09:07.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:06 smithi003 ceph-mon[107278]: pgmap v1635: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 661 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 605 op/s; 830/31761 objects misplaced (2.613%); 17 MiB/s, 274 objects/s recovering 2024-09-13T15:09:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:06 smithi003 ceph-mon[104007]: osdmap e461: 8 total, 8 up, 7 in 2024-09-13T15:09:07.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:06 smithi003 ceph-mon[104007]: pgmap v1635: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 661 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 605 op/s; 830/31761 objects misplaced (2.613%); 17 MiB/s, 274 objects/s recovering 2024-09-13T15:09:07.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:07.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:07.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:09:07.330 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:07.352 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:07.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:07.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:07.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:07.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:07.606 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:07.708 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:07.730 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:07.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:07.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:07.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:07.986 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:07 smithi019 ceph-mon[91296]: osdmap e462: 8 total, 8 up, 7 in 2024-09-13T15:09:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.990 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:07.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:08.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:08.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:08.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:07 smithi003 ceph-mon[107278]: osdmap e462: 8 total, 8 up, 7 in 2024-09-13T15:09:08.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:07 smithi003 ceph-mon[104007]: osdmap e462: 8 total, 8 up, 7 in 2024-09-13T15:09:08.231 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:08.270 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:08.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.417 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:08.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:08.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:08.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:08.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.699 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:08.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:08.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.834 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:08.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203841 13.17 13:ead20358:::benchmark_data_smithi003_293363_object84629:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261212+0000", 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 292.58051506999999, 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0753570539999999, 2024-09-13T15:09:08.846 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203841 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261212+0000", 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261212+0000", 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261210+0000", 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261217+0000", 2024-09-13T15:09:08.848 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8379999999999996e-06 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261229+0000", 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2378000000000001e-05 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261235+0000", 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9619999999999998e-06 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.849 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.439855+0000", 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.17862001 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.439884+0000", 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8668999999999999e-05 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.336495+0000", 2024-09-13T15:09:08.850 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.89661158600000002 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.336569+0000", 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3200000000000004e-05 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203845 13.10 13:0e2b4302:::benchmark_data_smithi003_293363_object84633:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261245+0000", 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 292.58048201999998, 2024-09-13T15:09:08.851 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1075082359999999, 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203845 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261245+0000", 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.852 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261245+0000", 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261247+0000", 2024-09-13T15:09:08.853 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.864e-06 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261254+0000", 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5459999999999997e-06 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261258+0000", 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.583e-06 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.854 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.473959+0000", 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21270049199999999 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.473967+0000", 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0390000000000005e-06 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.362740+0000", 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.88877301900000005 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.362967+0000", 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00022709499999999999 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.368452+0000", 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0054853269999999999 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.368753+0000", 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00030071399999999999 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.856 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:123396 14.f 14:f9a8006d:::benchmark_data_smithi003_327099_object11670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:13.317826+0000", 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 115.523900283, 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1804893710000002, 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.857 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123396 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317826+0000", 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.858 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317826+0000", 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317825+0000", 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317833+0000", 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7940000000000003e-06 2024-09-13T15:09:08.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317848+0000", 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5906e-05 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317855+0000", 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6599999999999998e-06 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.860 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317910+0000", 2024-09-13T15:09:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4688000000000002e-05 2024-09-13T15:09:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:09:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317918+0000", 2024-09-13T15:09:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2919999999999996e-06 2024-09-13T15:09:08.861 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.289086+0000", 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97116769599999997 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.289095+0000", 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1589999999999999e-06 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.862 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.469696+0000", 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.180600927 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.469835+0000", 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013882400000000001 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.470382+0000", 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00054688499999999999 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.863 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.480759+0000", 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010377001 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.480816+0000", 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7231000000000001e-05 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498032+0000", 2024-09-13T15:09:08.864 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.017215985999999999 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498090+0000", 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8216e-05 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498185+0000", 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5067000000000003e-05 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.865 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498316+0000", 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000130739 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:123403 14.f 14:fe42c978:::benchmark_data_smithi003_327099_object11677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:13.348752+0000", 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 115.492975046, 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1498335150000001, 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.866 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123403 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348752+0000", 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348752+0000", 2024-09-13T15:09:08.867 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348750+0000", 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348759+0000", 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1169999999999993e-06 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.868 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348776+0000", 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6937000000000001e-05 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348783+0000", 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3660000000000003e-06 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348843+0000", 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9824000000000002e-05 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:09:08.869 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348856+0000", 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3246e-05 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.289114+0000", 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.940257432 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.289119+0000", 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5369999999999997e-06 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.477574+0000", 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1884548669999999 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.477696+0000", 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012206600000000001 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.477839+0000", 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014236000000000001 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.871 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.480767+0000", 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002928047 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.497986+0000", 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.017219464 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498061+0000", 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5010000000000002e-05 2024-09-13T15:09:08.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498359+0000", 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00029802600000000001 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498456+0000", 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6972000000000006e-05 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498585+0000", 2024-09-13T15:09:08.873 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012900699999999999 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:175106 14.f 14:ff47b7c0:::benchmark_data_smithi003_327099_object63380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:21.106063+0000", 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 47.735664161999999, 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.18853082299999999, 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.874 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 175106 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106063+0000", 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106063+0000", 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106061+0000", 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106070+0000", 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.0299999999999999e-06 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106089+0000", 2024-09-13T15:09:08.876 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8567999999999999e-05 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106095+0000", 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.754e-06 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.107370+0000", 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0012743699999999999 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.877 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.107456+0000", 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6028999999999999e-05 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.107519+0000", 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2656000000000005e-05 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.113190+0000", 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.005671751 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.878 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.113215+0000", 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4484000000000001e-05 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.294459+0000", 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.18124454800000001 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.294551+0000", 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1846000000000001e-05 2024-09-13T15:09:08.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.294593+0000", 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2222999999999999e-05 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:175109 14.f 14:f0e99ae3:::benchmark_data_smithi003_327099_object63383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:21.106106+0000", 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 47.735620292999997, 2024-09-13T15:09:08.880 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19652803599999999, 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 175109 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106106+0000", 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106106+0000", 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106106+0000", 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.882 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106109+0000", 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6129999999999998e-06 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106115+0000", 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2759999999999997e-06 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106119+0000", 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0130000000000004e-06 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.883 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.110035+0000", 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039161839999999996 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.110172+0000", 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013690999999999999 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.110317+0000", 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000144711 2024-09-13T15:09:08.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.113211+0000", 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028946039999999998 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.113219+0000", 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2599999999999999e-06 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.885 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302526+0000", 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.189307743 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302589+0000", 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3149000000000001e-05 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302634+0000", 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5015999999999997e-05 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:175118 14.f 14:f9330baa:::benchmark_data_smithi003_327099_object63392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:21.171494+0000", 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 47.670232382999998, 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13128300100000001, 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 175118 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.887 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171494+0000", 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171494+0000", 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171492+0000", 2024-09-13T15:09:08.888 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:09:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.889 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171501+0000", 2024-09-13T15:09:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7029999999999995e-06 2024-09-13T15:09:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171521+0000", 2024-09-13T15:09:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9568e-05 2024-09-13T15:09:08.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171529+0000", 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1429999999999998e-06 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171599+0000", 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0655999999999996e-05 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.891 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171725+0000", 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012510399999999999 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171855+0000", 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000130667 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.174724+0000", 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028687780000000002 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.175303+0000", 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000578581 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302688+0000", 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.12738560099999999 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302739+0000", 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1057000000000003e-05 2024-09-13T15:09:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302777+0000", 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8124000000000003e-05 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:200935 14.f 14:f146a2d5:::benchmark_data_smithi003_327099_object89209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:48.867912+0000", 2024-09-13T15:09:08.894 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 19.973814298000001, 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.028222969000000001, 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 200935 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.867912+0000", 2024-09-13T15:09:08.895 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.867912+0000", 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.867911+0000", 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.896 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.867919+0000", 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.746e-06 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.867931+0000", 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.207e-05 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.867936+0000", 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2919999999999997e-06 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.869497+0000", 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001561366 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.869646+0000", 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014942699999999999 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.869756+0000", 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010991400000000001 2024-09-13T15:09:08.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.873162+0000", 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0034052430000000001 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.882140+0000", 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0089783689999999999 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.895796+0000", 2024-09-13T15:09:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013655660999999999 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.896010+0000", 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00021456900000000001 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:48.896135+0000", 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012525000000000001 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:203530 14.f 14:f91bb8a8:::benchmark_data_smithi003_327099_object91804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:51.423161+0000", 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 17.418566145, 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.029891602, 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203530 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.901 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423161+0000", 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423161+0000", 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423159+0000", 2024-09-13T15:09:08.902 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423166+0000", 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0169999999999999e-06 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423179+0000", 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3342e-05 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.903 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423188+0000", 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8049999999999996e-06 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.424206+0000", 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0010175550000000001 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.424404+0000", 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000198242 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.904 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.424549+0000", 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014546700000000001 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.432724+0000", 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0081748229999999995 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.432767+0000", 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2889999999999998e-05 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.905 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.452358+0000", 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.019591117000000002 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.452489+0000", 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013128099999999999 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.453052+0000", 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000562833 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:203536 14.f 14:fc530f54:::benchmark_data_smithi003_327099_object91810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:51.425626+0000", 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 17.416100868000001, 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.027845624999999999, 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203536 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.907 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.425626+0000", 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.425626+0000", 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.425625+0000", 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.908 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.425630+0000", 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7910000000000001e-06 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.425647+0000", 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.717e-05 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.425654+0000", 2024-09-13T15:09:08.909 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2940000000000002e-06 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.428921+0000", 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003267205 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.429068+0000", 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014717899999999999 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.429203+0000", 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013448299999999999 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.441683+0000", 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012480728 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.441832+0000", 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000148577 2024-09-13T15:09:08.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.453211+0000", 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011379501 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.453349+0000", 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000137832 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.912 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.453471+0000", 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012220300000000001 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:203756 14.f 14:f777587a:::benchmark_data_smithi003_327099_object92030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:51.672466+0000", 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 17.169260428000001, 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.03354791, 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.913 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203756 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672466+0000", 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672466+0000", 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672465+0000", 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672472+0000", 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6089999999999997e-06 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672497+0000", 2024-09-13T15:09:08.915 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5602999999999999e-05 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672506+0000", 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8659999999999995e-06 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.676715+0000", 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0042086850000000002 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.916 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.677072+0000", 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00035733 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.677418+0000", 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00034529599999999998 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.683082+0000", 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0056647870000000001 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.917 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.695913+0000", 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012830411 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.705840+0000", 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0099273199999999999 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.705977+0000", 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013695800000000001 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.706014+0000", 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7147999999999997e-05 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:205299 14.f 14:fc7adaae:::benchmark_data_smithi003_327099_object93573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:53.944365+0000", 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 14.897361496, 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.031709770999999998, 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.919 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 205299 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944365+0000", 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.920 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944365+0000", 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944364+0000", 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944370+0000", 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4860000000000002e-06 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944388+0000", 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7723999999999998e-05 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944394+0000", 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4160000000000001e-06 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.945950+0000", 2024-09-13T15:09:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001555489 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.946074+0000", 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012463900000000001 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.946197+0000", 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012310700000000001 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.960585+0000", 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014387539 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.960722+0000", 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000137007 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.975981+0000", 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015258595 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976038+0000", 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7379000000000001e-05 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976075+0000", 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6999999999999998e-05 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.925 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:205303 14.f 14:f4fb211a:::benchmark_data_smithi003_327099_object93577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:53.944407+0000", 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 14.897319261, 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.032453827999999997, 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 205303 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.926 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944407+0000", 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944407+0000", 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944407+0000", 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.927 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944410+0000", 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7539999999999998e-06 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944419+0000", 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.4970000000000007e-06 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944423+0000", 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8330000000000004e-06 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.952252+0000", 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0078291239999999998 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.952383+0000", 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013092400000000001 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.953362+0000", 2024-09-13T15:09:08.929 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0009784590000000001 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.960734+0000", 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.007372155 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.960790+0000", 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6113999999999998e-05 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976717+0000", 2024-09-13T15:09:08.930 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015926996999999998 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976818+0000", 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010168 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976861+0000", 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2781000000000003e-05 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.932 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:205304 14.f 14:f2f51a8f:::benchmark_data_smithi003_327099_object93578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-13T15:09:08.932 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:53.947897+0000", 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 14.893829327000001, 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.029458068, 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 205304 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.933 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947897+0000", 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947897+0000", 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947896+0000", 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.934 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947903+0000", 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6930000000000007e-06 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947920+0000", 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7173999999999999e-05 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947927+0000", 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3069999999999999e-06 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.956575+0000", 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0086487360000000006 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.956733+0000", 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015738300000000001 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.956862+0000", 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012953800000000001 2024-09-13T15:09:08.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.964608+0000", 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0077453640000000002 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.964936+0000", 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000328358 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.977192+0000", 2024-09-13T15:09:08.937 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012255406 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.977312+0000", 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120778 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.977355+0000", 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3140999999999999e-05 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.938 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:208713 14.f 14:fb1fed22:::benchmark_data_smithi003_327099_object96987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:57.418922+0000", 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 11.422805143, 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.027707038999999999, 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 208713 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.939 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.418922+0000", 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.418922+0000", 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.418920+0000", 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.940 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.418926+0000", 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.789e-06 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.418938+0000", 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2087e-05 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.418945+0000", 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.443e-06 2024-09-13T15:09:08.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.418986+0000", 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1597000000000002e-05 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.419094+0000", 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000107489 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.419185+0000", 2024-09-13T15:09:08.942 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1467999999999999e-05 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.421811+0000", 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026259320000000001 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.421860+0000", 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9147999999999998e-05 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.943 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.446574+0000", 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.024713434999999999 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.446605+0000", 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0824999999999997e-05 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:57.446629+0000", 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4100999999999998e-05 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.944 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:210474 14.f 14:fab60d5b:::benchmark_data_smithi003_327099_object98748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:59.315986+0000", 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 9.5257410769999993, 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.030457702, 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 210474 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.945 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.315986+0000", 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.315986+0000", 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.315984+0000", 2024-09-13T15:09:08.946 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.315991+0000", 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1910000000000003e-06 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.316003+0000", 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1573000000000001e-05 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.316009+0000", 2024-09-13T15:09:08.947 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9100000000000002e-06 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.323848+0000", 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.007838935 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.324237+0000", 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00038897200000000002 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.948 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.324715+0000", 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00047820399999999998 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.331861+0000", 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0071464019999999996 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.332012+0000", 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015039299999999999 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.949 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.346120+0000", 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014107977000000001 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.346293+0000", 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017336900000000001 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.346443+0000", 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015028599999999999 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.950 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:211262 14.f 14:f32cf6a9:::benchmark_data_smithi003_327099_object99536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:01.170748+0000", 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 7.6709784970000001, 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.043196911999999997, 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211262 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170748+0000", 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170748+0000", 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170747+0000", 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170752+0000", 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0370000000000002e-06 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170767+0000", 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4294000000000001e-05 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170772+0000", 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3619999999999997e-06 2024-09-13T15:09:08.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.173859+0000", 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0030865659999999998 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.174012+0000", 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015286799999999999 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.174137+0000", 2024-09-13T15:09:08.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000125718 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.176958+0000", 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028208759999999999 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177269+0000", 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00031072100000000002 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.955 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.213853+0000", 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.036584325000000001 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.213909+0000", 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5989000000000002e-05 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.213945+0000", 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5973999999999998e-05 2024-09-13T15:09:08.956 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:211267 14.f 14:fed515f8:::benchmark_data_smithi003_327099_object99541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:01.177346+0000", 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 7.664380918, 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.036696034000000002, 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211267 2024-09-13T15:09:08.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177346+0000", 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177346+0000", 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177345+0000", 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177350+0000", 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6899999999999997e-06 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177361+0000", 2024-09-13T15:09:08.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.022e-05 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177365+0000", 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0579999999999999e-06 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177399+0000", 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4694999999999999e-05 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.960 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177481+0000", 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1388000000000001e-05 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177555+0000", 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4338000000000006e-05 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.179821+0000", 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022655819999999999 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.961 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.180588+0000", 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00076705000000000002 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.213977+0000", 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.033389029000000001 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.214012+0000", 2024-09-13T15:09:08.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4688999999999999e-05 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.214042+0000", 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0304e-05 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:214330 14.f 14:fb3273f5:::benchmark_data_smithi003_327099_object102604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:05.109304+0000", 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 3.7324231810000001, 2024-09-13T15:09:08.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.12425335899999999, 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214330 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109304+0000", 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109304+0000", 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109302+0000", 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109309+0000", 2024-09-13T15:09:08.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2289999999999997e-06 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109319+0000", 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0277e-05 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109324+0000", 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7790000000000003e-06 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109364+0000", 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9369000000000002e-05 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109451+0000", 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7571000000000006e-05 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109525+0000", 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3669000000000002e-05 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.112116+0000", 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0025916569999999998 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.112346+0000", 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00022925800000000001 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.233449+0000", 2024-09-13T15:09:08.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.12110299200000001 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.233516+0000", 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7576999999999994e-05 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.233557+0000", 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0661999999999998e-05 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:215669 14.f 14:f7c3bda3:::benchmark_data_smithi003_327099_object103943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:08.087189+0000", 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 0.75453808899999997, 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.031969134000000003, 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 215669 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:09:08.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087189+0000", 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087189+0000", 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087188+0000", 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:09:08.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087193+0000", 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8499999999999999e-06 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087209+0000", 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5571000000000001e-05 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087216+0000", 2024-09-13T15:09:08.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0140000000000001e-06 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.101422+0000", 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014205751000000001 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.101558+0000", 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000136138 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.101692+0000", 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013428100000000001 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.106924+0000", 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0052315360000000002 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.107999+0000", 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0010749209999999999 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.119010+0000", 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011011549000000001 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.119115+0000", 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010494599999999999 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.119158+0000", 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2571999999999998e-05 2024-09-13T15:09:08.975 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.976 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.976 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.976 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:09:08.976 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:09:08.976 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:09:08.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:09.014 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:09.116 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:09.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:09.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:08 smithi003 ceph-mon[107278]: osdmap e463: 8 total, 8 up, 7 in 2024-09-13T15:09:09.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:08 smithi003 ceph-mon[107278]: pgmap v1638: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 622 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 440 op/s; 830/29886 objects misplaced (2.777%); 18 MiB/s, 284 objects/s recovering 2024-09-13T15:09:09.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:08 smithi003 ceph-mon[104007]: osdmap e463: 8 total, 8 up, 7 in 2024-09-13T15:09:09.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:08 smithi003 ceph-mon[104007]: pgmap v1638: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 622 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 440 op/s; 830/29886 objects misplaced (2.777%); 18 MiB/s, 284 objects/s recovering 2024-09-13T15:09:09.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:09.296 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:09.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:09.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:08 smithi019 ceph-mon[91296]: osdmap e463: 8 total, 8 up, 7 in 2024-09-13T15:09:09.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:08 smithi019 ceph-mon[91296]: pgmap v1638: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 622 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 440 op/s; 830/29886 objects misplaced (2.777%); 18 MiB/s, 284 objects/s recovering 2024-09-13T15:09:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:09.534 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:09.580 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:09.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:09.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:09.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:09.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:09.968 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:09.999 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:10.057 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:09:10.058 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:09:10.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:09:10.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:09:10.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:10.127 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:09:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:09:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:10.162 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:09:10.162 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:09:10.196 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:09:10.197 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:09:10.221 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:09:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:09:10.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:10.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:09:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:09:10.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:10 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1088791123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:10.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:10 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1088791123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:10.289 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:10.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:10.297 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:09:10.297 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:09:10.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:10.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:10.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:10 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1088791123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:10.433 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:09:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:09:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:09:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:10.535 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:10.573 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:10.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T15:09:10.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:10.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:10.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:10.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:10.993 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:11.036 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:11.138 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:11.163 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:11.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:11.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:11.401 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:11.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:11 smithi019 ceph-mon[91296]: pgmap v1639: 119 pgs: 119 active+clean; 447 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 568 op/s; 7.5 MiB/s, 119 objects/s recovering 2024-09-13T15:09:11.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:11 smithi003 ceph-mon[107278]: pgmap v1639: 119 pgs: 119 active+clean; 447 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 568 op/s; 7.5 MiB/s, 119 objects/s recovering 2024-09-13T15:09:11.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:11 smithi003 ceph-mon[104007]: pgmap v1639: 119 pgs: 119 active+clean; 447 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 568 op/s; 7.5 MiB/s, 119 objects/s recovering 2024-09-13T15:09:11.442 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:11.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:11.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:11.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:11.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:11.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:11.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:11.855 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:11.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:11.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.995 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:11.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:12.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.138 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:12.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:12.239 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:12.274 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:12.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:12.375 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:12.413 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:12.515 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:12.552 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:12.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:12.689 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:12.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:12.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:12.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:12.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:12.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:13.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:13.125 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:13.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T15:09:13.375 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.406 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:13.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:13.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:13.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:13.575 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:13 smithi003 ceph-mon[107278]: pgmap v1640: 119 pgs: 119 active+clean; 390 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 679 op/s; 7.0 MiB/s, 112 objects/s recovering 2024-09-13T15:09:13.575 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:13 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1401201106' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:09:13.575 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:13.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:13 smithi003 ceph-mon[104007]: pgmap v1640: 119 pgs: 119 active+clean; 390 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 679 op/s; 7.0 MiB/s, 112 objects/s recovering 2024-09-13T15:09:13.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:13 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1401201106' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:09:13.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:13.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:09:13.658 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:13 smithi019 ceph-mon[91296]: pgmap v1640: 119 pgs: 119 active+clean; 390 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 679 op/s; 7.0 MiB/s, 112 objects/s recovering 2024-09-13T15:09:13.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:13 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1401201106' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:09:13.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:13.691 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:13.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:13.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:13.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:13.981 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:14.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:14.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:14.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:14.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:14.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:14.344 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:09:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:09:14] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T15:09:14.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:14.398 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:14.500 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.533 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:14.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:14.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.673 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:14.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1401201106' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:09:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:14 smithi019 ceph-mon[91296]: osdmap e464: 8 total, 8 up, 7 in 2024-09-13T15:09:14.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1401201106' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:09:14.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:14 smithi003 ceph-mon[104007]: osdmap e464: 8 total, 8 up, 7 in 2024-09-13T15:09:14.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1401201106' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:09:14.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:14 smithi003 ceph-mon[107278]: osdmap e464: 8 total, 8 up, 7 in 2024-09-13T15:09:14.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:14.806 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:14.812 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 111723 objects 2024-09-13T15:09:14.812 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :139.136 2024-09-13T15:09:14.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T15:09:14.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:14.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:15.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:15.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:15.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:15.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:15.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:15.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:15.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:15.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:15.670 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:15 smithi019 ceph-mon[91296]: pgmap v1642: 119 pgs: 119 active+clean; 100 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.22k op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-13T15:09:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:15 smithi003 ceph-mon[107278]: pgmap v1642: 119 pgs: 119 active+clean; 100 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.22k op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-13T15:09:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:15 smithi003 ceph-mon[104007]: pgmap v1642: 119 pgs: 119 active+clean; 100 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.22k op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-13T15:09:15.695 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:15.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:15.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:15.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:15.949 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:15.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:16.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:16.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:16.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:16.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:16.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:16.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:16.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:16.355 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-13T15:09:16.417 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_9 2024-09-13T15:09:16.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_9 unique_pool_9 --yes-i-really-really-mean-it 2024-09-13T15:09:16.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:16.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:16.587 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:16.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:16 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3470270007' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:09:16.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:16 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:09:16.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:16 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3470270007' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:09:16.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:16 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:09:16.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:16 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3470270007' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:09:16.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:16 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:09:16.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:16.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:16.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:16.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:16.966 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:16.993 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:17.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:17.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:17.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:17.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:17.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:17.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:17.368 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:17.386 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:09:17.386 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:09:17.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:17.498 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:17.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:17.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:17.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:17 smithi019 ceph-mon[91296]: pgmap v1643: 119 pgs: 119 active+clean; 100 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 997 op/s; 5.4 MiB/s, 85 objects/s recovering 2024-09-13T15:09:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:17 smithi019 ceph-mon[91296]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:17 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:09:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:17 smithi019 ceph-mon[91296]: osdmap e465: 8 total, 8 up, 7 in 2024-09-13T15:09:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:17 smithi003 ceph-mon[107278]: pgmap v1643: 119 pgs: 119 active+clean; 100 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 997 op/s; 5.4 MiB/s, 85 objects/s recovering 2024-09-13T15:09:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:17 smithi003 ceph-mon[107278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:17 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:09:17.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:17 smithi003 ceph-mon[107278]: osdmap e465: 8 total, 8 up, 7 in 2024-09-13T15:09:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:17 smithi003 ceph-mon[104007]: pgmap v1643: 119 pgs: 119 active+clean; 100 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 997 op/s; 5.4 MiB/s, 85 objects/s recovering 2024-09-13T15:09:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:17 smithi003 ceph-mon[104007]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:17 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:09:17.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:17 smithi003 ceph-mon[104007]: osdmap e465: 8 total, 8 up, 7 in 2024-09-13T15:09:17.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:17.751 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:17.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:17.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:17.978 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:18.006 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:18.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:18.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.136 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:18.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:18.237 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:18.264 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:18.366 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:18.393 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:18.494 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:18.520 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:18.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:18.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:18.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:18.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:18.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:18.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:18.997 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:19.025 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:19.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:19.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:19.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:19.375 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:19.403 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:19.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:19.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:19.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:19.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:19.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:19 smithi003 ceph-mon[107278]: pgmap v1645: 119 pgs: 119 active+clean; 65 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-09-13T15:09:19.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:19 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:19.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:19 smithi003 ceph-mon[104007]: pgmap v1645: 119 pgs: 119 active+clean; 65 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-09-13T15:09:19.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:19 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:19.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:19.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:19.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:19.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T15:09:19.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:19.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:19 smithi019 ceph-mon[91296]: pgmap v1645: 119 pgs: 119 active+clean; 65 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-09-13T15:09:19.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:19 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:20.009 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:20.036 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:20.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:20.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:20.263 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.292 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:20.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:20.321 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:20.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:20.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:20.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:20.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:20.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:20.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:20.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:09:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:09:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:09:20.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:20.897 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:20.924 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:21.026 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:21.052 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:21.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:21.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:21.181 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:21.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:21.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:21.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:21.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:09:21.410 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:21.436 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:21.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:21.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:21 smithi003 ceph-mon[104007]: pgmap v1646: 119 pgs: 119 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 777 op/s 2024-09-13T15:09:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:21.663 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:21.689 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:21.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:21.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:21.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:21.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:21 smithi019 ceph-mon[91296]: pgmap v1646: 119 pgs: 119 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 777 op/s 2024-09-13T15:09:21.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:21 smithi003 ceph-mon[107278]: pgmap v1646: 119 pgs: 119 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 777 op/s 2024-09-13T15:09:21.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:22.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:22.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:22.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:22.298 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:22.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.327 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:22.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:22.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:22.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:22.553 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:22.580 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:22.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:22.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:22.807 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:22.834 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:22.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:22.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:23.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:23.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:23.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:23.090 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:23.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:23.213 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:23.315 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:23.341 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:23.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:23.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:23.549 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T15:09:23.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:23.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:23.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:23.720 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:23.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:23.759 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:23 smithi003 ceph-mon[104007]: pgmap v1647: 119 pgs: 119 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 698 op/s 2024-09-13T15:09:23.759 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:23 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/878828083' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:09:23.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:23 smithi003 ceph-mon[107278]: pgmap v1647: 119 pgs: 119 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 698 op/s 2024-09-13T15:09:23.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:23 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/878828083' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:09:23.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:23.848 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:23.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:23 smithi019 ceph-mon[91296]: pgmap v1647: 119 pgs: 119 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 698 op/s 2024-09-13T15:09:23.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:23 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/878828083' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:09:23.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:23.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:24.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:24.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:24.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:24.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:24.353 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:24.410 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:09:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:09:24] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T15:09:24.419 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:09:24.419 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:09:24.435 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:09:24.436 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:09:24.452 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:09:24.452 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:09:24.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:24.468 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:09:24.468 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:09:24.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:24.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:24.485 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:09:24.485 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:09:24.501 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:09:24.501 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:09:24.517 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:09:24.517 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:09:24.534 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:09:24.534 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:09:24.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:24.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:24 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/878828083' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:09:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:24 smithi003 ceph-mon[104007]: osdmap e466: 8 total, 8 up, 7 in 2024-09-13T15:09:24.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:24 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1299358989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:24.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:24 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/878828083' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:09:24.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:24 smithi003 ceph-mon[107278]: osdmap e466: 8 total, 8 up, 7 in 2024-09-13T15:09:24.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:24 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1299358989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:24.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:24.734 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:24.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:24.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:24.868 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:24.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:24 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/878828083' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:09:24.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:24 smithi019 ceph-mon[91296]: osdmap e466: 8 total, 8 up, 7 in 2024-09-13T15:09:24.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:24 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1299358989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:24.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:24.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:25.083 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:25.109 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:25.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:25.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:25.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:25.364 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:25.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T15:09:25.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:25.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:25.563 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_9' removed 2024-09-13T15:09:25.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:25.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:25 smithi003 ceph-mon[107278]: pgmap v1649: 119 pgs: 119 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 198 op/s 2024-09-13T15:09:25.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:25 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2812003096' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:09:25.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:25 smithi003 ceph-mon[104007]: pgmap v1649: 119 pgs: 119 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 198 op/s 2024-09-13T15:09:25.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:25 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2812003096' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:09:25.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:25.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:25.845 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:25.871 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:25 smithi019 ceph-mon[91296]: pgmap v1649: 119 pgs: 119 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 198 op/s 2024-09-13T15:09:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:25 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2812003096' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:09:25.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:25.999 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:26.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:26.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:26.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:26.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:26.305 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T15:09:26.306 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-13T15:09:26.306 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-13T15:09:26.312 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-09-13T15:09:26.312 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T15:09:26.313 DEBUG:teuthology.orchestra.run.smithi003:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:09:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.338 INFO:teuthology.orchestra.run.smithi003.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T15:09:26.338 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:09:26.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:26.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:26.420 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T15:09:26.420 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T15:09:26.477 INFO:tasks.workunit:timeout=3h 2024-09-13T15:09:26.477 INFO:tasks.workunit:cleanup=True 2024-09-13T15:09:26.478 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-13T15:09:26.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:26.533 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:26.542 INFO:tasks.workunit.client.0.smithi003.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T15:09:26.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:26.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:26.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:26.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:26.883 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2812003096' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:09:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:26 smithi019 ceph-mon[91296]: osdmap e467: 8 total, 8 up, 7 in 2024-09-13T15:09:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2812003096' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:09:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:26 smithi003 ceph-mon[107278]: osdmap e467: 8 total, 8 up, 7 in 2024-09-13T15:09:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2812003096' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:09:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:26 smithi003 ceph-mon[104007]: osdmap e467: 8 total, 8 up, 7 in 2024-09-13T15:09:27.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:27.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:27.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:27.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:27.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:27.280 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:27.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:27.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:27.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:27.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:27.630 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:27.656 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:27.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:27.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:27.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:27 smithi019 ceph-mon[91296]: pgmap v1651: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:27.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:27 smithi003 ceph-mon[107278]: pgmap v1651: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:27.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:27 smithi003 ceph-mon[104007]: pgmap v1651: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:28.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:28.142 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:28.169 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:28.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:28.297 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:28.399 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:28.426 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:28.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:28.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:28.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:28.681 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:28.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:28.888 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:09:28.888 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:09:28.889 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:09:28.889 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:09:28.889 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:09:28.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:29.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:29.039 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:29.139 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:29.139 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:29.139 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:28 smithi003 ceph-mon[107278]: pgmap v1652: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:29.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:29.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:29.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:28 smithi003 ceph-mon[104007]: pgmap v1652: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:29.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:29.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:28 smithi019 ceph-mon[91296]: pgmap v1652: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:29.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:29.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:29.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:29.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:29.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:29.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:29.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:09:29.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:29.666 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:29.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:29.768 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:29.795 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:29.897 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:29.923 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:30.025 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:30.051 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:30.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:09:30.138 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [3, 2]} : dispatch 2024-09-13T15:09:30.138 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:29 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]} : dispatch 2024-09-13T15:09:30.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [3, 2]} : dispatch 2024-09-13T15:09:30.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:29 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]} : dispatch 2024-09-13T15:09:30.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.180 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:30.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [3, 2]} : dispatch 2024-09-13T15:09:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:29 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]} : dispatch 2024-09-13T15:09:30.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:30.309 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:30.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:30.438 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:30.540 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:30.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.567 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:30.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:30.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:09:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:09:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:09:30.683 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:30.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:30.706 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:30.731 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:30.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:30.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:30.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:30.957 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:31.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:31.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:31.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [3, 2]}]': finished 2024-09-13T15:09:31.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]': finished 2024-09-13T15:09:31.160 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:30 smithi003 ceph-mon[107278]: osdmap e468: 8 total, 8 up, 7 in 2024-09-13T15:09:31.160 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:30 smithi003 ceph-mon[107278]: pgmap v1654: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:31.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [3, 2]}]': finished 2024-09-13T15:09:31.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]': finished 2024-09-13T15:09:31.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:30 smithi003 ceph-mon[104007]: osdmap e468: 8 total, 8 up, 7 in 2024-09-13T15:09:31.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:30 smithi003 ceph-mon[104007]: pgmap v1654: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [3, 2]}]': finished 2024-09-13T15:09:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]': finished 2024-09-13T15:09:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:30 smithi019 ceph-mon[91296]: osdmap e468: 8 total, 8 up, 7 in 2024-09-13T15:09:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:30 smithi019 ceph-mon[91296]: pgmap v1654: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:31.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:31.211 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:31.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:31.339 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:31.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:31.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:31.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:31.594 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:31.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:31.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:31.826 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:31.843 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-13T15:09:31.854 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:31.956 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:31.983 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:32.085 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:32.111 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:32.117 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T15:09:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:31 smithi019 ceph-mon[91296]: osdmap e469: 8 total, 8 up, 7 in 2024-09-13T15:09:32.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:31 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1553270638' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:32.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:31 smithi003 ceph-mon[107278]: osdmap e469: 8 total, 8 up, 7 in 2024-09-13T15:09:32.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:31 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1553270638' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:32.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:31 smithi003 ceph-mon[104007]: osdmap e469: 8 total, 8 up, 7 in 2024-09-13T15:09:32.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:31 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1553270638' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:32.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:32.222 INFO:tasks.workunit.client.0.smithi003.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T15:09:32.224 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T15:09:32.225 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T15:09:32.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:32.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:32.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:32.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:32.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:32.587 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:32.614 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:32.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:32.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:32.743 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:32.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:32.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:32.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:32.998 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:32.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:33.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:32 smithi003 ceph-mon[107278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:33.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1553270638' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:09:33.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:32 smithi003 ceph-mon[107278]: osdmap e470: 8 total, 8 up, 7 in 2024-09-13T15:09:33.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:32 smithi003 ceph-mon[107278]: pgmap v1657: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:33.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:32 smithi003 ceph-mon[104007]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:33.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1553270638' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:09:33.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:32 smithi003 ceph-mon[104007]: osdmap e470: 8 total, 8 up, 7 in 2024-09-13T15:09:33.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:32 smithi003 ceph-mon[104007]: pgmap v1657: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:33.099 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:33.126 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:32 smithi019 ceph-mon[91296]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1553270638' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:09:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:32 smithi019 ceph-mon[91296]: osdmap e470: 8 total, 8 up, 7 in 2024-09-13T15:09:33.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:32 smithi019 ceph-mon[91296]: pgmap v1657: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:33.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:33.255 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:33.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:33.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:33.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:33.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:33.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:33.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:33.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:33.767 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:33.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:33.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:33.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:34.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:34.122 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:34.150 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:34.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:34.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:34.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:34.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:09:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:09:34] "GET /metrics HTTP/1.1" 200 34113 "" "Prometheus/2.51.0" 2024-09-13T15:09:34.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:34.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:34.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:34.663 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:34.765 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:34.791 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:34.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:34.920 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:35.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:35.049 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:35.089 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T15:09:35.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:35.177 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:35.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:35.210 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T15:09:35.252 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T15:09:35.254 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T15:09:35.254 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T15:09:35.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:35.294 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T15:09:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:35.322 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-13T15:09:35.322 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T15:09:35.355 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-13T15:09:35.356 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-13T15:09:35.356 DEBUG:teuthology.orchestra.run.smithi003:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-13T15:09:35.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:35.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:35.512 INFO:tasks.workunit.client.0.smithi003.stdout:Running main() from gmock_main.cc 2024-09-13T15:09:35.512 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 70 tests from 1 test suite. 2024-09-13T15:09:35.512 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2024-09-13T15:09:35.512 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 70 tests from TestClsRbd 2024-09-13T15:09:35.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:35.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:35.605 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:35 smithi003 ceph-mon[107278]: pgmap v1658: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:35.605 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:35 smithi003 ceph-mon[104007]: pgmap v1658: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:35.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:35 smithi019 ceph-mon[91296]: pgmap v1658: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:35.686 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:35.713 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:35.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:35.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:35.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:35.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:09:35.968 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:36.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:36.196 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.224 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:36.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:36.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:36.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:36.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:36.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:36.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:36.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:36.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:36.730 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:36.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:36.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:36.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:36.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:37.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:37.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:37.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:37.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:37.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T15:09:37.339 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-13T15:09:37.366 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:37.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:37.432 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_all_features (93 ms) 2024-09-13T15:09:37.432 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.copyup 2024-09-13T15:09:37.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:37.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:37 smithi003 ceph-mon[107278]: pgmap v1659: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:37.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:37 smithi003 ceph-mon[107278]: osdmap e471: 8 total, 8 up, 7 in 2024-09-13T15:09:37.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/755456862' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-357922-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T15:09:37.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3228695915' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:09:37.587 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:37 smithi003 ceph-mon[104007]: pgmap v1659: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:37.587 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:37 smithi003 ceph-mon[104007]: osdmap e471: 8 total, 8 up, 7 in 2024-09-13T15:09:37.588 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/755456862' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-357922-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T15:09:37.588 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3228695915' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:09:37.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:37.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:37.637 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.copyup (205 ms) 2024-09-13T15:09:37.637 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-13T15:09:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:37 smithi019 ceph-mon[91296]: pgmap v1659: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:37 smithi019 ceph-mon[91296]: osdmap e471: 8 total, 8 up, 7 in 2024-09-13T15:09:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/755456862' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-357922-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T15:09:37.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3228695915' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:09:37.718 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:37.742 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:37.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:37.782 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.sparse_copyup (145 ms) 2024-09-13T15:09:37.782 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-13T15:09:37.790 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_and_set_id (8 ms) 2024-09-13T15:09:37.790 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-13T15:09:37.801 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.add_remove_child (11 ms) 2024-09-13T15:09:37.801 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-13T15:09:37.829 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.directory_methods (29 ms) 2024-09-13T15:09:37.830 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.create 2024-09-13T15:09:37.844 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:37.852 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.create (23 ms) 2024-09-13T15:09:37.852 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_features 2024-09-13T15:09:37.853 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_features (1 ms) 2024-09-13T15:09:37.854 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-13T15:09:37.855 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_object_prefix (1 ms) 2024-09-13T15:09:37.855 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-13T15:09:37.857 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_create_timestamp (2 ms) 2024-09-13T15:09:37.857 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-13T15:09:37.866 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:37.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:37.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:38.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:38.115 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_access_timestamp (258 ms) 2024-09-13T15:09:38.115 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-13T15:09:38.116 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_modify_timestamp (2 ms) 2024-09-13T15:09:38.116 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-13T15:09:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:38.119 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_data_pool (3 ms) 2024-09-13T15:09:38.120 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_size 2024-09-13T15:09:38.124 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_size (4 ms) 2024-09-13T15:09:38.124 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.set_size 2024-09-13T15:09:38.128 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.set_size (5 ms) 2024-09-13T15:09:38.129 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-13T15:09:38.147 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.protection_status (19 ms) 2024-09-13T15:09:38.147 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-13T15:09:38.155 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.snapshot_limits (7 ms) 2024-09-13T15:09:38.155 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-13T15:09:38.209 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.parents_v1 (54 ms) 2024-09-13T15:09:38.209 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-13T15:09:38.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:38.238 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.parents_v2 (29 ms) 2024-09-13T15:09:38.238 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-13T15:09:38.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:38.258 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.snapshots (21 ms) 2024-09-13T15:09:38.259 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-13T15:09:38.265 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.snapid_race (6 ms) 2024-09-13T15:09:38.265 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-13T15:09:38.275 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.stripingv2 (10 ms) 2024-09-13T15:09:38.275 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-13T15:09:38.277 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_save (2 ms) 2024-09-13T15:09:38.277 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-13T15:09:38.283 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_resize (7 ms) 2024-09-13T15:09:38.284 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-13T15:09:38.289 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_update (5 ms) 2024-09-13T15:09:38.289 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-13T15:09:38.289 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2024-09-13T15:09:38.289 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-13T15:09:38.296 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_snap_add (7 ms) 2024-09-13T15:09:38.296 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-13T15:09:38.304 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_snap_remove (7 ms) 2024-09-13T15:09:38.304 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.flags 2024-09-13T15:09:38.308 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.flags (6 ms) 2024-09-13T15:09:38.308 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.metadata 2024-09-13T15:09:38.332 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.metadata (24 ms) 2024-09-13T15:09:38.333 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.set_features 2024-09-13T15:09:38.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:38.345 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.set_features (12 ms) 2024-09-13T15:09:38.345 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror 2024-09-13T15:09:38.369 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:38.379 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror (35 ms) 2024-09-13T15:09:38.379 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-13T15:09:38.396 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_image (16 ms) 2024-09-13T15:09:38.396 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-13T15:09:38.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:38.494 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:38.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:38.618 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:38.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/755456862' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-357922-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T15:09:38.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3228695915' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:09:38.657 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:38 smithi003 ceph-mon[107278]: osdmap e472: 8 total, 8 up, 7 in 2024-09-13T15:09:38.657 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3619821581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:38.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/755456862' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-357922-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T15:09:38.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3228695915' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:09:38.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:38 smithi003 ceph-mon[104007]: osdmap e472: 8 total, 8 up, 7 in 2024-09-13T15:09:38.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3619821581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/755456862' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-357922-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3228695915' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:38 smithi019 ceph-mon[91296]: osdmap e472: 8 total, 8 up, 7 in 2024-09-13T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3619821581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:38.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:38.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.871 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:38.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:38.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:09:38.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:09:38.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:09:38.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:09:38.940 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:09:38.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:09:38.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:09:38.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:09:38.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:09:38.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:09:38.940 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:09:38.941 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:09:38.941 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:09:38.941 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:09:38.941 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:09:38.941 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:09:38.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:38.994 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:39.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:39.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:39.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:39.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:39.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:39.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:39.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:39.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:39.503 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:39.595 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:39 smithi003 ceph-mon[104007]: pgmap v1662: 135 pgs: 3 creating+peering, 29 unknown, 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:39.595 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:39 smithi003 ceph-mon[104007]: osdmap e473: 8 total, 8 up, 7 in 2024-09-13T15:09:39.595 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:39 smithi003 ceph-mon[107278]: pgmap v1662: 135 pgs: 3 creating+peering, 29 unknown, 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:39.595 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:39 smithi003 ceph-mon[107278]: osdmap e473: 8 total, 8 up, 7 in 2024-09-13T15:09:39.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:39.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:39 smithi019 ceph-mon[91296]: pgmap v1662: 135 pgs: 3 creating+peering, 29 unknown, 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:39 smithi019 ceph-mon[91296]: osdmap e473: 8 total, 8 up, 7 in 2024-09-13T15:09:39.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:39.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:39.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:39.989 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-13T15:09:39.989 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:09:39.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:40.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:40.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:40.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:40.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:40.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:40.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:40.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.397 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:40.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:40.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T15:09:40.498 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:40.520 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:40.582 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:09:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:09:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:09:40.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:40.644 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:40.717 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:40.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:40.766 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:40.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:40.877 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_image_status (2480 ms) 2024-09-13T15:09:40.877 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-13T15:09:40.878 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:40.900 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:40.959 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_image_map (83 ms) 2024-09-13T15:09:40.960 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-13T15:09:40.966 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_instances (8 ms) 2024-09-13T15:09:40.966 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-13T15:09:40.981 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_snapshot (13 ms) 2024-09-13T15:09:40.982 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-13T15:09:40.982 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_list (2 ms) 2024-09-13T15:09:40.982 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-13T15:09:40.984 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_add (3 ms) 2024-09-13T15:09:40.984 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-13T15:09:40.987 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.dir_add_already_existing (2 ms) 2024-09-13T15:09:40.987 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-13T15:09:40.993 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_rename (6 ms) 2024-09-13T15:09:40.993 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-13T15:09:40.996 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_remove (4 ms) 2024-09-13T15:09:40.996 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-13T15:09:41.000 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (3 ms) 2024-09-13T15:09:41.000 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-13T15:09:41.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:41.002 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_image_add (3 ms) 2024-09-13T15:09:41.002 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-13T15:09:41.006 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_image_remove (3 ms) 2024-09-13T15:09:41.006 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-13T15:09:41.008 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_image_list (3 ms) 2024-09-13T15:09:41.008 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-13T15:09:41.012 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_image_clean (4 ms) 2024-09-13T15:09:41.012 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-13T15:09:41.014 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-09-13T15:09:41.015 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-13T15:09:41.019 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.image_group_remove (4 ms) 2024-09-13T15:09:41.019 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-13T15:09:41.021 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.image_group_get (2 ms) 2024-09-13T15:09:41.021 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-13T15:09:41.023 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (2 ms) 2024-09-13T15:09:41.023 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-13T15:09:41.025 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (3 ms) 2024-09-13T15:09:41.025 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-13T15:09:41.025 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:41.028 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (3 ms) 2024-09-13T15:09:41.028 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-13T15:09:41.031 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (3 ms) 2024-09-13T15:09:41.031 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-13T15:09:41.034 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.7-1627-gbbdadfa3/rpm/el9/BUILD/ceph-17.2.7-1627-gbbdadfa3/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-09-13T15:09:41.034 INFO:tasks.workunit.client.0.smithi003.stdout:Expected equality of these values: 2024-09-13T15:09:41.034 INFO:tasks.workunit.client.0.smithi003.stdout: 1U 2024-09-13T15:09:41.034 INFO:tasks.workunit.client.0.smithi003.stdout: Which is: 1 2024-09-13T15:09:41.035 INFO:tasks.workunit.client.0.smithi003.stdout: keys.size() 2024-09-13T15:09:41.035 INFO:tasks.workunit.client.0.smithi003.stdout: Which is: 3 2024-09-13T15:09:41.035 INFO:tasks.workunit.client.0.smithi003.stdout:[ FAILED ] TestClsRbd.group_snap_set (3 ms) 2024-09-13T15:09:41.035 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-13T15:09:41.037 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_list (3 ms) 2024-09-13T15:09:41.038 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-13T15:09:41.055 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (18 ms) 2024-09-13T15:09:41.055 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-13T15:09:41.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:41.151 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:41.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:41.261 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (206 ms) 2024-09-13T15:09:41.262 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-13T15:09:41.265 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_remove (4 ms) 2024-09-13T15:09:41.266 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-13T15:09:41.267 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (2 ms) 2024-09-13T15:09:41.268 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-13T15:09:41.275 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.trash_methods (7 ms) 2024-09-13T15:09:41.275 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.op_features 2024-09-13T15:09:41.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.278 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:41.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:41.282 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.op_features (7 ms) 2024-09-13T15:09:41.282 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-13T15:09:41.301 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.clone_parent (20 ms) 2024-09-13T15:09:41.302 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-13T15:09:41.309 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.clone_parent_ns (8 ms) 2024-09-13T15:09:41.309 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-13T15:09:41.320 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.clone_child (10 ms) 2024-09-13T15:09:41.320 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-13T15:09:41.326 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.namespace_methods (7 ms) 2024-09-13T15:09:41.326 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.migration 2024-09-13T15:09:41.336 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.migration (9 ms) 2024-09-13T15:09:41.336 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-13T15:09:41.349 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.migration_v1 (13 ms) 2024-09-13T15:09:41.349 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-13T15:09:41.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:41 smithi003 ceph-mon[104007]: pgmap v1664: 135 pgs: 3 creating+peering, 132 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.6 MiB/s rd, 844 KiB/s wr, 482 op/s 2024-09-13T15:09:41.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:41 smithi003 ceph-mon[107278]: pgmap v1664: 135 pgs: 3 creating+peering, 132 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.6 MiB/s rd, 844 KiB/s wr, 482 op/s 2024-09-13T15:09:41.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:41.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:41.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:41.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:41.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:09:41.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:41 smithi019 ceph-mon[91296]: pgmap v1664: 135 pgs: 3 creating+peering, 132 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.6 MiB/s rd, 844 KiB/s wr, 482 op/s 2024-09-13T15:09:41.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:41.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:41.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:41.901 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:41.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:42.004 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:42.030 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:42.132 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:42.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T15:09:42.158 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:42.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:42.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:42.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:42.508 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:42.532 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:42.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:42.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:42.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:42.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:42.783 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:42.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:42.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:42.908 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:42.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:43.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:43.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:43.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:43.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:43.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:43.354 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.assert_snapc_seq (2006 ms) 2024-09-13T15:09:43.355 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-13T15:09:43.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:43.390 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.sparsify (35 ms) 2024-09-13T15:09:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:43.507 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:43.532 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:43.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:43.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:43.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:43 smithi003 ceph-mon[104007]: pgmap v1665: 135 pgs: 135 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.6 MiB/s rd, 846 KiB/s wr, 500 op/s 2024-09-13T15:09:43.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:43 smithi003 ceph-mon[104007]: osdmap e474: 8 total, 8 up, 7 in 2024-09-13T15:09:43.662 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:43.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:43 smithi003 ceph-mon[107278]: pgmap v1665: 135 pgs: 135 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.6 MiB/s rd, 846 KiB/s wr, 500 op/s 2024-09-13T15:09:43.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:43 smithi003 ceph-mon[107278]: osdmap e474: 8 total, 8 up, 7 in 2024-09-13T15:09:43.662 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:43 smithi019 ceph-mon[91296]: pgmap v1665: 135 pgs: 135 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.6 MiB/s rd, 846 KiB/s wr, 500 op/s 2024-09-13T15:09:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:43 smithi019 ceph-mon[91296]: osdmap e474: 8 total, 8 up, 7 in 2024-09-13T15:09:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:43.760 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:43.787 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:43.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:43.915 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:44.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:44.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:44.142 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:44.170 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:44.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:44.298 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:44.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:44.365 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:09:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:09:44] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-13T15:09:44.374 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 70 tests from TestClsRbd (6051 ms total) 2024-09-13T15:09:44.374 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-09-13T15:09:44.374 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2024-09-13T15:09:44.375 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 70 tests from 1 test suite ran. (8862 ms total) 2024-09-13T15:09:44.375 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 69 tests. 2024-09-13T15:09:44.375 INFO:tasks.workunit.client.0.smithi003.stdout:[ FAILED ] 1 test, listed below: 2024-09-13T15:09:44.375 INFO:tasks.workunit.client.0.smithi003.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-09-13T15:09:44.375 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-09-13T15:09:44.375 INFO:tasks.workunit.client.0.smithi003.stdout: 1 FAILED TEST 2024-09-13T15:09:44.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.399 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-09-13T15:09:44.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T15:09:44.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:44.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-09-13T15:09:44.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:44.530 DEBUG:teuthology.orchestra.run.smithi019:> sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:44.556 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: docker: command not found 2024-09-13T15:09:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi019 with status 1: 'sudo docker kill -s 1 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:44.639 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:44 smithi003 ceph-mon[107278]: osdmap e475: 8 total, 8 up, 7 in 2024-09-13T15:09:44.639 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:44 smithi003 ceph-mon[104007]: osdmap e475: 8 total, 8 up, 7 in 2024-09-13T15:09:44.641 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T15:09:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:44 smithi019 ceph-mon[91296]: osdmap e475: 8 total, 8 up, 7 in 2024-09-13T15:09:45.254 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:45.255 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:45.363 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T15:09:45.539 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:45 smithi003 ceph-mon[107278]: pgmap v1668: 135 pgs: 135 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 2.46k op/s 2024-09-13T15:09:45.539 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:45 smithi003 ceph-mon[107278]: osdmap e476: 8 total, 8 up, 7 in 2024-09-13T15:09:45.539 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:45 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1888645631' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:09:45.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:45 smithi003 ceph-mon[104007]: pgmap v1668: 135 pgs: 135 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 2.46k op/s 2024-09-13T15:09:45.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:45 smithi003 ceph-mon[104007]: osdmap e476: 8 total, 8 up, 7 in 2024-09-13T15:09:45.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:45 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1888645631' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:09:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:45 smithi019 ceph-mon[91296]: pgmap v1668: 135 pgs: 135 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 2.46k op/s 2024-09-13T15:09:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:45 smithi019 ceph-mon[91296]: osdmap e476: 8 total, 8 up, 7 in 2024-09-13T15:09:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:45 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1888645631' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:09:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:09:46.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T15:09:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:46 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1888645631' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:09:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:46 smithi019 ceph-mon[91296]: osdmap e477: 8 total, 8 up, 7 in 2024-09-13T15:09:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:46 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2200964473' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:46.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:46 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:46.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:46 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1888645631' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:09:46.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:46 smithi003 ceph-mon[107278]: osdmap e477: 8 total, 8 up, 7 in 2024-09-13T15:09:46.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:46 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2200964473' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:46.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:46 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:46.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:46 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1888645631' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:09:46.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:46 smithi003 ceph-mon[104007]: osdmap e477: 8 total, 8 up, 7 in 2024-09-13T15:09:46.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:46 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2200964473' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:46.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:46 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:47 smithi019 ceph-mon[91296]: pgmap v1671: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:47 smithi019 ceph-mon[91296]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:47 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:09:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:47 smithi019 ceph-mon[91296]: osdmap e478: 8 total, 8 up, 7 in 2024-09-13T15:09:47.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:47 smithi003 ceph-mon[107278]: pgmap v1671: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:47.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:47 smithi003 ceph-mon[107278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:47.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:47 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:09:47.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:47 smithi003 ceph-mon[107278]: osdmap e478: 8 total, 8 up, 7 in 2024-09-13T15:09:47.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:47 smithi003 ceph-mon[104007]: pgmap v1671: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:47.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:47 smithi003 ceph-mon[104007]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:47.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:47 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:09:47.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:47 smithi003 ceph-mon[104007]: osdmap e478: 8 total, 8 up, 7 in 2024-09-13T15:09:48.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:09:49.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:49 smithi019 ceph-mon[91296]: pgmap v1673: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:49.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:49 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:49 smithi003 ceph-mon[107278]: pgmap v1673: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:49 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:49 smithi003 ceph-mon[104007]: pgmap v1673: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:49 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:50.492 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-13T15:09:50.493 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-13T15:09:50.493 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:09:50.493 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:09:50.494 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:09:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:09:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:09:50.496 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-13T15:09:50.496 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:09:50.500 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:50.516 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-13T15:09:50.516 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:09:50.537 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-13T15:09:50.537 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:09:50.553 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-13T15:09:50.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:09:50.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-13T15:09:50.571 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:09:50.586 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-13T15:09:50.587 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:09:50.870 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:50 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:09:50.870 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:50 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2013814798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:50.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:50 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:09:50.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:50 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2013814798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:50.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:50 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:09:50.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:50 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2013814798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:51 smithi019 ceph-mon[91296]: pgmap v1674: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:51.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:51 smithi003 ceph-mon[107278]: pgmap v1674: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:51.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:51 smithi003 ceph-mon[104007]: pgmap v1674: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:52.185 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:09:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:09:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-13T15:09:52.186 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:09:52.379 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:52.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:09:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:53 smithi003 ceph-mon[107278]: pgmap v1675: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:53.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:53 smithi003 ceph-mon[104007]: pgmap v1675: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:53.825 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:53 smithi019 ceph-mon[91296]: pgmap v1675: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:09:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:09:54] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-13T15:09:55.295 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T15:09:55.298 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:55.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:55 smithi003 ceph-mon[104007]: pgmap v1676: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:55.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:55.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:55 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:55.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:55 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3136954555' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:09:55.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:55 smithi003 ceph-mon[107278]: pgmap v1676: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:55.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:55.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:55 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:55.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:55 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3136954555' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:09:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:55 smithi019 ceph-mon[91296]: pgmap v1676: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:55 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:09:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:55 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3136954555' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:09:56.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3136954555' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:09:56.416 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:56 smithi003 ceph-mon[107278]: osdmap e479: 8 total, 8 up, 7 in 2024-09-13T15:09:56.416 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3136954555' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:09:56.416 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:56 smithi003 ceph-mon[104007]: osdmap e479: 8 total, 8 up, 7 in 2024-09-13T15:09:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3136954555' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:09:56.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:56 smithi019 ceph-mon[91296]: osdmap e479: 8 total, 8 up, 7 in 2024-09-13T15:09:57.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T15:09:57.302 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-13T15:09:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:57 smithi003 ceph-mon[107278]: pgmap v1678: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:57.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:57 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3530182197' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:57.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:57 smithi003 ceph-mon[104007]: pgmap v1678: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:57.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:57 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3530182197' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:57 smithi019 ceph-mon[91296]: pgmap v1678: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:57 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3530182197' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:58.358 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:58.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:09:58.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:58 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3530182197' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:09:58.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:58 smithi003 ceph-mon[107278]: osdmap e480: 8 total, 8 up, 7 in 2024-09-13T15:09:58.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:58.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:58 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3530182197' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:09:58.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:58 smithi003 ceph-mon[104007]: osdmap e480: 8 total, 8 up, 7 in 2024-09-13T15:09:58.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:58 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3530182197' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:09:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:58 smithi019 ceph-mon[91296]: osdmap e480: 8 total, 8 up, 7 in 2024-09-13T15:09:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:59.667 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:09:59 smithi003 ceph-mon[104007]: pgmap v1680: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:59.667 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:09:59 smithi003 ceph-mon[107278]: pgmap v1680: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:59.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:09:59 smithi019 ceph-mon[91296]: pgmap v1680: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:09:59.715 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:00.539 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:10:00.539 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:10:00.539 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:10:00.539 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:10:00.539 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:10:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:00 smithi019 ceph-mon[91296]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-13T15:10:00.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:00 smithi019 ceph-mon[91296]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T15:10:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:10:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:10:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:10:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:00 smithi003 ceph-mon[104007]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-13T15:10:00.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:00 smithi003 ceph-mon[104007]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T15:10:00.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:00 smithi003 ceph-mon[107278]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-13T15:10:00.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:00 smithi003 ceph-mon[107278]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T15:10:01.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:10:01.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:01 smithi003 ceph-mon[107278]: pgmap v1681: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:01.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:01 smithi003 ceph-mon[104007]: pgmap v1681: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:01 smithi019 ceph-mon[91296]: pgmap v1681: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:03.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:03 smithi003 ceph-mon[107278]: pgmap v1682: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:03.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:03 smithi003 ceph-mon[104007]: pgmap v1682: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:03.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:03 smithi019 ceph-mon[91296]: pgmap v1682: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:04.110 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:04.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:10:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:10:04] "GET /metrics HTTP/1.1" 200 34116 "" "Prometheus/2.51.0" 2024-09-13T15:10:05.335 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-13T15:10:05.335 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T15:10:05.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[107278]: pgmap v1683: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:05.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1098213370' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:10:05.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:10:05.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3007881763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:05.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:05.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[104007]: pgmap v1683: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:05.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1098213370' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:10:05.495 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:10:05.495 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3007881763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:05.495 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:05 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:05 smithi019 ceph-mon[91296]: pgmap v1683: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1098213370' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:10:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:05 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:10:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:05 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3007881763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:05.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:05 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:06.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[104007]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:10:06.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[104007]: osdmap e481: 8 total, 8 up, 7 in 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[107278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[107278]: osdmap e481: 8 total, 8 up, 7 in 2024-09-13T15:10:06.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:06.657 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:06 smithi019 ceph-mon[91296]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:10:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:06 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:10:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:06 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:10:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:06 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:10:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:06 smithi019 ceph-mon[91296]: osdmap e481: 8 total, 8 up, 7 in 2024-09-13T15:10:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:06.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:07.076 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T15:10:07.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:10:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:07 smithi019 ceph-mon[91296]: pgmap v1685: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:07 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:10:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:07 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:10:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:07 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:07.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:07 smithi003 ceph-mon[104007]: pgmap v1685: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:07.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:07 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:10:07.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:07 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:10:07.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:07 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:07.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:07 smithi003 ceph-mon[107278]: pgmap v1685: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:07.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:07 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:10:07.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:07 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:10:07.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:07 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:08.523 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:10:08.523 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T15:10:08.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T15:10:08.523 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:10:08.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.523 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:203845 13.0 e394/362)", 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:16.252604+0000", 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 352.26053842099998, 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18934431500000001, 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.252604+0000", 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.252604+0000", 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.252606+0000", 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6530000000000001e-06 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.252617+0000", 2024-09-13T15:10:08.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1365999999999999e-05 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.252641+0000", 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3785999999999999e-05 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.252652+0000", 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0477e-05 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.441921+0000", 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18926980400000001 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.441949+0000", 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7229e-05 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:203840 13.4 e394/362 13:2a6c846f:::benchmark_data_smithi003_293363_object84628:head v 394'12916, mlcod=394'12913)", 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:16.255310+0000", 2024-09-13T15:10:08.527 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 352.257832592, 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23152400200000001, 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.255310+0000", 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.255310+0000", 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.255315+0000", 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9649999999999999e-06 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.255328+0000", 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2575000000000001e-05 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.255343+0000", 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5585000000000001e-05 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.255354+0000", 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1209000000000001e-05 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.255444+0000", 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9590000000000001e-05 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.255468+0000", 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4294000000000002e-05 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.486834+0000", 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23136578399999999 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.531 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:203858 13.d e395/362 13:b233ecd9:::benchmark_data_smithi003_293363_object84646:head v 395'12867, mlcod=395'12865)", 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:16.661896+0000", 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 351.85124664199998, 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.162828103, 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.661896+0000", 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.661896+0000", 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.661898+0000", 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5960000000000001e-06 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.661908+0000", 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0386e-05 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.661922+0000", 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4222e-05 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.661931+0000", 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6789999999999994e-06 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.661986+0000", 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5198999999999998e-05 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.662004+0000", 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7520000000000002e-05 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.824682+0000", 2024-09-13T15:10:08.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.162677981 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.824724+0000", 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2519999999999999e-05 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:230552 13.c e403/362)", 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:40.431122+0000", 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 328.08202030699999, 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.191484131, 2024-09-13T15:10:08.536 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.431122+0000", 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.431122+0000", 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.431124+0000", 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1060000000000001e-06 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.431135+0000", 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1151999999999999e-05 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.431146+0000", 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1652e-05 2024-09-13T15:10:08.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.431154+0000", 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7859999999999997e-06 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.622589+0000", 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.191434732 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.622607+0000", 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7703e-05 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:230557 13.4 e403/395 13:2562e8b0:::benchmark_data_smithi003_293363_object111345:head v 403'13715, mlcod=403'13713)", 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:40.433465+0000", 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 328.07967804499998, 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19280091199999999, 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.540 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.433465+0000", 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.433465+0000", 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.433466+0000", 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5349999999999999e-06 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.433478+0000", 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1481000000000001e-05 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.433492+0000", 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3926e-05 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.433502+0000", 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9059999999999997e-06 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.622669+0000", 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.189167854 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.622693+0000", 2024-09-13T15:10:08.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3383e-05 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.626218+0000", 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035254169999999999 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.626266+0000", 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7410000000000002e-05 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.544 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:230558 13.c e403/362)", 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:40.440638+0000", 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 328.07250440899998, 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18433087000000001, 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.440638+0000", 2024-09-13T15:10:08.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.440638+0000", 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.440640+0000", 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9230000000000001e-06 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.440652+0000", 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2048999999999999e-05 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.440674+0000", 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1387e-05 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.440684+0000", 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0832999999999999e-05 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.624954+0000", 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18426943200000001 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.624969+0000", 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5245999999999999e-05 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36909.0:230568 13.10 e403/395 13:0d7cc342:::benchmark_data_smithi003_293363_object111356:head v 403'13627, mlcod=403'13625)", 2024-09-13T15:10:08.548 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:40.461287+0000", 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 328.05185572099998, 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.095575526999999993, 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.461287+0000", 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.461287+0000", 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.461288+0000", 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.308e-06 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.461295+0000", 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1770000000000001e-06 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.461317+0000", 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1821e-05 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.461330+0000", 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2247e-05 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.461460+0000", 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130791 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.461495+0000", 2024-09-13T15:10:08.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4677e-05 2024-09-13T15:10:08.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.556535+0000", 2024-09-13T15:10:08.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.095039955999999995 2024-09-13T15:10:08.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.556863+0000", 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032755000000000002 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:95154 14.f e424/416 14:f50e0830:::benchmark_data_smithi003_327099_object95153:head v 424'5908, mlcod=424'5906)", 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:06:32.058468+0000", 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 216.45467519499999, 2024-09-13T15:10:08.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.178584245, 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058468+0000", 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058468+0000", 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058469+0000", 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1400000000000001e-06 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058549+0000", 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9870999999999995e-05 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058562+0000", 2024-09-13T15:10:08.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3930999999999999e-05 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058570+0000", 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4460000000000004e-06 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058643+0000", 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2798000000000003e-05 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058663+0000", 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0636999999999999e-05 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.064026+0000", 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053626790000000004 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.237052+0000", 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17302574300000001 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.557 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:95153 14.8 e424/416 14:14d8c6a7:::benchmark_data_smithi003_327099_object95152:head v 424'6019, mlcod=424'6016)", 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:06:32.058888+0000", 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 216.45425467999999, 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.178155598, 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058888+0000", 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058888+0000", 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058889+0000", 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.085e-06 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058912+0000", 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3337999999999999e-05 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058925+0000", 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2437e-05 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.058932+0000", 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9450000000000004e-06 2024-09-13T15:10:08.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.059006+0000", 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4595e-05 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.059024+0000", 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8048999999999999e-05 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.065810+0000", 2024-09-13T15:10:08.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067859870000000003 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.237044+0000", 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17123316199999999 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:95159 14.2 e424/416 14:4f6c0648:::benchmark_data_smithi003_327099_object95158:head v 424'5939, mlcod=424'5937)", 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:06:32.059584+0000", 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 216.45355895700001, 2024-09-13T15:10:08.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.177462645, 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.059584+0000", 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.059584+0000", 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.059585+0000", 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8199999999999998e-07 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.059619+0000", 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4406000000000002e-05 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.059630+0000", 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1299e-05 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.059637+0000", 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5420000000000002e-06 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.061742+0000", 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021046459999999999 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.061762+0000", 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0347999999999999e-05 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.236930+0000", 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17516842499999999 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.237046+0000", 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116097 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.566 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.567 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.567 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:95157 14.b e424/416 14:d81b51ca:::benchmark_data_smithi003_327099_object95156:head v 424'5818, mlcod=424'5816)", 2024-09-13T15:10:08.567 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:06:32.065759+0000", 2024-09-13T15:10:08.567 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 216.44738379399999, 2024-09-13T15:10:08.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18075617899999999, 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.065759+0000", 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.065759+0000", 2024-09-13T15:10:08.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.065760+0000", 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3960000000000001e-06 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.065788+0000", 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7427999999999999e-05 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.236883+0000", 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17109521499999999 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.236917+0000", 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4356000000000001e-05 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.237287+0000", 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036926299999999999 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.237306+0000", 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8970999999999998e-05 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.246491+0000", 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0091854250000000005 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:06:32.246515+0000", 2024-09-13T15:10:08.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4125e-05 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:139659 14.3 e437/435 14:cc8301c0:::benchmark_data_smithi003_327099_object27933:head v 437'8778, mlcod=437'8776)", 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:38.568482+0000", 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 149.94466049600001, 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21203757500000001, 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.572 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568482+0000", 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568482+0000", 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568483+0000", 2024-09-13T15:10:08.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5799999999999998e-07 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568488+0000", 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.519e-06 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568501+0000", 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2102e-05 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568508+0000", 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7530000000000001e-06 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568565+0000", 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6906e-05 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568589+0000", 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.34e-05 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.780485+0000", 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21189677600000001 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.780520+0000", 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4360999999999998e-05 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:139657 14.1 e437/432 14:8acffc44:::benchmark_data_smithi003_327099_object27931:head v 437'8699, mlcod=437'8695)", 2024-09-13T15:10:08.576 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:38.568736+0000", 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 149.944406531, 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20929757800000001, 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568736+0000", 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568736+0000", 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568737+0000", 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8999999999999996e-07 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.568742+0000", 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2000000000000002e-06 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.775139+0000", 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.206397258 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.775149+0000", 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5619999999999993e-06 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.775264+0000", 2024-09-13T15:10:08.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011521100000000001 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.775288+0000", 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3949000000000001e-05 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.777973+0000", 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026852550000000001 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.778034+0000", 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0452999999999999e-05 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:139664 14.f e437/432 14:f51c2d25:::benchmark_data_smithi003_327099_object27938:head v 437'8716, mlcod=437'8714)", 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:38.573518+0000", 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 149.93962486500001, 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20674246800000001, 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.581 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.582 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.573518+0000", 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.573518+0000", 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.573519+0000", 2024-09-13T15:10:08.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1850000000000001e-06 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.573528+0000", 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5949999999999999e-06 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.774822+0000", 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.201294054 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.774836+0000", 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4789e-05 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.774905+0000", 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8769000000000005e-05 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.774935+0000", 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9462e-05 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.780170+0000", 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052352930000000002 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.780260+0000", 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0321000000000005e-05 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.586 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:139676 14.2 e437/432 14:4e1bbe06:::benchmark_data_smithi003_327099_object27950:head v 437'8740, mlcod=437'8738)", 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:38.590417+0000", 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 149.92272551900001, 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.189590271, 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.590417+0000", 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.590417+0000", 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.590418+0000", 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.263e-06 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.590428+0000", 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4919999999999999e-06 2024-09-13T15:10:08.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.775182+0000", 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.184754258 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.775191+0000", 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4800000000000001e-06 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.775249+0000", 2024-09-13T15:10:08.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8201000000000001e-05 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.775274+0000", 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5588e-05 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.779971+0000", 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004696169 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:38.780007+0000", 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6820000000000003e-05 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:140539 14.b e437/432 14:d8a0ccba:::benchmark_data_smithi003_327099_object28813:head v 437'8583, mlcod=437'8579)", 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:39.782504+0000", 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 148.73063850599999, 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.068840213999999997, 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.591 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.782504+0000", 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.782504+0000", 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.782504+0000", 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9399999999999999e-07 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.782507+0000", 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6089999999999999e-06 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.782512+0000", 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6369999999999998e-06 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.782515+0000", 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6789999999999998e-06 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.784761+0000", 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002245735 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.784784+0000", 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2538999999999999e-05 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.799145+0000", 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014361807000000001 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:39.851344+0000", 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.052199014000000002 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.595 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:169704 14.3 e447/435 14:c99eee29:::benchmark_data_smithi003_327099_object57978:head v 447'10619, mlcod=447'10617)", 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:08:14.308555+0000", 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.204587268, 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.057993208999999997, 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308555+0000", 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308555+0000", 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308556+0000", 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7599999999999998e-07 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308559+0000", 2024-09-13T15:10:08.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2140000000000001e-06 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308567+0000", 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7749999999999993e-06 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308571+0000", 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3680000000000004e-06 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308642+0000", 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1098000000000003e-05 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308666+0000", 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3312e-05 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.366396+0000", 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.057729950000000002 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.366549+0000", 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015311599999999999 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:169707 14.8 e447/432 14:18973624:::benchmark_data_smithi003_327099_object57981:head v 447'10741, mlcod=447'10739)", 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:08:14.308848+0000", 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.204294678, 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.064303807000000004, 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.600 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308848+0000", 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308848+0000", 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308854+0000", 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6420000000000004e-06 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.308859+0000", 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0259999999999998e-06 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.365051+0000", 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.056192541999999998 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.365066+0000", 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4951e-05 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.365123+0000", 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6990000000000002e-05 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.365150+0000", 2024-09-13T15:10:08.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6835999999999999e-05 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.373107+0000", 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079567909999999995 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:14.373152+0000", 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5028999999999999e-05 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:213078 14.8 e460/432 14:1e37ab36:::benchmark_data_smithi003_327099_object101352:head v 460'13450, mlcod=459'13448)", 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:09:03.925779+0000", 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 64.587363756000002, 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12089955300000001, 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.925779+0000", 2024-09-13T15:10:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.925779+0000", 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.925780+0000", 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3859999999999999e-06 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.925787+0000", 2024-09-13T15:10:08.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0790000000000003e-06 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.925799+0000", 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1918e-05 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.925807+0000", 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4959999999999999e-06 2024-09-13T15:10:08.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.925889+0000", 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2581000000000001e-05 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.925914+0000", 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4341999999999998e-05 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:04.046515+0000", 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.120600864 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:04.046679+0000", 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016388700000000001 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:213097 14.8 e460/432 14:13bb5124:::benchmark_data_smithi003_327099_object101371:head v 460'13451, mlcod=459'13448)", 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:09:03.945322+0000", 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 64.567820604999994, 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10420533799999999, 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:08.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.945322+0000", 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.945322+0000", 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:08.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.945323+0000", 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.395e-06 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:03.945333+0000", 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5349999999999993e-06 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:04.038325+0000", 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.09299172 2024-09-13T15:10:08.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:04.038338+0000", 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2945999999999999e-05 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:04.041147+0000", 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028088050000000002 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:08.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:04.041177+0000", 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0056000000000002e-05 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:04.049481+0000", 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0083041290000000004 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:04.049527+0000", 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6752000000000002e-05 2024-09-13T15:10:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.618 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:08.618 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:08.618 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:10:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:09 smithi019 ceph-mon[91296]: pgmap v1686: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:09 smithi003 ceph-mon[104007]: pgmap v1686: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:09 smithi003 ceph-mon[107278]: pgmap v1686: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:09.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:10:10.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:10:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:10:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:10:11.350 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:10:11.476 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:10:11.595 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:11 smithi019 ceph-mon[91296]: pgmap v1687: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:11.610 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:11 smithi003 ceph-mon[107278]: pgmap v1687: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:11.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:11 smithi003 ceph-mon[104007]: pgmap v1687: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:12.307 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1687,"stamp":"2024-09-13T15:10:10.274829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":989337,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3159,"ondisk_log_size":3159,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9813456,"kb_used_data":47356,"kb_used_omap":23,"kb_used_meta":9765992,"kb_avail":646402608,"statfs":{"total":671965249536,"available":661916270592,"internally_reserved":0,"allocated":48492544,"data_stored":28631280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24286,"internal_metadata":10000376098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002550"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":225,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368246+0000","last_change":"2024-09-13T15:07:34.471107+0000","last_active":"2024-09-13T15:10:05.368246+0000","last_peered":"2024-09-13T15:10:05.368246+0000","last_clean":"2024-09-13T15:10:05.368246+0000","last_became_active":"2024-09-13T15:07:34.470229+0000","last_became_peered":"2024-09-13T15:07:34.470229+0000","last_unstale":"2024-09-13T15:10:05.368246+0000","last_undegraded":"2024-09-13T15:10:05.368246+0000","last_fullsized":"2024-09-13T15:10:05.368246+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:48.428315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:48.428315+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:08.746827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":231,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313324+0000","last_change":"2024-09-13T15:02:19.429607+0000","last_active":"2024-09-13T15:10:06.313324+0000","last_peered":"2024-09-13T15:10:06.313324+0000","last_clean":"2024-09-13T15:10:06.313324+0000","last_became_active":"2024-09-13T15:02:19.418793+0000","last_became_peered":"2024-09-13T15:02:19.418793+0000","last_unstale":"2024-09-13T15:10:06.313324+0000","last_undegraded":"2024-09-13T15:10:06.313324+0000","last_fullsized":"2024-09-13T15:10:06.313324+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:03.520208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":65,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.812796+0000","last_change":"2024-09-13T15:09:31.313952+0000","last_active":"2024-09-13T15:09:57.812796+0000","last_peered":"2024-09-13T15:09:57.812796+0000","last_clean":"2024-09-13T15:09:57.812796+0000","last_became_active":"2024-09-13T15:09:31.313402+0000","last_became_peered":"2024-09-13T15:09:31.313402+0000","last_unstale":"2024-09-13T15:09:57.812796+0000","last_undegraded":"2024-09-13T15:09:57.812796+0000","last_fullsized":"2024-09-13T15:09:57.812796+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:08.823474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":189,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313599+0000","last_change":"2024-09-13T15:07:14.303688+0000","last_active":"2024-09-13T15:09:58.313599+0000","last_peered":"2024-09-13T15:09:58.313599+0000","last_clean":"2024-09-13T15:09:58.313599+0000","last_became_active":"2024-09-13T15:07:14.303310+0000","last_became_peered":"2024-09-13T15:07:14.303310+0000","last_unstale":"2024-09-13T15:09:58.313599+0000","last_undegraded":"2024-09-13T15:09:58.313599+0000","last_fullsized":"2024-09-13T15:09:58.313599+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:26.066822+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":197,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368609+0000","last_change":"2024-09-13T15:03:46.871160+0000","last_active":"2024-09-13T15:10:05.368609+0000","last_peered":"2024-09-13T15:10:05.368609+0000","last_clean":"2024-09-13T15:10:05.368609+0000","last_became_active":"2024-09-13T15:03:46.870825+0000","last_became_peered":"2024-09-13T15:03:46.870825+0000","last_unstale":"2024-09-13T15:10:05.368609+0000","last_undegraded":"2024-09-13T15:10:05.368609+0000","last_fullsized":"2024-09-13T15:10:05.368609+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:54.702996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367678+0000","last_change":"2024-09-13T15:03:46.873412+0000","last_active":"2024-09-13T15:10:05.367678+0000","last_peered":"2024-09-13T15:10:05.367678+0000","last_clean":"2024-09-13T15:10:05.367678+0000","last_became_active":"2024-09-13T15:03:46.873094+0000","last_became_peered":"2024-09-13T15:03:46.873094+0000","last_unstale":"2024-09-13T15:10:05.367678+0000","last_undegraded":"2024-09-13T15:10:05.367678+0000","last_fullsized":"2024-09-13T15:10:05.367678+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:45.283467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":59,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813701+0000","last_change":"2024-09-13T15:07:34.593081+0000","last_active":"2024-09-13T15:09:57.813701+0000","last_peered":"2024-09-13T15:09:57.813701+0000","last_clean":"2024-09-13T15:09:57.813701+0000","last_became_active":"2024-09-13T15:07:34.471710+0000","last_became_peered":"2024-09-13T15:07:34.471710+0000","last_unstale":"2024-09-13T15:09:57.813701+0000","last_undegraded":"2024-09-13T15:09:57.813701+0000","last_fullsized":"2024-09-13T15:09:57.813701+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:26.250353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813215+0000","last_change":"2024-09-13T15:07:34.580859+0000","last_active":"2024-09-13T15:09:57.813215+0000","last_peered":"2024-09-13T15:09:57.813215+0000","last_clean":"2024-09-13T15:09:57.813215+0000","last_became_active":"2024-09-13T15:07:34.580521+0000","last_became_peered":"2024-09-13T15:07:34.580521+0000","last_unstale":"2024-09-13T15:09:57.813215+0000","last_undegraded":"2024-09-13T15:09:57.813215+0000","last_fullsized":"2024-09-13T15:09:57.813215+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:01.252401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":205,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368555+0000","last_change":"2024-09-13T15:03:47.469254+0000","last_active":"2024-09-13T15:10:05.368555+0000","last_peered":"2024-09-13T15:10:05.368555+0000","last_clean":"2024-09-13T15:10:05.368555+0000","last_became_active":"2024-09-13T15:03:46.854242+0000","last_became_peered":"2024-09-13T15:03:46.854242+0000","last_unstale":"2024-09-13T15:10:05.368555+0000","last_undegraded":"2024-09-13T15:10:05.368555+0000","last_fullsized":"2024-09-13T15:10:05.368555+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:47.469211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:47.469211+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:59.563316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":59,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813298+0000","last_change":"2024-09-13T15:07:34.575982+0000","last_active":"2024-09-13T15:09:57.813298+0000","last_peered":"2024-09-13T15:09:57.813298+0000","last_clean":"2024-09-13T15:09:57.813298+0000","last_became_active":"2024-09-13T15:07:34.470622+0000","last_became_peered":"2024-09-13T15:07:34.470622+0000","last_unstale":"2024-09-13T15:09:57.813298+0000","last_undegraded":"2024-09-13T15:09:57.813298+0000","last_fullsized":"2024-09-13T15:09:57.813298+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:10.751863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813488+0000","last_change":"2024-09-13T15:07:34.459131+0000","last_active":"2024-09-13T15:09:57.813488+0000","last_peered":"2024-09-13T15:09:57.813488+0000","last_clean":"2024-09-13T15:09:57.813488+0000","last_became_active":"2024-09-13T15:07:34.457225+0000","last_became_peered":"2024-09-13T15:07:34.457225+0000","last_unstale":"2024-09-13T15:09:57.813488+0000","last_undegraded":"2024-09-13T15:09:57.813488+0000","last_fullsized":"2024-09-13T15:09:57.813488+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:57:25.961911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367769+0000","last_change":"2024-09-13T15:03:46.864596+0000","last_active":"2024-09-13T15:10:05.367769+0000","last_peered":"2024-09-13T15:10:05.367769+0000","last_clean":"2024-09-13T15:10:05.367769+0000","last_became_active":"2024-09-13T15:03:46.863107+0000","last_became_peered":"2024-09-13T15:03:46.863107+0000","last_unstale":"2024-09-13T15:10:05.367769+0000","last_undegraded":"2024-09-13T15:10:05.367769+0000","last_fullsized":"2024-09-13T15:10:05.367769+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:07.821841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":246,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313206+0000","last_change":"2024-09-13T15:07:34.489594+0000","last_active":"2024-09-13T15:10:06.313206+0000","last_peered":"2024-09-13T15:10:06.313206+0000","last_clean":"2024-09-13T15:10:06.313206+0000","last_became_active":"2024-09-13T15:07:34.488980+0000","last_became_peered":"2024-09-13T15:07:34.488980+0000","last_unstale":"2024-09-13T15:10:06.313206+0000","last_undegraded":"2024-09-13T15:10:06.313206+0000","last_fullsized":"2024-09-13T15:10:06.313206+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:23.320438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":859,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.993216+0000","last_change":"2024-09-13T15:09:31.178585+0000","last_active":"2024-09-13T15:10:06.993216+0000","last_peered":"2024-09-13T15:10:06.993216+0000","last_clean":"2024-09-13T15:10:06.993216+0000","last_became_active":"2024-09-13T15:09:31.178256+0000","last_became_peered":"2024-09-13T15:09:31.178256+0000","last_unstale":"2024-09-13T15:10:06.993216+0000","last_undegraded":"2024-09-13T15:10:06.993216+0000","last_fullsized":"2024-09-13T15:10:06.993216+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:31.137253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":176,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053319+0000","last_change":"2024-09-13T14:58:09.011981+0000","last_active":"2024-09-13T15:09:58.053319+0000","last_peered":"2024-09-13T15:09:58.053319+0000","last_clean":"2024-09-13T15:09:58.053319+0000","last_became_active":"2024-09-13T14:57:53.357674+0000","last_became_peered":"2024-09-13T14:57:53.357674+0000","last_unstale":"2024-09-13T15:09:58.053319+0000","last_undegraded":"2024-09-13T15:09:58.053319+0000","last_fullsized":"2024-09-13T15:09:58.053319+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.701912+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813057+0000","last_change":"2024-09-13T15:07:34.467512+0000","last_active":"2024-09-13T15:09:57.813057+0000","last_peered":"2024-09-13T15:09:57.813057+0000","last_clean":"2024-09-13T15:09:57.813057+0000","last_became_active":"2024-09-13T15:07:34.467061+0000","last_became_peered":"2024-09-13T15:07:34.467061+0000","last_unstale":"2024-09-13T15:09:57.813057+0000","last_undegraded":"2024-09-13T15:09:57.813057+0000","last_fullsized":"2024-09-13T15:09:57.813057+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:43.721513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":217,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368236+0000","last_change":"2024-09-13T15:07:34.470666+0000","last_active":"2024-09-13T15:10:05.368236+0000","last_peered":"2024-09-13T15:10:05.368236+0000","last_clean":"2024-09-13T15:10:05.368236+0000","last_became_active":"2024-09-13T15:07:34.470346+0000","last_became_peered":"2024-09-13T15:07:34.470346+0000","last_unstale":"2024-09-13T15:10:05.368236+0000","last_undegraded":"2024-09-13T15:10:05.368236+0000","last_fullsized":"2024-09-13T15:10:05.368236+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:21.848356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":212,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053642+0000","last_change":"2024-09-13T15:07:34.492713+0000","last_active":"2024-09-13T15:09:58.053642+0000","last_peered":"2024-09-13T15:09:58.053642+0000","last_clean":"2024-09-13T15:09:58.053642+0000","last_became_active":"2024-09-13T15:07:34.492318+0000","last_became_peered":"2024-09-13T15:07:34.492318+0000","last_unstale":"2024-09-13T15:09:58.053642+0000","last_undegraded":"2024-09-13T15:09:58.053642+0000","last_fullsized":"2024-09-13T15:09:58.053642+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:02.258576+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":509,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367371+0000","last_change":"2024-09-13T15:07:34.452596+0000","last_active":"2024-09-13T15:10:05.367371+0000","last_peered":"2024-09-13T15:10:05.367371+0000","last_clean":"2024-09-13T15:10:05.367371+0000","last_became_active":"2024-09-13T15:07:34.452257+0000","last_became_peered":"2024-09-13T15:07:34.452257+0000","last_unstale":"2024-09-13T15:10:05.367371+0000","last_undegraded":"2024-09-13T15:10:05.367371+0000","last_fullsized":"2024-09-13T15:10:05.367371+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:05:40.025880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":239,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313177+0000","last_change":"2024-09-13T15:03:46.864512+0000","last_active":"2024-09-13T15:10:06.313177+0000","last_peered":"2024-09-13T15:10:06.313177+0000","last_clean":"2024-09-13T15:10:06.313177+0000","last_became_active":"2024-09-13T15:03:46.864120+0000","last_became_peered":"2024-09-13T15:03:46.864120+0000","last_unstale":"2024-09-13T15:10:06.313177+0000","last_undegraded":"2024-09-13T15:10:06.313177+0000","last_fullsized":"2024-09-13T15:10:06.313177+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:36.450217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813062+0000","last_change":"2024-09-13T15:07:34.478273+0000","last_active":"2024-09-13T15:09:57.813062+0000","last_peered":"2024-09-13T15:09:57.813062+0000","last_clean":"2024-09-13T15:09:57.813062+0000","last_became_active":"2024-09-13T15:07:34.477980+0000","last_became_peered":"2024-09-13T15:07:34.477980+0000","last_unstale":"2024-09-13T15:09:57.813062+0000","last_undegraded":"2024-09-13T15:09:57.813062+0000","last_fullsized":"2024-09-13T15:09:57.813062+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:06:47.046963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:06:47.046963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:53.376898+0000","scrub_duration":0.018211127000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":203,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367209+0000","last_change":"2024-09-13T15:07:14.305884+0000","last_active":"2024-09-13T15:10:05.367209+0000","last_peered":"2024-09-13T15:10:05.367209+0000","last_clean":"2024-09-13T15:10:05.367209+0000","last_became_active":"2024-09-13T15:07:14.303053+0000","last_became_peered":"2024-09-13T15:07:14.303053+0000","last_unstale":"2024-09-13T15:10:05.367209+0000","last_undegraded":"2024-09-13T15:10:05.367209+0000","last_fullsized":"2024-09-13T15:10:05.367209+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:56.705465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":473,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313825+0000","last_change":"2024-09-13T15:07:14.323822+0000","last_active":"2024-09-13T15:09:58.313825+0000","last_peered":"2024-09-13T15:09:58.313825+0000","last_clean":"2024-09-13T15:09:58.313825+0000","last_became_active":"2024-09-13T15:07:14.323535+0000","last_became_peered":"2024-09-13T15:07:14.323535+0000","last_unstale":"2024-09-13T15:09:58.313825+0000","last_undegraded":"2024-09-13T15:09:58.313825+0000","last_fullsized":"2024-09-13T15:09:58.313825+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:02.779356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":215,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314274+0000","last_change":"2024-09-13T15:07:14.307345+0000","last_active":"2024-09-13T15:09:58.314274+0000","last_peered":"2024-09-13T15:09:58.314274+0000","last_clean":"2024-09-13T15:09:58.314274+0000","last_became_active":"2024-09-13T15:07:14.306790+0000","last_became_peered":"2024-09-13T15:07:14.306790+0000","last_unstale":"2024-09-13T15:09:58.314274+0000","last_undegraded":"2024-09-13T15:09:58.314274+0000","last_fullsized":"2024-09-13T15:09:58.314274+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:53.006962+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":210,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367317+0000","last_change":"2024-09-13T15:07:34.458459+0000","last_active":"2024-09-13T15:10:05.367317+0000","last_peered":"2024-09-13T15:10:05.367317+0000","last_clean":"2024-09-13T15:10:05.367317+0000","last_became_active":"2024-09-13T15:07:34.458134+0000","last_became_peered":"2024-09-13T15:07:34.458134+0000","last_unstale":"2024-09-13T15:10:05.367317+0000","last_undegraded":"2024-09-13T15:10:05.367317+0000","last_fullsized":"2024-09-13T15:10:05.367317+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:00.244169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":232,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313316+0000","last_change":"2024-09-13T15:03:46.871342+0000","last_active":"2024-09-13T15:10:06.313316+0000","last_peered":"2024-09-13T15:10:06.313316+0000","last_clean":"2024-09-13T15:10:06.313316+0000","last_became_active":"2024-09-13T15:03:46.863406+0000","last_became_peered":"2024-09-13T15:03:46.863406+0000","last_unstale":"2024-09-13T15:10:06.313316+0000","last_undegraded":"2024-09-13T15:10:06.313316+0000","last_fullsized":"2024-09-13T15:10:06.313316+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:53.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367533+0000","last_change":"2024-09-13T15:03:46.859350+0000","last_active":"2024-09-13T15:10:05.367533+0000","last_peered":"2024-09-13T15:10:05.367533+0000","last_clean":"2024-09-13T15:10:05.367533+0000","last_became_active":"2024-09-13T15:03:46.853202+0000","last_became_peered":"2024-09-13T15:03:46.853202+0000","last_unstale":"2024-09-13T15:10:05.367533+0000","last_undegraded":"2024-09-13T15:10:05.367533+0000","last_fullsized":"2024-09-13T15:10:05.367533+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:42.594537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":172,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053749+0000","last_change":"2024-09-13T15:07:14.309051+0000","last_active":"2024-09-13T15:09:58.053749+0000","last_peered":"2024-09-13T15:09:58.053749+0000","last_clean":"2024-09-13T15:09:58.053749+0000","last_became_active":"2024-09-13T15:07:14.308645+0000","last_became_peered":"2024-09-13T15:07:14.308645+0000","last_unstale":"2024-09-13T15:09:58.053749+0000","last_undegraded":"2024-09-13T15:09:58.053749+0000","last_fullsized":"2024-09-13T15:09:58.053749+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:04:05.335587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":554,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.993071+0000","last_change":"2024-09-13T15:07:34.456969+0000","last_active":"2024-09-13T15:10:06.993071+0000","last_peered":"2024-09-13T15:10:06.993071+0000","last_clean":"2024-09-13T15:10:06.993071+0000","last_became_active":"2024-09-13T15:07:34.456596+0000","last_became_peered":"2024-09-13T15:07:34.456596+0000","last_unstale":"2024-09-13T15:10:06.993071+0000","last_undegraded":"2024-09-13T15:10:06.993071+0000","last_fullsized":"2024-09-13T15:10:06.993071+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:19.319199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":182,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053971+0000","last_change":"2024-09-13T15:02:19.406264+0000","last_active":"2024-09-13T15:09:58.053971+0000","last_peered":"2024-09-13T15:09:58.053971+0000","last_clean":"2024-09-13T15:09:58.053971+0000","last_became_active":"2024-09-13T15:02:19.405876+0000","last_became_peered":"2024-09-13T15:02:19.405876+0000","last_unstale":"2024-09-13T15:09:58.053971+0000","last_undegraded":"2024-09-13T15:09:58.053971+0000","last_fullsized":"2024-09-13T15:09:58.053971+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:19.208307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":308,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313919+0000","last_change":"2024-09-13T15:07:34.622674+0000","last_active":"2024-09-13T15:09:58.313919+0000","last_peered":"2024-09-13T15:09:58.313919+0000","last_clean":"2024-09-13T15:09:58.313919+0000","last_became_active":"2024-09-13T15:07:34.474941+0000","last_became_peered":"2024-09-13T15:07:34.474941+0000","last_unstale":"2024-09-13T15:09:58.313919+0000","last_undegraded":"2024-09-13T15:09:58.313919+0000","last_fullsized":"2024-09-13T15:09:58.313919+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:14.657841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":392,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313411+0000","last_change":"2024-09-13T15:07:14.294018+0000","last_active":"2024-09-13T15:09:58.313411+0000","last_peered":"2024-09-13T15:09:58.313411+0000","last_clean":"2024-09-13T15:09:58.313411+0000","last_became_active":"2024-09-13T15:07:14.293678+0000","last_became_peered":"2024-09-13T15:07:14.293678+0000","last_unstale":"2024-09-13T15:09:58.313411+0000","last_undegraded":"2024-09-13T15:09:58.313411+0000","last_fullsized":"2024-09-13T15:09:58.313411+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:29.584863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":263,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312966+0000","last_change":"2024-09-13T15:07:34.482157+0000","last_active":"2024-09-13T15:10:06.312966+0000","last_peered":"2024-09-13T15:10:06.312966+0000","last_clean":"2024-09-13T15:10:06.312966+0000","last_became_active":"2024-09-13T15:07:34.475831+0000","last_became_peered":"2024-09-13T15:07:34.475831+0000","last_unstale":"2024-09-13T15:10:06.312966+0000","last_undegraded":"2024-09-13T15:10:06.312966+0000","last_fullsized":"2024-09-13T15:10:06.312966+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:08.311777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":65,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813107+0000","last_change":"2024-09-13T15:07:34.603715+0000","last_active":"2024-09-13T15:09:57.813107+0000","last_peered":"2024-09-13T15:09:57.813107+0000","last_clean":"2024-09-13T15:09:57.813107+0000","last_became_active":"2024-09-13T15:07:34.476556+0000","last_became_peered":"2024-09-13T15:07:34.476556+0000","last_unstale":"2024-09-13T15:09:57.813107+0000","last_undegraded":"2024-09-13T15:09:57.813107+0000","last_fullsized":"2024-09-13T15:09:57.813107+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:49:06.015543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813337+0000","last_change":"2024-09-13T15:07:34.473602+0000","last_active":"2024-09-13T15:09:57.813337+0000","last_peered":"2024-09-13T15:09:57.813337+0000","last_clean":"2024-09-13T15:09:57.813337+0000","last_became_active":"2024-09-13T15:07:34.473208+0000","last_became_peered":"2024-09-13T15:07:34.473208+0000","last_unstale":"2024-09-13T15:09:57.813337+0000","last_undegraded":"2024-09-13T15:09:57.813337+0000","last_fullsized":"2024-09-13T15:09:57.813337+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:30.687198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.812785+0000","last_change":"2024-09-13T15:07:34.470575+0000","last_active":"2024-09-13T15:09:57.812785+0000","last_peered":"2024-09-13T15:09:57.812785+0000","last_clean":"2024-09-13T15:09:57.812785+0000","last_became_active":"2024-09-13T15:07:34.470231+0000","last_became_peered":"2024-09-13T15:07:34.470231+0000","last_unstale":"2024-09-13T15:09:57.812785+0000","last_undegraded":"2024-09-13T15:09:57.812785+0000","last_fullsized":"2024-09-13T15:09:57.812785+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:34.618632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":476,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313899+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T15:09:58.313899+0000","last_peered":"2024-09-13T15:09:58.313899+0000","last_clean":"2024-09-13T15:09:58.313899+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T15:09:58.313899+0000","last_undegraded":"2024-09-13T15:09:58.313899+0000","last_fullsized":"2024-09-13T15:09:58.313899+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":262,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313321+0000","last_change":"2024-09-13T15:07:34.605821+0000","last_active":"2024-09-13T15:10:06.313321+0000","last_peered":"2024-09-13T15:10:06.313321+0000","last_clean":"2024-09-13T15:10:06.313321+0000","last_became_active":"2024-09-13T15:07:34.477436+0000","last_became_peered":"2024-09-13T15:07:34.477436+0000","last_unstale":"2024-09-13T15:10:06.313321+0000","last_undegraded":"2024-09-13T15:10:06.313321+0000","last_fullsized":"2024-09-13T15:10:06.313321+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:16.983267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":197,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367976+0000","last_change":"2024-09-13T15:03:46.868597+0000","last_active":"2024-09-13T15:10:05.367976+0000","last_peered":"2024-09-13T15:10:05.367976+0000","last_clean":"2024-09-13T15:10:05.367976+0000","last_became_active":"2024-09-13T15:03:46.866939+0000","last_became_peered":"2024-09-13T15:03:46.866939+0000","last_unstale":"2024-09-13T15:10:05.367976+0000","last_undegraded":"2024-09-13T15:10:05.367976+0000","last_fullsized":"2024-09-13T15:10:05.367976+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.060483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":197,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367115+0000","last_change":"2024-09-13T15:03:46.865877+0000","last_active":"2024-09-13T15:10:05.367115+0000","last_peered":"2024-09-13T15:10:05.367115+0000","last_clean":"2024-09-13T15:10:05.367115+0000","last_became_active":"2024-09-13T15:03:46.865029+0000","last_became_peered":"2024-09-13T15:03:46.865029+0000","last_unstale":"2024-09-13T15:10:05.367115+0000","last_undegraded":"2024-09-13T15:10:05.367115+0000","last_fullsized":"2024-09-13T15:10:05.367115+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:28.871714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":347,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313797+0000","last_change":"2024-09-13T15:07:34.472379+0000","last_active":"2024-09-13T15:09:58.313797+0000","last_peered":"2024-09-13T15:09:58.313797+0000","last_clean":"2024-09-13T15:09:58.313797+0000","last_became_active":"2024-09-13T15:07:34.472027+0000","last_became_peered":"2024-09-13T15:07:34.472027+0000","last_unstale":"2024-09-13T15:09:58.313797+0000","last_undegraded":"2024-09-13T15:09:58.313797+0000","last_fullsized":"2024-09-13T15:09:58.313797+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:23.006884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":200,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368547+0000","last_change":"2024-09-13T15:03:46.879528+0000","last_active":"2024-09-13T15:10:05.368547+0000","last_peered":"2024-09-13T15:10:05.368547+0000","last_clean":"2024-09-13T15:10:05.368547+0000","last_became_active":"2024-09-13T15:03:46.860753+0000","last_became_peered":"2024-09-13T15:03:46.860753+0000","last_unstale":"2024-09-13T15:10:05.368547+0000","last_undegraded":"2024-09-13T15:10:05.368547+0000","last_fullsized":"2024-09-13T15:10:05.368547+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:28.561501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813601+0000","last_change":"2024-09-13T15:07:34.471374+0000","last_active":"2024-09-13T15:09:57.813601+0000","last_peered":"2024-09-13T15:09:57.813601+0000","last_clean":"2024-09-13T15:09:57.813601+0000","last_became_active":"2024-09-13T15:07:34.469293+0000","last_became_peered":"2024-09-13T15:07:34.469293+0000","last_unstale":"2024-09-13T15:09:57.813601+0000","last_undegraded":"2024-09-13T15:09:57.813601+0000","last_fullsized":"2024-09-13T15:09:57.813601+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:06:46.602703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:06:46.602703+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:55.225137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":225,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313197+0000","last_change":"2024-09-13T15:02:19.423088+0000","last_active":"2024-09-13T15:10:06.313197+0000","last_peered":"2024-09-13T15:10:06.313197+0000","last_clean":"2024-09-13T15:10:06.313197+0000","last_became_active":"2024-09-13T15:02:19.422675+0000","last_became_peered":"2024-09-13T15:02:19.422675+0000","last_unstale":"2024-09-13T15:10:06.313197+0000","last_undegraded":"2024-09-13T15:10:06.313197+0000","last_fullsized":"2024-09-13T15:10:06.313197+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:58.764834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":506,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314311+0000","last_change":"2024-09-13T15:07:34.516311+0000","last_active":"2024-09-13T15:09:58.314311+0000","last_peered":"2024-09-13T15:09:58.314311+0000","last_clean":"2024-09-13T15:09:58.314311+0000","last_became_active":"2024-09-13T15:07:34.516030+0000","last_became_peered":"2024-09-13T15:07:34.516030+0000","last_unstale":"2024-09-13T15:09:58.314311+0000","last_undegraded":"2024-09-13T15:09:58.314311+0000","last_fullsized":"2024-09-13T15:09:58.314311+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:28.151366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":812,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.993101+0000","last_change":"2024-09-13T15:07:14.329958+0000","last_active":"2024-09-13T15:10:06.993101+0000","last_peered":"2024-09-13T15:10:06.993101+0000","last_clean":"2024-09-13T15:10:06.993101+0000","last_became_active":"2024-09-13T15:07:14.329538+0000","last_became_peered":"2024-09-13T15:07:14.329538+0000","last_unstale":"2024-09-13T15:10:06.993101+0000","last_undegraded":"2024-09-13T15:10:06.993101+0000","last_fullsized":"2024-09-13T15:10:06.993101+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:58.643216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":477,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313449+0000","last_change":"2024-09-13T15:03:46.869628+0000","last_active":"2024-09-13T15:09:58.313449+0000","last_peered":"2024-09-13T15:09:58.313449+0000","last_clean":"2024-09-13T15:09:58.313449+0000","last_became_active":"2024-09-13T15:03:46.869274+0000","last_became_peered":"2024-09-13T15:03:46.869274+0000","last_unstale":"2024-09-13T15:09:58.313449+0000","last_undegraded":"2024-09-13T15:09:58.313449+0000","last_fullsized":"2024-09-13T15:09:58.313449+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:21.999130+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":192,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053557+0000","last_change":"2024-09-13T15:07:34.451764+0000","last_active":"2024-09-13T15:09:58.053557+0000","last_peered":"2024-09-13T15:09:58.053557+0000","last_clean":"2024-09-13T15:09:58.053557+0000","last_became_active":"2024-09-13T15:07:34.451452+0000","last_became_peered":"2024-09-13T15:07:34.451452+0000","last_unstale":"2024-09-13T15:09:58.053557+0000","last_undegraded":"2024-09-13T15:09:58.053557+0000","last_fullsized":"2024-09-13T15:09:58.053557+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:02.786773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":232,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313022+0000","last_change":"2024-09-13T15:07:14.318434+0000","last_active":"2024-09-13T15:10:06.313022+0000","last_peered":"2024-09-13T15:10:06.313022+0000","last_clean":"2024-09-13T15:10:06.313022+0000","last_became_active":"2024-09-13T15:07:14.318017+0000","last_became_peered":"2024-09-13T15:07:14.318017+0000","last_unstale":"2024-09-13T15:10:06.313022+0000","last_undegraded":"2024-09-13T15:10:06.313022+0000","last_fullsized":"2024-09-13T15:10:06.313022+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:26.290395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":182,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053766+0000","last_change":"2024-09-13T15:03:46.866800+0000","last_active":"2024-09-13T15:09:58.053766+0000","last_peered":"2024-09-13T15:09:58.053766+0000","last_clean":"2024-09-13T15:09:58.053766+0000","last_became_active":"2024-09-13T15:03:46.866557+0000","last_became_peered":"2024-09-13T15:03:46.866557+0000","last_unstale":"2024-09-13T15:09:58.053766+0000","last_undegraded":"2024-09-13T15:09:58.053766+0000","last_fullsized":"2024-09-13T15:09:58.053766+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:28.181919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":477,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313730+0000","last_change":"2024-09-13T15:03:46.851975+0000","last_active":"2024-09-13T15:09:58.313730+0000","last_peered":"2024-09-13T15:09:58.313730+0000","last_clean":"2024-09-13T15:09:58.313730+0000","last_became_active":"2024-09-13T15:03:46.851616+0000","last_became_peered":"2024-09-13T15:03:46.851616+0000","last_unstale":"2024-09-13T15:09:58.313730+0000","last_undegraded":"2024-09-13T15:09:58.313730+0000","last_fullsized":"2024-09-13T15:09:58.313730+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:01.819924+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":724,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.993224+0000","last_change":"2024-09-13T15:07:14.312909+0000","last_active":"2024-09-13T15:10:06.993224+0000","last_peered":"2024-09-13T15:10:06.993224+0000","last_clean":"2024-09-13T15:10:06.993224+0000","last_became_active":"2024-09-13T15:07:14.312606+0000","last_became_peered":"2024-09-13T15:07:14.312606+0000","last_unstale":"2024-09-13T15:10:06.993224+0000","last_undegraded":"2024-09-13T15:10:06.993224+0000","last_fullsized":"2024-09-13T15:10:06.993224+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:41.599239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813330+0000","last_change":"2024-09-13T15:07:34.598991+0000","last_active":"2024-09-13T15:09:57.813330+0000","last_peered":"2024-09-13T15:09:57.813330+0000","last_clean":"2024-09-13T15:09:57.813330+0000","last_became_active":"2024-09-13T15:07:34.598343+0000","last_became_peered":"2024-09-13T15:07:34.598343+0000","last_unstale":"2024-09-13T15:09:57.813330+0000","last_undegraded":"2024-09-13T15:09:57.813330+0000","last_fullsized":"2024-09-13T15:09:57.813330+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:31.371179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":81,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313606+0000","last_change":"2024-09-13T15:07:14.345712+0000","last_active":"2024-09-13T15:09:58.313606+0000","last_peered":"2024-09-13T15:09:58.313606+0000","last_clean":"2024-09-13T15:09:58.313606+0000","last_became_active":"2024-09-13T15:07:14.302650+0000","last_became_peered":"2024-09-13T15:07:14.302650+0000","last_unstale":"2024-09-13T15:09:58.313606+0000","last_undegraded":"2024-09-13T15:09:58.313606+0000","last_fullsized":"2024-09-13T15:09:58.313606+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:50.212668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":225,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312674+0000","last_change":"2024-09-13T15:02:19.420193+0000","last_active":"2024-09-13T15:10:06.312674+0000","last_peered":"2024-09-13T15:10:06.312674+0000","last_clean":"2024-09-13T15:10:06.312674+0000","last_became_active":"2024-09-13T15:02:19.419807+0000","last_became_peered":"2024-09-13T15:02:19.419807+0000","last_unstale":"2024-09-13T15:10:06.312674+0000","last_undegraded":"2024-09-13T15:10:06.312674+0000","last_fullsized":"2024-09-13T15:10:06.312674+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:13.929284+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":446,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313143+0000","last_change":"2024-09-13T15:07:14.324008+0000","last_active":"2024-09-13T15:10:06.313143+0000","last_peered":"2024-09-13T15:10:06.313143+0000","last_clean":"2024-09-13T15:10:06.313143+0000","last_became_active":"2024-09-13T15:07:14.323650+0000","last_became_peered":"2024-09-13T15:07:14.323650+0000","last_unstale":"2024-09-13T15:10:06.313143+0000","last_undegraded":"2024-09-13T15:10:06.313143+0000","last_fullsized":"2024-09-13T15:10:06.313143+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:59.889970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":651,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.992857+0000","last_change":"2024-09-13T15:07:14.301469+0000","last_active":"2024-09-13T15:10:06.992857+0000","last_peered":"2024-09-13T15:10:06.992857+0000","last_clean":"2024-09-13T15:10:06.992857+0000","last_became_active":"2024-09-13T15:07:14.300482+0000","last_became_peered":"2024-09-13T15:07:14.300482+0000","last_unstale":"2024-09-13T15:10:06.992857+0000","last_undegraded":"2024-09-13T15:10:06.992857+0000","last_fullsized":"2024-09-13T15:10:06.992857+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:30.889776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":862,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.992916+0000","last_change":"2024-09-13T15:03:46.871298+0000","last_active":"2024-09-13T15:10:06.992916+0000","last_peered":"2024-09-13T15:10:06.992916+0000","last_clean":"2024-09-13T15:10:06.992916+0000","last_became_active":"2024-09-13T15:03:46.870966+0000","last_became_peered":"2024-09-13T15:03:46.870966+0000","last_unstale":"2024-09-13T15:10:06.992916+0000","last_undegraded":"2024-09-13T15:10:06.992916+0000","last_fullsized":"2024-09-13T15:10:06.992916+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:22.463434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":278,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312796+0000","last_change":"2024-09-13T15:07:34.464473+0000","last_active":"2024-09-13T15:09:58.312796+0000","last_peered":"2024-09-13T15:09:58.312796+0000","last_clean":"2024-09-13T15:09:58.312796+0000","last_became_active":"2024-09-13T15:07:34.464122+0000","last_became_peered":"2024-09-13T15:07:34.464122+0000","last_unstale":"2024-09-13T15:09:58.312796+0000","last_undegraded":"2024-09-13T15:09:58.312796+0000","last_fullsized":"2024-09-13T15:09:58.312796+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:55.352605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":232,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313225+0000","last_change":"2024-09-13T15:07:14.326411+0000","last_active":"2024-09-13T15:10:06.313225+0000","last_peered":"2024-09-13T15:10:06.313225+0000","last_clean":"2024-09-13T15:10:06.313225+0000","last_became_active":"2024-09-13T15:07:14.326053+0000","last_became_peered":"2024-09-13T15:07:14.326053+0000","last_unstale":"2024-09-13T15:10:06.313225+0000","last_undegraded":"2024-09-13T15:10:06.313225+0000","last_fullsized":"2024-09-13T15:10:06.313225+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:59.937117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":253,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312626+0000","last_change":"2024-09-13T15:07:34.460652+0000","last_active":"2024-09-13T15:10:06.312626+0000","last_peered":"2024-09-13T15:10:06.312626+0000","last_clean":"2024-09-13T15:10:06.312626+0000","last_became_active":"2024-09-13T15:07:34.458909+0000","last_became_peered":"2024-09-13T15:07:34.458909+0000","last_unstale":"2024-09-13T15:10:06.312626+0000","last_undegraded":"2024-09-13T15:10:06.312626+0000","last_fullsized":"2024-09-13T15:10:06.312626+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:05.444737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312648+0000","last_change":"2024-09-13T15:07:14.312738+0000","last_active":"2024-09-13T15:09:58.312648+0000","last_peered":"2024-09-13T15:09:58.312648+0000","last_clean":"2024-09-13T15:09:58.312648+0000","last_became_active":"2024-09-13T15:07:14.312418+0000","last_became_peered":"2024-09-13T15:07:14.312418+0000","last_unstale":"2024-09-13T15:09:58.312648+0000","last_undegraded":"2024-09-13T15:09:58.312648+0000","last_fullsized":"2024-09-13T15:09:58.312648+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:45.706331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":447,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312719+0000","last_change":"2024-09-13T15:07:34.458019+0000","last_active":"2024-09-13T15:09:58.312719+0000","last_peered":"2024-09-13T15:09:58.312719+0000","last_clean":"2024-09-13T15:09:58.312719+0000","last_became_active":"2024-09-13T15:07:34.457720+0000","last_became_peered":"2024-09-13T15:07:34.457720+0000","last_unstale":"2024-09-13T15:09:58.312719+0000","last_undegraded":"2024-09-13T15:09:58.312719+0000","last_fullsized":"2024-09-13T15:09:58.312719+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:44.330131+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":446,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313559+0000","last_change":"2024-09-13T15:07:34.462911+0000","last_active":"2024-09-13T15:09:58.313559+0000","last_peered":"2024-09-13T15:09:58.313559+0000","last_clean":"2024-09-13T15:09:58.313559+0000","last_became_active":"2024-09-13T15:07:34.462577+0000","last_became_peered":"2024-09-13T15:07:34.462577+0000","last_unstale":"2024-09-13T15:09:58.313559+0000","last_undegraded":"2024-09-13T15:09:58.313559+0000","last_fullsized":"2024-09-13T15:09:58.313559+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:51.431494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":197,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367703+0000","last_change":"2024-09-13T15:03:46.875410+0000","last_active":"2024-09-13T15:10:05.367703+0000","last_peered":"2024-09-13T15:10:05.367703+0000","last_clean":"2024-09-13T15:10:05.367703+0000","last_became_active":"2024-09-13T15:03:46.875103+0000","last_became_peered":"2024-09-13T15:03:46.875103+0000","last_unstale":"2024-09-13T15:10:05.367703+0000","last_undegraded":"2024-09-13T15:10:05.367703+0000","last_fullsized":"2024-09-13T15:10:05.367703+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:53.941854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":166,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053623+0000","last_change":"2024-09-13T14:57:53.360296+0000","last_active":"2024-09-13T15:09:58.053623+0000","last_peered":"2024-09-13T15:09:58.053623+0000","last_clean":"2024-09-13T15:09:58.053623+0000","last_became_active":"2024-09-13T14:57:53.347710+0000","last_became_peered":"2024-09-13T14:57:53.347710+0000","last_unstale":"2024-09-13T15:09:58.053623+0000","last_undegraded":"2024-09-13T15:09:58.053623+0000","last_fullsized":"2024-09-13T15:09:58.053623+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:43.099884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":258,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312384+0000","last_change":"2024-09-13T15:02:19.407022+0000","last_active":"2024-09-13T15:09:58.312384+0000","last_peered":"2024-09-13T15:09:58.312384+0000","last_clean":"2024-09-13T15:09:58.312384+0000","last_became_active":"2024-09-13T15:02:19.406620+0000","last_became_peered":"2024-09-13T15:02:19.406620+0000","last_unstale":"2024-09-13T15:09:58.312384+0000","last_undegraded":"2024-09-13T15:09:58.312384+0000","last_fullsized":"2024-09-13T15:09:58.312384+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:37.100188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":510,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368424+0000","last_change":"2024-09-13T15:07:34.484543+0000","last_active":"2024-09-13T15:10:05.368424+0000","last_peered":"2024-09-13T15:10:05.368424+0000","last_clean":"2024-09-13T15:10:05.368424+0000","last_became_active":"2024-09-13T15:07:34.484218+0000","last_became_peered":"2024-09-13T15:07:34.484218+0000","last_unstale":"2024-09-13T15:10:05.368424+0000","last_undegraded":"2024-09-13T15:10:05.368424+0000","last_fullsized":"2024-09-13T15:10:05.368424+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:12.280906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":210,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367267+0000","last_change":"2024-09-13T15:07:34.460538+0000","last_active":"2024-09-13T15:10:05.367267+0000","last_peered":"2024-09-13T15:10:05.367267+0000","last_clean":"2024-09-13T15:10:05.367267+0000","last_became_active":"2024-09-13T15:07:34.460274+0000","last_became_peered":"2024-09-13T15:07:34.460274+0000","last_unstale":"2024-09-13T15:10:05.367267+0000","last_undegraded":"2024-09-13T15:10:05.367267+0000","last_fullsized":"2024-09-13T15:10:05.367267+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:19.022732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":330,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.992692+0000","last_change":"2024-09-13T14:58:28.428608+0000","last_active":"2024-09-13T15:10:06.992692+0000","last_peered":"2024-09-13T15:10:06.992692+0000","last_clean":"2024-09-13T15:10:06.992692+0000","last_became_active":"2024-09-13T14:58:28.428282+0000","last_became_peered":"2024-09-13T14:58:28.428282+0000","last_unstale":"2024-09-13T15:10:06.992692+0000","last_undegraded":"2024-09-13T15:10:06.992692+0000","last_fullsized":"2024-09-13T15:10:06.992692+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.603606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.603606+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:10.695417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813644+0000","last_change":"2024-09-13T15:07:34.475401+0000","last_active":"2024-09-13T15:09:57.813644+0000","last_peered":"2024-09-13T15:09:57.813644+0000","last_clean":"2024-09-13T15:09:57.813644+0000","last_became_active":"2024-09-13T15:07:34.469994+0000","last_became_peered":"2024-09-13T15:07:34.469994+0000","last_unstale":"2024-09-13T15:09:57.813644+0000","last_undegraded":"2024-09-13T15:09:57.813644+0000","last_fullsized":"2024-09-13T15:09:57.813644+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:24.460236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.812831+0000","last_change":"2024-09-13T15:07:34.478488+0000","last_active":"2024-09-13T15:09:57.812831+0000","last_peered":"2024-09-13T15:09:57.812831+0000","last_clean":"2024-09-13T15:09:57.812831+0000","last_became_active":"2024-09-13T15:07:34.477918+0000","last_became_peered":"2024-09-13T15:07:34.477918+0000","last_unstale":"2024-09-13T15:09:57.812831+0000","last_undegraded":"2024-09-13T15:09:57.812831+0000","last_fullsized":"2024-09-13T15:09:57.812831+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:59.742232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":249,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313070+0000","last_change":"2024-09-13T15:07:14.302370+0000","last_active":"2024-09-13T15:10:06.313070+0000","last_peered":"2024-09-13T15:10:06.313070+0000","last_clean":"2024-09-13T15:10:06.313070+0000","last_became_active":"2024-09-13T15:07:14.301636+0000","last_became_peered":"2024-09-13T15:07:14.301636+0000","last_unstale":"2024-09-13T15:10:06.313070+0000","last_undegraded":"2024-09-13T15:10:06.313070+0000","last_fullsized":"2024-09-13T15:10:06.313070+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:02:36.128767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:02:36.128767+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:34.619009+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813582+0000","last_change":"2024-09-13T15:07:34.472070+0000","last_active":"2024-09-13T15:09:57.813582+0000","last_peered":"2024-09-13T15:09:57.813582+0000","last_clean":"2024-09-13T15:09:57.813582+0000","last_became_active":"2024-09-13T15:07:34.470958+0000","last_became_peered":"2024-09-13T15:07:34.470958+0000","last_unstale":"2024-09-13T15:09:57.813582+0000","last_undegraded":"2024-09-13T15:09:57.813582+0000","last_fullsized":"2024-09-13T15:09:57.813582+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:17.779488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":494,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313505+0000","last_change":"2024-09-13T15:07:34.460856+0000","last_active":"2024-09-13T15:09:58.313505+0000","last_peered":"2024-09-13T15:09:58.313505+0000","last_clean":"2024-09-13T15:09:58.313505+0000","last_became_active":"2024-09-13T15:07:34.460540+0000","last_became_peered":"2024-09-13T15:07:34.460540+0000","last_unstale":"2024-09-13T15:09:58.313505+0000","last_undegraded":"2024-09-13T15:09:58.313505+0000","last_fullsized":"2024-09-13T15:09:58.313505+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:08.451991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":399,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.992762+0000","last_change":"2024-09-13T15:03:46.869405+0000","last_active":"2024-09-13T15:10:06.992762+0000","last_peered":"2024-09-13T15:10:06.992762+0000","last_clean":"2024-09-13T15:10:06.992762+0000","last_became_active":"2024-09-13T15:03:46.868913+0000","last_became_peered":"2024-09-13T15:03:46.868913+0000","last_unstale":"2024-09-13T15:10:06.992762+0000","last_undegraded":"2024-09-13T15:10:06.992762+0000","last_fullsized":"2024-09-13T15:10:06.992762+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:02.098981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813378+0000","last_change":"2024-09-13T15:07:34.473797+0000","last_active":"2024-09-13T15:09:57.813378+0000","last_peered":"2024-09-13T15:09:57.813378+0000","last_clean":"2024-09-13T15:09:57.813378+0000","last_became_active":"2024-09-13T15:07:34.473444+0000","last_became_peered":"2024-09-13T15:07:34.473444+0000","last_unstale":"2024-09-13T15:09:57.813378+0000","last_undegraded":"2024-09-13T15:09:57.813378+0000","last_fullsized":"2024-09-13T15:09:57.813378+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:04.278156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":233,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367941+0000","last_change":"2024-09-13T15:07:34.609901+0000","last_active":"2024-09-13T15:10:05.367941+0000","last_peered":"2024-09-13T15:10:05.367941+0000","last_clean":"2024-09-13T15:10:05.367941+0000","last_became_active":"2024-09-13T15:07:34.490242+0000","last_became_peered":"2024-09-13T15:07:34.490242+0000","last_unstale":"2024-09-13T15:10:05.367941+0000","last_undegraded":"2024-09-13T15:10:05.367941+0000","last_fullsized":"2024-09-13T15:10:05.367941+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:23.303360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":304,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312495+0000","last_change":"2024-09-13T15:03:46.863264+0000","last_active":"2024-09-13T15:09:58.312495+0000","last_peered":"2024-09-13T15:09:58.312495+0000","last_clean":"2024-09-13T15:09:58.312495+0000","last_became_active":"2024-09-13T15:03:46.862850+0000","last_became_peered":"2024-09-13T15:03:46.862850+0000","last_unstale":"2024-09-13T15:09:58.312495+0000","last_undegraded":"2024-09-13T15:09:58.312495+0000","last_fullsized":"2024-09-13T15:09:58.312495+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:57.707268+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":246,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313074+0000","last_change":"2024-09-13T15:07:34.489295+0000","last_active":"2024-09-13T15:10:06.313074+0000","last_peered":"2024-09-13T15:10:06.313074+0000","last_clean":"2024-09-13T15:10:06.313074+0000","last_became_active":"2024-09-13T15:07:34.488579+0000","last_became_peered":"2024-09-13T15:07:34.488579+0000","last_unstale":"2024-09-13T15:10:06.313074+0000","last_undegraded":"2024-09-13T15:10:06.313074+0000","last_fullsized":"2024-09-13T15:10:06.313074+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:36.942324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367847+0000","last_change":"2024-09-13T15:03:46.869406+0000","last_active":"2024-09-13T15:10:05.367847+0000","last_peered":"2024-09-13T15:10:05.367847+0000","last_clean":"2024-09-13T15:10:05.367847+0000","last_became_active":"2024-09-13T15:03:46.866230+0000","last_became_peered":"2024-09-13T15:03:46.866230+0000","last_unstale":"2024-09-13T15:10:05.367847+0000","last_undegraded":"2024-09-13T15:10:05.367847+0000","last_fullsized":"2024-09-13T15:10:05.367847+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:40.914289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":65,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813634+0000","last_change":"2024-09-13T15:08:59.871028+0000","last_active":"2024-09-13T15:09:57.813634+0000","last_peered":"2024-09-13T15:09:57.813634+0000","last_clean":"2024-09-13T15:09:57.813634+0000","last_became_active":"2024-09-13T15:07:34.478006+0000","last_became_peered":"2024-09-13T15:07:34.478006+0000","last_unstale":"2024-09-13T15:09:57.813634+0000","last_undegraded":"2024-09-13T15:09:57.813634+0000","last_fullsized":"2024-09-13T15:09:57.813634+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:08:59.870786+0000","last_deep_scrub":"79'100","last_deep_scrub_stamp":"2024-09-13T15:08:59.870786+0000","last_clean_scrub_stamp":"2024-09-13T15:08:59.870786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:43:55.603202+0000","scrub_duration":0.010767947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":476,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313644+0000","last_change":"2024-09-13T15:07:14.302821+0000","last_active":"2024-09-13T15:09:58.313644+0000","last_peered":"2024-09-13T15:09:58.313644+0000","last_clean":"2024-09-13T15:09:58.313644+0000","last_became_active":"2024-09-13T15:07:14.302553+0000","last_became_peered":"2024-09-13T15:07:14.302553+0000","last_unstale":"2024-09-13T15:09:58.313644+0000","last_undegraded":"2024-09-13T15:09:58.313644+0000","last_fullsized":"2024-09-13T15:09:58.313644+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:32:21.630096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":485,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314257+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:09:58.314257+0000","last_peered":"2024-09-13T15:09:58.314257+0000","last_clean":"2024-09-13T15:09:58.314257+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:09:58.314257+0000","last_undegraded":"2024-09-13T15:09:58.314257+0000","last_fullsized":"2024-09-13T15:09:58.314257+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":457,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313787+0000","last_change":"2024-09-13T15:07:34.463017+0000","last_active":"2024-09-13T15:09:58.313787+0000","last_peered":"2024-09-13T15:09:58.313787+0000","last_clean":"2024-09-13T15:09:58.313787+0000","last_became_active":"2024-09-13T15:07:34.462570+0000","last_became_peered":"2024-09-13T15:07:34.462570+0000","last_unstale":"2024-09-13T15:09:58.313787+0000","last_undegraded":"2024-09-13T15:09:58.313787+0000","last_fullsized":"2024-09-13T15:09:58.313787+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:56.696358+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.812984+0000","last_change":"2024-09-13T15:07:34.471456+0000","last_active":"2024-09-13T15:09:57.812984+0000","last_peered":"2024-09-13T15:09:57.812984+0000","last_clean":"2024-09-13T15:09:57.812984+0000","last_became_active":"2024-09-13T15:07:34.471201+0000","last_became_peered":"2024-09-13T15:07:34.471201+0000","last_unstale":"2024-09-13T15:09:57.812984+0000","last_undegraded":"2024-09-13T15:09:57.812984+0000","last_fullsized":"2024-09-13T15:09:57.812984+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:18:48.572515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":489,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314292+0000","last_change":"2024-09-13T15:07:34.615859+0000","last_active":"2024-09-13T15:09:58.314292+0000","last_peered":"2024-09-13T15:09:58.314292+0000","last_clean":"2024-09-13T15:09:58.314292+0000","last_became_active":"2024-09-13T15:07:34.474503+0000","last_became_peered":"2024-09-13T15:07:34.474503+0000","last_unstale":"2024-09-13T15:09:58.314292+0000","last_undegraded":"2024-09-13T15:09:58.314292+0000","last_fullsized":"2024-09-13T15:09:58.314292+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:29.217608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":200,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368045+0000","last_change":"2024-09-13T15:03:46.882026+0000","last_active":"2024-09-13T15:10:05.368045+0000","last_peered":"2024-09-13T15:10:05.368045+0000","last_clean":"2024-09-13T15:10:05.368045+0000","last_became_active":"2024-09-13T15:03:46.859557+0000","last_became_peered":"2024-09-13T15:03:46.859557+0000","last_unstale":"2024-09-13T15:10:05.368045+0000","last_undegraded":"2024-09-13T15:10:05.368045+0000","last_fullsized":"2024-09-13T15:10:05.368045+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:00.933755+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":162,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053267+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:09:58.053267+0000","last_peered":"2024-09-13T15:09:58.053267+0000","last_clean":"2024-09-13T15:09:58.053267+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:09:58.053267+0000","last_undegraded":"2024-09-13T15:09:58.053267+0000","last_fullsized":"2024-09-13T15:09:58.053267+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":312,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313697+0000","last_change":"2024-09-13T15:07:34.635697+0000","last_active":"2024-09-13T15:09:58.313697+0000","last_peered":"2024-09-13T15:09:58.313697+0000","last_clean":"2024-09-13T15:09:58.313697+0000","last_became_active":"2024-09-13T15:07:34.568238+0000","last_became_peered":"2024-09-13T15:07:34.568238+0000","last_unstale":"2024-09-13T15:09:58.313697+0000","last_undegraded":"2024-09-13T15:09:58.313697+0000","last_fullsized":"2024-09-13T15:09:58.313697+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:55.995331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":298,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314030+0000","last_change":"2024-09-13T15:07:34.475965+0000","last_active":"2024-09-13T15:09:58.314030+0000","last_peered":"2024-09-13T15:09:58.314030+0000","last_clean":"2024-09-13T15:09:58.314030+0000","last_became_active":"2024-09-13T15:07:34.475412+0000","last_became_peered":"2024-09-13T15:07:34.475412+0000","last_unstale":"2024-09-13T15:09:58.314030+0000","last_undegraded":"2024-09-13T15:09:58.314030+0000","last_fullsized":"2024-09-13T15:09:58.314030+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:09:38.928248+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":572,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313929+0000","last_change":"2024-09-13T15:02:19.426511+0000","last_active":"2024-09-13T15:09:58.313929+0000","last_peered":"2024-09-13T15:09:58.313929+0000","last_clean":"2024-09-13T15:09:58.313929+0000","last_became_active":"2024-09-13T15:02:19.391448+0000","last_became_peered":"2024-09-13T15:02:19.391448+0000","last_unstale":"2024-09-13T15:09:58.313929+0000","last_undegraded":"2024-09-13T15:09:58.313929+0000","last_fullsized":"2024-09-13T15:09:58.313929+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:40.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368649+0000","last_change":"2024-09-13T15:03:46.870536+0000","last_active":"2024-09-13T15:10:05.368649+0000","last_peered":"2024-09-13T15:10:05.368649+0000","last_clean":"2024-09-13T15:10:05.368649+0000","last_became_active":"2024-09-13T15:03:46.869736+0000","last_became_peered":"2024-09-13T15:03:46.869736+0000","last_unstale":"2024-09-13T15:10:05.368649+0000","last_undegraded":"2024-09-13T15:10:05.368649+0000","last_fullsized":"2024-09-13T15:10:05.368649+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:45.432825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":172,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053480+0000","last_change":"2024-09-13T15:07:14.305701+0000","last_active":"2024-09-13T15:09:58.053480+0000","last_peered":"2024-09-13T15:09:58.053480+0000","last_clean":"2024-09-13T15:09:58.053480+0000","last_became_active":"2024-09-13T15:07:14.304628+0000","last_became_peered":"2024-09-13T15:07:14.304628+0000","last_unstale":"2024-09-13T15:09:58.053480+0000","last_undegraded":"2024-09-13T15:09:58.053480+0000","last_fullsized":"2024-09-13T15:09:58.053480+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:11.433108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":468,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312830+0000","last_change":"2024-09-13T15:07:34.466871+0000","last_active":"2024-09-13T15:09:58.312830+0000","last_peered":"2024-09-13T15:09:58.312830+0000","last_clean":"2024-09-13T15:09:58.312830+0000","last_became_active":"2024-09-13T15:07:34.466559+0000","last_became_peered":"2024-09-13T15:07:34.466559+0000","last_unstale":"2024-09-13T15:09:58.312830+0000","last_undegraded":"2024-09-13T15:09:58.312830+0000","last_fullsized":"2024-09-13T15:09:58.312830+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:48:24.377353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":239,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313295+0000","last_change":"2024-09-13T15:03:46.864291+0000","last_active":"2024-09-13T15:10:06.313295+0000","last_peered":"2024-09-13T15:10:06.313295+0000","last_clean":"2024-09-13T15:10:06.313295+0000","last_became_active":"2024-09-13T15:03:46.863671+0000","last_became_peered":"2024-09-13T15:03:46.863671+0000","last_unstale":"2024-09-13T15:10:06.313295+0000","last_undegraded":"2024-09-13T15:10:06.313295+0000","last_fullsized":"2024-09-13T15:10:06.313295+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:28.672041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":217,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367811+0000","last_change":"2024-09-13T15:07:34.489714+0000","last_active":"2024-09-13T15:10:05.367811+0000","last_peered":"2024-09-13T15:10:05.367811+0000","last_clean":"2024-09-13T15:10:05.367811+0000","last_became_active":"2024-09-13T15:07:34.489373+0000","last_became_peered":"2024-09-13T15:07:34.489373+0000","last_unstale":"2024-09-13T15:10:05.367811+0000","last_undegraded":"2024-09-13T15:10:05.367811+0000","last_fullsized":"2024-09-13T15:10:05.367811+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:40.264920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":239,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312944+0000","last_change":"2024-09-13T15:03:46.863848+0000","last_active":"2024-09-13T15:10:06.312944+0000","last_peered":"2024-09-13T15:10:06.312944+0000","last_clean":"2024-09-13T15:10:06.312944+0000","last_became_active":"2024-09-13T15:03:46.863523+0000","last_became_peered":"2024-09-13T15:03:46.863523+0000","last_unstale":"2024-09-13T15:10:06.312944+0000","last_undegraded":"2024-09-13T15:10:06.312944+0000","last_fullsized":"2024-09-13T15:10:06.312944+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:39.303427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":359,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314105+0000","last_change":"2024-09-13T15:03:46.860138+0000","last_active":"2024-09-13T15:09:58.314105+0000","last_peered":"2024-09-13T15:09:58.314105+0000","last_clean":"2024-09-13T15:09:58.314105+0000","last_became_active":"2024-09-13T15:03:46.859819+0000","last_became_peered":"2024-09-13T15:03:46.859819+0000","last_unstale":"2024-09-13T15:09:58.314105+0000","last_undegraded":"2024-09-13T15:09:58.314105+0000","last_fullsized":"2024-09-13T15:09:58.314105+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:28.081805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":170,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312561+0000","last_change":"2024-09-13T14:58:28.428896+0000","last_active":"2024-09-13T15:09:58.312561+0000","last_peered":"2024-09-13T15:09:58.312561+0000","last_clean":"2024-09-13T15:09:58.312561+0000","last_became_active":"2024-09-13T14:58:28.428522+0000","last_became_peered":"2024-09-13T14:58:28.428522+0000","last_unstale":"2024-09-13T15:09:58.312561+0000","last_undegraded":"2024-09-13T15:09:58.312561+0000","last_fullsized":"2024-09-13T15:09:58.312561+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:05.418586+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":495,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312994+0000","last_change":"2024-09-13T15:07:34.601477+0000","last_active":"2024-09-13T15:09:58.312994+0000","last_peered":"2024-09-13T15:09:58.312994+0000","last_clean":"2024-09-13T15:09:58.312994+0000","last_became_active":"2024-09-13T15:07:34.464188+0000","last_became_peered":"2024-09-13T15:07:34.464188+0000","last_unstale":"2024-09-13T15:09:58.312994+0000","last_undegraded":"2024-09-13T15:09:58.312994+0000","last_fullsized":"2024-09-13T15:09:58.312994+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:57.481351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":203,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368089+0000","last_change":"2024-09-13T15:07:14.312451+0000","last_active":"2024-09-13T15:10:05.368089+0000","last_peered":"2024-09-13T15:10:05.368089+0000","last_clean":"2024-09-13T15:10:05.368089+0000","last_became_active":"2024-09-13T15:07:14.307807+0000","last_became_peered":"2024-09-13T15:07:14.307807+0000","last_unstale":"2024-09-13T15:10:05.368089+0000","last_undegraded":"2024-09-13T15:10:05.368089+0000","last_fullsized":"2024-09-13T15:10:05.368089+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:56:58.706128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":239,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312882+0000","last_change":"2024-09-13T15:03:46.864604+0000","last_active":"2024-09-13T15:10:06.312882+0000","last_peered":"2024-09-13T15:10:06.312882+0000","last_clean":"2024-09-13T15:10:06.312882+0000","last_became_active":"2024-09-13T15:03:46.864085+0000","last_became_peered":"2024-09-13T15:03:46.864085+0000","last_unstale":"2024-09-13T15:10:06.312882+0000","last_undegraded":"2024-09-13T15:10:06.312882+0000","last_fullsized":"2024-09-13T15:10:06.312882+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:32.524052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":6278,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365405,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1520272,"kb_used_data":6788,"kb_used_omap":1,"kb_used_meta":1513470,"kb_avail":92224880,"statfs":{"total":95995035648,"available":94438277120,"internally_reserved":0,"allocated":6950912,"data_stored":4102507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1549793624},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561635,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499076,"kb_used_data":6840,"kb_used_omap":1,"kb_used_meta":1492222,"kb_avail":92246076,"statfs":{"total":95995035648,"available":94459981824,"internally_reserved":0,"allocated":7004160,"data_stored":4167322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1528035672},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790567,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497340,"kb_used_data":6768,"kb_used_omap":1,"kb_used_meta":1490558,"kb_avail":92247812,"statfs":{"total":95995035648,"available":94461759488,"internally_reserved":0,"allocated":6930432,"data_stored":4097904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1526331736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986795,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1508812,"kb_used_data":6528,"kb_used_omap":1,"kb_used_meta":1502270,"kb_avail":92236340,"statfs":{"total":95995035648,"available":94450012160,"internally_reserved":0,"allocated":6684672,"data_stored":3840014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1538324824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519950,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266808,"kb_used_data":6828,"kb_used_omap":4,"kb_used_meta":1259963,"kb_avail":92478344,"statfs":{"total":95995035648,"available":94697824256,"internally_reserved":0,"allocated":6991872,"data_stored":4162906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":1290202560},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304866,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1264960,"kb_used_data":6772,"kb_used_omap":6,"kb_used_meta":1258169,"kb_avail":92480192,"statfs":{"total":95995035648,"available":94699716608,"internally_reserved":0,"allocated":6934528,"data_stored":4098428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":1288365511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":416611828169,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482286034,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1256188,"kb_used_data":6832,"kb_used_omap":5,"kb_used_meta":1249338,"kb_avail":92488964,"statfs":{"total":95995035648,"available":94708699136,"internally_reserved":0,"allocated":6995968,"data_stored":4162199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":1279322171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:10:12.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:10:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:12 smithi019 ceph-mon[91296]: from='client.47552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:10:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:12 smithi003 ceph-mon[107278]: from='client.47552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:10:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:12 smithi003 ceph-mon[104007]: from='client.47552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:10:13.242 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:13.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:13 smithi003 ceph-mon[104007]: pgmap v1688: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:13.487 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:13.487 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:13.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:13 smithi003 ceph-mon[107278]: pgmap v1688: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:13.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:13.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:13 smithi019 ceph-mon[91296]: pgmap v1688: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:14.426 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:10:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:10:14] "GET /metrics HTTP/1.1" 200 34110 "" "Prometheus/2.51.0" 2024-09-13T15:10:14.449 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:10:14.449 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:10:14.449 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-13T15:10:14.449 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:10:15.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:10:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:15 smithi003 ceph-mon[107278]: pgmap v1689: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:15.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:15 smithi003 ceph-mon[104007]: pgmap v1689: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:15 smithi019 ceph-mon[91296]: pgmap v1689: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:16.283 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:16.453 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:10:16.454 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:10:17.203 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1690,"stamp":"2024-09-13T15:10:16.276622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":989337,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3159,"ondisk_log_size":3159,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9813456,"kb_used_data":47356,"kb_used_omap":23,"kb_used_meta":9765992,"kb_avail":646402608,"statfs":{"total":671965249536,"available":661916270592,"internally_reserved":0,"allocated":48492544,"data_stored":28631280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24286,"internal_metadata":10000376098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.944916"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":225,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368246+0000","last_change":"2024-09-13T15:07:34.471107+0000","last_active":"2024-09-13T15:10:05.368246+0000","last_peered":"2024-09-13T15:10:05.368246+0000","last_clean":"2024-09-13T15:10:05.368246+0000","last_became_active":"2024-09-13T15:07:34.470229+0000","last_became_peered":"2024-09-13T15:07:34.470229+0000","last_unstale":"2024-09-13T15:10:05.368246+0000","last_undegraded":"2024-09-13T15:10:05.368246+0000","last_fullsized":"2024-09-13T15:10:05.368246+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:48.428315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:48.428315+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:08.746827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":231,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313324+0000","last_change":"2024-09-13T15:02:19.429607+0000","last_active":"2024-09-13T15:10:06.313324+0000","last_peered":"2024-09-13T15:10:06.313324+0000","last_clean":"2024-09-13T15:10:06.313324+0000","last_became_active":"2024-09-13T15:02:19.418793+0000","last_became_peered":"2024-09-13T15:02:19.418793+0000","last_unstale":"2024-09-13T15:10:06.313324+0000","last_undegraded":"2024-09-13T15:10:06.313324+0000","last_fullsized":"2024-09-13T15:10:06.313324+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:03.520208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":65,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.812796+0000","last_change":"2024-09-13T15:09:31.313952+0000","last_active":"2024-09-13T15:09:57.812796+0000","last_peered":"2024-09-13T15:09:57.812796+0000","last_clean":"2024-09-13T15:09:57.812796+0000","last_became_active":"2024-09-13T15:09:31.313402+0000","last_became_peered":"2024-09-13T15:09:31.313402+0000","last_unstale":"2024-09-13T15:09:57.812796+0000","last_undegraded":"2024-09-13T15:09:57.812796+0000","last_fullsized":"2024-09-13T15:09:57.812796+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:08.823474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":189,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313599+0000","last_change":"2024-09-13T15:07:14.303688+0000","last_active":"2024-09-13T15:09:58.313599+0000","last_peered":"2024-09-13T15:09:58.313599+0000","last_clean":"2024-09-13T15:09:58.313599+0000","last_became_active":"2024-09-13T15:07:14.303310+0000","last_became_peered":"2024-09-13T15:07:14.303310+0000","last_unstale":"2024-09-13T15:09:58.313599+0000","last_undegraded":"2024-09-13T15:09:58.313599+0000","last_fullsized":"2024-09-13T15:09:58.313599+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:26.066822+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":197,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368609+0000","last_change":"2024-09-13T15:03:46.871160+0000","last_active":"2024-09-13T15:10:05.368609+0000","last_peered":"2024-09-13T15:10:05.368609+0000","last_clean":"2024-09-13T15:10:05.368609+0000","last_became_active":"2024-09-13T15:03:46.870825+0000","last_became_peered":"2024-09-13T15:03:46.870825+0000","last_unstale":"2024-09-13T15:10:05.368609+0000","last_undegraded":"2024-09-13T15:10:05.368609+0000","last_fullsized":"2024-09-13T15:10:05.368609+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:54.702996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367678+0000","last_change":"2024-09-13T15:03:46.873412+0000","last_active":"2024-09-13T15:10:05.367678+0000","last_peered":"2024-09-13T15:10:05.367678+0000","last_clean":"2024-09-13T15:10:05.367678+0000","last_became_active":"2024-09-13T15:03:46.873094+0000","last_became_peered":"2024-09-13T15:03:46.873094+0000","last_unstale":"2024-09-13T15:10:05.367678+0000","last_undegraded":"2024-09-13T15:10:05.367678+0000","last_fullsized":"2024-09-13T15:10:05.367678+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:45.283467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":59,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813701+0000","last_change":"2024-09-13T15:07:34.593081+0000","last_active":"2024-09-13T15:09:57.813701+0000","last_peered":"2024-09-13T15:09:57.813701+0000","last_clean":"2024-09-13T15:09:57.813701+0000","last_became_active":"2024-09-13T15:07:34.471710+0000","last_became_peered":"2024-09-13T15:07:34.471710+0000","last_unstale":"2024-09-13T15:09:57.813701+0000","last_undegraded":"2024-09-13T15:09:57.813701+0000","last_fullsized":"2024-09-13T15:09:57.813701+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:26.250353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813215+0000","last_change":"2024-09-13T15:07:34.580859+0000","last_active":"2024-09-13T15:09:57.813215+0000","last_peered":"2024-09-13T15:09:57.813215+0000","last_clean":"2024-09-13T15:09:57.813215+0000","last_became_active":"2024-09-13T15:07:34.580521+0000","last_became_peered":"2024-09-13T15:07:34.580521+0000","last_unstale":"2024-09-13T15:09:57.813215+0000","last_undegraded":"2024-09-13T15:09:57.813215+0000","last_fullsized":"2024-09-13T15:09:57.813215+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:01.252401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":205,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368555+0000","last_change":"2024-09-13T15:03:47.469254+0000","last_active":"2024-09-13T15:10:05.368555+0000","last_peered":"2024-09-13T15:10:05.368555+0000","last_clean":"2024-09-13T15:10:05.368555+0000","last_became_active":"2024-09-13T15:03:46.854242+0000","last_became_peered":"2024-09-13T15:03:46.854242+0000","last_unstale":"2024-09-13T15:10:05.368555+0000","last_undegraded":"2024-09-13T15:10:05.368555+0000","last_fullsized":"2024-09-13T15:10:05.368555+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:47.469211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:47.469211+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:59.563316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":59,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813298+0000","last_change":"2024-09-13T15:07:34.575982+0000","last_active":"2024-09-13T15:09:57.813298+0000","last_peered":"2024-09-13T15:09:57.813298+0000","last_clean":"2024-09-13T15:09:57.813298+0000","last_became_active":"2024-09-13T15:07:34.470622+0000","last_became_peered":"2024-09-13T15:07:34.470622+0000","last_unstale":"2024-09-13T15:09:57.813298+0000","last_undegraded":"2024-09-13T15:09:57.813298+0000","last_fullsized":"2024-09-13T15:09:57.813298+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:10.751863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813488+0000","last_change":"2024-09-13T15:07:34.459131+0000","last_active":"2024-09-13T15:09:57.813488+0000","last_peered":"2024-09-13T15:09:57.813488+0000","last_clean":"2024-09-13T15:09:57.813488+0000","last_became_active":"2024-09-13T15:07:34.457225+0000","last_became_peered":"2024-09-13T15:07:34.457225+0000","last_unstale":"2024-09-13T15:09:57.813488+0000","last_undegraded":"2024-09-13T15:09:57.813488+0000","last_fullsized":"2024-09-13T15:09:57.813488+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:57:25.961911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367769+0000","last_change":"2024-09-13T15:03:46.864596+0000","last_active":"2024-09-13T15:10:05.367769+0000","last_peered":"2024-09-13T15:10:05.367769+0000","last_clean":"2024-09-13T15:10:05.367769+0000","last_became_active":"2024-09-13T15:03:46.863107+0000","last_became_peered":"2024-09-13T15:03:46.863107+0000","last_unstale":"2024-09-13T15:10:05.367769+0000","last_undegraded":"2024-09-13T15:10:05.367769+0000","last_fullsized":"2024-09-13T15:10:05.367769+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:07.821841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":246,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313206+0000","last_change":"2024-09-13T15:07:34.489594+0000","last_active":"2024-09-13T15:10:06.313206+0000","last_peered":"2024-09-13T15:10:06.313206+0000","last_clean":"2024-09-13T15:10:06.313206+0000","last_became_active":"2024-09-13T15:07:34.488980+0000","last_became_peered":"2024-09-13T15:07:34.488980+0000","last_unstale":"2024-09-13T15:10:06.313206+0000","last_undegraded":"2024-09-13T15:10:06.313206+0000","last_fullsized":"2024-09-13T15:10:06.313206+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:23.320438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":859,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.993216+0000","last_change":"2024-09-13T15:09:31.178585+0000","last_active":"2024-09-13T15:10:06.993216+0000","last_peered":"2024-09-13T15:10:06.993216+0000","last_clean":"2024-09-13T15:10:06.993216+0000","last_became_active":"2024-09-13T15:09:31.178256+0000","last_became_peered":"2024-09-13T15:09:31.178256+0000","last_unstale":"2024-09-13T15:10:06.993216+0000","last_undegraded":"2024-09-13T15:10:06.993216+0000","last_fullsized":"2024-09-13T15:10:06.993216+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:31.137253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":176,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053319+0000","last_change":"2024-09-13T14:58:09.011981+0000","last_active":"2024-09-13T15:09:58.053319+0000","last_peered":"2024-09-13T15:09:58.053319+0000","last_clean":"2024-09-13T15:09:58.053319+0000","last_became_active":"2024-09-13T14:57:53.357674+0000","last_became_peered":"2024-09-13T14:57:53.357674+0000","last_unstale":"2024-09-13T15:09:58.053319+0000","last_undegraded":"2024-09-13T15:09:58.053319+0000","last_fullsized":"2024-09-13T15:09:58.053319+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.701912+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813057+0000","last_change":"2024-09-13T15:07:34.467512+0000","last_active":"2024-09-13T15:09:57.813057+0000","last_peered":"2024-09-13T15:09:57.813057+0000","last_clean":"2024-09-13T15:09:57.813057+0000","last_became_active":"2024-09-13T15:07:34.467061+0000","last_became_peered":"2024-09-13T15:07:34.467061+0000","last_unstale":"2024-09-13T15:09:57.813057+0000","last_undegraded":"2024-09-13T15:09:57.813057+0000","last_fullsized":"2024-09-13T15:09:57.813057+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:43.721513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":217,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368236+0000","last_change":"2024-09-13T15:07:34.470666+0000","last_active":"2024-09-13T15:10:05.368236+0000","last_peered":"2024-09-13T15:10:05.368236+0000","last_clean":"2024-09-13T15:10:05.368236+0000","last_became_active":"2024-09-13T15:07:34.470346+0000","last_became_peered":"2024-09-13T15:07:34.470346+0000","last_unstale":"2024-09-13T15:10:05.368236+0000","last_undegraded":"2024-09-13T15:10:05.368236+0000","last_fullsized":"2024-09-13T15:10:05.368236+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:21.848356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":212,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053642+0000","last_change":"2024-09-13T15:07:34.492713+0000","last_active":"2024-09-13T15:09:58.053642+0000","last_peered":"2024-09-13T15:09:58.053642+0000","last_clean":"2024-09-13T15:09:58.053642+0000","last_became_active":"2024-09-13T15:07:34.492318+0000","last_became_peered":"2024-09-13T15:07:34.492318+0000","last_unstale":"2024-09-13T15:09:58.053642+0000","last_undegraded":"2024-09-13T15:09:58.053642+0000","last_fullsized":"2024-09-13T15:09:58.053642+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:02.258576+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":509,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367371+0000","last_change":"2024-09-13T15:07:34.452596+0000","last_active":"2024-09-13T15:10:05.367371+0000","last_peered":"2024-09-13T15:10:05.367371+0000","last_clean":"2024-09-13T15:10:05.367371+0000","last_became_active":"2024-09-13T15:07:34.452257+0000","last_became_peered":"2024-09-13T15:07:34.452257+0000","last_unstale":"2024-09-13T15:10:05.367371+0000","last_undegraded":"2024-09-13T15:10:05.367371+0000","last_fullsized":"2024-09-13T15:10:05.367371+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:05:40.025880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":239,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313177+0000","last_change":"2024-09-13T15:03:46.864512+0000","last_active":"2024-09-13T15:10:06.313177+0000","last_peered":"2024-09-13T15:10:06.313177+0000","last_clean":"2024-09-13T15:10:06.313177+0000","last_became_active":"2024-09-13T15:03:46.864120+0000","last_became_peered":"2024-09-13T15:03:46.864120+0000","last_unstale":"2024-09-13T15:10:06.313177+0000","last_undegraded":"2024-09-13T15:10:06.313177+0000","last_fullsized":"2024-09-13T15:10:06.313177+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:36.450217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813062+0000","last_change":"2024-09-13T15:07:34.478273+0000","last_active":"2024-09-13T15:09:57.813062+0000","last_peered":"2024-09-13T15:09:57.813062+0000","last_clean":"2024-09-13T15:09:57.813062+0000","last_became_active":"2024-09-13T15:07:34.477980+0000","last_became_peered":"2024-09-13T15:07:34.477980+0000","last_unstale":"2024-09-13T15:09:57.813062+0000","last_undegraded":"2024-09-13T15:09:57.813062+0000","last_fullsized":"2024-09-13T15:09:57.813062+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:06:47.046963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:06:47.046963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:53.376898+0000","scrub_duration":0.018211127000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":203,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367209+0000","last_change":"2024-09-13T15:07:14.305884+0000","last_active":"2024-09-13T15:10:05.367209+0000","last_peered":"2024-09-13T15:10:05.367209+0000","last_clean":"2024-09-13T15:10:05.367209+0000","last_became_active":"2024-09-13T15:07:14.303053+0000","last_became_peered":"2024-09-13T15:07:14.303053+0000","last_unstale":"2024-09-13T15:10:05.367209+0000","last_undegraded":"2024-09-13T15:10:05.367209+0000","last_fullsized":"2024-09-13T15:10:05.367209+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:56.705465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":473,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313825+0000","last_change":"2024-09-13T15:07:14.323822+0000","last_active":"2024-09-13T15:09:58.313825+0000","last_peered":"2024-09-13T15:09:58.313825+0000","last_clean":"2024-09-13T15:09:58.313825+0000","last_became_active":"2024-09-13T15:07:14.323535+0000","last_became_peered":"2024-09-13T15:07:14.323535+0000","last_unstale":"2024-09-13T15:09:58.313825+0000","last_undegraded":"2024-09-13T15:09:58.313825+0000","last_fullsized":"2024-09-13T15:09:58.313825+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:02.779356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":215,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314274+0000","last_change":"2024-09-13T15:07:14.307345+0000","last_active":"2024-09-13T15:09:58.314274+0000","last_peered":"2024-09-13T15:09:58.314274+0000","last_clean":"2024-09-13T15:09:58.314274+0000","last_became_active":"2024-09-13T15:07:14.306790+0000","last_became_peered":"2024-09-13T15:07:14.306790+0000","last_unstale":"2024-09-13T15:09:58.314274+0000","last_undegraded":"2024-09-13T15:09:58.314274+0000","last_fullsized":"2024-09-13T15:09:58.314274+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:53.006962+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":210,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367317+0000","last_change":"2024-09-13T15:07:34.458459+0000","last_active":"2024-09-13T15:10:05.367317+0000","last_peered":"2024-09-13T15:10:05.367317+0000","last_clean":"2024-09-13T15:10:05.367317+0000","last_became_active":"2024-09-13T15:07:34.458134+0000","last_became_peered":"2024-09-13T15:07:34.458134+0000","last_unstale":"2024-09-13T15:10:05.367317+0000","last_undegraded":"2024-09-13T15:10:05.367317+0000","last_fullsized":"2024-09-13T15:10:05.367317+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:00.244169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":232,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313316+0000","last_change":"2024-09-13T15:03:46.871342+0000","last_active":"2024-09-13T15:10:06.313316+0000","last_peered":"2024-09-13T15:10:06.313316+0000","last_clean":"2024-09-13T15:10:06.313316+0000","last_became_active":"2024-09-13T15:03:46.863406+0000","last_became_peered":"2024-09-13T15:03:46.863406+0000","last_unstale":"2024-09-13T15:10:06.313316+0000","last_undegraded":"2024-09-13T15:10:06.313316+0000","last_fullsized":"2024-09-13T15:10:06.313316+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:53.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367533+0000","last_change":"2024-09-13T15:03:46.859350+0000","last_active":"2024-09-13T15:10:05.367533+0000","last_peered":"2024-09-13T15:10:05.367533+0000","last_clean":"2024-09-13T15:10:05.367533+0000","last_became_active":"2024-09-13T15:03:46.853202+0000","last_became_peered":"2024-09-13T15:03:46.853202+0000","last_unstale":"2024-09-13T15:10:05.367533+0000","last_undegraded":"2024-09-13T15:10:05.367533+0000","last_fullsized":"2024-09-13T15:10:05.367533+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:42.594537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":172,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053749+0000","last_change":"2024-09-13T15:07:14.309051+0000","last_active":"2024-09-13T15:09:58.053749+0000","last_peered":"2024-09-13T15:09:58.053749+0000","last_clean":"2024-09-13T15:09:58.053749+0000","last_became_active":"2024-09-13T15:07:14.308645+0000","last_became_peered":"2024-09-13T15:07:14.308645+0000","last_unstale":"2024-09-13T15:09:58.053749+0000","last_undegraded":"2024-09-13T15:09:58.053749+0000","last_fullsized":"2024-09-13T15:09:58.053749+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:04:05.335587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":554,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.993071+0000","last_change":"2024-09-13T15:07:34.456969+0000","last_active":"2024-09-13T15:10:06.993071+0000","last_peered":"2024-09-13T15:10:06.993071+0000","last_clean":"2024-09-13T15:10:06.993071+0000","last_became_active":"2024-09-13T15:07:34.456596+0000","last_became_peered":"2024-09-13T15:07:34.456596+0000","last_unstale":"2024-09-13T15:10:06.993071+0000","last_undegraded":"2024-09-13T15:10:06.993071+0000","last_fullsized":"2024-09-13T15:10:06.993071+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:19.319199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":182,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053971+0000","last_change":"2024-09-13T15:02:19.406264+0000","last_active":"2024-09-13T15:09:58.053971+0000","last_peered":"2024-09-13T15:09:58.053971+0000","last_clean":"2024-09-13T15:09:58.053971+0000","last_became_active":"2024-09-13T15:02:19.405876+0000","last_became_peered":"2024-09-13T15:02:19.405876+0000","last_unstale":"2024-09-13T15:09:58.053971+0000","last_undegraded":"2024-09-13T15:09:58.053971+0000","last_fullsized":"2024-09-13T15:09:58.053971+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:19.208307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":308,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313919+0000","last_change":"2024-09-13T15:07:34.622674+0000","last_active":"2024-09-13T15:09:58.313919+0000","last_peered":"2024-09-13T15:09:58.313919+0000","last_clean":"2024-09-13T15:09:58.313919+0000","last_became_active":"2024-09-13T15:07:34.474941+0000","last_became_peered":"2024-09-13T15:07:34.474941+0000","last_unstale":"2024-09-13T15:09:58.313919+0000","last_undegraded":"2024-09-13T15:09:58.313919+0000","last_fullsized":"2024-09-13T15:09:58.313919+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:14.657841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":392,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313411+0000","last_change":"2024-09-13T15:07:14.294018+0000","last_active":"2024-09-13T15:09:58.313411+0000","last_peered":"2024-09-13T15:09:58.313411+0000","last_clean":"2024-09-13T15:09:58.313411+0000","last_became_active":"2024-09-13T15:07:14.293678+0000","last_became_peered":"2024-09-13T15:07:14.293678+0000","last_unstale":"2024-09-13T15:09:58.313411+0000","last_undegraded":"2024-09-13T15:09:58.313411+0000","last_fullsized":"2024-09-13T15:09:58.313411+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:29.584863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":263,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312966+0000","last_change":"2024-09-13T15:07:34.482157+0000","last_active":"2024-09-13T15:10:06.312966+0000","last_peered":"2024-09-13T15:10:06.312966+0000","last_clean":"2024-09-13T15:10:06.312966+0000","last_became_active":"2024-09-13T15:07:34.475831+0000","last_became_peered":"2024-09-13T15:07:34.475831+0000","last_unstale":"2024-09-13T15:10:06.312966+0000","last_undegraded":"2024-09-13T15:10:06.312966+0000","last_fullsized":"2024-09-13T15:10:06.312966+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:08.311777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":65,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813107+0000","last_change":"2024-09-13T15:07:34.603715+0000","last_active":"2024-09-13T15:09:57.813107+0000","last_peered":"2024-09-13T15:09:57.813107+0000","last_clean":"2024-09-13T15:09:57.813107+0000","last_became_active":"2024-09-13T15:07:34.476556+0000","last_became_peered":"2024-09-13T15:07:34.476556+0000","last_unstale":"2024-09-13T15:09:57.813107+0000","last_undegraded":"2024-09-13T15:09:57.813107+0000","last_fullsized":"2024-09-13T15:09:57.813107+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:49:06.015543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813337+0000","last_change":"2024-09-13T15:07:34.473602+0000","last_active":"2024-09-13T15:09:57.813337+0000","last_peered":"2024-09-13T15:09:57.813337+0000","last_clean":"2024-09-13T15:09:57.813337+0000","last_became_active":"2024-09-13T15:07:34.473208+0000","last_became_peered":"2024-09-13T15:07:34.473208+0000","last_unstale":"2024-09-13T15:09:57.813337+0000","last_undegraded":"2024-09-13T15:09:57.813337+0000","last_fullsized":"2024-09-13T15:09:57.813337+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:30.687198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.812785+0000","last_change":"2024-09-13T15:07:34.470575+0000","last_active":"2024-09-13T15:09:57.812785+0000","last_peered":"2024-09-13T15:09:57.812785+0000","last_clean":"2024-09-13T15:09:57.812785+0000","last_became_active":"2024-09-13T15:07:34.470231+0000","last_became_peered":"2024-09-13T15:07:34.470231+0000","last_unstale":"2024-09-13T15:09:57.812785+0000","last_undegraded":"2024-09-13T15:09:57.812785+0000","last_fullsized":"2024-09-13T15:09:57.812785+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:34.618632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":476,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313899+0000","last_change":"2024-09-13T14:49:05.833563+0000","last_active":"2024-09-13T15:09:58.313899+0000","last_peered":"2024-09-13T15:09:58.313899+0000","last_clean":"2024-09-13T15:09:58.313899+0000","last_became_active":"2024-09-13T14:49:05.832780+0000","last_became_peered":"2024-09-13T14:49:05.832780+0000","last_unstale":"2024-09-13T15:09:58.313899+0000","last_undegraded":"2024-09-13T15:09:58.313899+0000","last_fullsized":"2024-09-13T15:09:58.313899+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:36.468693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":262,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313321+0000","last_change":"2024-09-13T15:07:34.605821+0000","last_active":"2024-09-13T15:10:06.313321+0000","last_peered":"2024-09-13T15:10:06.313321+0000","last_clean":"2024-09-13T15:10:06.313321+0000","last_became_active":"2024-09-13T15:07:34.477436+0000","last_became_peered":"2024-09-13T15:07:34.477436+0000","last_unstale":"2024-09-13T15:10:06.313321+0000","last_undegraded":"2024-09-13T15:10:06.313321+0000","last_fullsized":"2024-09-13T15:10:06.313321+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:16.983267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":197,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367976+0000","last_change":"2024-09-13T15:03:46.868597+0000","last_active":"2024-09-13T15:10:05.367976+0000","last_peered":"2024-09-13T15:10:05.367976+0000","last_clean":"2024-09-13T15:10:05.367976+0000","last_became_active":"2024-09-13T15:03:46.866939+0000","last_became_peered":"2024-09-13T15:03:46.866939+0000","last_unstale":"2024-09-13T15:10:05.367976+0000","last_undegraded":"2024-09-13T15:10:05.367976+0000","last_fullsized":"2024-09-13T15:10:05.367976+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.060483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":197,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367115+0000","last_change":"2024-09-13T15:03:46.865877+0000","last_active":"2024-09-13T15:10:05.367115+0000","last_peered":"2024-09-13T15:10:05.367115+0000","last_clean":"2024-09-13T15:10:05.367115+0000","last_became_active":"2024-09-13T15:03:46.865029+0000","last_became_peered":"2024-09-13T15:03:46.865029+0000","last_unstale":"2024-09-13T15:10:05.367115+0000","last_undegraded":"2024-09-13T15:10:05.367115+0000","last_fullsized":"2024-09-13T15:10:05.367115+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:28.871714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":347,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313797+0000","last_change":"2024-09-13T15:07:34.472379+0000","last_active":"2024-09-13T15:09:58.313797+0000","last_peered":"2024-09-13T15:09:58.313797+0000","last_clean":"2024-09-13T15:09:58.313797+0000","last_became_active":"2024-09-13T15:07:34.472027+0000","last_became_peered":"2024-09-13T15:07:34.472027+0000","last_unstale":"2024-09-13T15:09:58.313797+0000","last_undegraded":"2024-09-13T15:09:58.313797+0000","last_fullsized":"2024-09-13T15:09:58.313797+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:23.006884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":200,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368547+0000","last_change":"2024-09-13T15:03:46.879528+0000","last_active":"2024-09-13T15:10:05.368547+0000","last_peered":"2024-09-13T15:10:05.368547+0000","last_clean":"2024-09-13T15:10:05.368547+0000","last_became_active":"2024-09-13T15:03:46.860753+0000","last_became_peered":"2024-09-13T15:03:46.860753+0000","last_unstale":"2024-09-13T15:10:05.368547+0000","last_undegraded":"2024-09-13T15:10:05.368547+0000","last_fullsized":"2024-09-13T15:10:05.368547+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:28.561501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813601+0000","last_change":"2024-09-13T15:07:34.471374+0000","last_active":"2024-09-13T15:09:57.813601+0000","last_peered":"2024-09-13T15:09:57.813601+0000","last_clean":"2024-09-13T15:09:57.813601+0000","last_became_active":"2024-09-13T15:07:34.469293+0000","last_became_peered":"2024-09-13T15:07:34.469293+0000","last_unstale":"2024-09-13T15:09:57.813601+0000","last_undegraded":"2024-09-13T15:09:57.813601+0000","last_fullsized":"2024-09-13T15:09:57.813601+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:06:46.602703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:06:46.602703+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:55.225137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":225,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313197+0000","last_change":"2024-09-13T15:02:19.423088+0000","last_active":"2024-09-13T15:10:06.313197+0000","last_peered":"2024-09-13T15:10:06.313197+0000","last_clean":"2024-09-13T15:10:06.313197+0000","last_became_active":"2024-09-13T15:02:19.422675+0000","last_became_peered":"2024-09-13T15:02:19.422675+0000","last_unstale":"2024-09-13T15:10:06.313197+0000","last_undegraded":"2024-09-13T15:10:06.313197+0000","last_fullsized":"2024-09-13T15:10:06.313197+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:58.764834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":506,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314311+0000","last_change":"2024-09-13T15:07:34.516311+0000","last_active":"2024-09-13T15:09:58.314311+0000","last_peered":"2024-09-13T15:09:58.314311+0000","last_clean":"2024-09-13T15:09:58.314311+0000","last_became_active":"2024-09-13T15:07:34.516030+0000","last_became_peered":"2024-09-13T15:07:34.516030+0000","last_unstale":"2024-09-13T15:09:58.314311+0000","last_undegraded":"2024-09-13T15:09:58.314311+0000","last_fullsized":"2024-09-13T15:09:58.314311+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:28.151366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":812,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.993101+0000","last_change":"2024-09-13T15:07:14.329958+0000","last_active":"2024-09-13T15:10:06.993101+0000","last_peered":"2024-09-13T15:10:06.993101+0000","last_clean":"2024-09-13T15:10:06.993101+0000","last_became_active":"2024-09-13T15:07:14.329538+0000","last_became_peered":"2024-09-13T15:07:14.329538+0000","last_unstale":"2024-09-13T15:10:06.993101+0000","last_undegraded":"2024-09-13T15:10:06.993101+0000","last_fullsized":"2024-09-13T15:10:06.993101+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:58.643216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":477,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313449+0000","last_change":"2024-09-13T15:03:46.869628+0000","last_active":"2024-09-13T15:09:58.313449+0000","last_peered":"2024-09-13T15:09:58.313449+0000","last_clean":"2024-09-13T15:09:58.313449+0000","last_became_active":"2024-09-13T15:03:46.869274+0000","last_became_peered":"2024-09-13T15:03:46.869274+0000","last_unstale":"2024-09-13T15:09:58.313449+0000","last_undegraded":"2024-09-13T15:09:58.313449+0000","last_fullsized":"2024-09-13T15:09:58.313449+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:21.999130+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":192,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053557+0000","last_change":"2024-09-13T15:07:34.451764+0000","last_active":"2024-09-13T15:09:58.053557+0000","last_peered":"2024-09-13T15:09:58.053557+0000","last_clean":"2024-09-13T15:09:58.053557+0000","last_became_active":"2024-09-13T15:07:34.451452+0000","last_became_peered":"2024-09-13T15:07:34.451452+0000","last_unstale":"2024-09-13T15:09:58.053557+0000","last_undegraded":"2024-09-13T15:09:58.053557+0000","last_fullsized":"2024-09-13T15:09:58.053557+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:02.786773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":232,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313022+0000","last_change":"2024-09-13T15:07:14.318434+0000","last_active":"2024-09-13T15:10:06.313022+0000","last_peered":"2024-09-13T15:10:06.313022+0000","last_clean":"2024-09-13T15:10:06.313022+0000","last_became_active":"2024-09-13T15:07:14.318017+0000","last_became_peered":"2024-09-13T15:07:14.318017+0000","last_unstale":"2024-09-13T15:10:06.313022+0000","last_undegraded":"2024-09-13T15:10:06.313022+0000","last_fullsized":"2024-09-13T15:10:06.313022+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:26.290395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":182,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053766+0000","last_change":"2024-09-13T15:03:46.866800+0000","last_active":"2024-09-13T15:09:58.053766+0000","last_peered":"2024-09-13T15:09:58.053766+0000","last_clean":"2024-09-13T15:09:58.053766+0000","last_became_active":"2024-09-13T15:03:46.866557+0000","last_became_peered":"2024-09-13T15:03:46.866557+0000","last_unstale":"2024-09-13T15:09:58.053766+0000","last_undegraded":"2024-09-13T15:09:58.053766+0000","last_fullsized":"2024-09-13T15:09:58.053766+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:28.181919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":477,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313730+0000","last_change":"2024-09-13T15:03:46.851975+0000","last_active":"2024-09-13T15:09:58.313730+0000","last_peered":"2024-09-13T15:09:58.313730+0000","last_clean":"2024-09-13T15:09:58.313730+0000","last_became_active":"2024-09-13T15:03:46.851616+0000","last_became_peered":"2024-09-13T15:03:46.851616+0000","last_unstale":"2024-09-13T15:09:58.313730+0000","last_undegraded":"2024-09-13T15:09:58.313730+0000","last_fullsized":"2024-09-13T15:09:58.313730+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:01.819924+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":724,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.993224+0000","last_change":"2024-09-13T15:07:14.312909+0000","last_active":"2024-09-13T15:10:06.993224+0000","last_peered":"2024-09-13T15:10:06.993224+0000","last_clean":"2024-09-13T15:10:06.993224+0000","last_became_active":"2024-09-13T15:07:14.312606+0000","last_became_peered":"2024-09-13T15:07:14.312606+0000","last_unstale":"2024-09-13T15:10:06.993224+0000","last_undegraded":"2024-09-13T15:10:06.993224+0000","last_fullsized":"2024-09-13T15:10:06.993224+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:41.599239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813330+0000","last_change":"2024-09-13T15:07:34.598991+0000","last_active":"2024-09-13T15:09:57.813330+0000","last_peered":"2024-09-13T15:09:57.813330+0000","last_clean":"2024-09-13T15:09:57.813330+0000","last_became_active":"2024-09-13T15:07:34.598343+0000","last_became_peered":"2024-09-13T15:07:34.598343+0000","last_unstale":"2024-09-13T15:09:57.813330+0000","last_undegraded":"2024-09-13T15:09:57.813330+0000","last_fullsized":"2024-09-13T15:09:57.813330+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:31.371179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":81,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313606+0000","last_change":"2024-09-13T15:07:14.345712+0000","last_active":"2024-09-13T15:09:58.313606+0000","last_peered":"2024-09-13T15:09:58.313606+0000","last_clean":"2024-09-13T15:09:58.313606+0000","last_became_active":"2024-09-13T15:07:14.302650+0000","last_became_peered":"2024-09-13T15:07:14.302650+0000","last_unstale":"2024-09-13T15:09:58.313606+0000","last_undegraded":"2024-09-13T15:09:58.313606+0000","last_fullsized":"2024-09-13T15:09:58.313606+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:50.212668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":225,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312674+0000","last_change":"2024-09-13T15:02:19.420193+0000","last_active":"2024-09-13T15:10:06.312674+0000","last_peered":"2024-09-13T15:10:06.312674+0000","last_clean":"2024-09-13T15:10:06.312674+0000","last_became_active":"2024-09-13T15:02:19.419807+0000","last_became_peered":"2024-09-13T15:02:19.419807+0000","last_unstale":"2024-09-13T15:10:06.312674+0000","last_undegraded":"2024-09-13T15:10:06.312674+0000","last_fullsized":"2024-09-13T15:10:06.312674+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:13.929284+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":446,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313143+0000","last_change":"2024-09-13T15:07:14.324008+0000","last_active":"2024-09-13T15:10:06.313143+0000","last_peered":"2024-09-13T15:10:06.313143+0000","last_clean":"2024-09-13T15:10:06.313143+0000","last_became_active":"2024-09-13T15:07:14.323650+0000","last_became_peered":"2024-09-13T15:07:14.323650+0000","last_unstale":"2024-09-13T15:10:06.313143+0000","last_undegraded":"2024-09-13T15:10:06.313143+0000","last_fullsized":"2024-09-13T15:10:06.313143+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:59.889970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":651,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.992857+0000","last_change":"2024-09-13T15:07:14.301469+0000","last_active":"2024-09-13T15:10:06.992857+0000","last_peered":"2024-09-13T15:10:06.992857+0000","last_clean":"2024-09-13T15:10:06.992857+0000","last_became_active":"2024-09-13T15:07:14.300482+0000","last_became_peered":"2024-09-13T15:07:14.300482+0000","last_unstale":"2024-09-13T15:10:06.992857+0000","last_undegraded":"2024-09-13T15:10:06.992857+0000","last_fullsized":"2024-09-13T15:10:06.992857+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:30.889776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":862,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.992916+0000","last_change":"2024-09-13T15:03:46.871298+0000","last_active":"2024-09-13T15:10:06.992916+0000","last_peered":"2024-09-13T15:10:06.992916+0000","last_clean":"2024-09-13T15:10:06.992916+0000","last_became_active":"2024-09-13T15:03:46.870966+0000","last_became_peered":"2024-09-13T15:03:46.870966+0000","last_unstale":"2024-09-13T15:10:06.992916+0000","last_undegraded":"2024-09-13T15:10:06.992916+0000","last_fullsized":"2024-09-13T15:10:06.992916+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:22.463434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":278,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312796+0000","last_change":"2024-09-13T15:07:34.464473+0000","last_active":"2024-09-13T15:09:58.312796+0000","last_peered":"2024-09-13T15:09:58.312796+0000","last_clean":"2024-09-13T15:09:58.312796+0000","last_became_active":"2024-09-13T15:07:34.464122+0000","last_became_peered":"2024-09-13T15:07:34.464122+0000","last_unstale":"2024-09-13T15:09:58.312796+0000","last_undegraded":"2024-09-13T15:09:58.312796+0000","last_fullsized":"2024-09-13T15:09:58.312796+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:55.352605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":232,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313225+0000","last_change":"2024-09-13T15:07:14.326411+0000","last_active":"2024-09-13T15:10:06.313225+0000","last_peered":"2024-09-13T15:10:06.313225+0000","last_clean":"2024-09-13T15:10:06.313225+0000","last_became_active":"2024-09-13T15:07:14.326053+0000","last_became_peered":"2024-09-13T15:07:14.326053+0000","last_unstale":"2024-09-13T15:10:06.313225+0000","last_undegraded":"2024-09-13T15:10:06.313225+0000","last_fullsized":"2024-09-13T15:10:06.313225+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:59.937117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":253,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312626+0000","last_change":"2024-09-13T15:07:34.460652+0000","last_active":"2024-09-13T15:10:06.312626+0000","last_peered":"2024-09-13T15:10:06.312626+0000","last_clean":"2024-09-13T15:10:06.312626+0000","last_became_active":"2024-09-13T15:07:34.458909+0000","last_became_peered":"2024-09-13T15:07:34.458909+0000","last_unstale":"2024-09-13T15:10:06.312626+0000","last_undegraded":"2024-09-13T15:10:06.312626+0000","last_fullsized":"2024-09-13T15:10:06.312626+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:05.444737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312648+0000","last_change":"2024-09-13T15:07:14.312738+0000","last_active":"2024-09-13T15:09:58.312648+0000","last_peered":"2024-09-13T15:09:58.312648+0000","last_clean":"2024-09-13T15:09:58.312648+0000","last_became_active":"2024-09-13T15:07:14.312418+0000","last_became_peered":"2024-09-13T15:07:14.312418+0000","last_unstale":"2024-09-13T15:09:58.312648+0000","last_undegraded":"2024-09-13T15:09:58.312648+0000","last_fullsized":"2024-09-13T15:09:58.312648+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:45.706331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":447,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312719+0000","last_change":"2024-09-13T15:07:34.458019+0000","last_active":"2024-09-13T15:09:58.312719+0000","last_peered":"2024-09-13T15:09:58.312719+0000","last_clean":"2024-09-13T15:09:58.312719+0000","last_became_active":"2024-09-13T15:07:34.457720+0000","last_became_peered":"2024-09-13T15:07:34.457720+0000","last_unstale":"2024-09-13T15:09:58.312719+0000","last_undegraded":"2024-09-13T15:09:58.312719+0000","last_fullsized":"2024-09-13T15:09:58.312719+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:44.330131+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":446,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313559+0000","last_change":"2024-09-13T15:07:34.462911+0000","last_active":"2024-09-13T15:09:58.313559+0000","last_peered":"2024-09-13T15:09:58.313559+0000","last_clean":"2024-09-13T15:09:58.313559+0000","last_became_active":"2024-09-13T15:07:34.462577+0000","last_became_peered":"2024-09-13T15:07:34.462577+0000","last_unstale":"2024-09-13T15:09:58.313559+0000","last_undegraded":"2024-09-13T15:09:58.313559+0000","last_fullsized":"2024-09-13T15:09:58.313559+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:51.431494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":197,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367703+0000","last_change":"2024-09-13T15:03:46.875410+0000","last_active":"2024-09-13T15:10:05.367703+0000","last_peered":"2024-09-13T15:10:05.367703+0000","last_clean":"2024-09-13T15:10:05.367703+0000","last_became_active":"2024-09-13T15:03:46.875103+0000","last_became_peered":"2024-09-13T15:03:46.875103+0000","last_unstale":"2024-09-13T15:10:05.367703+0000","last_undegraded":"2024-09-13T15:10:05.367703+0000","last_fullsized":"2024-09-13T15:10:05.367703+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:53.941854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":166,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053623+0000","last_change":"2024-09-13T14:57:53.360296+0000","last_active":"2024-09-13T15:09:58.053623+0000","last_peered":"2024-09-13T15:09:58.053623+0000","last_clean":"2024-09-13T15:09:58.053623+0000","last_became_active":"2024-09-13T14:57:53.347710+0000","last_became_peered":"2024-09-13T14:57:53.347710+0000","last_unstale":"2024-09-13T15:09:58.053623+0000","last_undegraded":"2024-09-13T15:09:58.053623+0000","last_fullsized":"2024-09-13T15:09:58.053623+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:43.099884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":258,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312384+0000","last_change":"2024-09-13T15:02:19.407022+0000","last_active":"2024-09-13T15:09:58.312384+0000","last_peered":"2024-09-13T15:09:58.312384+0000","last_clean":"2024-09-13T15:09:58.312384+0000","last_became_active":"2024-09-13T15:02:19.406620+0000","last_became_peered":"2024-09-13T15:02:19.406620+0000","last_unstale":"2024-09-13T15:09:58.312384+0000","last_undegraded":"2024-09-13T15:09:58.312384+0000","last_fullsized":"2024-09-13T15:09:58.312384+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:37.100188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":510,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368424+0000","last_change":"2024-09-13T15:07:34.484543+0000","last_active":"2024-09-13T15:10:05.368424+0000","last_peered":"2024-09-13T15:10:05.368424+0000","last_clean":"2024-09-13T15:10:05.368424+0000","last_became_active":"2024-09-13T15:07:34.484218+0000","last_became_peered":"2024-09-13T15:07:34.484218+0000","last_unstale":"2024-09-13T15:10:05.368424+0000","last_undegraded":"2024-09-13T15:10:05.368424+0000","last_fullsized":"2024-09-13T15:10:05.368424+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:12.280906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":210,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367267+0000","last_change":"2024-09-13T15:07:34.460538+0000","last_active":"2024-09-13T15:10:05.367267+0000","last_peered":"2024-09-13T15:10:05.367267+0000","last_clean":"2024-09-13T15:10:05.367267+0000","last_became_active":"2024-09-13T15:07:34.460274+0000","last_became_peered":"2024-09-13T15:07:34.460274+0000","last_unstale":"2024-09-13T15:10:05.367267+0000","last_undegraded":"2024-09-13T15:10:05.367267+0000","last_fullsized":"2024-09-13T15:10:05.367267+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:19.022732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":330,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.992692+0000","last_change":"2024-09-13T14:58:28.428608+0000","last_active":"2024-09-13T15:10:06.992692+0000","last_peered":"2024-09-13T15:10:06.992692+0000","last_clean":"2024-09-13T15:10:06.992692+0000","last_became_active":"2024-09-13T14:58:28.428282+0000","last_became_peered":"2024-09-13T14:58:28.428282+0000","last_unstale":"2024-09-13T15:10:06.992692+0000","last_undegraded":"2024-09-13T15:10:06.992692+0000","last_fullsized":"2024-09-13T15:10:06.992692+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.603606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.603606+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:10.695417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813644+0000","last_change":"2024-09-13T15:07:34.475401+0000","last_active":"2024-09-13T15:09:57.813644+0000","last_peered":"2024-09-13T15:09:57.813644+0000","last_clean":"2024-09-13T15:09:57.813644+0000","last_became_active":"2024-09-13T15:07:34.469994+0000","last_became_peered":"2024-09-13T15:07:34.469994+0000","last_unstale":"2024-09-13T15:09:57.813644+0000","last_undegraded":"2024-09-13T15:09:57.813644+0000","last_fullsized":"2024-09-13T15:09:57.813644+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:24.460236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.812831+0000","last_change":"2024-09-13T15:07:34.478488+0000","last_active":"2024-09-13T15:09:57.812831+0000","last_peered":"2024-09-13T15:09:57.812831+0000","last_clean":"2024-09-13T15:09:57.812831+0000","last_became_active":"2024-09-13T15:07:34.477918+0000","last_became_peered":"2024-09-13T15:07:34.477918+0000","last_unstale":"2024-09-13T15:09:57.812831+0000","last_undegraded":"2024-09-13T15:09:57.812831+0000","last_fullsized":"2024-09-13T15:09:57.812831+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:59.742232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":249,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313070+0000","last_change":"2024-09-13T15:07:14.302370+0000","last_active":"2024-09-13T15:10:06.313070+0000","last_peered":"2024-09-13T15:10:06.313070+0000","last_clean":"2024-09-13T15:10:06.313070+0000","last_became_active":"2024-09-13T15:07:14.301636+0000","last_became_peered":"2024-09-13T15:07:14.301636+0000","last_unstale":"2024-09-13T15:10:06.313070+0000","last_undegraded":"2024-09-13T15:10:06.313070+0000","last_fullsized":"2024-09-13T15:10:06.313070+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:02:36.128767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:02:36.128767+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:34.619009+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813582+0000","last_change":"2024-09-13T15:07:34.472070+0000","last_active":"2024-09-13T15:09:57.813582+0000","last_peered":"2024-09-13T15:09:57.813582+0000","last_clean":"2024-09-13T15:09:57.813582+0000","last_became_active":"2024-09-13T15:07:34.470958+0000","last_became_peered":"2024-09-13T15:07:34.470958+0000","last_unstale":"2024-09-13T15:09:57.813582+0000","last_undegraded":"2024-09-13T15:09:57.813582+0000","last_fullsized":"2024-09-13T15:09:57.813582+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:17.779488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":494,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313505+0000","last_change":"2024-09-13T15:07:34.460856+0000","last_active":"2024-09-13T15:09:58.313505+0000","last_peered":"2024-09-13T15:09:58.313505+0000","last_clean":"2024-09-13T15:09:58.313505+0000","last_became_active":"2024-09-13T15:07:34.460540+0000","last_became_peered":"2024-09-13T15:07:34.460540+0000","last_unstale":"2024-09-13T15:09:58.313505+0000","last_undegraded":"2024-09-13T15:09:58.313505+0000","last_fullsized":"2024-09-13T15:09:58.313505+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:08.451991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":399,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.992762+0000","last_change":"2024-09-13T15:03:46.869405+0000","last_active":"2024-09-13T15:10:06.992762+0000","last_peered":"2024-09-13T15:10:06.992762+0000","last_clean":"2024-09-13T15:10:06.992762+0000","last_became_active":"2024-09-13T15:03:46.868913+0000","last_became_peered":"2024-09-13T15:03:46.868913+0000","last_unstale":"2024-09-13T15:10:06.992762+0000","last_undegraded":"2024-09-13T15:10:06.992762+0000","last_fullsized":"2024-09-13T15:10:06.992762+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:02.098981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813378+0000","last_change":"2024-09-13T15:07:34.473797+0000","last_active":"2024-09-13T15:09:57.813378+0000","last_peered":"2024-09-13T15:09:57.813378+0000","last_clean":"2024-09-13T15:09:57.813378+0000","last_became_active":"2024-09-13T15:07:34.473444+0000","last_became_peered":"2024-09-13T15:07:34.473444+0000","last_unstale":"2024-09-13T15:09:57.813378+0000","last_undegraded":"2024-09-13T15:09:57.813378+0000","last_fullsized":"2024-09-13T15:09:57.813378+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:04.278156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":233,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367941+0000","last_change":"2024-09-13T15:07:34.609901+0000","last_active":"2024-09-13T15:10:05.367941+0000","last_peered":"2024-09-13T15:10:05.367941+0000","last_clean":"2024-09-13T15:10:05.367941+0000","last_became_active":"2024-09-13T15:07:34.490242+0000","last_became_peered":"2024-09-13T15:07:34.490242+0000","last_unstale":"2024-09-13T15:10:05.367941+0000","last_undegraded":"2024-09-13T15:10:05.367941+0000","last_fullsized":"2024-09-13T15:10:05.367941+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:23.303360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":304,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312495+0000","last_change":"2024-09-13T15:03:46.863264+0000","last_active":"2024-09-13T15:09:58.312495+0000","last_peered":"2024-09-13T15:09:58.312495+0000","last_clean":"2024-09-13T15:09:58.312495+0000","last_became_active":"2024-09-13T15:03:46.862850+0000","last_became_peered":"2024-09-13T15:03:46.862850+0000","last_unstale":"2024-09-13T15:09:58.312495+0000","last_undegraded":"2024-09-13T15:09:58.312495+0000","last_fullsized":"2024-09-13T15:09:58.312495+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:57.707268+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":246,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313074+0000","last_change":"2024-09-13T15:07:34.489295+0000","last_active":"2024-09-13T15:10:06.313074+0000","last_peered":"2024-09-13T15:10:06.313074+0000","last_clean":"2024-09-13T15:10:06.313074+0000","last_became_active":"2024-09-13T15:07:34.488579+0000","last_became_peered":"2024-09-13T15:07:34.488579+0000","last_unstale":"2024-09-13T15:10:06.313074+0000","last_undegraded":"2024-09-13T15:10:06.313074+0000","last_fullsized":"2024-09-13T15:10:06.313074+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:36.942324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367847+0000","last_change":"2024-09-13T15:03:46.869406+0000","last_active":"2024-09-13T15:10:05.367847+0000","last_peered":"2024-09-13T15:10:05.367847+0000","last_clean":"2024-09-13T15:10:05.367847+0000","last_became_active":"2024-09-13T15:03:46.866230+0000","last_became_peered":"2024-09-13T15:03:46.866230+0000","last_unstale":"2024-09-13T15:10:05.367847+0000","last_undegraded":"2024-09-13T15:10:05.367847+0000","last_fullsized":"2024-09-13T15:10:05.367847+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:40.914289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":65,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.813634+0000","last_change":"2024-09-13T15:08:59.871028+0000","last_active":"2024-09-13T15:09:57.813634+0000","last_peered":"2024-09-13T15:09:57.813634+0000","last_clean":"2024-09-13T15:09:57.813634+0000","last_became_active":"2024-09-13T15:07:34.478006+0000","last_became_peered":"2024-09-13T15:07:34.478006+0000","last_unstale":"2024-09-13T15:09:57.813634+0000","last_undegraded":"2024-09-13T15:09:57.813634+0000","last_fullsized":"2024-09-13T15:09:57.813634+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:08:59.870786+0000","last_deep_scrub":"79'100","last_deep_scrub_stamp":"2024-09-13T15:08:59.870786+0000","last_clean_scrub_stamp":"2024-09-13T15:08:59.870786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:43:55.603202+0000","scrub_duration":0.010767947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":476,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313644+0000","last_change":"2024-09-13T15:07:14.302821+0000","last_active":"2024-09-13T15:09:58.313644+0000","last_peered":"2024-09-13T15:09:58.313644+0000","last_clean":"2024-09-13T15:09:58.313644+0000","last_became_active":"2024-09-13T15:07:14.302553+0000","last_became_peered":"2024-09-13T15:07:14.302553+0000","last_unstale":"2024-09-13T15:09:58.313644+0000","last_undegraded":"2024-09-13T15:09:58.313644+0000","last_fullsized":"2024-09-13T15:09:58.313644+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:32:21.630096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":485,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314257+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:09:58.314257+0000","last_peered":"2024-09-13T15:09:58.314257+0000","last_clean":"2024-09-13T15:09:58.314257+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:09:58.314257+0000","last_undegraded":"2024-09-13T15:09:58.314257+0000","last_fullsized":"2024-09-13T15:09:58.314257+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":457,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313787+0000","last_change":"2024-09-13T15:07:34.463017+0000","last_active":"2024-09-13T15:09:58.313787+0000","last_peered":"2024-09-13T15:09:58.313787+0000","last_clean":"2024-09-13T15:09:58.313787+0000","last_became_active":"2024-09-13T15:07:34.462570+0000","last_became_peered":"2024-09-13T15:07:34.462570+0000","last_unstale":"2024-09-13T15:09:58.313787+0000","last_undegraded":"2024-09-13T15:09:58.313787+0000","last_fullsized":"2024-09-13T15:09:58.313787+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:56.696358+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":55,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:57.812984+0000","last_change":"2024-09-13T15:07:34.471456+0000","last_active":"2024-09-13T15:09:57.812984+0000","last_peered":"2024-09-13T15:09:57.812984+0000","last_clean":"2024-09-13T15:09:57.812984+0000","last_became_active":"2024-09-13T15:07:34.471201+0000","last_became_peered":"2024-09-13T15:07:34.471201+0000","last_unstale":"2024-09-13T15:09:57.812984+0000","last_undegraded":"2024-09-13T15:09:57.812984+0000","last_fullsized":"2024-09-13T15:09:57.812984+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:18:48.572515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":489,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314292+0000","last_change":"2024-09-13T15:07:34.615859+0000","last_active":"2024-09-13T15:09:58.314292+0000","last_peered":"2024-09-13T15:09:58.314292+0000","last_clean":"2024-09-13T15:09:58.314292+0000","last_became_active":"2024-09-13T15:07:34.474503+0000","last_became_peered":"2024-09-13T15:07:34.474503+0000","last_unstale":"2024-09-13T15:09:58.314292+0000","last_undegraded":"2024-09-13T15:09:58.314292+0000","last_fullsized":"2024-09-13T15:09:58.314292+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:29.217608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":200,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368045+0000","last_change":"2024-09-13T15:03:46.882026+0000","last_active":"2024-09-13T15:10:05.368045+0000","last_peered":"2024-09-13T15:10:05.368045+0000","last_clean":"2024-09-13T15:10:05.368045+0000","last_became_active":"2024-09-13T15:03:46.859557+0000","last_became_peered":"2024-09-13T15:03:46.859557+0000","last_unstale":"2024-09-13T15:10:05.368045+0000","last_undegraded":"2024-09-13T15:10:05.368045+0000","last_fullsized":"2024-09-13T15:10:05.368045+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:00.933755+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":162,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053267+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:09:58.053267+0000","last_peered":"2024-09-13T15:09:58.053267+0000","last_clean":"2024-09-13T15:09:58.053267+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:09:58.053267+0000","last_undegraded":"2024-09-13T15:09:58.053267+0000","last_fullsized":"2024-09-13T15:09:58.053267+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":312,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313697+0000","last_change":"2024-09-13T15:07:34.635697+0000","last_active":"2024-09-13T15:09:58.313697+0000","last_peered":"2024-09-13T15:09:58.313697+0000","last_clean":"2024-09-13T15:09:58.313697+0000","last_became_active":"2024-09-13T15:07:34.568238+0000","last_became_peered":"2024-09-13T15:07:34.568238+0000","last_unstale":"2024-09-13T15:09:58.313697+0000","last_undegraded":"2024-09-13T15:09:58.313697+0000","last_fullsized":"2024-09-13T15:09:58.313697+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:55.995331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":298,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314030+0000","last_change":"2024-09-13T15:07:34.475965+0000","last_active":"2024-09-13T15:09:58.314030+0000","last_peered":"2024-09-13T15:09:58.314030+0000","last_clean":"2024-09-13T15:09:58.314030+0000","last_became_active":"2024-09-13T15:07:34.475412+0000","last_became_peered":"2024-09-13T15:07:34.475412+0000","last_unstale":"2024-09-13T15:09:58.314030+0000","last_undegraded":"2024-09-13T15:09:58.314030+0000","last_fullsized":"2024-09-13T15:09:58.314030+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:09:38.928248+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":572,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.313929+0000","last_change":"2024-09-13T15:02:19.426511+0000","last_active":"2024-09-13T15:09:58.313929+0000","last_peered":"2024-09-13T15:09:58.313929+0000","last_clean":"2024-09-13T15:09:58.313929+0000","last_became_active":"2024-09-13T15:02:19.391448+0000","last_became_peered":"2024-09-13T15:02:19.391448+0000","last_unstale":"2024-09-13T15:09:58.313929+0000","last_undegraded":"2024-09-13T15:09:58.313929+0000","last_fullsized":"2024-09-13T15:09:58.313929+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:40.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":196,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368649+0000","last_change":"2024-09-13T15:03:46.870536+0000","last_active":"2024-09-13T15:10:05.368649+0000","last_peered":"2024-09-13T15:10:05.368649+0000","last_clean":"2024-09-13T15:10:05.368649+0000","last_became_active":"2024-09-13T15:03:46.869736+0000","last_became_peered":"2024-09-13T15:03:46.869736+0000","last_unstale":"2024-09-13T15:10:05.368649+0000","last_undegraded":"2024-09-13T15:10:05.368649+0000","last_fullsized":"2024-09-13T15:10:05.368649+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:45.432825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":172,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.053480+0000","last_change":"2024-09-13T15:07:14.305701+0000","last_active":"2024-09-13T15:09:58.053480+0000","last_peered":"2024-09-13T15:09:58.053480+0000","last_clean":"2024-09-13T15:09:58.053480+0000","last_became_active":"2024-09-13T15:07:14.304628+0000","last_became_peered":"2024-09-13T15:07:14.304628+0000","last_unstale":"2024-09-13T15:09:58.053480+0000","last_undegraded":"2024-09-13T15:09:58.053480+0000","last_fullsized":"2024-09-13T15:09:58.053480+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:11.433108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":468,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312830+0000","last_change":"2024-09-13T15:07:34.466871+0000","last_active":"2024-09-13T15:09:58.312830+0000","last_peered":"2024-09-13T15:09:58.312830+0000","last_clean":"2024-09-13T15:09:58.312830+0000","last_became_active":"2024-09-13T15:07:34.466559+0000","last_became_peered":"2024-09-13T15:07:34.466559+0000","last_unstale":"2024-09-13T15:09:58.312830+0000","last_undegraded":"2024-09-13T15:09:58.312830+0000","last_fullsized":"2024-09-13T15:09:58.312830+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:48:24.377353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":239,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.313295+0000","last_change":"2024-09-13T15:03:46.864291+0000","last_active":"2024-09-13T15:10:06.313295+0000","last_peered":"2024-09-13T15:10:06.313295+0000","last_clean":"2024-09-13T15:10:06.313295+0000","last_became_active":"2024-09-13T15:03:46.863671+0000","last_became_peered":"2024-09-13T15:03:46.863671+0000","last_unstale":"2024-09-13T15:10:06.313295+0000","last_undegraded":"2024-09-13T15:10:06.313295+0000","last_fullsized":"2024-09-13T15:10:06.313295+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:28.672041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":217,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.367811+0000","last_change":"2024-09-13T15:07:34.489714+0000","last_active":"2024-09-13T15:10:05.367811+0000","last_peered":"2024-09-13T15:10:05.367811+0000","last_clean":"2024-09-13T15:10:05.367811+0000","last_became_active":"2024-09-13T15:07:34.489373+0000","last_became_peered":"2024-09-13T15:07:34.489373+0000","last_unstale":"2024-09-13T15:10:05.367811+0000","last_undegraded":"2024-09-13T15:10:05.367811+0000","last_fullsized":"2024-09-13T15:10:05.367811+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:40.264920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":239,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312944+0000","last_change":"2024-09-13T15:03:46.863848+0000","last_active":"2024-09-13T15:10:06.312944+0000","last_peered":"2024-09-13T15:10:06.312944+0000","last_clean":"2024-09-13T15:10:06.312944+0000","last_became_active":"2024-09-13T15:03:46.863523+0000","last_became_peered":"2024-09-13T15:03:46.863523+0000","last_unstale":"2024-09-13T15:10:06.312944+0000","last_undegraded":"2024-09-13T15:10:06.312944+0000","last_fullsized":"2024-09-13T15:10:06.312944+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:39.303427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":359,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.314105+0000","last_change":"2024-09-13T15:03:46.860138+0000","last_active":"2024-09-13T15:09:58.314105+0000","last_peered":"2024-09-13T15:09:58.314105+0000","last_clean":"2024-09-13T15:09:58.314105+0000","last_became_active":"2024-09-13T15:03:46.859819+0000","last_became_peered":"2024-09-13T15:03:46.859819+0000","last_unstale":"2024-09-13T15:09:58.314105+0000","last_undegraded":"2024-09-13T15:09:58.314105+0000","last_fullsized":"2024-09-13T15:09:58.314105+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:28.081805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":170,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312561+0000","last_change":"2024-09-13T14:58:28.428896+0000","last_active":"2024-09-13T15:09:58.312561+0000","last_peered":"2024-09-13T15:09:58.312561+0000","last_clean":"2024-09-13T15:09:58.312561+0000","last_became_active":"2024-09-13T14:58:28.428522+0000","last_became_peered":"2024-09-13T14:58:28.428522+0000","last_unstale":"2024-09-13T15:09:58.312561+0000","last_undegraded":"2024-09-13T15:09:58.312561+0000","last_fullsized":"2024-09-13T15:09:58.312561+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:05.418586+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":495,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T15:09:58.312994+0000","last_change":"2024-09-13T15:07:34.601477+0000","last_active":"2024-09-13T15:09:58.312994+0000","last_peered":"2024-09-13T15:09:58.312994+0000","last_clean":"2024-09-13T15:09:58.312994+0000","last_became_active":"2024-09-13T15:07:34.464188+0000","last_became_peered":"2024-09-13T15:07:34.464188+0000","last_unstale":"2024-09-13T15:09:58.312994+0000","last_undegraded":"2024-09-13T15:09:58.312994+0000","last_fullsized":"2024-09-13T15:09:58.312994+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:57.481351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":203,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:05.368089+0000","last_change":"2024-09-13T15:07:14.312451+0000","last_active":"2024-09-13T15:10:05.368089+0000","last_peered":"2024-09-13T15:10:05.368089+0000","last_clean":"2024-09-13T15:10:05.368089+0000","last_became_active":"2024-09-13T15:07:14.307807+0000","last_became_peered":"2024-09-13T15:07:14.307807+0000","last_unstale":"2024-09-13T15:10:05.368089+0000","last_undegraded":"2024-09-13T15:10:05.368089+0000","last_fullsized":"2024-09-13T15:10:05.368089+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:56:58.706128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":239,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-13T15:10:06.312882+0000","last_change":"2024-09-13T15:03:46.864604+0000","last_active":"2024-09-13T15:10:06.312882+0000","last_peered":"2024-09-13T15:10:06.312882+0000","last_clean":"2024-09-13T15:10:06.312882+0000","last_became_active":"2024-09-13T15:03:46.864085+0000","last_became_peered":"2024-09-13T15:03:46.864085+0000","last_unstale":"2024-09-13T15:10:06.312882+0000","last_undegraded":"2024-09-13T15:10:06.312882+0000","last_fullsized":"2024-09-13T15:10:06.312882+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:32.524052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":6278,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365406,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1520272,"kb_used_data":6788,"kb_used_omap":1,"kb_used_meta":1513470,"kb_avail":92224880,"statfs":{"total":95995035648,"available":94438277120,"internally_reserved":0,"allocated":6950912,"data_stored":4102507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1549793624},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561636,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499076,"kb_used_data":6840,"kb_used_omap":1,"kb_used_meta":1492222,"kb_avail":92246076,"statfs":{"total":95995035648,"available":94459981824,"internally_reserved":0,"allocated":7004160,"data_stored":4167322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1528035672},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790568,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497340,"kb_used_data":6768,"kb_used_omap":1,"kb_used_meta":1490558,"kb_avail":92247812,"statfs":{"total":95995035648,"available":94461759488,"internally_reserved":0,"allocated":6930432,"data_stored":4097904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1526331736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986796,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1508812,"kb_used_data":6528,"kb_used_omap":1,"kb_used_meta":1502270,"kb_avail":92236340,"statfs":{"total":95995035648,"available":94450012160,"internally_reserved":0,"allocated":6684672,"data_stored":3840014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1538324824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519951,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266808,"kb_used_data":6828,"kb_used_omap":4,"kb_used_meta":1259963,"kb_avail":92478344,"statfs":{"total":95995035648,"available":94697824256,"internally_reserved":0,"allocated":6991872,"data_stored":4162906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":1290202560},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304867,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1264960,"kb_used_data":6772,"kb_used_omap":6,"kb_used_meta":1258169,"kb_avail":92480192,"statfs":{"total":95995035648,"available":94699716608,"internally_reserved":0,"allocated":6934528,"data_stored":4098428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":1288365511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":416611828170,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482286035,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1256188,"kb_used_data":6832,"kb_used_omap":5,"kb_used_meta":1249338,"kb_avail":92488964,"statfs":{"total":95995035648,"available":94708699136,"internally_reserved":0,"allocated":6995968,"data_stored":4162199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":1279322171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:10:17.208 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T15:10:17.208 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:10:17.208 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [2, 6, 7, 5, 0, 3, 4] 2024-09-13T15:10:17.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd out 2 2024-09-13T15:10:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:17 smithi003 ceph-mon[107278]: pgmap v1690: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:17.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:17 smithi003 ceph-mon[107278]: from='client.47558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:10:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:17 smithi003 ceph-mon[104007]: pgmap v1690: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:17.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:17 smithi003 ceph-mon[104007]: from='client.47558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:10:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:17 smithi019 ceph-mon[91296]: pgmap v1690: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:17 smithi019 ceph-mon[91296]: from='client.47558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:10:17.720 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:18.968 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:10:18.968 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:10:18.969 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:10:18.969 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:10:18.969 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:10:19.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:19 smithi019 ceph-mon[91296]: pgmap v1691: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:19.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:19 smithi003 ceph-mon[104007]: pgmap v1691: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:19.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:19 smithi003 ceph-mon[107278]: pgmap v1691: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:10:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:10:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:10:20.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:10:21.061 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:21.377 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.2. 2024-09-13T15:10:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:21 smithi019 ceph-mon[91296]: pgmap v1692: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:21 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/818568088' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:10:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:21 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:10:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:21 smithi019 ceph-mon[91296]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T15:10:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:21 smithi003 ceph-mon[107278]: pgmap v1692: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:21 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/818568088' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:10:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:21 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:10:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:21 smithi003 ceph-mon[107278]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T15:10:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:21 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:10:21.195+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e481 definitely_dead 0 2024-09-13T15:10:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:21 smithi003 ceph-mon[104007]: pgmap v1692: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail 2024-09-13T15:10:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:21 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/818568088' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:10:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:21 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:10:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:21 smithi003 ceph-mon[104007]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T15:10:22.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:22 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T15:10:22.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:22 smithi003 ceph-mon[107278]: osdmap e482: 8 total, 8 up, 6 in 2024-09-13T15:10:22.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:22 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T15:10:22.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:22 smithi003 ceph-mon[104007]: osdmap e482: 8 total, 8 up, 6 in 2024-09-13T15:10:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:22 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T15:10:22.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:22 smithi019 ceph-mon[91296]: osdmap e482: 8 total, 8 up, 6 in 2024-09-13T15:10:23.456 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127130 13.b 13:d90b32dc:::benchmark_data_smithi003_293363_object7918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.241318+0000", 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 455.21176860700001, 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0674569620000001, 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.457 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127130 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241318+0000", 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241318+0000", 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241320+0000", 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0640000000000001e-06 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241326+0000", 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.754e-06 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241340+0000", 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3370000000000001e-05 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.241350+0000", 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0139000000000001e-05 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.242933+0000", 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015835319999999999 2024-09-13T15:10:23.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.242944+0000", 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1032999999999999e-05 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.197291+0000", 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.954346743 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.197302+0000", 2024-09-13T15:10:23.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0978999999999999e-05 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.289014+0000", 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.091712282 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.290488+0000", 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001473591 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4,5", 2024-09-13T15:10:23.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.290633+0000", 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014513300000000001 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.293633+0000", 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00299957 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297084+0000", 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034512610000000002 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297110+0000", 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5732999999999999e-05 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.308591+0000", 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011481460000000001 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.308652+0000", 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1274999999999999e-05 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.308775+0000", 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122043 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127131 13.b 13:d278da57:::benchmark_data_smithi003_293363_object7919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:23.465 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.243218+0000", 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 455.20986844800001, 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0678335479999999, 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127131 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243218+0000", 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243218+0000", 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243219+0000", 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.426e-06 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243234+0000", 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4453999999999999e-05 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243246+0000", 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1983e-05 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243254+0000", 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4740000000000005e-06 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243309+0000", 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5216999999999997e-05 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.243334+0000", 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4329999999999999e-05 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.197325+0000", 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.953991547 2024-09-13T15:10:23.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.197333+0000", 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5920000000000003e-06 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.293680+0000", 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0963472320000001 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.293777+0000", 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7516000000000006e-05 2024-09-13T15:10:23.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4,5", 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.293867+0000", 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9382000000000006e-05 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297117+0000", 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032505009999999998 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297305+0000", 2024-09-13T15:10:23.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018791000000000001 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.297361+0000", 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.613e-05 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.310877+0000", 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013515160999999999 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.310939+0000", 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2268999999999996e-05 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:50.311051+0000", 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011242400000000001 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.473 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176522 13.b 13:dd5b9f0f:::benchmark_data_smithi003_293363_object57310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.828681+0000", 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 398.624404728, 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0237406389999999, 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176522 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828681+0000", 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828681+0000", 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828682+0000", 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9200000000000001e-07 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828684+0000", 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2639999999999998e-06 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828691+0000", 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.545e-06 2024-09-13T15:10:23.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828696+0000", 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9219999999999996e-06 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828744+0000", 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8066999999999999e-05 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828873+0000", 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000129395 2024-09-13T15:10:23.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4,5", 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.828994+0000", 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121124 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.836675+0000", 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076807539999999997 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.840494+0000", 2024-09-13T15:10:23.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038188470000000002 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.841971+0000", 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014773620000000001 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.869201+0000", 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027230061999999999 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.869211+0000", 2024-09-13T15:10:23.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5140000000000007e-06 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852408+0000", 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98319757399999996 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852422+0000", 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3716999999999999e-05 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176536 13.b 13:ddb01560:::benchmark_data_smithi003_293363_object57324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.855022+0000", 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 398.59806413699999, 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99743847200000002, 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176536 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.481 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855022+0000", 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855022+0000", 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855023+0000", 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.252e-06 2024-09-13T15:10:23.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855029+0000", 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3109999999999996e-06 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855042+0000", 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3067999999999999e-05 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855050+0000", 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3010000000000007e-06 2024-09-13T15:10:23.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855254+0000", 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020365800000000001 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855386+0000", 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000132547 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4,5", 2024-09-13T15:10:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.855517+0000", 2024-09-13T15:10:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131326 2024-09-13T15:10:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.871590+0000", 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016072521999999999 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.871619+0000", 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9026999999999998e-05 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852451+0000", 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.980831699 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852460+0000", 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7610000000000002e-06 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176559 13.b 13.8239baab (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.878857+0000", 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 398.57422959799999, 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97363370699999996, 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.487 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176559 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878857+0000", 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878857+0000", 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878858+0000", 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.359e-06 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878864+0000", 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6620000000000002e-06 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878873+0000", 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8849999999999994e-06 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878880+0000", 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3169999999999998e-06 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878939+0000", 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9685000000000002e-05 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.878947+0000", 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9980000000000003e-06 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852482+0000", 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97353494100000004 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852490+0000", 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8599999999999993e-06 2024-09-13T15:10:23.491 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176618 13.b 13.834c39db (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.954483+0000", 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 398.49860332399999, 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.898035413, 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176618 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954483+0000", 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954483+0000", 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954484+0000", 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.64e-06 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954490+0000", 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4820000000000002e-06 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954507+0000", 2024-09-13T15:10:23.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7000999999999999e-05 2024-09-13T15:10:23.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954516+0000", 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9570000000000008e-06 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954585+0000", 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8709999999999998e-05 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.954595+0000", 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9420000000000007e-06 2024-09-13T15:10:23.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852510+0000", 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89791587799999995 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852518+0000", 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8029999999999997e-06 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.498 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176637 13.b 13.fc35e09b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:10:23.498 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.976426+0000", 2024-09-13T15:10:23.498 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 398.476660235, 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87612334999999997, 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176637 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976426+0000", 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976426+0000", 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976428+0000", 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.762e-06 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976434+0000", 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1709999999999999e-06 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976448+0000", 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4541e-05 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976457+0000", 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3729999999999993e-06 2024-09-13T15:10:23.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976534+0000", 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7105000000000001e-05 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.976544+0000", 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0036e-05 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852541+0000", 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87599744099999999 2024-09-13T15:10:23.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852549+0000", 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9209999999999992e-06 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176642 13.b 13.92db103b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:44.979747+0000", 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 398.47333878500001, 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87283178299999997, 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176642 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979747+0000", 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979747+0000", 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979749+0000", 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.376e-06 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979754+0000", 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.558e-06 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979767+0000", 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.254e-05 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979774+0000", 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2989999999999999e-06 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979836+0000", 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1471999999999997e-05 2024-09-13T15:10:23.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:44.979844+0000", 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3529999999999995e-06 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852571+0000", 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87272746199999995 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852579+0000", 2024-09-13T15:10:23.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7230000000000004e-06 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176681 13.b 13.a6fc3bab (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.025138+0000", 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 398.427948098, 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.827471603, 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.508 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176681 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025138+0000", 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025138+0000", 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025139+0000", 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2130000000000001e-06 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025145+0000", 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2399999999999998e-06 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025155+0000", 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0327e-05 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025162+0000", 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8360000000000003e-06 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025284+0000", 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121896 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.025291+0000", 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7759999999999997e-06 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852602+0000", 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82731021599999999 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852610+0000", 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0989999999999999e-06 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176684 13.b 13.cf4c3d6b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.026384+0000", 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 398.42670257700001, 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82625494399999999, 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.513 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176684 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026384+0000", 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026384+0000", 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026384+0000", 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0999999999999997e-07 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026388+0000", 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.399e-06 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026397+0000", 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6149999999999997e-06 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026402+0000", 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7860000000000002e-06 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026465+0000", 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2617e-05 2024-09-13T15:10:23.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.026474+0000", 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.653e-06 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852631+0000", 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82615713899999998 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852639+0000", 2024-09-13T15:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8250000000000005e-06 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176690 13.b 13.a2ad84cb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.031425+0000", 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 398.42166151700002, 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82124366999999998, 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.518 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176690 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031425+0000", 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031425+0000", 2024-09-13T15:10:23.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031425+0000", 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1500000000000003e-07 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031430+0000", 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3059999999999999e-06 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031445+0000", 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5152e-05 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031453+0000", 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8290000000000008e-06 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031509+0000", 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6678999999999997e-05 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.031519+0000", 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0030999999999999e-05 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852661+0000", 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82114151800000001 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.852668+0000", 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.34e-06 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:203844 13.8 13:1625f442:::benchmark_data_smithi003_293363_object84632:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:16.261053+0000", 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 367.19203323800002, 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.068389952, 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.523 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 203844 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261053+0000", 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261053+0000", 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261055+0000", 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1299999999999999e-06 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261063+0000", 2024-09-13T15:10:23.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1499999999999999e-06 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261080+0000", 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6597999999999999e-05 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.261090+0000", 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0103e-05 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.524007+0000", 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26291679200000001 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:16.524037+0000", 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9847000000000001e-05 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.291818+0000", 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.76778101099999996 2024-09-13T15:10:23.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.291981+0000", 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016377500000000001 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.292454+0000", 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000473144 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.297532+0000", 2024-09-13T15:10:23.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005077285 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.297546+0000", 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3783e-05 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.329324+0000", 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.031778907000000002 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.329397+0000", 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2489000000000003e-05 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:17.329443+0000", 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5938e-05 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:23 smithi003 ceph-mon[104007]: pgmap v1694: 103 pgs: 9 peering, 94 active+clean; 582 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail 2024-09-13T15:10:23.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:23 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:10:23.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:23 smithi003 ceph-mon[104007]: osdmap e483: 8 total, 8 up, 6 in 2024-09-13T15:10:23.531 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:23 smithi003 ceph-mon[107278]: pgmap v1694: 103 pgs: 9 peering, 94 active+clean; 582 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail 2024-09-13T15:10:23.531 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:23 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:10:23.531 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:23 smithi003 ceph-mon[107278]: osdmap e483: 8 total, 8 up, 6 in 2024-09-13T15:10:23.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.532 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:230553 13.a 13:55a3516a:::benchmark_data_smithi003_293363_object111341:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:23.532 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:40.523014+0000", 2024-09-13T15:10:23.532 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 342.930071672, 2024-09-13T15:10:23.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0051218200000001, 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 230553 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523014+0000", 2024-09-13T15:10:23.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523014+0000", 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523016+0000", 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5719999999999999e-06 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523021+0000", 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8670000000000002e-06 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523033+0000", 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1586999999999999e-05 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523044+0000", 2024-09-13T15:10:23.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1484000000000001e-05 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.669000+0000", 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.145956051 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.669044+0000", 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3952000000000002e-05 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.487776+0000", 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.81873235099999997 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.487983+0000", 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020640799999999999 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.488376+0000", 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039276799999999999 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.491093+0000", 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027174759999999999 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.525660+0000", 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.034567154000000003 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.528041+0000", 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002380418 2024-09-13T15:10:23.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.528097+0000", 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6279000000000001e-05 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.528136+0000", 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9453000000000003e-05 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:230565 13.1a 13:5e506c0d:::benchmark_data_smithi003_293363_object111353:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:40.523058+0000", 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 342.93002855499998, 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.009851184, 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 230565 2024-09-13T15:10:23.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523058+0000", 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523058+0000", 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523058+0000", 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3799999999999998e-07 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523060+0000", 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.001e-06 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523066+0000", 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2020000000000001e-06 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523071+0000", 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1719999999999998e-06 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.669099+0000", 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14602759900000001 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.669105+0000", 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6579999999999999e-06 2024-09-13T15:10:23.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.525684+0000", 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.85657905099999998 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.525809+0000", 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012483000000000001 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.526065+0000", 2024-09-13T15:10:23.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025653900000000002 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.528588+0000", 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025232290000000001 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.529038+0000", 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000450035 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.532795+0000", 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003756526 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.532864+0000", 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9552000000000001e-05 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.532909+0000", 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4351999999999998e-05 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.546 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:230567 13.a 13:57e709a8:::benchmark_data_smithi003_293363_object111355:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:04:40.523077+0000", 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 342.93000873699998, 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.005947527, 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 230567 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.547 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523077+0000", 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523077+0000", 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523078+0000", 2024-09-13T15:10:23.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6600000000000002e-07 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523079+0000", 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6789999999999999e-06 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523085+0000", 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4949999999999999e-06 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.523090+0000", 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.916e-06 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.669119+0000", 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14602916799999999 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:40.669128+0000", 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6710000000000005e-06 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.491122+0000", 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82199413600000004 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.491243+0000", 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121409 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.491505+0000", 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026223499999999999 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.527923+0000", 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036417502999999997 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.527998+0000", 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4512999999999996e-05 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.528573+0000", 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00057551200000000003 2024-09-13T15:10:23.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.528939+0000", 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036629799999999998 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:04:41.529025+0000", 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5625999999999996e-05 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.553 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:123391 14.e 14:773617bc:::benchmark_data_smithi003_327099_object11665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:13.295906+0000", 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 190.15718007699999, 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.130988053, 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123391 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.295906+0000", 2024-09-13T15:10:23.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.295906+0000", 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.295908+0000", 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5969999999999999e-06 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.295914+0000", 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9989999999999997e-06 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.295927+0000", 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3672999999999999e-05 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.295938+0000", 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0709e-05 2024-09-13T15:10:23.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.296432+0000", 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049366600000000003 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.296456+0000", 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3915000000000001e-05 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.417066+0000", 2024-09-13T15:10:23.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.120610468 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.417148+0000", 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2206999999999994e-05 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,6", 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.417602+0000", 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000454105 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.425929+0000", 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0083263580000000007 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.426122+0000", 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019341700000000001 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.426171+0000", 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9214e-05 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.426780+0000", 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00060832799999999999 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.426817+0000", 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7129000000000003e-05 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.426894+0000", 2024-09-13T15:10:23.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7268000000000006e-05 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:136675 14.e 14:7c62bf51:::benchmark_data_smithi003_327099_object24949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:33.398723+0000", 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 170.05436276, 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.040141094, 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.561 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136675 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.398723+0000", 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.398723+0000", 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.398725+0000", 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.469e-06 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.398730+0000", 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.87e-06 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.398742+0000", 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.217e-05 2024-09-13T15:10:23.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.398749+0000", 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5440000000000001e-06 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.398805+0000", 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5606000000000003e-05 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.398981+0000", 2024-09-13T15:10:23.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017568 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,6", 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.399135+0000", 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015438800000000001 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.403227+0000", 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040916099999999999 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.403241+0000", 2024-09-13T15:10:23.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4543e-05 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.403741+0000", 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049991499999999997 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.446539+0000", 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.042798165999999999 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.446554+0000", 2024-09-13T15:10:23.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4904e-05 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:34.438839+0000", 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99228457800000003 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:34.438864+0000", 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5650999999999999e-05 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:136689 14.e 14.8f84682e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:33.585116+0000", 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 169.86797065299999, 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.85383145900000001, 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136689 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.585116+0000", 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.585116+0000", 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.585118+0000", 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3140000000000002e-06 2024-09-13T15:10:23.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.585124+0000", 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3690000000000003e-06 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.585143+0000", 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8955999999999999e-05 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.585152+0000", 2024-09-13T15:10:23.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8179999999999993e-06 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.586340+0000", 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001187603 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.586353+0000", 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3704e-05 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:34.438925+0000", 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.85257220499999997 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:34.438947+0000", 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1489999999999999e-05 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:175035 14.b e449/432 14:dab7da8c:::benchmark_data_smithi003_327099_object63309:head v 449'10702, mlcod=449'10700)", 2024-09-13T15:10:23.572 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:08:20.883703+0000", 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 122.569383364, 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19781968799999999, 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.883703+0000", 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.883703+0000", 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.883704+0000", 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4099999999999997e-07 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.883726+0000", 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.19e-05 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.883739+0000", 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2889e-05 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.883746+0000", 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5079999999999999e-06 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.883817+0000", 2024-09-13T15:10:23.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0868e-05 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.883839+0000", 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2299000000000001e-05 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:21.081485+0000", 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19764547499999999 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:21.081522+0000", 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7808000000000001e-05 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.47273.0:175037 14.b e449/432 14:dd17d983:::benchmark_data_smithi003_327099_object63311:head v 449'10703, mlcod=449'10700)", 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:08:20.885235+0000", 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 122.567851232, 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.196321367, 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.885235+0000", 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.885235+0000", 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:23.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.885236+0000", 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8699999999999995e-07 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.885241+0000", 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3129999999999998e-06 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.885251+0000", 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9899999999999992e-06 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.885257+0000", 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2269999999999998e-06 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.886578+0000", 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013205199999999999 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:20.886600+0000", 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1996000000000001e-05 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:21.081527+0000", 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19492698999999999 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:21.081556+0000", 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9743999999999998e-05 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:10:23.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:23 smithi019 ceph-mon[91296]: pgmap v1694: 103 pgs: 9 peering, 94 active+clean; 582 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail 2024-09-13T15:10:23.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:23 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:10:23.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:23 smithi019 ceph-mon[91296]: osdmap e483: 8 total, 8 up, 6 in 2024-09-13T15:10:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:10:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:10:24] "GET /metrics HTTP/1.1" 200 34110 "" "Prometheus/2.51.0" 2024-09-13T15:10:24.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:10:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:25 smithi019 ceph-mon[91296]: 1.a scrub starts 2024-09-13T15:10:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:25 smithi019 ceph-mon[91296]: 1.a scrub ok 2024-09-13T15:10:25.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:25 smithi019 ceph-mon[91296]: pgmap v1696: 103 pgs: 9 peering, 94 active+clean; 608 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:25 smithi003 ceph-mon[107278]: 1.a scrub starts 2024-09-13T15:10:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:25 smithi003 ceph-mon[107278]: 1.a scrub ok 2024-09-13T15:10:25.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:25 smithi003 ceph-mon[107278]: pgmap v1696: 103 pgs: 9 peering, 94 active+clean; 608 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:25 smithi003 ceph-mon[104007]: 1.a scrub starts 2024-09-13T15:10:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:25 smithi003 ceph-mon[104007]: 1.a scrub ok 2024-09-13T15:10:25.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:25 smithi003 ceph-mon[104007]: pgmap v1696: 103 pgs: 9 peering, 94 active+clean; 608 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:26.500 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:10:26.500 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:10:26.500 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-13T15:10:26.500 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:10:27.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:10:27.479 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:27.588 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T15:10:27.588 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-13T15:10:27.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd in 1 2024-09-13T15:10:27.601 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:27 smithi003 ceph-mon[107278]: pgmap v1697: 103 pgs: 9 peering, 94 active+clean; 608 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:27.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:27 smithi003 ceph-mon[104007]: pgmap v1697: 103 pgs: 9 peering, 94 active+clean; 608 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:27 smithi019 ceph-mon[91296]: pgmap v1697: 103 pgs: 9 peering, 94 active+clean; 608 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:28.318 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:29.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:29.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:29.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:28 smithi003 ceph-mon[107278]: pgmap v1698: 103 pgs: 103 active+clean; 608 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:29.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:29.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:29.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:28 smithi003 ceph-mon[104007]: pgmap v1698: 103 pgs: 103 active+clean; 608 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:29.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:28 smithi019 ceph-mon[91296]: pgmap v1698: 103 pgs: 103 active+clean; 608 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:29.949 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:29 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-13T15:10:29.949 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:29 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:10:29.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:29 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-13T15:10:29.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:29 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:10:30.057 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:10:30.057 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-13T15:10:30.057 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-13T15:10:30.058 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-13T15:10:30.058 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:10:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:29 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-13T15:10:30.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:29 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:10:30.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:10:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:10:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:10:30.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:10:31.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:30 smithi003 ceph-mon[104007]: pgmap v1699: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 29 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:31.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:30 smithi003 ceph-mon[107278]: pgmap v1699: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 29 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:30 smithi019 ceph-mon[91296]: pgmap v1699: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 29 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:32.330 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.1. 2024-09-13T15:10:32.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2582185963' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T15:10:32.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:32 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T15:10:32.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:32 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:10:32.096+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e483 definitely_dead 0 2024-09-13T15:10:32.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2582185963' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T15:10:32.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:32 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T15:10:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2582185963' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T15:10:32.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:32 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T15:10:33.051 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-13T15:10:33.051 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T15:10:33.051 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-13T15:10:33.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd in 2 2024-09-13T15:10:33.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:33 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T15:10:33.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:33 smithi003 ceph-mon[104007]: osdmap e484: 8 total, 8 up, 7 in 2024-09-13T15:10:33.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:33 smithi003 ceph-mon[104007]: pgmap v1701: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 26 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:33.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:33 smithi003 ceph-mon[104007]: osdmap e485: 8 total, 8 up, 7 in 2024-09-13T15:10:33.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:33 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T15:10:33.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:33 smithi003 ceph-mon[107278]: osdmap e484: 8 total, 8 up, 7 in 2024-09-13T15:10:33.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:33 smithi003 ceph-mon[107278]: pgmap v1701: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 26 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:33.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:33 smithi003 ceph-mon[107278]: osdmap e485: 8 total, 8 up, 7 in 2024-09-13T15:10:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:33 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T15:10:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:33 smithi019 ceph-mon[91296]: osdmap e484: 8 total, 8 up, 7 in 2024-09-13T15:10:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:33 smithi019 ceph-mon[91296]: pgmap v1701: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 26 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:33.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:33 smithi019 ceph-mon[91296]: osdmap e485: 8 total, 8 up, 7 in 2024-09-13T15:10:34.074 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127114 13.3 13:c1660045:::benchmark_data_smithi003_293363_object7902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.202037+0000", 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 465.838013746, 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.135729252, 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.075 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127114 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202037+0000", 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202037+0000", 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202039+0000", 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.875e-06 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202046+0000", 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4459999999999998e-06 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202060+0000", 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4323e-05 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.202068+0000", 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3270000000000001e-06 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.203713+0000", 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001644413 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.203830+0000", 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116867 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.203956+0000", 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012679299999999999 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244242+0000", 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040285728 2024-09-13T15:10:34.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244254+0000", 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1476e-05 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.328915+0000", 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0846615079999999 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for missing object", 2024-09-13T15:10:34.080 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329047+0000", 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013149400000000001 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.337707+0000", 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0086602120000000005 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.081 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.337767+0000", 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9790000000000002e-05 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127123 13.3 13:cfeb70cd:::benchmark_data_smithi003_293363_object7911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.232129+0000", 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 465.80792201899999, 2024-09-13T15:10:34.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1073596480000001, 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127123 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232129+0000", 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232129+0000", 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232131+0000", 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.699e-06 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232137+0000", 2024-09-13T15:10:34.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.224e-06 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232150+0000", 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3455000000000001e-05 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.232158+0000", 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6029999999999999e-06 2024-09-13T15:10:34.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244280+0000", 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012121709 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.244287+0000", 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8340000000000001e-06 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329294+0000", 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0850076449999999 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329431+0000", 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013655399999999999 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.329578+0000", 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014753499999999999 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336227+0000", 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066491049999999998 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336286+0000", 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9038e-05 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336352+0000", 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5582999999999999e-05 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339347+0000", 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029945530000000001 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339394+0000", 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7123000000000003e-05 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339489+0000", 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4988000000000006e-05 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.089 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:127141 13.3 13:c8a2a879:::benchmark_data_smithi003_293363_object7929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:02:48.256678+0000", 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 465.78337328600003, 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.082947579, 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.090 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 127141 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256678+0000", 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256678+0000", 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256680+0000", 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9769999999999999e-06 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256687+0000", 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2980000000000003e-06 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256708+0000", 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1121999999999999e-05 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256716+0000", 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0460000000000006e-06 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256778+0000", 2024-09-13T15:10:34.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2205999999999994e-05 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:48.256787+0000", 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4100000000000008e-06 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.332032+0000", 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0752451359999999 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.332117+0000", 2024-09-13T15:10:34.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4950000000000005e-05 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.332198+0000", 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0968000000000001e-05 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336368+0000", 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041701589999999997 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336418+0000", 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9521999999999998e-05 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.336452+0000", 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4790999999999999e-05 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339529+0000", 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030764569999999999 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339561+0000", 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2246999999999998e-05 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:02:49.339625+0000", 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4289999999999996e-05 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176637 13.b 13:d907ac3f:::benchmark_data_smithi003_293363_object57425:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.097 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865081+0000", 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17496958200002, 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.097294284, 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176637 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865081+0000", 2024-09-13T15:10:34.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865081+0000", 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865082+0000", 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8099999999999998e-07 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865084+0000", 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.182e-06 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865091+0000", 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5649999999999998e-06 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865096+0000", 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6420000000000004e-06 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865251+0000", 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015508899999999999 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865258+0000", 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2219999999999999e-06 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935706+0000", 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0704488969999999 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935808+0000", 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010106 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.935897+0000", 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9606999999999998e-05 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944419+0000", 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0085219030000000008 2024-09-13T15:10:34.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944552+0000", 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133387 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962201+0000", 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017648440000000001 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962260+0000", 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9416999999999997e-05 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962376+0000", 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115492 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176642 13.b 13:dc08db49:::benchmark_data_smithi003_293363_object57430:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:10:34.104 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17494121200002, 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.09749176, 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176642 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:10:34.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865110+0000", 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0200000000000003e-07 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865113+0000", 2024-09-13T15:10:34.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.61e-06 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865120+0000", 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8870000000000003e-06 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865125+0000", 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4920000000000001e-06 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:10:34.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153487 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865285+0000", 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3799999999999999e-06 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.938324+0000", 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0730388900000001 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.938395+0000", 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0896999999999996e-05 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.938507+0000", 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112172 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944565+0000", 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006058066 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944611+0000", 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6084999999999997e-05 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962436+0000", 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017824815000000001 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962496+0000", 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0374999999999997e-05 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962602+0000", 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105202 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176681 13.b 13:d5dc3f65:::benchmark_data_smithi003_293363_object57469:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17491846299998, 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0977554540000001, 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.111 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176681 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865133+0000", 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9999999999999998e-07 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865135+0000", 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2790000000000001e-06 2024-09-13T15:10:34.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865142+0000", 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3849999999999999e-06 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865147+0000", 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2109999999999997e-06 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865307+0000", 2024-09-13T15:10:34.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000159815 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865313+0000", 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1129999999999997e-06 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.941097+0000", 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0757844999999999 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.941212+0000", 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011456 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.941347+0000", 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013543100000000001 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944571+0000", 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032240150000000002 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944667+0000", 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5983999999999994e-05 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962729+0000", 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018061628999999999 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962782+0000", 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2652999999999999e-05 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962888+0000", 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010647899999999999 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176684 13.b 13:d6bc32f3:::benchmark_data_smithi003_293363_object57472:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17489737900002, 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.098160848, 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.118 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176684 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865154+0000", 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7899999999999999e-07 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865156+0000", 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2120000000000002e-06 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865163+0000", 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2979999999999997e-06 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865168+0000", 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.938e-06 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865339+0000", 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000171719 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865345+0000", 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.122e-06 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944711+0000", 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0793655870000001 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.944815+0000", 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010432199999999999 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.945061+0000", 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024528700000000001 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955429+0000", 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010368297 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955546+0000", 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117385 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963159+0000", 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076123060000000001 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963223+0000", 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4424000000000005e-05 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963315+0000", 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1571999999999997e-05 2024-09-13T15:10:34.124 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176690 13.b 13:d321b545:::benchmark_data_smithi003_293363_object57478:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17487719799999, 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.098341236, 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.125 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176690 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865174+0000", 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0400000000000002e-07 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865176+0000", 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.12e-06 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865191+0000", 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4345000000000001e-05 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865196+0000", 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6949999999999997e-06 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865366+0000", 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016998100000000001 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865372+0000", 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9170000000000003e-06 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.948302+0000", 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0829294700000001 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.948432+0000", 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013043500000000001 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.948566+0000", 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013371600000000001 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955473+0000", 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069065769999999997 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962132+0000", 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006659954 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963377+0000", 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012440940000000001 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963425+0000", 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8899000000000001e-05 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963515+0000", 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9629000000000005e-05 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.131 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176701 13.b 13:d317b68b:::benchmark_data_smithi003_293363_object57489:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17484719999999, 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0984964370000001, 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176701 2024-09-13T15:10:34.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865204+0000", 2024-09-13T15:10:34.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0100000000000005e-07 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865207+0000", 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4959999999999999e-06 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865213+0000", 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6279999999999999e-06 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865219+0000", 2024-09-13T15:10:34.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1059999999999999e-06 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865442+0000", 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022335700000000001 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865449+0000", 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1289999999999998e-06 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.952217+0000", 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.086767781 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.952348+0000", 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130698 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.952485+0000", 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013729000000000001 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955532+0000", 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030474899999999999 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962140+0000", 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066075830000000002 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963560+0000", 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014204339999999999 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963608+0000", 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7970000000000003e-05 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963700+0000", 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1973999999999997e-05 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176716 13.b 13:d4377599:::benchmark_data_smithi003_293363_object57504:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17482598200002, 2024-09-13T15:10:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0986561319999999, 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176716 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865225+0000", 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7599999999999998e-07 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865228+0000", 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0619999999999999e-06 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865234+0000", 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3949999999999998e-06 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865239+0000", 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1699999999999996e-06 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865472+0000", 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023298500000000001 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865478+0000", 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9479999999999996e-06 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955574+0000", 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0900960959999999 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955659+0000", 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4608e-05 2024-09-13T15:10:34.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.955737+0000", 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8046000000000005e-05 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962615+0000", 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068780509999999996 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962671+0000", 2024-09-13T15:10:34.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6412999999999997e-05 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963738+0000", 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010668470000000001 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963787+0000", 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8671000000000002e-05 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963881+0000", 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4463999999999995e-05 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176737 13.b 13:ddd89c0c:::benchmark_data_smithi003_293363_object57525:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865245+0000", 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17480580199998, 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.098807461, 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176737 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865245+0000", 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865245+0000", 2024-09-13T15:10:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865246+0000", 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6300000000000001e-07 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865259+0000", 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3437e-05 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865266+0000", 2024-09-13T15:10:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0720000000000002e-06 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865272+0000", 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6500000000000001e-06 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865499+0000", 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022734299999999999 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865505+0000", 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1879999999999999e-06 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.958889+0000", 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.093383966 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.959023+0000", 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133664 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.959159+0000", 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013578400000000001 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.962904+0000", 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037453019999999998 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963100+0000", 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000195565 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963924+0000", 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000824711 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.963971+0000", 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6295000000000003e-05 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964053+0000", 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2120999999999995e-05 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.151 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176756 13.b 13:d9b260c5:::benchmark_data_smithi003_293363_object57544:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17477161400001, 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1185605869999999, 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.152 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176756 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865279+0000", 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865280+0000", 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0800000000000005e-07 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865282+0000", 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2139999999999999e-06 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865288+0000", 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.297e-06 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865293+0000", 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9450000000000001e-06 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865527+0000", 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023310799999999999 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865532+0000", 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9569999999999999e-06 2024-09-13T15:10:34.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964086+0000", 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0985532360000001 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964196+0000", 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110356 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.964312+0000", 2024-09-13T15:10:34.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116267 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976732+0000", 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012419902999999999 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976741+0000", 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8419999999999994e-06 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.983604+0000", 2024-09-13T15:10:34.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006862511 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.983720+0000", 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116086 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.983840+0000", 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120357 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176775 13.b 13:d5416e9e:::benchmark_data_smithi003_293363_object57563:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17475131999998, 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1439602769999999, 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176775 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865300+0000", 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.77e-07 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865302+0000", 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8989999999999999e-06 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865308+0000", 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3040000000000001e-06 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865313+0000", 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0599999999999998e-06 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866806+0000", 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001492581 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866814+0000", 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5619999999999998e-06 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.967407+0000", 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.100593173 2024-09-13T15:10:34.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.967483+0000", 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6363000000000001e-05 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.967581+0000", 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7825000000000006e-05 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976749+0000", 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0091678430000000002 2024-09-13T15:10:34.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976804+0000", 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5241000000000002e-05 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009025+0000", 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032221078 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009088+0000", 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2738999999999997e-05 2024-09-13T15:10:34.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009260+0000", 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000172232 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176784 13.b 13:d9e88943:::benchmark_data_smithi003_293363_object57572:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17436550500003, 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.143711921, 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.165 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176784 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865686+0000", 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1600000000000001e-07 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865689+0000", 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7190000000000001e-06 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865697+0000", 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7959999999999996e-06 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865703+0000", 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7750000000000004e-06 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866835+0000", 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001132128 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866842+0000", 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6200000000000001e-06 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.970297+0000", 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1034550540000001 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.970390+0000", 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3011999999999996e-05 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.970471+0000", 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1049000000000003e-05 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976847+0000", 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063753589999999997 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976866+0000", 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.897e-05 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009300+0000", 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032434642 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009336+0000", 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6041000000000003e-05 2024-09-13T15:10:34.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.009397+0000", 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1240000000000003e-05 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36909.0:176811 13.b 13:d3189ffb:::benchmark_data_smithi003_293363_object57599:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:03:45.865711+0000", 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 408.17433978999998, 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1460795859999999, 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.172 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36909", 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176811 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865711+0000", 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865711+0000", 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865712+0000", 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1100000000000001e-07 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865714+0000", 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0679999999999999e-06 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865720+0000", 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3029999999999996e-06 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.865725+0000", 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.989e-06 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866873+0000", 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011484360000000001 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:45.866880+0000", 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5289999999999997e-06 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.972880+0000", 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.105999841 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.973094+0000", 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000213888 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.973209+0000", 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011561300000000001 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:46.976918+0000", 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037089139999999998 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.000375+0000", 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023456290000000001 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.011630+0000", 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011255292 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.011734+0000", 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104122 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:03:47.011791+0000", 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6889999999999999e-05 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.178 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:123386 14.1 14:880e9125:::benchmark_data_smithi003_327099_object11660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:13.271860+0000", 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 200.76819061099999, 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1641921220000002, 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123386 2024-09-13T15:10:34.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271860+0000", 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271860+0000", 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271862+0000", 2024-09-13T15:10:34.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.686e-06 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271869+0000", 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2949999999999996e-06 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271887+0000", 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7829000000000002e-05 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.271896+0000", 2024-09-13T15:10:34.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3950000000000001e-06 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.272239+0000", 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034359100000000002 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.272371+0000", 2024-09-13T15:10:34.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013179500000000001 2024-09-13T15:10:34.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:10:34.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.272500+0000", 2024-09-13T15:10:34.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012896399999999999 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.279137+0000", 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066367229999999998 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.291636+0000", 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012499148999999999 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.312212+0000", 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020576522 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.312229+0000", 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6231000000000001e-05 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.315906+0000", 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0036772819999999 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.315921+0000", 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5559000000000001e-05 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.435973+0000", 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1200512170000001 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.436053+0000", 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9883999999999996e-05 2024-09-13T15:10:34.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:123398 14.1 14:89cc6598:::benchmark_data_smithi003_327099_object11672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:13.319706+0000", 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 200.72034494299999, 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1364959680000002, 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:10:34.187 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 123398 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319706+0000", 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319706+0000", 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319708+0000", 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8670000000000001e-06 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319716+0000", 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0139999999999998e-06 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319736+0000", 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0196000000000001e-05 2024-09-13T15:10:34.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319749+0000", 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2745000000000001e-05 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319869+0000", 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120412 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:13.319882+0000", 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2543e-05 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.315954+0000", 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99607232400000001 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:14.315963+0000", 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9730000000000003e-06 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.436320+0000", 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.120356753 2024-09-13T15:10:34.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.436436+0000", 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011609599999999999 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.438424+0000", 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019879099999999999 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.450917+0000", 2024-09-13T15:10:34.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012492771999999999 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.451974+0000", 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010571580000000001 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.452675+0000", 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070099099999999998 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.456061+0000", 2024-09-13T15:10:34.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033863970000000002 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.456107+0000", 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5420000000000002e-05 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:15.456202+0000", 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5396999999999995e-05 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:136678 14.9 14:9c1f4eb4:::benchmark_data_smithi003_327099_object24952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:07:33.401566+0000", 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 180.638485409, 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0827195519999999, 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136678 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401566+0000", 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401566+0000", 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401567+0000", 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6360000000000001e-06 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401573+0000", 2024-09-13T15:10:34.197 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:34.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5879999999999997e-06 2024-09-13T15:10:34.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401586+0000", 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3542e-05 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401594+0000", 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8849999999999999e-06 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401653+0000", 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8292999999999999e-05 2024-09-13T15:10:34.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401761+0000", 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010793000000000001 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.401885+0000", 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012484400000000001 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.404245+0000", 2024-09-13T15:10:34.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002359407 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.405002+0000", 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000757312 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.405090+0000", 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7656000000000002e-05 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.448173+0000", 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.043083370000000003 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:33.448187+0000", 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3782e-05 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:34.484215+0000", 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0360284209999999 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:07:34.484285+0000", 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9886000000000001e-05 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:210863 14.a 14:552142d9:::benchmark_data_smithi003_327099_object99137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:08:59.839731+0000", 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.200319726999993, 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.065348049, 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.202 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 210863 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.839731+0000", 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.839731+0000", 2024-09-13T15:10:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.839733+0000", 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6619999999999999e-06 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.839739+0000", 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9349999999999999e-06 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.839752+0000", 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2595999999999999e-05 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.839759+0000", 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1330000000000001e-06 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.845687+0000", 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059281560000000004 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.845698+0000", 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0688e-05 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.901340+0000", 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.055642212 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.901478+0000", 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013810800000000001 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:34.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.901630+0000", 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015199300000000001 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.904928+0000", 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032982829999999999 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.904973+0000", 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4478999999999999e-05 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.904991+0000", 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8068e-05 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.905042+0000", 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1455000000000001e-05 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.905079+0000", 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7280999999999998e-05 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.47273.0:210896 14.a 14:5fa38d65:::benchmark_data_smithi003_327099_object99170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e457)", 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-13T15:08:59.868918+0000", 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.171133061000006, 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.043929962, 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.47273", 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 210896 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-13T15:10:34.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.868918+0000", 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.868918+0000", 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.868919+0000", 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.269e-06 2024-09-13T15:10:34.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.868924+0000", 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0540000000000002e-06 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.868940+0000", 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5739999999999998e-05 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.868948+0000", 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1699999999999997e-06 2024-09-13T15:10:34.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.869003+0000", 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4565999999999999e-05 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:08:59.869011+0000", 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2930000000000001e-06 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.905130+0000", 2024-09-13T15:10:34.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.036118554 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.905233+0000", 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102988 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.905296+0000", 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3100000000000002e-05 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.912626+0000", 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073297809999999996 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.912680+0000", 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4966999999999997e-05 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.912764+0000", 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3147999999999997e-05 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.912820+0000", 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6331e-05 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-13T15:09:00.912848+0000", 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8000999999999998e-05 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-13T15:10:34.215 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-13T15:10:34.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:10:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:10:34] "GET /metrics HTTP/1.1" 200 34128 "" "Prometheus/2.51.0" 2024-09-13T15:10:34.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:34 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:34.898 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:34 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:34 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:35.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:10:35.827 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:35.829 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:35 smithi003 ceph-mon[107278]: pgmap v1703: 103 pgs: 5 peering, 98 active+clean; 582 KiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 29 KiB/s, 0 objects/s recovering 2024-09-13T15:10:35.829 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:35 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:10:35.830 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:35 smithi003 ceph-mon[104007]: pgmap v1703: 103 pgs: 5 peering, 98 active+clean; 582 KiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 29 KiB/s, 0 objects/s recovering 2024-09-13T15:10:35.830 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:35 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:10:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:35 smithi019 ceph-mon[91296]: pgmap v1703: 103 pgs: 5 peering, 98 active+clean; 582 KiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 29 KiB/s, 0 objects/s recovering 2024-09-13T15:10:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:35 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:10:37.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:37 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:10:37.177+0000 7fdb0b3ca640 -1 mon.a@0(leader).osd e485 definitely_dead 0 2024-09-13T15:10:37.552 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:10:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:10:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T15:10:37.553 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:10:37.577 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.2. 2024-09-13T15:10:37.865 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:37 smithi019 ceph-mon[91296]: pgmap v1704: 103 pgs: 5 peering, 98 active+clean; 582 KiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 29 KiB/s, 0 objects/s recovering 2024-09-13T15:10:37.865 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:37 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2648119777' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T15:10:37.890 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:37 smithi003 ceph-mon[107278]: pgmap v1704: 103 pgs: 5 peering, 98 active+clean; 582 KiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 29 KiB/s, 0 objects/s recovering 2024-09-13T15:10:37.890 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:37 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2648119777' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T15:10:37.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:37 smithi003 ceph-mon[104007]: pgmap v1704: 103 pgs: 5 peering, 98 active+clean; 582 KiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 29 KiB/s, 0 objects/s recovering 2024-09-13T15:10:37.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:37 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2648119777' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T15:10:38.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:10:38.297 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-13T15:10:38.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T15:10:38.519 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:38.565 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:38.877 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:38 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2648119777' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T15:10:38.877 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:38 smithi019 ceph-mon[91296]: osdmap e486: 8 total, 8 up, 8 in 2024-09-13T15:10:38.877 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:38 smithi019 ceph-mon[91296]: osdmap e487: 8 total, 8 up, 8 in 2024-09-13T15:10:38.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:38 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2648119777' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T15:10:38.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:38 smithi003 ceph-mon[107278]: osdmap e486: 8 total, 8 up, 8 in 2024-09-13T15:10:38.892 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:38 smithi003 ceph-mon[107278]: osdmap e487: 8 total, 8 up, 8 in 2024-09-13T15:10:38.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:38 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2648119777' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T15:10:38.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:38 smithi003 ceph-mon[104007]: osdmap e486: 8 total, 8 up, 8 in 2024-09-13T15:10:38.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:38 smithi003 ceph-mon[104007]: osdmap e487: 8 total, 8 up, 8 in 2024-09-13T15:10:39.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[104007]: pgmap v1706: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 42 KiB/s, 0 objects/s recovering 2024-09-13T15:10:39.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T15:10:39.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:10:39.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[104007]: 1.a scrub starts 2024-09-13T15:10:39.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[104007]: 1.a scrub ok 2024-09-13T15:10:39.876 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[107278]: pgmap v1706: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 42 KiB/s, 0 objects/s recovering 2024-09-13T15:10:39.876 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T15:10:39.876 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:10:39.876 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[107278]: 1.a scrub starts 2024-09-13T15:10:39.876 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:39 smithi003 ceph-mon[107278]: 1.a scrub ok 2024-09-13T15:10:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:39 smithi019 ceph-mon[91296]: pgmap v1706: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 42 KiB/s, 0 objects/s recovering 2024-09-13T15:10:39.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:39 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T15:10:39.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:39 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:10:39.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:39 smithi019 ceph-mon[91296]: 1.a scrub starts 2024-09-13T15:10:39.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:39 smithi019 ceph-mon[91296]: 1.a scrub ok 2024-09-13T15:10:40.175 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:10:40.175 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:10:40.176 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:10:40.176 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:10:40.176 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:10:40.513 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:10:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:10:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:10:40.535 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a: {} 2024-09-13T15:10:40.535 INFO:teuthology.orchestra.run.smithi003.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T15:10:40.543 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b: {} 2024-09-13T15:10:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T15:10:40.552 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c: {} 2024-09-13T15:10:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T15:10:40.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:40 smithi003 ceph-mon[107278]: pgmap v1708: 103 pgs: 8 peering, 95 active+clean; 608 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 4.8 KiB/s rd, 10 KiB/s wr, 2 op/s; 76 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:10:40.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:40 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1960550078' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T15:10:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:40 smithi003 ceph-mon[104007]: pgmap v1708: 103 pgs: 8 peering, 95 active+clean; 608 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 4.8 KiB/s rd, 10 KiB/s wr, 2 op/s; 76 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:10:40.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:40 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1960550078' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T15:10:40.888 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:40 smithi019 ceph-mon[91296]: pgmap v1708: 103 pgs: 8 peering, 95 active+clean; 608 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 4.8 KiB/s rd, 10 KiB/s wr, 2 op/s; 76 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:10:40.888 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:40 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1960550078' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T15:10:40.915 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:10:41.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T15:10:41.174 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:41.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:41 smithi003 ceph-mon[107278]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:10:41.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:41 smithi003 ceph-mon[104007]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:10:41.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:41 smithi019 ceph-mon[91296]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:10:42.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:42 smithi019 ceph-mon[91296]: pgmap v1709: 103 pgs: 8 peering, 95 active+clean; 608 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:42 smithi003 ceph-mon[107278]: pgmap v1709: 103 pgs: 8 peering, 95 active+clean; 608 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:42 smithi003 ceph-mon[104007]: pgmap v1709: 103 pgs: 8 peering, 95 active+clean; 608 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s; 35 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:42.948 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:10:42.948 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:10:42.948 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:10:42.948 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:10:42.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127109 13.2 13:48fb3bf8:::benchmark_data_smithi003_293363_object7897:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.214533+0000", 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.724442393, 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1467311059999998, 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127109 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:42.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.214533+0000", 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.214533+0000", 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.214531+0000", 2024-09-13T15:10:42.950 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.214539+0000", 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9100000000000005e-06 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.214552+0000", 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2164000000000001e-05 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:42.951 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.214559+0000", 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9630000000000001e-06 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.214743+0000", 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00018414899999999999 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.214751+0000", 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0299999999999994e-06 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.952 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206671+0000", 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99191989599999997 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206680+0000", 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8689999999999999e-06 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321017+0000", 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.114336446 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321115+0000", 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7806000000000005e-05 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361207+0000", 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040092020999999999 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361264+0000", 2024-09-13T15:10:42.954 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7516999999999998e-05 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127124 13.2 13:428b7a1f:::benchmark_data_smithi003_293363_object7912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.232210+0000", 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.70676521799999, 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.129350627, 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:42.955 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127124 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.232210+0000", 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.232210+0000", 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:42.956 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.232209+0000", 2024-09-13T15:10:42.957 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:42.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.957 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.957 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:42.957 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.232219+0000", 2024-09-13T15:10:42.957 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0511e-05 2024-09-13T15:10:42.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.957 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.957 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.232237+0000", 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7575e-05 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.232244+0000", 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8900000000000001e-06 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.232777+0000", 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00053304800000000005 2024-09-13T15:10:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.232786+0000", 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3950000000000007e-06 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206712+0000", 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97392622699999998 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206720+0000", 2024-09-13T15:10:42.959 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9629999999999998e-06 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321179+0000", 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1144582949999999 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321300+0000", 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120995 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5,6", 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321846+0000", 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00054655100000000005 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.357583+0000", 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.035737275999999998 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361417+0000", 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0038340520000000001 2024-09-13T15:10:42.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361452+0000", 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4523000000000001e-05 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361481+0000", 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9425999999999999e-05 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:42.962 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361506+0000", 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5049e-05 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361561+0000", 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4549999999999998e-05 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127128 13.2 13:4a15310b:::benchmark_data_smithi003_293363_object7916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:42.963 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.241080+0000", 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.69789574499998, 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1206325279999998, 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127128 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:42.964 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241080+0000", 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241080+0000", 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241078+0000", 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.965 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241085+0000", 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5019999999999998e-06 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241094+0000", 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3929999999999997e-06 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241099+0000", 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8949999999999997e-06 2024-09-13T15:10:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241658+0000", 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00055860199999999995 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241667+0000", 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5389999999999996e-06 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206749+0000", 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96508120900000005 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206757+0000", 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4290000000000001e-06 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.337851+0000", 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1310936659999999 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.337971+0000", 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120221 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5,6", 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.338102+0000", 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013134099999999999 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.357591+0000", 2024-09-13T15:10:42.969 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.019489263 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361437+0000", 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0038451190000000001 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361467+0000", 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0543000000000002e-05 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.970 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361640+0000", 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017239499999999999 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361661+0000", 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1143999999999999e-05 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361712+0000", 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1662000000000001e-05 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127129 13.6 13:64ec7890:::benchmark_data_smithi003_293363_object7917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.241109+0000", 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.69786663799999, 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0665491789999999, 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127129 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.972 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241109+0000", 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241109+0000", 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:42.973 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241108+0000", 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241111+0000", 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8380000000000002e-06 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241118+0000", 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.438e-06 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.974 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241122+0000", 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0879999999999997e-06 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241223+0000", 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010135400000000001 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.241233+0000", 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0091999999999999e-05 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.975 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206006+0000", 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96477289399999999 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206019+0000", 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3339000000000001e-05 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.976 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.293514+0000", 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0874945140000001 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.293648+0000", 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013426700000000001 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.294261+0000", 2024-09-13T15:10:42.977 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00061329699999999995 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.301676+0000", 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0074145299999999999 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.301735+0000", 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9138999999999997e-05 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.978 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307418+0000", 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0056825779999999998 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307427+0000", 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7850000000000003e-06 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307528+0000", 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010035399999999999 2024-09-13T15:10:42.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307658+0000", 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013026800000000001 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127133 13.2 13:420a011e:::benchmark_data_smithi003_293363_object7921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.243423+0000", 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.69555221399997, 2024-09-13T15:10:42.980 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.118485342, 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127133 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.243423+0000", 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.243423+0000", 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.243422+0000", 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.243430+0000", 2024-09-13T15:10:42.982 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2779999999999993e-06 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.243445+0000", 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4705000000000001e-05 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.243453+0000", 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3329999999999997e-06 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.983 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.243944+0000", 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00049106599999999996 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.243955+0000", 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1324e-05 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206782+0000", 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96282627300000001 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.984 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206790+0000", 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8429999999999993e-06 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.344895+0000", 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1381051280000001 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.345028+0000", 2024-09-13T15:10:42.985 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013301200000000001 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5,6", 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.345183+0000", 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015503700000000001 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361177+0000", 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015994435000000001 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.986 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361579+0000", 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00040214899999999997 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361610+0000", 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0855999999999997e-05 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361742+0000", 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013143900000000001 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361789+0000", 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7481999999999997e-05 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361909+0000", 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000119609 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127136 13.6 13:657871e8:::benchmark_data_smithi003_293363_object7924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:42.988 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.247406+0000", 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.69156942900003, 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0605212069999999, 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127136 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.247406+0000", 2024-09-13T15:10:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.247406+0000", 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.247405+0000", 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.247412+0000", 2024-09-13T15:10:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9940000000000003e-06 2024-09-13T15:10:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.247424+0000", 2024-09-13T15:10:42.993 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2315e-05 2024-09-13T15:10:42.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.993 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:42.993 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.247430+0000", 2024-09-13T15:10:42.993 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5550000000000002e-06 2024-09-13T15:10:42.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.247484+0000", 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4895999999999998e-05 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.247494+0000", 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2769999999999994e-06 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206074+0000", 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95857999599999999 2024-09-13T15:10:42.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206081+0000", 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1130000000000003e-06 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.301764+0000", 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0956830449999999 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.301852+0000", 2024-09-13T15:10:42.995 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7762000000000004e-05 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.301960+0000", 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000107926 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307721+0000", 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057617959999999996 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.996 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307768+0000", 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6730999999999997e-05 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307774+0000", 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4299999999999997e-06 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307806+0000", 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2852999999999999e-05 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307845+0000", 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8726000000000002e-05 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307927+0000", 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2200000000000006e-05 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127153 13.6 13:6dd53aab:::benchmark_data_smithi003_293363_object7941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.276918+0000", 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.66205716500002, 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0312779700000001, 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127153 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:42.999 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.276918+0000", 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.276918+0000", 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.276918+0000", 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.000 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.276920+0000", 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4030000000000001e-06 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.276931+0000", 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0509999999999999e-05 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.276935+0000", 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.912e-06 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.276981+0000", 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5904e-05 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.276989+0000", 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1200000000000002e-06 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206103+0000", 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92911364299999999 2024-09-13T15:10:43.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206109+0000", 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8210000000000004e-06 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.304284+0000", 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0981744980000001 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.003 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.304425+0000", 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014083300000000001 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.304549+0000", 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000124265 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.004 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307940+0000", 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003391317 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.307971+0000", 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0809999999999998e-05 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.308005+0000", 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3835999999999999e-05 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.005 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.308055+0000", 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0430999999999999e-05 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.308109+0000", 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3733000000000002e-05 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.308196+0000", 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7276999999999998e-05 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.006 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127160 13.6 13:63e97ea4:::benchmark_data_smithi003_293363_object7948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.287798+0000", 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.65117768099998, 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.032559542, 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127160 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.287798+0000", 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.287798+0000", 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.287796+0000", 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.287804+0000", 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2470000000000002e-06 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.287818+0000", 2024-09-13T15:10:43.009 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.484e-05 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.287825+0000", 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3199999999999996e-06 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.287872+0000", 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7012999999999999e-05 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.010 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.287879+0000", 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8730000000000001e-06 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.209331+0000", 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92145280200000002 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.209342+0000", 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0793e-05 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.011 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.308244+0000", 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.098901511 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.308366+0000", 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012213 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.308489+0000", 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012279000000000001 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.012 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.314209+0000", 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.005720602 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.320048+0000", 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0058390220000000001 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.320121+0000", 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3028000000000006e-05 2024-09-13T15:10:43.013 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.320160+0000", 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8679e-05 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.320247+0000", 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6666e-05 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.320357+0000", 2024-09-13T15:10:43.014 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000110711 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127173 13.2 13:4e457058:::benchmark_data_smithi003_293363_object7961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.324142+0000", 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.61483381900001, 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0508069340000001, 2024-09-13T15:10:43.015 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127173 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.324142+0000", 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.016 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.324142+0000", 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.324140+0000", 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.324148+0000", 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4730000000000003e-06 2024-09-13T15:10:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.324161+0000", 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3474e-05 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.324167+0000", 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5319999999999997e-06 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.324230+0000", 2024-09-13T15:10:43.018 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3013000000000001e-05 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.324239+0000", 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9050000000000003e-06 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206816+0000", 2024-09-13T15:10:43.019 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.88257683499999995 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206824+0000", 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0409999999999998e-06 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.348337+0000", 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.141513673 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.020 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.348450+0000", 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011297 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5,6", 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.348576+0000", 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012547599999999999 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361184+0000", 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012607801 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361594+0000", 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000410789 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361975+0000", 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000380119 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.374789+0000", 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.01281445 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.374836+0000", 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7457000000000003e-05 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.374949+0000", 2024-09-13T15:10:43.023 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000112229 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127177 13.2 13:4ab97587:::benchmark_data_smithi003_293363_object7965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:48.344112+0000", 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 474.59486329700002, 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.0310430820000001, 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:43.024 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127177 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.344112+0000", 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.344112+0000", 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.344111+0000", 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.344120+0000", 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9800000000000004e-06 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.344137+0000", 2024-09-13T15:10:43.026 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7513999999999999e-05 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.344143+0000", 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1399999999999997e-06 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.344204+0000", 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0770999999999998e-05 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.027 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:48.344213+0000", 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1700000000000003e-06 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206847+0000", 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.86263423500000003 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.206855+0000", 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7300000000000005e-06 2024-09-13T15:10:43.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.351174+0000", 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1443186400000001 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.351292+0000", 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011859 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.029 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5,6", 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.351427+0000", 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013460600000000001 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361187+0000", 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0097603399999999993 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361625+0000", 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00043757300000000002 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.362003+0000", 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00037801700000000002 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.374988+0000", 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012984878 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.375034+0000", 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6533999999999999e-05 2024-09-13T15:10:43.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.375155+0000", 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000121066 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127187 13.6 13:6b170565:::benchmark_data_smithi003_293363_object7975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:49.360864+0000", 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 473.57811194499999, 2024-09-13T15:10:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96066441499999999, 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127187 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.360864+0000", 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.360864+0000", 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.360862+0000", 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.360869+0000", 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3509999999999998e-06 2024-09-13T15:10:43.034 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.360883+0000", 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4068e-05 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.360894+0000", 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1549000000000001e-05 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.360961+0000", 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6994000000000003e-05 2024-09-13T15:10:43.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.360972+0000", 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0696000000000001e-05 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.311520+0000", 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95054853299999997 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.036 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.311635+0000", 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011500199999999999 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.311761+0000", 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012511799999999999 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.320078+0000", 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0083171140000000005 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.037 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321283+0000", 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001205451 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321318+0000", 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4944000000000003e-05 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321348+0000", 2024-09-13T15:10:43.038 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0349999999999999e-05 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321426+0000", 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7960000000000006e-05 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.321528+0000", 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101605 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.039 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127209 13.6 13:6138a2a9:::benchmark_data_smithi003_293363_object7997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:49.418874+0000", 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 473.520101549, 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.90587283600000001, 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127209 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.040 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.418874+0000", 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.418874+0000", 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.418873+0000", 2024-09-13T15:10:43.041 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.418878+0000", 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7010000000000004e-06 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.418899+0000", 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1041000000000001e-05 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.042 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.418903+0000", 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5159999999999996e-06 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.418958+0000", 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5044999999999999e-05 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.418969+0000", 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0645999999999999e-05 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.314261+0000", 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.89529223999999996 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.314398+0000", 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013617499999999999 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.314559+0000", 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000161084 2024-09-13T15:10:43.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.320086+0000", 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0055277249999999998 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.324088+0000", 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0040016080000000002 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.045 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.324323+0000", 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000235326 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.324357+0000", 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3559000000000002e-05 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.324639+0000", 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00028183199999999998 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.324747+0000", 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000108177 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127210 13.6 13:6a21611c:::benchmark_data_smithi003_293363_object7998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:49.420753+0000", 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 473.51822226000002, 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.90427315100000005, 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.047 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127210 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.420753+0000", 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.048 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.420753+0000", 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.420753+0000", 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.420756+0000", 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.703e-06 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.420764+0000", 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0209999999999999e-06 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.420769+0000", 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2490000000000003e-06 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.420805+0000", 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5882e-05 2024-09-13T15:10:43.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.420811+0000", 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.596e-06 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.317061+0000", 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.89624995299999999 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.317192+0000", 2024-09-13T15:10:43.051 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013101399999999999 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.317336+0000", 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014372000000000001 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.320128+0000", 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00279225 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.052 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.324783+0000", 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0046550510000000003 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.324810+0000", 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7101999999999999e-05 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.324838+0000", 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7282000000000001e-05 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.324922+0000", 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4939999999999997e-05 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.325026+0000", 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010396200000000001 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.054 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:127216 13.2 13:488b8d62:::benchmark_data_smithi003_293363_object8004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-13T15:10:43.055 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:02:49.453831+0000", 2024-09-13T15:10:43.055 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 473.485144537, 2024-09-13T15:10:43.055 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92154708799999996, 2024-09-13T15:10:43.055 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.055 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.055 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.055 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:43.055 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:43.055 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 127216 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.453831+0000", 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.453831+0000", 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.453829+0000", 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.453838+0000", 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2989999999999997e-06 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.453855+0000", 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7541999999999999e-05 2024-09-13T15:10:43.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.453861+0000", 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.879e-06 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.453923+0000", 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1705999999999995e-05 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:49.453932+0000", 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5680000000000005e-06 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.354462+0000", 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.90052928600000004 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.354583+0000", 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012182500000000001 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5,6", 2024-09-13T15:10:43.059 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.354715+0000", 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013171600000000001 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361191+0000", 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0064753579999999996 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.361946+0000", 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000755473 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.060 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.375192+0000", 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013246437 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.375220+0000", 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7300999999999998e-05 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.375265+0000", 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4753999999999999e-05 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:02:50.375378+0000", 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000113583 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203835 13.16 13:6bbc74f7:::benchmark_data_smithi003_293363_object84623:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.265392+0000", 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 386.67358333999999, 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0829750950000001, 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.062 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203835 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.265392+0000", 2024-09-13T15:10:43.063 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.265392+0000", 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.265392+0000", 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.265394+0000", 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2139999999999999e-06 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.265399+0000", 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3630000000000002e-06 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.265403+0000", 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2109999999999998e-06 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.065 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.628525+0000", 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.36312217400000002 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.628534+0000", 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9579999999999996e-06 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.348169+0000", 2024-09-13T15:10:43.066 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.71963556799999995 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.348367+0000", 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00019790899999999999 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230569 13.1d 13:beaa889f:::benchmark_data_smithi003_293363_object111357:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.523536+0000", 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 362.41543938699999, 2024-09-13T15:10:43.067 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99258905799999997, 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230569 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523536+0000", 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523536+0000", 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523534+0000", 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.069 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523543+0000", 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.3850000000000008e-06 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523570+0000", 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6475e-05 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523579+0000", 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5529999999999993e-06 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.070 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.769874+0000", 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24629593999999999 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.769882+0000", 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7190000000000001e-06 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.512425+0000", 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.742543218 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.512545+0000", 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011998900000000001 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,7", 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.512867+0000", 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00032161800000000002 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.514626+0000", 2024-09-13T15:10:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0017587830000000001 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515300+0000", 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00067378399999999997 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.516038+0000", 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00073801200000000002 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.073 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.516087+0000", 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9465999999999997e-05 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.516125+0000", 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8198000000000003e-05 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:136675 14.e 14:7c62bf51:::benchmark_data_smithi003_327099_object24949:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:10:43.074 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:34.439489+0000", 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 188.49948687700001, 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.040437906, 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 136675 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439489+0000", 2024-09-13T15:10:43.075 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439489+0000", 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439487+0000", 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.076 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439495+0000", 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6490000000000008e-06 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439509+0000", 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4494e-05 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439516+0000", 2024-09-13T15:10:43.077 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9399999999999996e-06 2024-09-13T15:10:43.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.078 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.078 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439601+0000", 2024-09-13T15:10:43.078 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4303999999999998e-05 2024-09-13T15:10:43.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.078 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op must wait for map", 2024-09-13T15:10:43.079 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439609+0000", 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.388e-06 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.450499+0000", 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010890086 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.450508+0000", 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7909999999999993e-06 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.479819+0000", 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.029310819 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.479927+0000", 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010798099999999999 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:136689 14.e 14:741621f1:::benchmark_data_smithi003_327099_object24963:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:34.439530+0000", 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 188.49944522999999, 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.053548895, 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 136689 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.082 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439530+0000", 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439530+0000", 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439530+0000", 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439533+0000", 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9900000000000002e-06 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439540+0000", 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8750000000000002e-06 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439544+0000", 2024-09-13T15:10:43.084 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9550000000000002e-06 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439704+0000", 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016067499999999999 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.439716+0000", 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2064e-05 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.085 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.460655+0000", 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.020938934999999999 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.460664+0000", 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.037e-06 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.480389+0000", 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0197248999999999 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.086 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.480485+0000", 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5339000000000003e-05 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.481046+0000", 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00056115099999999997 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.492867+0000", 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011820957999999999 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.492931+0000", 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4374000000000004e-05 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.492983+0000", 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1888000000000002e-05 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.493039+0000", 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6416999999999999e-05 2024-09-13T15:10:43.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.493079+0000", 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9771999999999998e-05 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:136723 14.e 14:72a85920:::benchmark_data_smithi003_327099_object24997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:34.647157+0000", 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 188.291818689, 2024-09-13T15:10:43.089 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.84608397300000004, 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 136723 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.647157+0000", 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.090 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.647157+0000", 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.647155+0000", 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.647164+0000", 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7050000000000005e-06 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.647177+0000", 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3613e-05 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.647183+0000", 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.852e-06 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.647230+0000", 2024-09-13T15:10:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7015999999999998e-05 2024-09-13T15:10:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.647238+0000", 2024-09-13T15:10:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2439999999999993e-06 2024-09-13T15:10:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.488844+0000", 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.84160597699999995 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.488988+0000", 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000143435 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.489123+0000", 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013463 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.492923+0000", 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0038003139999999999 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.493123+0000", 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00019991599999999999 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.493164+0000", 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1396999999999997e-05 2024-09-13T15:10:43.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.493208+0000", 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3693999999999999e-05 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.493241+0000", 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2988999999999999e-05 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:136738 14.e 14:762c1793:::benchmark_data_smithi003_327099_object25012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:34.667238+0000", 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 188.271737476, 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.83806198700000001, 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 136738 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:43.097 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.667238+0000", 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.667238+0000", 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.667236+0000", 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.098 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.667243+0000", 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1619999999999996e-06 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.667254+0000", 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1965e-05 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.667260+0000", 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5280000000000003e-06 2024-09-13T15:10:43.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.667303+0000", 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3090999999999998e-05 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.667311+0000", 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4200000000000001e-06 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.493284+0000", 2024-09-13T15:10:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.82597383599999996 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.493373+0000", 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8745999999999994e-05 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.493471+0000", 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8196999999999995e-05 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.101 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.499269+0000", 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0057981550000000001 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.499280+0000", 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0556e-05 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.505212+0000", 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0059317780000000004 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:43.102 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.505256+0000", 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3732000000000001e-05 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:35.505300+0000", 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4552999999999998e-05 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:43.103 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:10:43.617 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T15:10:43.912 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:43.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:43 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2928432797' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:10:43.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:43 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:10:43.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:43.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:43.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:43 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2928432797' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:10:43.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:43 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:10:43.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:43 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2928432797' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:10:43.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:43 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:10:43.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:43.997 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:10:44.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:10:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:10:44] "GET /metrics HTTP/1.1" 200 34133 "" "Prometheus/2.51.0" 2024-09-13T15:10:44.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-13T15:10:44.264 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:44.489 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:44.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:44 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:10:44.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:44 smithi019 ceph-mon[91296]: osdmap e488: 8 total, 8 up, 8 in 2024-09-13T15:10:44.683 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:44 smithi019 ceph-mon[91296]: pgmap v1711: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 38 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:10:44.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:44 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:10:44.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:44 smithi003 ceph-mon[107278]: osdmap e488: 8 total, 8 up, 8 in 2024-09-13T15:10:44.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:44 smithi003 ceph-mon[107278]: pgmap v1711: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 38 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:10:44.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:44 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:10:44.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:44 smithi003 ceph-mon[104007]: osdmap e488: 8 total, 8 up, 8 in 2024-09-13T15:10:44.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:44 smithi003 ceph-mon[104007]: pgmap v1711: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 38 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:10:45.787 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:45 smithi019 ceph-mon[91296]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T15:10:45.787 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:45 smithi019 ceph-mon[91296]: Cluster is now healthy 2024-09-13T15:10:45.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:45 smithi003 ceph-mon[107278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T15:10:45.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:45 smithi003 ceph-mon[107278]: Cluster is now healthy 2024-09-13T15:10:45.871 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:45 smithi003 ceph-mon[104007]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T15:10:45.871 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:45 smithi003 ceph-mon[104007]: Cluster is now healthy 2024-09-13T15:10:46.037 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:10:46.037 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:10:46.037 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T15:10:46.038 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:10:46.627 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T15:10:46.788 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:10:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:46 smithi019 ceph-mon[91296]: pgmap v1712: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:46.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:46 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1741564675' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:10:46.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:46 smithi003 ceph-mon[107278]: pgmap v1712: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:46.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:46 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1741564675' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:10:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:46 smithi003 ceph-mon[104007]: pgmap v1712: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:10:46.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:46 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1741564675' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:10:47.048 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:47.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T15:10:47.632 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:47.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:47 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1741564675' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:10:47.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:47 smithi003 ceph-mon[107278]: osdmap e489: 8 total, 8 up, 8 in 2024-09-13T15:10:47.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:47 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1741564675' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:10:47.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:47 smithi003 ceph-mon[104007]: osdmap e489: 8 total, 8 up, 8 in 2024-09-13T15:10:47.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:47 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1741564675' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:10:47.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:47 smithi019 ceph-mon[91296]: osdmap e489: 8 total, 8 up, 8 in 2024-09-13T15:10:48.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:48 smithi003 ceph-mon[107278]: pgmap v1714: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:48.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:48 smithi003 ceph-mon[104007]: pgmap v1714: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:48.805 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:10:48.805 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:10:48.806 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:10:48.806 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:10:48.806 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:10:48.808 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:48 smithi019 ceph-mon[91296]: pgmap v1714: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:49.393 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:10:49.635 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:49 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3342705117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:49.636 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:49 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:49.644 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T15:10:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:49 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3342705117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:49 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:49 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3342705117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:49 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:50.466 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:10:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:10:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:10:50.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-13T15:10:50.750 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:50.815 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:50 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:10:50.815 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:50 smithi019 ceph-mon[91296]: osdmap e490: 8 total, 8 up, 8 in 2024-09-13T15:10:50.815 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:50 smithi019 ceph-mon[91296]: pgmap v1716: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:50.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:50 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:10:50.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:50 smithi003 ceph-mon[107278]: osdmap e490: 8 total, 8 up, 8 in 2024-09-13T15:10:50.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:50 smithi003 ceph-mon[107278]: pgmap v1716: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:50.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:50 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:10:50.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:50 smithi003 ceph-mon[104007]: osdmap e490: 8 total, 8 up, 8 in 2024-09-13T15:10:50.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:50 smithi003 ceph-mon[104007]: pgmap v1716: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:51.304 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203838 13.9 13:90916903:::benchmark_data_smithi003_293363_object84626:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261654+0000", 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 395.03944150199999, 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.138641585, 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203838 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261654+0000", 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261654+0000", 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261653+0000", 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261656+0000", 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8909999999999999e-06 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261663+0000", 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7730000000000002e-06 2024-09-13T15:10:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261668+0000", 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3789999999999999e-06 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.665613+0000", 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.40394590800000002 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.665621+0000", 2024-09-13T15:10:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6629999999999993e-06 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341383+0000", 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.67576234199999996 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341551+0000", 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016705999999999999 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.400228+0000", 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.058677894000000001 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.400296+0000", 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7154000000000001e-05 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203848 13.9 13:93c75d08:::benchmark_data_smithi003_293363_object84636:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261672+0000", 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 395.03942333999998, 2024-09-13T15:10:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1376833749999999, 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203848 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261672+0000", 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261672+0000", 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261672+0000", 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261674+0000", 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.159e-06 2024-09-13T15:10:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261680+0000", 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9649999999999996e-06 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261684+0000", 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1559999999999997e-06 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.665646+0000", 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.40396186099999998 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.665653+0000", 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2749999999999998e-06 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341577+0000", 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.67592373500000003 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341702+0000", 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000124645 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.342154+0000", 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00045249999999999999 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.345844+0000", 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0036899749999999999 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.398353+0000", 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.052508949999999999 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.398931+0000", 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00057793600000000003 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.398979+0000", 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8279999999999999e-05 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.399356+0000", 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00037627 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:223946 13.9 13:9078f912:::benchmark_data_smithi003_293363_object104734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.394207+0000", 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.90688850700002, 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0222590629999999, 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.317 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 223946 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.394207+0000", 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.394207+0000", 2024-09-13T15:10:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.394205+0000", 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.394212+0000", 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8390000000000001e-06 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.319 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.394615+0000", 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00040292799999999999 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.394623+0000", 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1170000000000004e-06 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.397445+0000", 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028218079999999999 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.320 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.397453+0000", 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8860000000000004e-06 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.396444+0000", 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99899174099999999 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.396531+0000", 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6583999999999996e-05 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.396821+0000", 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00028951200000000002 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.401865+0000", 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0050445289999999999 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416401+0000", 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014535935 2024-09-13T15:10:51.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416415+0000", 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.436e-05 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416445+0000", 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9539999999999998e-05 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416466+0000", 2024-09-13T15:10:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1081e-05 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:223994 13.9 13:9146e8ff:::benchmark_data_smithi003_293363_object104782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.426130+0000", 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.87496592500003, 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99044978699999997, 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.324 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 223994 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.426130+0000", 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.426130+0000", 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.426129+0000", 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.426131+0000", 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.305e-06 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.426137+0000", 2024-09-13T15:10:51.326 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.643e-06 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.426141+0000", 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.721e-06 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.426760+0000", 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00061904500000000003 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.426769+0000", 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.4669999999999993e-06 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.398215+0000", 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97144558599999997 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.398280+0000", 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5142999999999996e-05 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.328 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.398449+0000", 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016908200000000001 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.404145+0000", 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0056960880000000002 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416497+0000", 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012352227 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.329 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416530+0000", 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2691000000000003e-05 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416562+0000", 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1439000000000002e-05 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416579+0000", 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7790000000000001e-05 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.330 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:224007 13.9 13:92f010a1:::benchmark_data_smithi003_293363_object104795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.436733+0000", 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.86436233900002, 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97989731000000002, 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.331 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.332 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 224007 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.436733+0000", 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.436733+0000", 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.333 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.436731+0000", 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.436738+0000", 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7120000000000003e-06 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.436751+0000", 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2576e-05 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.334 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.436757+0000", 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.801e-06 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.436813+0000", 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6471999999999997e-05 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.436822+0000", 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2050000000000008e-06 2024-09-13T15:10:51.335 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.399318+0000", 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96249536300000005 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.399432+0000", 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000114024 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.336 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.399681+0000", 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000249807 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.404149+0000", 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0044680910000000004 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416511+0000", 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012361693 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.337 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416597+0000", 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5499000000000003e-05 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416616+0000", 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8909000000000001e-05 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416630+0000", 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4859e-05 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.338 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:224036 13.9 13:9304152d:::benchmark_data_smithi003_293363_object104824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.457022+0000", 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.84407303900002, 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95975340099999995, 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 224036 2024-09-13T15:10:51.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.457022+0000", 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.457022+0000", 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.457022+0000", 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.457026+0000", 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3100000000000002e-06 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.457036+0000", 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8800000000000003e-06 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.457041+0000", 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1379999999999998e-06 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.457267+0000", 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00022554399999999999 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.457275+0000", 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6109999999999994e-06 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.400694+0000", 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.94341860799999999 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.400797+0000", 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010293600000000001 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.401010+0000", 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000213 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.405354+0000", 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0043438330000000001 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416652+0000", 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011297971 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416723+0000", 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1077999999999999e-05 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416756+0000", 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3510999999999998e-05 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416776+0000", 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9579999999999999e-05 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:224049 13.9 13:94dc15ef:::benchmark_data_smithi003_293363_object104837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.467881+0000", 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.83321458699999, 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.949027383, 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 224049 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.346 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.467881+0000", 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.467881+0000", 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.467879+0000", 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.467891+0000", 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1704e-05 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.467903+0000", 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1996e-05 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.467909+0000", 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0650000000000004e-06 2024-09-13T15:10:51.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.467948+0000", 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8819000000000002e-05 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.467954+0000", 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1e-06 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.401900+0000", 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.9339459 2024-09-13T15:10:51.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.401963+0000", 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3090999999999996e-05 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.402153+0000", 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00019003499999999999 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.405358+0000", 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0032052819999999998 2024-09-13T15:10:51.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416676+0000", 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011318448 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416823+0000", 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014625700000000001 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416858+0000", 2024-09-13T15:10:51.351 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5385000000000001e-05 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416908+0000", 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0157000000000001e-05 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:224060 13.9 13:964af792:::benchmark_data_smithi003_293363_object104848:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.475249+0000", 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.825846345, 2024-09-13T15:10:51.352 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.941879943, 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 224060 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.475249+0000", 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.475249+0000", 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.475232+0000", 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999824 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.475275+0000", 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3133000000000002e-05 2024-09-13T15:10:51.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.475306+0000", 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1463e-05 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.475378+0000", 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1596999999999999e-05 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.475962+0000", 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00058459099999999997 2024-09-13T15:10:51.355 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.475970+0000", 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1300000000000001e-06 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.403005+0000", 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92703432699999999 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.403071+0000", 2024-09-13T15:10:51.356 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5834999999999996e-05 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.403248+0000", 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017746099999999999 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.408778+0000", 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0055298439999999999 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.416992+0000", 2024-09-13T15:10:51.357 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0082136689999999998 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417058+0000", 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6549000000000003e-05 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417095+0000", 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6770000000000002e-05 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.358 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417129+0000", 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4195e-05 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:224131 13.9 13:950c3a3d:::benchmark_data_smithi003_293363_object104919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.534550+0000", 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.76654529000001, 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.88270739799999998, 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.359 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 224131 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.534550+0000", 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.534550+0000", 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.360 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.534549+0000", 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.534554+0000", 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3079999999999998e-06 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.534563+0000", 2024-09-13T15:10:51.361 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.2199999999999998e-06 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.534568+0000", 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0459999999999996e-06 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.534682+0000", 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000113899 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.534691+0000", 2024-09-13T15:10:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.0049999999999993e-06 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.404165+0000", 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.86947426100000003 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.404240+0000", 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4235999999999999e-05 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.363 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.404504+0000", 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00026398699999999998 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.412173+0000", 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00766952 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417166+0000", 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0049924130000000002 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.364 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417194+0000", 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8449000000000002e-05 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417225+0000", 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1217000000000003e-05 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417258+0000", 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.2311999999999998e-05 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.365 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:224174 13.9 13:95634b9f:::benchmark_data_smithi003_293363_object104962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.568788+0000", 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.73230725600001, 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.84862940099999995, 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 224174 2024-09-13T15:10:51.366 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.568788+0000", 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.568788+0000", 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.568786+0000", 2024-09-13T15:10:51.367 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.568797+0000", 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.099e-05 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.568811+0000", 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4086e-05 2024-09-13T15:10:51.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.568818+0000", 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7630000000000003e-06 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.568899+0000", 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0693999999999997e-05 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.568907+0000", 2024-09-13T15:10:51.369 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.055e-06 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.405383+0000", 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.83647626799999997 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.405445+0000", 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1969000000000002e-05 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.370 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.405623+0000", 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017806000000000001 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.412176+0000", 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0065527789999999999 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417299+0000", 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.005122766 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.371 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417331+0000", 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1993999999999999e-05 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417386+0000", 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5646000000000002e-05 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417418+0000", 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1208999999999999e-05 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.372 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:224225 13.9 13:90b218c0:::benchmark_data_smithi003_293363_object105013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.612035+0000", 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.68906065300001, 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.80550371899999995, 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 224225 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.373 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.612035+0000", 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.612035+0000", 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.612033+0000", 2024-09-13T15:10:51.374 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.612039+0000", 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.2689999999999999e-06 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.612052+0000", 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3043e-05 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.375 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.612058+0000", 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.353e-06 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.612203+0000", 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014511299999999999 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.612210+0000", 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0879999999999997e-06 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.376 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.406456+0000", 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.79424633099999997 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.406517+0000", 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0770999999999998e-05 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.406698+0000", 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00018042000000000001 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.377 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.412181+0000", 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0054836199999999998 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417452+0000", 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0052705119999999998 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417483+0000", 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1259e-05 2024-09-13T15:10:51.378 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417507+0000", 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4032999999999998e-05 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417539+0000", 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1449000000000003e-05 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:224228 13.9 13:90979a27:::benchmark_data_smithi003_293363_object105016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:34.614462+0000", 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 376.68663370399997, 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.80318654300000003, 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 224228 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.380 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.614462+0000", 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.614462+0000", 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.614460+0000", 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.614467+0000", 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8990000000000004e-06 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.614483+0000", 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5557999999999999e-05 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.614489+0000", 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8919999999999997e-06 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.616472+0000", 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001983032 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:34.616480+0000", 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4430000000000003e-06 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.407556+0000", 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.79107566100000004 2024-09-13T15:10:51.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.407624+0000", 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8164999999999995e-05 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,2", 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.407862+0000", 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000237698 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.415687+0000", 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0078254629999999995 2024-09-13T15:10:51.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417570+0000", 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0018828779999999999 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417596+0000", 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6270000000000001e-05 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417618+0000", 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1370999999999999e-05 2024-09-13T15:10:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:35.417648+0000", 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0601000000000001e-05 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230552 13.c 13:31fae71f:::benchmark_data_smithi003_293363_object111340:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.522912+0000", 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 370.77818392699999, 2024-09-13T15:10:51.386 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.023088843, 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230552 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522912+0000", 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522912+0000", 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522910+0000", 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522921+0000", 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1062e-05 2024-09-13T15:10:51.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522937+0000", 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6514000000000001e-05 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522946+0000", 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1949999999999992e-06 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523044+0000", 2024-09-13T15:10:51.389 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7279000000000002e-05 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op must wait for map", 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523051+0000", 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3690000000000001e-06 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.733346+0000", 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21029541199999999 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.390 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.733355+0000", 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8119999999999997e-06 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.512268+0000", 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.77891241499999997 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.512430+0000", 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016201799999999999 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.545833+0000", 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.033403263000000002 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.546000+0000", 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016754000000000001 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.392 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230558 13.c 13:34ce7135:::benchmark_data_smithi003_293363_object111346:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.522969+0000", 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 370.778126848, 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.023345975, 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230558 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.393 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522969+0000", 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522969+0000", 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522968+0000", 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.394 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522971+0000", 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8260000000000001e-06 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522978+0000", 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6259999999999997e-06 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.522982+0000", 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0570000000000003e-06 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.395 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523096+0000", 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011435200000000001 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523102+0000", 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.48e-06 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.733376+0000", 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21027401100000001 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.733381+0000", 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8549999999999998e-06 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.512464+0000", 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.77908266500000001 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.512585+0000", 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000121258 2024-09-13T15:10:51.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.546225+0000", 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.033639806000000001 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.546315+0000", 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9491000000000001e-05 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.398 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:136673 14.7 14:ec3d4014:::benchmark_data_smithi003_327099_object24947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:33.398516+0000", 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 197.90257950500001, 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.206717606, 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 136673 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.399 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.400 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.398516+0000", 2024-09-13T15:10:51.400 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.400 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.400 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.400 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.400 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.398516+0000", 2024-09-13T15:10:51.400 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.401 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.401 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.401 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.401 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.398514+0000", 2024-09-13T15:10:51.401 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:51.401 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.401 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.398522+0000", 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8900000000000007e-06 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.398540+0000", 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8063e-05 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.398546+0000", 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7320000000000003e-06 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.405711+0000", 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0071652 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.405850+0000", 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013868800000000001 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.406026+0000", 2024-09-13T15:10:51.403 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000176546 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.423940+0000", 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.017913563 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.423997+0000", 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6576000000000002e-05 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.404 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.468820+0000", 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.044823727000000001 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.468835+0000", 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4499e-05 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.590618+0000", 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.12178344 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.405 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.590767+0000", 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014880899999999999 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.605143+0000", 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014376145999999999 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.605234+0000", 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.0328999999999995e-05 2024-09-13T15:10:51.406 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:172127 14.7 14:e72d3227:::benchmark_data_smithi003_327099_object60401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:18.066512+0000", 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 153.23458345399999, 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.091417479999999995, 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:51.407 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 172127 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066512+0000", 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066512+0000", 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066511+0000", 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066516+0000", 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4700000000000001e-06 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066536+0000", 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9446e-05 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.409 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066543+0000", 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0249999999999997e-06 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.069924+0000", 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0033810480000000002 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.070038+0000", 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011391399999999999 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.070229+0000", 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000191769 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.077644+0000", 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0074147120000000004 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.078387+0000", 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00074236799999999998 2024-09-13T15:10:51.411 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.157799+0000", 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.079411930000000006 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.157855+0000", 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6681999999999997e-05 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.157930+0000", 2024-09-13T15:10:51.412 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4312000000000003e-05 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:172128 14.7 14:e170ef17:::benchmark_data_smithi003_327099_object60402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:18.066930+0000", 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 153.23416558700001, 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.100028819, 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:51.413 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 172128 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066930+0000", 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066930+0000", 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.414 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066929+0000", 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066935+0000", 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7540000000000003e-06 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066946+0000", 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1348999999999999e-05 2024-09-13T15:10:51.415 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.066952+0000", 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7169999999999996e-06 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.073766+0000", 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0068142840000000003 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.416 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.073873+0000", 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000107426 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.074083+0000", 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00020995200000000001 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.077647+0000", 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0035636740000000002 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.417 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.078451+0000", 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00080380899999999995 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.166860+0000", 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.088408682000000002 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.166914+0000", 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4085000000000001e-05 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.418 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:18.166959+0000", 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5117000000000002e-05 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:209670 14.7 14:e066fca5:::benchmark_data_smithi003_327099_object97944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:58.456802+0000", 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 112.84429328100001, 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.080389266000000001, 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.419 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 209670 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.456802+0000", 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.420 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.456802+0000", 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.456802+0000", 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.456805+0000", 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5899999999999999e-06 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.421 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.456812+0000", 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9199999999999998e-06 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.456817+0000", 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6269999999999999e-06 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.467648+0000", 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010831624 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.422 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.467778+0000", 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012963699999999999 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.467939+0000", 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016074399999999999 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.480603+0000", 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012663974999999999 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.423 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.480858+0000", 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00025559999999999998 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.537119+0000", 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.056261064999999999 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.537151+0000", 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1121000000000002e-05 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.424 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.537191+0000", 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0908999999999997e-05 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:209674 14.7 14:e638bee0:::benchmark_data_smithi003_327099_object97948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:58.459084+0000", 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 112.842011402, 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.078239241000000001, 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.425 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 209674 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.459084+0000", 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.459084+0000", 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.459083+0000", 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.459088+0000", 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5530000000000003e-06 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.459096+0000", 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1009999999999992e-06 2024-09-13T15:10:51.427 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.459100+0000", 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.211e-06 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.471218+0000", 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012118375000000001 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.428 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.471335+0000", 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00011634899999999999 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.471459+0000", 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012377699999999999 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.480830+0000", 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0093718250000000003 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.429 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.480911+0000", 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1128e-05 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.537237+0000", 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.056325205000000003 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.537285+0000", 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8202999999999999e-05 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:58.537323+0000", 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8442999999999998e-05 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:210568 14.7 14:ec1eb915:::benchmark_data_smithi003_327099_object98842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:59.413277+0000", 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 111.88781798399999, 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.115652414, 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:51.431 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 210568 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.413277+0000", 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.413277+0000", 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:51.432 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.413276+0000", 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.413284+0000", 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7989999999999994e-06 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.413301+0000", 2024-09-13T15:10:51.433 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7244e-05 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.413308+0000", 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.781e-06 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.413469+0000", 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00016173500000000001 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.413808+0000", 2024-09-13T15:10:51.434 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00033915099999999999 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.414108+0000", 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00029952100000000002 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.418305+0000", 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0041970649999999998 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.418359+0000", 2024-09-13T15:10:51.435 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4011000000000002e-05 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.528702+0000", 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.110343467 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.528766+0000", 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3664000000000006e-05 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:51.436 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.528930+0000", 2024-09-13T15:10:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000163754 2024-09-13T15:10:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:51.437 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:10:52.504 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:10:52.761 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:53.346 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T15:10:53.566 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:53 smithi019 ceph-mon[91296]: pgmap v1717: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:53.566 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:53 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1016826001' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:53.566 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:53 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:53 smithi003 ceph-mon[107278]: pgmap v1717: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:53 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1016826001' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:53.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:53 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:53.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:53 smithi003 ceph-mon[104007]: pgmap v1717: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:53.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:53 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1016826001' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:53.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:53 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:10:54.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T15:10:54.355 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:10:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:10:54] "GET /metrics HTTP/1.1" 200 34133 "" "Prometheus/2.51.0" 2024-09-13T15:10:54.369 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:54.394 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:10:54.394 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:10:54.394 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T15:10:54.394 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:10:54.395 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:54 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:10:54.395 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:54 smithi019 ceph-mon[91296]: osdmap e491: 8 total, 8 up, 8 in 2024-09-13T15:10:54.395 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:54 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:54.395 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:54 smithi019 ceph-mon[91296]: osdmap e492: 8 total, 8 up, 8 in 2024-09-13T15:10:54.608 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:54 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:10:54.608 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:54 smithi003 ceph-mon[107278]: osdmap e491: 8 total, 8 up, 8 in 2024-09-13T15:10:54.608 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:54 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:54.608 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:54 smithi003 ceph-mon[107278]: osdmap e492: 8 total, 8 up, 8 in 2024-09-13T15:10:54.608 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:54 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:10:54.608 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:54 smithi003 ceph-mon[104007]: osdmap e491: 8 total, 8 up, 8 in 2024-09-13T15:10:54.608 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:54 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:54.609 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:54 smithi003 ceph-mon[104007]: osdmap e492: 8 total, 8 up, 8 in 2024-09-13T15:10:54.990 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:10:55.246 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:55.635 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:55 smithi019 ceph-mon[91296]: pgmap v1719: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:55.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:55 smithi003 ceph-mon[107278]: pgmap v1719: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:55.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:55 smithi003 ceph-mon[104007]: pgmap v1719: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:56.595 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T15:10:56.677 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:56 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1547333668' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:10:56.902 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:10:56.902 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:10:56.902 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:10:56.902 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:10:56.902 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:10:56.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:56 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1547333668' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:10:56.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:56 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1547333668' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:10:57.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-13T15:10:57.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:57.856 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:57.878 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:57 smithi003 ceph-mon[107278]: pgmap v1721: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:57.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:57 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1547333668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:10:57.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:57 smithi003 ceph-mon[107278]: osdmap e493: 8 total, 8 up, 8 in 2024-09-13T15:10:57.879 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:57 smithi003 ceph-mon[104007]: pgmap v1721: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:57.879 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:57 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1547333668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:10:57.879 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:57 smithi003 ceph-mon[104007]: osdmap e493: 8 total, 8 up, 8 in 2024-09-13T15:10:57.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:57 smithi019 ceph-mon[91296]: pgmap v1721: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:57.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:57 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1547333668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:10:57.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:57 smithi019 ceph-mon[91296]: osdmap e493: 8 total, 8 up, 8 in 2024-09-13T15:10:58.993 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:58.993 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:58.993 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:58 smithi019 ceph-mon[91296]: pgmap v1723: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:59.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:59.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:59.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:58 smithi003 ceph-mon[107278]: pgmap v1723: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:59.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:10:59.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:59.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:58 smithi003 ceph-mon[104007]: pgmap v1723: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:10:59.486 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:10:59.486 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:10:59.486 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:10:59.486 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:10:59.486 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203839 13.11 13:8c49368b:::benchmark_data_smithi003_293363_object84627:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 403.22058065800002, 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.123564244, 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203839 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.487 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261243+0000", 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:59.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261250+0000", 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7020000000000001e-06 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261263+0000", 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2676e-05 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261269+0000", 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8490000000000002e-06 2024-09-13T15:10:59.489 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.502730+0000", 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.241461017 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.502806+0000", 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5870000000000004e-05 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.490 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.379709+0000", 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.876903561 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.379851+0000", 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014137500000000001 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.384603+0000", 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0047521660000000004 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.491 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.384809+0000", 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00020574999999999999 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203840 13.14 13:2a6c846f:::benchmark_data_smithi003_293363_object84628:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261277+0000", 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 403.22054833300001, 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.089639778, 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:59.492 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203840 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261277+0000", 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261277+0000", 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.493 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261276+0000", 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261279+0000", 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7089999999999998e-06 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261284+0000", 2024-09-13T15:10:59.494 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3399999999999997e-06 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261288+0000", 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.45e-06 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612519+0000", 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.35123135700000002 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612526+0000", 2024-09-13T15:10:59.495 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3339999999999996e-06 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341783+0000", 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.72925640000000003 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341904+0000", 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120869 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.496 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350844+0000", 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0089401470000000007 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350916+0000", 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2562000000000001e-05 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203847 13.11 13:8898f2cd:::benchmark_data_smithi003_293363_object84635:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261293+0000", 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 403.22053222400001, 2024-09-13T15:10:59.497 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1261630679999999, 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203847 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261293+0000", 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.498 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261293+0000", 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261292+0000", 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261295+0000", 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.193e-06 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.499 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.500 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.500 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261299+0000", 2024-09-13T15:10:59.500 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6909999999999996e-06 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261302+0000", 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.19e-06 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.503068+0000", 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.24176531500000001 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.501 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.503987+0000", 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00091926099999999999 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.379883+0000", 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.87589635300000002 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.380025+0000", 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014113400000000001 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.502 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.387391+0000", 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0073666950000000004 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.387456+0000", 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4535000000000005e-05 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203853 13.14 13:2dc48b45:::benchmark_data_smithi003_293363_object84641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:10:59.503 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.310940+0000", 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 403.17088487400002, 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0401961900000001, 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203853 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310940+0000", 2024-09-13T15:10:59.504 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310940+0000", 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310937+0000", 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999971 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310952+0000", 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4966e-05 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.310990+0000", 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.756e-05 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.311000+0000", 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0484e-05 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.506 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612544+0000", 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.30154405099999998 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.612549+0000", 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4390000000000002e-06 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.341929+0000", 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.72938062400000003 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.342019+0000", 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9975000000000005e-05 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.342293+0000", 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00027399400000000001 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.345037+0000", 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002743255 2024-09-13T15:10:59.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.347766+0000", 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027296099999999999 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.350999+0000", 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0032323930000000001 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.351040+0000", 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.1807999999999997e-05 2024-09-13T15:10:59.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.351136+0000", 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5866000000000007e-05 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230539 13.f 13:f205da92:::benchmark_data_smithi003_293363_object111327:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521183+0000", 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 378.96064145700001, 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96228324200000004, 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.510 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230539 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521183+0000", 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521183+0000", 2024-09-13T15:10:59.511 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521182+0000", 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521192+0000", 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0701e-05 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.512 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521207+0000", 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4569000000000001e-05 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521213+0000", 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4409999999999998e-06 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561286+0000", 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040072137000000001 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.513 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561291+0000", 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9390000000000002e-06 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483382+0000", 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92209029899999995 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483467+0000", 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4948000000000001e-05 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.514 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230542 13.1f 13:fe3a49e0:::benchmark_data_smithi003_293363_object111330:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 378.96060258, 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.99456133199999996, 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230542 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.515 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521222+0000", 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:59.516 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521225+0000", 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7439999999999999e-06 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521230+0000", 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.558e-06 2024-09-13T15:10:59.517 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521233+0000", 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3809999999999999e-06 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561254+0000", 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040020864000000003 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561262+0000", 2024-09-13T15:10:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3529999999999997e-06 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515697+0000", 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95443503600000001 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515784+0000", 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6948999999999997e-05 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.519 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230548 13.f 13:f2b282bc:::benchmark_data_smithi003_293363_object111336:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521238+0000", 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 378.96058741799999, 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96231032900000002, 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230548 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.520 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521238+0000", 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521238+0000", 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521237+0000", 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.521 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521239+0000", 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9539999999999998e-06 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521244+0000", 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.8770000000000001e-06 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521247+0000", 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1329999999999999e-06 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.522 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.563020+0000", 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.041772769000000001 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.563051+0000", 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1293e-05 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483490+0000", 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.92043902 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.523 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.483548+0000", 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7589e-05 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230559 13.1f 13:feac4672:::benchmark_data_smithi003_293363_object111347:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 378.96055999999999, 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.000411124, 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.524 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230559 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.525 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521265+0000", 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521266+0000", 2024-09-13T15:10:59.526 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8840000000000001e-06 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521271+0000", 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.6990000000000002e-06 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521274+0000", 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1939999999999998e-06 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.527 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561310+0000", 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.040035846 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.561316+0000", 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.0349999999999998e-06 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515819+0000", 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95450267600000005 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.515931+0000", 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000111594 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.516267+0000", 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000336163 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.519875+0000", 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003608385 2024-09-13T15:10:59.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.519881+0000", 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3010000000000003e-06 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.521593+0000", 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0017120709999999999 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.521641+0000", 2024-09-13T15:10:59.530 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7216000000000003e-05 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.521676+0000", 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5380999999999999e-05 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:230561 13.1 13:826700a8:::benchmark_data_smithi003_293363_object111349:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:40.521278+0000", 2024-09-13T15:10:59.531 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 378.960547179, 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.95642722800000002, 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 230561 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521278+0000", 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.532 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521278+0000", 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521277+0000", 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.533 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521279+0000", 2024-09-13T15:10:59.534 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9659999999999999e-06 2024-09-13T15:10:59.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.534 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.534 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.534 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521284+0000", 2024-09-13T15:10:59.534 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5759999999999999e-06 2024-09-13T15:10:59.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.534 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.534 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.521287+0000", 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.96e-06 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523964+0000", 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026767449999999999 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:40.523988+0000", 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4579e-05 2024-09-13T15:10:59.535 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460069+0000", 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93608031000000003 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460199+0000", 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013029900000000001 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,4", 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.460537+0000", 2024-09-13T15:10:59.536 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00033807200000000003 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477550+0000", 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.017013495999999999 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477558+0000", 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2479999999999999e-06 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.537 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477616+0000", 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7852000000000001e-05 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477669+0000", 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3695e-05 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:41.477705+0000", 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5771e-05 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.538 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:123397 14.2 14:450a99e6:::benchmark_data_smithi003_327099_object11671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:13.318784+0000", 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 226.16304084699999, 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1419773559999999, 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123397 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318784+0000", 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318784+0000", 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318783+0000", 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.540 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318789+0000", 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7660000000000001e-06 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318802+0000", 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2238e-05 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318807+0000", 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.366e-06 2024-09-13T15:10:59.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.542 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.542 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.542 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318865+0000", 2024-09-13T15:10:59.543 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7877000000000002e-05 2024-09-13T15:10:59.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.543 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.543 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.543 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.318876+0000", 2024-09-13T15:10:59.543 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1472e-05 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.283892+0000", 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.96501579199999998 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.283906+0000", 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.428e-05 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.544 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.438947+0000", 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1550406930000001 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.439080+0000", 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013328999999999999 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.439655+0000", 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00057495500000000004 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.545 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.451726+0000", 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012070292999999999 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.454406+0000", 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0026805399999999999 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.456100+0000", 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0016936970000000001 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.546 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460596+0000", 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.004495995 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460694+0000", 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7740999999999998e-05 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460761+0000", 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7727999999999999e-05 2024-09-13T15:10:59.547 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:123404 14.2 14:476c8c71:::benchmark_data_smithi003_327099_object11678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:13.351072+0000", 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 226.130752532, 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1098517659999998, 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.548 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123404 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351072+0000", 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351072+0000", 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.549 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351071+0000", 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351079+0000", 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1149999999999994e-06 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.550 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351091+0000", 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1989e-05 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351096+0000", 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4099999999999999e-06 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351159+0000", 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2504999999999999e-05 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.351169+0000", 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0475999999999999e-05 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.283932+0000", 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.93276217100000003 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.283941+0000", 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6940000000000007e-06 2024-09-13T15:10:59.552 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.447343+0000", 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.163402015 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.447471+0000", 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012805600000000001 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.447606+0000", 2024-09-13T15:10:59.553 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013456399999999999 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.454100+0000", 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.006493993 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.454434+0000", 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00033445099999999999 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.554 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.456132+0000", 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001697282 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460787+0000", 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0046557869999999998 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460815+0000", 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7246e-05 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.555 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.460924+0000", 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010948899999999999 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:136685 14.d 14:b60d3feb:::benchmark_data_smithi003_327099_object24959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:33.424329+0000", 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 206.057495821, 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1967100070000001, 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:10:59.556 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 136685 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424329+0000", 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.557 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424329+0000", 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424328+0000", 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424336+0000", 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4840000000000004e-06 2024-09-13T15:10:59.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424350+0000", 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4416999999999999e-05 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.424356+0000", 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6749999999999999e-06 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.433661+0000", 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0093052819999999998 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.433790+0000", 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012886899999999999 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.433958+0000", 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000167768 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.560 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.462602+0000", 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.028644284999999998 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.462611+0000", 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1900000000000001e-06 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.471863+0000", 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0092517629999999997 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.471873+0000", 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.8719999999999997e-06 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.610659+0000", 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.138785876 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.610801+0000", 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014245999999999999 2024-09-13T15:10:59.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.620973+0000", 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010171556 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.621039+0000", 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6094999999999997e-05 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.563 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:136694 14.d 14:b9a67a18:::benchmark_data_smithi003_327099_object24968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:33.597900+0000", 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 205.88392490499999, 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.022468412, 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 136694 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.564 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597900+0000", 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597900+0000", 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597899+0000", 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:59.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597906+0000", 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.046e-06 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597919+0000", 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3482999999999999e-05 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597926+0000", 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5289999999999997e-06 2024-09-13T15:10:59.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597980+0000", 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4045000000000002e-05 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:33.597988+0000", 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0819999999999999e-06 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.610835+0000", 2024-09-13T15:10:59.567 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.012847584 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.611001+0000", 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000165401 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.611107+0000", 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000106289 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.568 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.614125+0000", 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.003018195 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.614161+0000", 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6185e-05 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.619810+0000", 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0056483030000000004 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.569 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.619849+0000", 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8782999999999998e-05 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:34.620368+0000", 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00051989600000000005 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183555 14.d 14:b5c49246:::benchmark_data_smithi003_327099_object71829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:10:59.570 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.585663+0000", 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 149.89616215300001, 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.061434201, 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183555 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585663+0000", 2024-09-13T15:10:59.571 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585663+0000", 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585661+0000", 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.572 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585668+0000", 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6279999999999999e-06 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585680+0000", 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2451e-05 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.573 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.585687+0000", 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7159999999999997e-06 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.590533+0000", 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.004846342 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.590542+0000", 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2770000000000005e-06 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.574 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.609871+0000", 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.019329073 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.609991+0000", 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120322 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.610071+0000", 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.9628000000000004e-05 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621753+0000", 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011681904 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.646936+0000", 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.025183245 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647004+0000", 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8289999999999998e-05 2024-09-13T15:10:59.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647058+0000", 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3816000000000001e-05 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647097+0000", 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9104999999999999e-05 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.577 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183556 14.d 14:b3561416:::benchmark_data_smithi003_327099_object71830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.587770+0000", 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 149.89405466900001, 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.059504521, 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183556 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587770+0000", 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587770+0000", 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587769+0000", 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.579 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587778+0000", 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9849999999999995e-06 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587806+0000", 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8422999999999999e-05 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.587814+0000", 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4819999999999997e-06 2024-09-13T15:10:59.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.590563+0000", 2024-09-13T15:10:59.581 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0027496199999999999 2024-09-13T15:10:59.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.581 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.581 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.590569+0000", 2024-09-13T15:10:59.581 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.699e-06 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.612536+0000", 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0219669389999999 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.612636+0000", 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.9932000000000005e-05 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:59.582 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.612714+0000", 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8164999999999994e-05 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621757+0000", 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0090431339999999995 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647147+0000", 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.025390341 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.583 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647190+0000", 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2422999999999997e-05 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647237+0000", 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7064999999999997e-05 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647275+0000", 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7935000000000002e-05 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.584 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183564 14.d 14:baefa26f:::benchmark_data_smithi003_327099_object71838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.593574+0000", 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 149.888250524, 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0538798110000001, 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.585 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183564 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593574+0000", 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593574+0000", 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.586 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593574+0000", 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593580+0000", 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8769999999999998e-06 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593594+0000", 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4511e-05 2024-09-13T15:10:59.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593600+0000", 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.1229999999999997e-06 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.594261+0000", 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00066062200000000003 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.594269+0000", 2024-09-13T15:10:59.588 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.4109999999999996e-06 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.615210+0000", 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.02094094 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.615293+0000", 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.3270999999999996e-05 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.589 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.615370+0000", 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6464999999999994e-05 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.626464+0000", 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011094405999999999 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647299+0000", 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.020834386999999999 2024-09-13T15:10:59.590 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647399+0000", 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000100677 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647429+0000", 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0159e-05 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647454+0000", 2024-09-13T15:10:59.591 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4732999999999999e-05 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183565 14.d 14:bc68576d:::benchmark_data_smithi003_327099_object71839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.593626+0000", 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 149.888198579, 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.053956822, 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.592 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183565 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593626+0000", 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593626+0000", 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593626+0000", 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593630+0000", 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8120000000000001e-06 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.594 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593637+0000", 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8029999999999997e-06 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.593642+0000", 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2150000000000003e-06 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.594288+0000", 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000646669 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.595 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.594294+0000", 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3040000000000004e-06 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.617719+0000", 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0234252319999999 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.617800+0000", 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1549000000000001e-05 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.596 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.617875+0000", 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4802000000000007e-05 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.626471+0000", 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0085958059999999992 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647501+0000", 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.021029835 2024-09-13T15:10:59.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647528+0000", 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7654e-05 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647557+0000", 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8319000000000001e-05 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.598 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647583+0000", 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.641e-05 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183571 14.d 14:b36892d7:::benchmark_data_smithi003_327099_object71845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.598281+0000", 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 149.883543973, 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0494433430000001, 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.599 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183571 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598281+0000", 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.600 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598281+0000", 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598280+0000", 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598285+0000", 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0209999999999999e-06 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598296+0000", 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0852999999999999e-05 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598302+0000", 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9370000000000001e-06 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598384+0000", 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1879000000000007e-05 2024-09-13T15:10:59.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.598392+0000", 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.5199999999999997e-06 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.619787+0000", 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.021394374 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.619860+0000", 2024-09-13T15:10:59.603 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3506999999999999e-05 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.619962+0000", 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000101642 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.630766+0000", 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010804309 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.604 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647628+0000", 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016861830000000001 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647654+0000", 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5999999999999998e-05 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647700+0000", 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5797000000000002e-05 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647724+0000", 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4471e-05 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183614 14.d 14:b95de5de:::benchmark_data_smithi003_327099_object71888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.659145+0000", 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 149.822679693, 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98874907499999998, 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.606 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183614 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659145+0000", 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.607 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659145+0000", 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659144+0000", 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659151+0000", 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9630000000000001e-06 2024-09-13T15:10:59.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659166+0000", 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4878000000000001e-05 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659172+0000", 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.852e-06 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659230+0000", 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8239000000000003e-05 2024-09-13T15:10:59.609 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.659243+0000", 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2908000000000001e-05 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621788+0000", 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.962545659 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621858+0000", 2024-09-13T15:10:59.610 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.9383999999999998e-05 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.621956+0000", 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.7905999999999994e-05 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.630772+0000", 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0088158490000000006 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.611 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647765+0000", 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.016993726000000001 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647842+0000", 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6320000000000001e-05 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647864+0000", 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2785e-05 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647894+0000", 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.9887999999999999e-05 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:183619 14.d 14:bd455a57:::benchmark_data_smithi003_327099_object71893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:29.663599+0000", 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 149.81822581700001, 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.98442653800000002, 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:10:59.613 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 183619 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663599+0000", 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:10:59.614 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663599+0000", 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663597+0000", 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663605+0000", 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6720000000000004e-06 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:10:59.615 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663621+0000", 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6157000000000001e-05 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663629+0000", 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0849999999999997e-06 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663689+0000", 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9589000000000002e-05 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.616 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:29.663700+0000", 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1355000000000001e-05 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.623787+0000", 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.960086471 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.623907+0000", 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012057600000000001 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.624030+0000", 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012242700000000001 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.632872+0000", 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0088422350000000004 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647794+0000", 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014921719 2024-09-13T15:10:59.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647944+0000", 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015033399999999999 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.647988+0000", 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.4246999999999998e-05 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:30.648026+0000", 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7292000000000001e-05 2024-09-13T15:10:59.619 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.620 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.620 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.620 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:10:59.620 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:10:59.620 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:10:59.843 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T15:11:00.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:59 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/904091798' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:11:00.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:10:59 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:11:00.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:59 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/904091798' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:11:00.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:10:59 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:11:00.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:59 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/904091798' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:11:00.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:10:59 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:11:00.260 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:11:00.518 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:11:00.639 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:11:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:11:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:11:00.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T15:11:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:00 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:11:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:00 smithi019 ceph-mon[91296]: osdmap e494: 8 total, 8 up, 8 in 2024-09-13T15:11:00.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:00 smithi019 ceph-mon[91296]: pgmap v1725: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:00 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:11:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:00 smithi003 ceph-mon[107278]: osdmap e494: 8 total, 8 up, 8 in 2024-09-13T15:11:00.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:00 smithi003 ceph-mon[107278]: pgmap v1725: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:00 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:11:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:00 smithi003 ceph-mon[104007]: osdmap e494: 8 total, 8 up, 8 in 2024-09-13T15:11:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:00 smithi003 ceph-mon[104007]: pgmap v1725: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:02.076 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:11:02.076 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:11:02.076 INFO:teuthology.orchestra.run.smithi019.stdout: "num_ops": 0 2024-09-13T15:11:02.076 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:11:02.882 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:11:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:11:03.349 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T15:11:03.550 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:03 smithi019 ceph-mon[91296]: pgmap v1726: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:03.550 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:03 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2038332602' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:11:03.550 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:03 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:11:03.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:03 smithi003 ceph-mon[107278]: pgmap v1726: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:03.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:03 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2038332602' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:11:03.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:03 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:11:03.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:03 smithi003 ceph-mon[104007]: pgmap v1726: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:03.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:03 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2038332602' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:11:03.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:03 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:11:04.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-13T15:11:04.355 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:11:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:11:04] "GET /metrics HTTP/1.1" 200 34104 "" "Prometheus/2.51.0" 2024-09-13T15:11:04.425 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:04.548 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:04 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:11:04.548 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:04 smithi019 ceph-mon[91296]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T15:11:04.635 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:04 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:11:04.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:04 smithi003 ceph-mon[107278]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T15:11:04.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:04 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:11:04.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:04 smithi003 ceph-mon[104007]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T15:11:04.787 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:11:04.788 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [], 2024-09-13T15:11:04.788 INFO:teuthology.orchestra.run.smithi019.stdout: "complaint_time": 30, 2024-09-13T15:11:04.788 INFO:teuthology.orchestra.run.smithi019.stdout: "num_blocked_ops": 0 2024-09-13T15:11:04.788 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:11:05.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:11:05.392 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:05 smithi019 ceph-mon[91296]: pgmap v1728: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:05.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:05 smithi003 ceph-mon[107278]: pgmap v1728: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:05.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:05 smithi003 ceph-mon[104007]: pgmap v1728: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:05.625 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:11:06.961 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T15:11:07.080 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:06 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2171575756' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:11:07.080 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:06 smithi019 ceph-mon[91296]: pgmap v1729: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:07.080 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:06 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:11:07.337 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "size": 20, 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 600, 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "ops": [ 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203841 13.17 13:ead20358:::benchmark_data_smithi003_293363_object84629:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261212+0000", 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 411.06693087600001, 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0753570539999999, 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:11:07.338 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203841 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261212+0000", 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.339 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261212+0000", 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261210+0000", 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261217+0000", 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.8379999999999996e-06 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261229+0000", 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.2378000000000001e-05 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261235+0000", 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9619999999999998e-06 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.439855+0000", 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.17862001 2024-09-13T15:11:07.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.439884+0000", 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.8668999999999999e-05 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.336495+0000", 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.89661158600000002 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.342 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.336569+0000", 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3200000000000004e-05 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.36909.0:203845 13.10 13:0e2b4302:::benchmark_data_smithi003_293363_object84633:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e395)", 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:04:16.261245+0000", 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 411.066897826, 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1075082359999999, 2024-09-13T15:11:07.343 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "reached pg", 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.36909", 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/367932699", 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203845 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261245+0000", 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261245+0000", 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261244+0000", 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.345 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261247+0000", 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.864e-06 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261254+0000", 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.5459999999999997e-06 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.261258+0000", 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.583e-06 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.346 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.473959+0000", 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.21270049199999999 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:16.473967+0000", 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.0390000000000005e-06 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.347 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.362740+0000", 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.88877301900000005 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for missing object", 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.362967+0000", 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00022709499999999999 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.368452+0000", 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0054853269999999999 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.348 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:04:17.368753+0000", 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00030071399999999999 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:123396 14.f 14:f9a8006d:::benchmark_data_smithi003_327099_object11670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:13.317826+0000", 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 234.01031608900001, 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1804893710000002, 2024-09-13T15:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123396 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317826+0000", 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.350 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317826+0000", 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317825+0000", 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317833+0000", 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7940000000000003e-06 2024-09-13T15:11:07.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317848+0000", 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5906e-05 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317855+0000", 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6599999999999998e-06 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.352 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317910+0000", 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.4688000000000002e-05 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.317918+0000", 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2919999999999996e-06 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.289086+0000", 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.97116769599999997 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.353 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.289095+0000", 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1589999999999999e-06 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.469696+0000", 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.180600927 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.469835+0000", 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013882400000000001 2024-09-13T15:11:07.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.470382+0000", 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00054688499999999999 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.480759+0000", 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010377001 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.355 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.480816+0000", 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7231000000000001e-05 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498032+0000", 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.017215985999999999 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498090+0000", 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8216e-05 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498185+0000", 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.5067000000000003e-05 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498316+0000", 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000130739 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.357 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:123403 14.f 14:fe42c978:::benchmark_data_smithi003_327099_object11677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:07:13.348752+0000", 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 233.97939085199999, 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1498335150000001, 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 123403 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.358 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348752+0000", 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348752+0000", 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348750+0000", 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.359 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348759+0000", 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1169999999999993e-06 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348776+0000", 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.6937000000000001e-05 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348783+0000", 2024-09-13T15:11:07.360 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3660000000000003e-06 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348843+0000", 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9824000000000002e-05 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:13.348856+0000", 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3246e-05 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.361 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.289114+0000", 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.940257432 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for peered", 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:14.289119+0000", 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5369999999999997e-06 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.477574+0000", 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1884548669999999 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.362 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.477696+0000", 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012206600000000001 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.477839+0000", 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014236000000000001 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.480767+0000", 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.002928047 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.363 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.497986+0000", 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.017219464 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498061+0000", 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5010000000000002e-05 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498359+0000", 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00029802600000000001 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.364 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498456+0000", 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.6972000000000006e-05 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:07:15.498585+0000", 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012900699999999999 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.365 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:175106 14.f 14:ff47b7c0:::benchmark_data_smithi003_327099_object63380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:21.106063+0000", 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 166.222079968, 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.18853082299999999, 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 175106 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106063+0000", 2024-09-13T15:11:07.366 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106063+0000", 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106061+0000", 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106070+0000", 2024-09-13T15:11:07.367 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.0299999999999999e-06 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106089+0000", 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.8567999999999999e-05 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106095+0000", 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.754e-06 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.368 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.107370+0000", 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0012743699999999999 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.107456+0000", 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.6028999999999999e-05 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.107519+0000", 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2656000000000005e-05 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.369 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.113190+0000", 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.005671751 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.113215+0000", 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.4484000000000001e-05 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.294459+0000", 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.18124454800000001 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.370 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.294551+0000", 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.1846000000000001e-05 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.294593+0000", 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2222999999999999e-05 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:175109 14.f 14:f0e99ae3:::benchmark_data_smithi003_327099_object63383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-13T15:11:07.371 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:21.106106+0000", 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 166.22203609900001, 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.19652803599999999, 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 175109 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106106+0000", 2024-09-13T15:11:07.372 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106106+0000", 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106106+0000", 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106109+0000", 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.6129999999999998e-06 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106115+0000", 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.2759999999999997e-06 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.106119+0000", 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0130000000000004e-06 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.374 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.110035+0000", 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0039161839999999996 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.110172+0000", 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013690999999999999 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.375 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.110317+0000", 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000144711 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.113211+0000", 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028946039999999998 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.113219+0000", 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2599999999999999e-06 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.376 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302526+0000", 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.189307743 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302589+0000", 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3149000000000001e-05 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302634+0000", 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.5015999999999997e-05 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.377 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:175118 14.f 14:f9330baa:::benchmark_data_smithi003_327099_object63392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:21.171494+0000", 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 166.15664818900001, 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.13128300100000001, 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 175118 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.378 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171494+0000", 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171494+0000", 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171492+0000", 2024-09-13T15:11:07.379 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:11:07.380 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.380 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.380 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.381 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171501+0000", 2024-09-13T15:11:07.381 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7029999999999995e-06 2024-09-13T15:11:07.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171521+0000", 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.9568e-05 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171529+0000", 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1429999999999998e-06 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171599+0000", 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0655999999999996e-05 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171725+0000", 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012510399999999999 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.171855+0000", 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000130667 2024-09-13T15:11:07.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.174724+0000", 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028687780000000002 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.175303+0000", 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000578581 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.384 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302688+0000", 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.12738560099999999 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302739+0000", 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.1057000000000003e-05 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:21.302777+0000", 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8124000000000003e-05 2024-09-13T15:11:07.385 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:203530 14.f 14:f91bb8a8:::benchmark_data_smithi003_327099_object91804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:51.423161+0000", 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 135.904981951, 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.029891602, 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.386 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203530 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423161+0000", 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.387 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423161+0000", 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423159+0000", 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423166+0000", 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0169999999999999e-06 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423179+0000", 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.3342e-05 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.423188+0000", 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8049999999999996e-06 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.389 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.424206+0000", 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0010175550000000001 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.424404+0000", 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000198242 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.424549+0000", 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014546700000000001 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.432724+0000", 2024-09-13T15:11:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0081748229999999995 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.432767+0000", 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2889999999999998e-05 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.452358+0000", 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.019591117000000002 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.452489+0000", 2024-09-13T15:11:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013128099999999999 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.453052+0000", 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000562833 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:203756 14.f 14:f777587a:::benchmark_data_smithi003_327099_object92030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:51.672466+0000", 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 135.655676234, 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.03354791, 2024-09-13T15:11:07.392 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 203756 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672466+0000", 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.393 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672466+0000", 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672465+0000", 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672472+0000", 2024-09-13T15:11:07.394 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.6089999999999997e-06 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672497+0000", 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.5602999999999999e-05 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.672506+0000", 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.8659999999999995e-06 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.676715+0000", 2024-09-13T15:11:07.395 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0042086850000000002 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.677072+0000", 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00035733 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.677418+0000", 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00034529599999999998 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.396 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.683082+0000", 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0056647870000000001 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.695913+0000", 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012830411 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.705840+0000", 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0099273199999999999 2024-09-13T15:11:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.705977+0000", 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013695800000000001 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:51.706014+0000", 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.7147999999999997e-05 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:205299 14.f 14:fc7adaae:::benchmark_data_smithi003_327099_object93573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:53.944365+0000", 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 133.383777302, 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.031709770999999998, 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 205299 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.399 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944365+0000", 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944365+0000", 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944364+0000", 2024-09-13T15:11:07.400 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944370+0000", 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4860000000000002e-06 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944388+0000", 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7723999999999998e-05 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.401 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944394+0000", 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.4160000000000001e-06 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.945950+0000", 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.001555489 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.946074+0000", 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012463900000000001 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.402 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.946197+0000", 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012310700000000001 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.960585+0000", 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014387539 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.960722+0000", 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000137007 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.403 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.975981+0000", 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015258595 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976038+0000", 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.7379000000000001e-05 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976075+0000", 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6999999999999998e-05 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.404 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:205303 14.f 14:f4fb211a:::benchmark_data_smithi003_327099_object93577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:53.944407+0000", 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 133.383735067, 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.032453827999999997, 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 205303 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944407+0000", 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944407+0000", 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944407+0000", 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:11:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944410+0000", 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.7539999999999998e-06 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944419+0000", 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 9.4970000000000007e-06 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.407 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.944423+0000", 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.8330000000000004e-06 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.952252+0000", 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0078291239999999998 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.952383+0000", 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013092400000000001 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.953362+0000", 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0009784590000000001 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.960734+0000", 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.007372155 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.960790+0000", 2024-09-13T15:11:07.409 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6113999999999998e-05 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976717+0000", 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.015926996999999998 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976818+0000", 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010168 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.410 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.976861+0000", 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2781000000000003e-05 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:205304 14.f 14:f2f51a8f:::benchmark_data_smithi003_327099_object93578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:53.947897+0000", 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 133.38024513299999, 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.029458068, 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.411 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 205304 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947897+0000", 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.412 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947897+0000", 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947896+0000", 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947903+0000", 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.6930000000000007e-06 2024-09-13T15:11:07.413 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947920+0000", 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.7173999999999999e-05 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.947927+0000", 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.3069999999999999e-06 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.414 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.956575+0000", 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0086487360000000006 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.956733+0000", 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015738300000000001 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.956862+0000", 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012953800000000001 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.415 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.964608+0000", 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0077453640000000002 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.964936+0000", 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000328358 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.977192+0000", 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012255406 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.977312+0000", 2024-09-13T15:11:07.416 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000120778 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:53.977355+0000", 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.3140999999999999e-05 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:210474 14.f 14:fab60d5b:::benchmark_data_smithi003_327099_object98748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:08:59.315986+0000", 2024-09-13T15:11:07.417 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 128.01215688299999, 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.030457702, 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 210474 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.315986+0000", 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.418 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.315986+0000", 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.315984+0000", 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.315991+0000", 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.1910000000000003e-06 2024-09-13T15:11:07.419 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.316003+0000", 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.1573000000000001e-05 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.316009+0000", 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.9100000000000002e-06 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.323848+0000", 2024-09-13T15:11:07.420 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.007838935 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.324237+0000", 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00038897200000000002 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.324715+0000", 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00047820399999999998 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.421 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.331861+0000", 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0071464019999999996 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.332012+0000", 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015039299999999999 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.346120+0000", 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014107977000000001 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.422 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.346293+0000", 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00017336900000000001 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:08:59.346443+0000", 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015028599999999999 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:211262 14.f 14:f32cf6a9:::benchmark_data_smithi003_327099_object99536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:01.170748+0000", 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 126.157394303, 2024-09-13T15:11:07.423 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.043196911999999997, 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211262 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170748+0000", 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.424 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170748+0000", 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170747+0000", 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170752+0000", 2024-09-13T15:11:07.425 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0370000000000002e-06 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170767+0000", 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.4294000000000001e-05 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.170772+0000", 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.3619999999999997e-06 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.173859+0000", 2024-09-13T15:11:07.426 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0030865659999999998 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.174012+0000", 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00015286799999999999 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.174137+0000", 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000125718 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.176958+0000", 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0028208759999999999 2024-09-13T15:11:07.427 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177269+0000", 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00031072100000000002 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.213853+0000", 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.036584325000000001 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.213909+0000", 2024-09-13T15:11:07.428 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.5989000000000002e-05 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.213945+0000", 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.5973999999999998e-05 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:211267 14.f 14:fed515f8:::benchmark_data_smithi003_327099_object99541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:01.177346+0000", 2024-09-13T15:11:07.429 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 126.150796724, 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.036696034000000002, 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 211267 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177346+0000", 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.430 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177346+0000", 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177345+0000", 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.431 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177350+0000", 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.6899999999999997e-06 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177361+0000", 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.022e-05 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177365+0000", 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0579999999999999e-06 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.432 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177399+0000", 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4694999999999999e-05 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177481+0000", 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.1388000000000001e-05 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.177555+0000", 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.4338000000000006e-05 2024-09-13T15:11:07.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.179821+0000", 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0022655819999999999 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.180588+0000", 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00076705000000000002 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.213977+0000", 2024-09-13T15:11:07.434 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.033389029000000001 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.214012+0000", 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.4688999999999999e-05 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:01.214042+0000", 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.0304e-05 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:214330 14.f 14:fb3273f5:::benchmark_data_smithi003_327099_object102604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:05.109304+0000", 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 122.218838987, 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.12425335899999999, 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 214330 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.437 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.437 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109304+0000", 2024-09-13T15:11:07.437 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.437 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.437 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.437 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109304+0000", 2024-09-13T15:11:07.437 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109302+0000", 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109309+0000", 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.2289999999999997e-06 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109319+0000", 2024-09-13T15:11:07.438 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.0277e-05 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109324+0000", 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.7790000000000003e-06 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109364+0000", 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.9369000000000002e-05 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.439 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109451+0000", 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.7571000000000006e-05 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.109525+0000", 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.3669000000000002e-05 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.112116+0000", 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0025916569999999998 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.112346+0000", 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00022925800000000001 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.233449+0000", 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.12110299200000001 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.233516+0000", 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 6.7576999999999994e-05 2024-09-13T15:11:07.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:05.233557+0000", 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.0661999999999998e-05 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:215669 14.f 14:f7c3bda3:::benchmark_data_smithi003_327099_object103943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:08.087189+0000", 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 119.240953895, 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.031969134000000003, 2024-09-13T15:11:07.442 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 215669 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087189+0000", 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.443 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087189+0000", 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087188+0000", 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.999999 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087193+0000", 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.8499999999999999e-06 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087209+0000", 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 1.5571000000000001e-05 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.087216+0000", 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.0140000000000001e-06 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.101422+0000", 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.014205751000000001 2024-09-13T15:11:07.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.101558+0000", 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000136138 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.101692+0000", 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013428100000000001 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.106924+0000", 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0052315360000000002 2024-09-13T15:11:07.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.107999+0000", 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0010749209999999999 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.119010+0000", 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.011011549000000001 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.119115+0000", 2024-09-13T15:11:07.447 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010494599999999999 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:08.119158+0000", 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2571999999999998e-05 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:217149 14.f 14:f342aaad:::benchmark_data_smithi003_327099_object105423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:09.315565+0000", 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 118.01257727700001, 2024-09-13T15:11:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.030454561000000002, 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 217149 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.315565+0000", 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.315565+0000", 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.315563+0000", 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999981 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.315572+0000", 2024-09-13T15:11:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.9430000000000006e-06 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.315593+0000", 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.1086000000000001e-05 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.315601+0000", 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.8050000000000007e-06 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.322331+0000", 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0067297540000000001 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.322472+0000", 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014104800000000001 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.322595+0000", 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012280200000000001 2024-09-13T15:11:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.335212+0000", 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.012617229000000001 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.335248+0000", 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6202000000000003e-05 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.345721+0000", 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.010473059999999999 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.345878+0000", 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.000157095 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.346020+0000", 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00014130800000000001 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:217860 14.f 14:fb7df391:::benchmark_data_smithi003_327099_object106134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:09.908992+0000", 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 117.419151001, 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.029581340000000001, 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 217860 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.908992+0000", 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.908992+0000", 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.908990+0000", 2024-09-13T15:11:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999986 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.908998+0000", 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7489999999999998e-06 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.909021+0000", 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 2.2849000000000001e-05 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.909028+0000", 2024-09-13T15:11:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.5290000000000003e-06 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.914361+0000", 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0053331710000000003 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.914462+0000", 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010124 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.914567+0000", 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00010448 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.924350+0000", 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0097835079999999998 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.924486+0000", 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013560199999999999 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.938486+0000", 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.013999587000000001 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.938536+0000", 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 5.0355999999999997e-05 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.938573+0000", 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.6856000000000001e-05 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "description": "osd_op(client.47273.0:217862 14.f 14:fa2deaa9:::benchmark_data_smithi003_327099_object106136:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "initiated_at": "2024-09-13T15:09:09.909048+0000", 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "age": 117.419094828, 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.029630666, 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "type_data": { 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "client_info": { 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "client": "client.47273", 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "client_addr": "172.21.15.3:0/2017051447", 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "tid": 217862 2024-09-13T15:11:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: "events": [ 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "initiated", 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.909048+0000", 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "throttled", 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.909048+0000", 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "header_read", 2024-09-13T15:11:07.462 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.909047+0000", 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4294967295.9999995 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "all_read", 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.909051+0000", 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.2350000000000001e-06 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "dispatched", 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.909059+0000", 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 7.7789999999999996e-06 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.463 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "queued_for_pg", 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.909064+0000", 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 4.9710000000000003e-06 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "reached_pg", 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.917811+0000", 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0087472999999999995 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "started", 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.917951+0000", 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00013953899999999999 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.464 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.918076+0000", 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.00012500799999999999 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.924410+0000", 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.0063340669999999996 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "op_commit", 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.924492+0000", 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 8.2301999999999999e-05 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.465 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.938614+0000", 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 0.01412156 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "commit_sent", 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.938647+0000", 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.3136999999999998e-05 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: "event": "done", 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: "time": "2024-09-13T15:09:09.938678+0000", 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: "duration": 3.1401e-05 2024-09-13T15:11:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.467 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-09-13T15:11:07.467 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-09-13T15:11:07.467 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-09-13T15:11:07.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:06 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2171575756' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:11:07.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:06 smithi003 ceph-mon[107278]: pgmap v1729: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:07.468 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:06 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:11:07.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:06 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2171575756' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:11:07.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:06 smithi003 ceph-mon[104007]: pgmap v1729: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:07.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:06 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:11:07.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T15:11:08.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:07 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2171575756' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:11:08.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:07 smithi003 ceph-mon[107278]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T15:11:08.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:07 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2171575756' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:11:08.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:07 smithi003 ceph-mon[104007]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T15:11:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:07 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2171575756' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:11:08.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:07 smithi019 ceph-mon[91296]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T15:11:08.479 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:09.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:08 smithi019 ceph-mon[91296]: pgmap v1731: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:09.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:08 smithi003 ceph-mon[104007]: pgmap v1731: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:09.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:08 smithi003 ceph-mon[107278]: pgmap v1731: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:10.535 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:11:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:11:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:11:11.353 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T15:11:11.631 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:11 smithi003 ceph-mon[107278]: pgmap v1732: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:11.631 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:11 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/637969751' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:11:11.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:11 smithi003 ceph-mon[104007]: pgmap v1732: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:11.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:11 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/637969751' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:11:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:11 smithi019 ceph-mon[91296]: pgmap v1732: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:11 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/637969751' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:11:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:12 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/637969751' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:11:12.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:12 smithi019 ceph-mon[91296]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T15:11:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:12 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/637969751' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:11:12.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:12 smithi003 ceph-mon[107278]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T15:11:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:12 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/637969751' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:11:12.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:12 smithi003 ceph-mon[104007]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T15:11:12.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-13T15:11:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:13.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:13 smithi003 ceph-mon[107278]: pgmap v1734: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:13.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:13.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:13.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:13.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:13 smithi003 ceph-mon[104007]: pgmap v1734: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:13.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:13.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:13.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:13 smithi019 ceph-mon[91296]: pgmap v1734: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:13.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:14.417 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:11:14] "GET /metrics HTTP/1.1" 200 34112 "" "Prometheus/2.51.0" 2024-09-13T15:11:14.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:14.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:14.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:11:14.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:11:14.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:14.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:14.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:14.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:11:14.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:11:14.684 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:14 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:14 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:14 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:14 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:11:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:14 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:11:14.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:14 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:11:15.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:15 smithi019 ceph-mon[91296]: pgmap v1735: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:15.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:15 smithi003 ceph-mon[104007]: pgmap v1735: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:15.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:15 smithi003 ceph-mon[107278]: pgmap v1735: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:16.556 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T15:11:16.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:16 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2925371573' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:11:16.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:16 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:11:16.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:16 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2925371573' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:11:16.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:16 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:11:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:16 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2925371573' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:11:16.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:16 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:11:17.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T15:11:17.559 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:17.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:17 smithi003 ceph-mon[107278]: pgmap v1736: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:17.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:17 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:11:17.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:17 smithi003 ceph-mon[107278]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T15:11:17.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:17 smithi003 ceph-mon[104007]: pgmap v1736: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:17.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:17 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:11:17.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:17 smithi003 ceph-mon[104007]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T15:11:17.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:17 smithi019 ceph-mon[91296]: pgmap v1736: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:17.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:17 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:11:17.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:17 smithi019 ceph-mon[91296]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T15:11:19.560 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T15:11:19.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:19 smithi003 ceph-mon[107278]: pgmap v1738: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:19.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:19 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2509322693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:11:19.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:19 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:11:19.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:19 smithi003 ceph-mon[104007]: pgmap v1738: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:19.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:19 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2509322693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:11:19.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:19 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:11:19.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:19 smithi019 ceph-mon[91296]: pgmap v1738: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:19.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:19 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2509322693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:11:19.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:19 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:11:20.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-13T15:11:20.527 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:20.568 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:11:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:11:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:11:20.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:20 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:11:20.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:20 smithi003 ceph-mon[107278]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T15:11:20.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:20 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:11:20.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:20 smithi003 ceph-mon[104007]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T15:11:20.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:20 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:11:20.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:20 smithi019 ceph-mon[91296]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T15:11:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:21 smithi019 ceph-mon[91296]: pgmap v1740: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:21.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:21 smithi003 ceph-mon[104007]: pgmap v1740: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:21.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:21 smithi003 ceph-mon[107278]: pgmap v1740: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:22.575 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T15:11:22.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:22 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/217580654' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:11:22.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:22 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/217580654' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:11:22.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:22 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/217580654' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:11:23.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T15:11:23.545 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:23.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:23 smithi003 ceph-mon[107278]: pgmap v1741: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:23.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:23 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/217580654' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:11:23.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:23 smithi003 ceph-mon[107278]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T15:11:23.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:23 smithi003 ceph-mon[104007]: pgmap v1741: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:23.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:23 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/217580654' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:11:23.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:23 smithi003 ceph-mon[104007]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T15:11:23.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:23 smithi019 ceph-mon[91296]: pgmap v1741: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:23.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:23 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/217580654' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:11:23.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:23 smithi019 ceph-mon[91296]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T15:11:24.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:11:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:11:24] "GET /metrics HTTP/1.1" 200 34112 "" "Prometheus/2.51.0" 2024-09-13T15:11:24.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:24 smithi003 ceph-mon[107278]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T15:11:24.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:24 smithi003 ceph-mon[104007]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T15:11:24.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:24 smithi019 ceph-mon[91296]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T15:11:25.588 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T15:11:25.911 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:25 smithi003 ceph-mon[107278]: pgmap v1744: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:25.911 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:25 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/464090948' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:11:25.911 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:25 smithi003 ceph-mon[104007]: pgmap v1744: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:25.911 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:25 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/464090948' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:11:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:25 smithi019 ceph-mon[91296]: pgmap v1744: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:25 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/464090948' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:11:26.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-13T15:11:26.474 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:26 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/464090948' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:11:26.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:26 smithi019 ceph-mon[91296]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T15:11:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:26 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/464090948' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:11:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:26 smithi003 ceph-mon[104007]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T15:11:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:26 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/464090948' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:11:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:26 smithi003 ceph-mon[107278]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T15:11:27.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:27 smithi003 ceph-mon[107278]: pgmap v1746: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:27.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:27 smithi003 ceph-mon[104007]: pgmap v1746: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:27.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:27 smithi019 ceph-mon[91296]: pgmap v1746: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:28.609 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T15:11:28.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:28 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:28.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:28 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1064770647' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:11:28.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:28 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:28.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:28 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1064770647' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:11:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:28 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:28.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:28 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1064770647' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:11:29.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T15:11:29.622 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:29.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:29 smithi003 ceph-mon[104007]: pgmap v1747: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:29.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:29 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1064770647' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:11:29.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:29 smithi003 ceph-mon[104007]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T15:11:29.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:29 smithi003 ceph-mon[107278]: pgmap v1747: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:29.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:29 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1064770647' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:11:29.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:29 smithi003 ceph-mon[107278]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T15:11:29.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:29 smithi019 ceph-mon[91296]: pgmap v1747: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:29.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:29 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1064770647' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:11:29.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:29 smithi019 ceph-mon[91296]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T15:11:30.615 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:11:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:11:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:11:30.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-09-13T15:11:30.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-13T15:11:30.887 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:30 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]} : dispatch 2024-09-13T15:11:30.887 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:30 smithi003 ceph-mon[107278]: pgmap v1749: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:30.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-09-13T15:11:30.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-13T15:11:30.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:30 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]} : dispatch 2024-09-13T15:11:30.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:30 smithi003 ceph-mon[104007]: pgmap v1749: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-09-13T15:11:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-13T15:11:30.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:30 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]} : dispatch 2024-09-13T15:11:30.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:30 smithi019 ceph-mon[91296]: pgmap v1749: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-13T15:11:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-13T15:11:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:31 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-13T15:11:31.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:31 smithi019 ceph-mon[91296]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T15:11:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-13T15:11:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-13T15:11:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:31 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-13T15:11:31.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:31 smithi003 ceph-mon[104007]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T15:11:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-13T15:11:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-13T15:11:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:31 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-13T15:11:31.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:31 smithi003 ceph-mon[107278]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T15:11:32.627 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T15:11:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:32 smithi019 ceph-mon[91296]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T15:11:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:32 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/765303687' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:32 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:32.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:32 smithi019 ceph-mon[91296]: pgmap v1752: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:32.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:32 smithi003 ceph-mon[104007]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T15:11:32.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:32 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/765303687' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:32.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:32 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:32.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:32 smithi003 ceph-mon[104007]: pgmap v1752: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:32.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:32 smithi003 ceph-mon[107278]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T15:11:32.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:32 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/765303687' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:32.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:32 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:32.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:32 smithi003 ceph-mon[107278]: pgmap v1752: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:33.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-13T15:11:33.572 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:33.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:33 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:11:33.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:33 smithi003 ceph-mon[107278]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T15:11:33.861 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:33 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:11:33.862 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:33 smithi003 ceph-mon[104007]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T15:11:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:33 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:11:33.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:33 smithi019 ceph-mon[91296]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T15:11:34.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:11:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:11:34] "GET /metrics HTTP/1.1" 200 34116 "" "Prometheus/2.51.0" 2024-09-13T15:11:34.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:34 smithi003 ceph-mon[107278]: pgmap v1754: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:34.752 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:34 smithi003 ceph-mon[104007]: pgmap v1754: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:34.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:34 smithi019 ceph-mon[91296]: pgmap v1754: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:35.645 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T15:11:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:35 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/3502474501' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:35.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:35 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:35.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:35 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/3502474501' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:35.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:35 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:35.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:35 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/3502474501' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:35.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:35 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:11:36.414 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T15:11:36.414 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T15:11:36.414 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T15:11:36.414 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T15:11:36.415 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T15:11:36.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:11:36.678 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:36.905 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:36 smithi003 ceph-mon[107278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:11:36.905 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:36 smithi003 ceph-mon[107278]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T15:11:36.905 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:36 smithi003 ceph-mon[107278]: pgmap v1756: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:36.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:36 smithi003 ceph-mon[104007]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:11:36.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:36 smithi003 ceph-mon[104007]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T15:11:36.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:36 smithi003 ceph-mon[104007]: pgmap v1756: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:36.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:36 smithi019 ceph-mon[91296]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:11:36.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:36 smithi019 ceph-mon[91296]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T15:11:36.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:36 smithi019 ceph-mon[91296]: pgmap v1756: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:38.565 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:11:38.566 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":507,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:15:00.204452+0000","modified":"2024-09-13T15:11:35.635461+0000","last_up_change":"2024-09-13T14:37:07.731579+0000","last_in_change":"2024-09-13T15:10:37.574286+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:18:02.574675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"370","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"319","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:20:35.870236+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:21:51.432728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:21:52.047402+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a153ec29-6e01-4063-aa90-01d1de00e53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":79,"up_from":83,"up_thru":504,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6803","nonce":2281086808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6805","nonce":2281086808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6809","nonce":2281086808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6807","nonce":2281086808}]},"public_addr":"172.21.15.3:6803/2281086808","cluster_addr":"172.21.15.3:6805/2281086808","heartbeat_back_addr":"172.21.15.3:6809/2281086808","heartbeat_front_addr":"172.21.15.3:6807/2281086808","state":["exists","up"]},{"osd":1,"uuid":"9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":93,"up_from":97,"up_thru":491,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6811","nonce":3208418243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6813","nonce":3208418243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6817","nonce":3208418243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6815","nonce":3208418243}]},"public_addr":"172.21.15.3:6811/3208418243","cluster_addr":"172.21.15.3:6813/3208418243","heartbeat_back_addr":"172.21.15.3:6817/3208418243","heartbeat_front_addr":"172.21.15.3:6815/3208418243","state":["exists","up"]},{"osd":2,"uuid":"676ec1e5-071c-47ba-86ae-2193f448fe8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":107,"up_from":113,"up_thru":504,"down_at":108,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6819","nonce":1260576691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6821","nonce":1260576691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6825","nonce":1260576691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6823","nonce":1260576691}]},"public_addr":"172.21.15.3:6819/1260576691","cluster_addr":"172.21.15.3:6821/1260576691","heartbeat_back_addr":"172.21.15.3:6825/1260576691","heartbeat_front_addr":"172.21.15.3:6823/1260576691","state":["exists","up"]},{"osd":3,"uuid":"7ec0ad16-0df3-4319-b475-72cfaaa5edae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":133,"up_from":137,"up_thru":486,"down_at":134,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6827","nonce":1196691419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6829","nonce":1196691419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6833","nonce":1196691419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6831","nonce":1196691419}]},"public_addr":"172.21.15.3:6827/1196691419","cluster_addr":"172.21.15.3:6829/1196691419","heartbeat_back_addr":"172.21.15.3:6833/1196691419","heartbeat_front_addr":"172.21.15.3:6831/1196691419","state":["exists","up"]},{"osd":4,"uuid":"0ef54d09-4000-4695-942b-1b1cf9659983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6801","nonce":4204660131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6803","nonce":4204660131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6807","nonce":4204660131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6805","nonce":4204660131}]},"public_addr":"172.21.15.19:6801/4204660131","cluster_addr":"172.21.15.19:6803/4204660131","heartbeat_back_addr":"172.21.15.19:6807/4204660131","heartbeat_front_addr":"172.21.15.19:6805/4204660131","state":["exists","up"]},{"osd":5,"uuid":"6dfd8fce-4d3d-4a8b-90ac-556447045378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":500,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6809","nonce":3591244104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6811","nonce":3591244104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6815","nonce":3591244104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6813","nonce":3591244104}]},"public_addr":"172.21.15.19:6809/3591244104","cluster_addr":"172.21.15.19:6811/3591244104","heartbeat_back_addr":"172.21.15.19:6815/3591244104","heartbeat_front_addr":"172.21.15.19:6813/3591244104","state":["exists","up"]},{"osd":6,"uuid":"17e27318-c0ce-413d-b660-d756731ea743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6817","nonce":2162145215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6819","nonce":2162145215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6823","nonce":2162145215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6821","nonce":2162145215}]},"public_addr":"172.21.15.19:6817/2162145215","cluster_addr":"172.21.15.19:6819/2162145215","heartbeat_back_addr":"172.21.15.19:6823/2162145215","heartbeat_front_addr":"172.21.15.19:6821/2162145215","state":["exists","up"]},{"osd":7,"uuid":"1edab2d1-dad5-45c1-977c-6911541c282e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6825","nonce":2814206175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6827","nonce":2814206175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6831","nonce":2814206175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6829","nonce":2814206175}]},"public_addr":"172.21.15.19:6825/2814206175","cluster_addr":"172.21.15.19:6827/2814206175","heartbeat_back_addr":"172.21.15.19:6831/2814206175","heartbeat_front_addr":"172.21.15.19:6829/2814206175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:11.323986+0000","dead_epoch":79},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:34.429745+0000","dead_epoch":93},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:59.523592+0000","dead_epoch":107},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:24.423713+0000","dead_epoch":133},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:46.452261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:10.483064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:34.315912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:58.203878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":3,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/2886196962":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1330279387":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/2143031132":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6801/3309089360":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1361669764":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/738145194":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2893689034":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/1118901319":"2024-09-14T14:24:43.914607+0000","172.21.15.3:6800/3309089360":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6801/70918773":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/1087248546":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1863609487":"2024-09-14T14:24:43.914607+0000","172.21.15.19:0/211301439":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/1018276975":"2024-09-14T14:20:33.329796+0000","172.21.15.19:6834/2089501540":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/253916983":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/3529413292":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/3505363388":"2024-09-14T14:15:28.360446+0000","172.21.15.19:6835/2089501540":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6800/70918773":"2024-09-14T14:24:43.914607+0000","172.21.15.19:0/3762108460":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6801/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.19:0/255629335":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/2346548909":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/3615653824":"2024-09-14T14:20:33.329796+0000","172.21.15.19:0/1270823610":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6800/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6800/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6801/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2723658197":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/1688297948":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/416475212":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6800/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/2497820386":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/3631040762":"2024-09-14T14:15:52.081305+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:11:39.192 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T15:11:39.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:11:39.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:39 smithi003 ceph-mon[107278]: pgmap v1757: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:39.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:39 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1440278994' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:11:39.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:39 smithi003 ceph-mon[104007]: pgmap v1757: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:39.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:39 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1440278994' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:11:39.451 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:39 smithi019 ceph-mon[91296]: pgmap v1757: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:39.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:39 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1440278994' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:11:40.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:11:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:11:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:11:41.330 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:11:41.330 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":507,"fsid":"77a3f016-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:15:00.204452+0000","modified":"2024-09-13T15:11:35.635461+0000","last_up_change":"2024-09-13T14:37:07.731579+0000","last_in_change":"2024-09-13T15:10:37.574286+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:18:02.574675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"370","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"319","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:20:35.870236+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:21:51.432728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:21:52.047402+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a153ec29-6e01-4063-aa90-01d1de00e53d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":79,"up_from":83,"up_thru":504,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6803","nonce":2281086808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6805","nonce":2281086808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6809","nonce":2281086808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":2281086808},{"type":"v1","addr":"172.21.15.3:6807","nonce":2281086808}]},"public_addr":"172.21.15.3:6803/2281086808","cluster_addr":"172.21.15.3:6805/2281086808","heartbeat_back_addr":"172.21.15.3:6809/2281086808","heartbeat_front_addr":"172.21.15.3:6807/2281086808","state":["exists","up"]},{"osd":1,"uuid":"9b69ab74-8ceb-4da4-bd4c-5b3ed0a30c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":93,"up_from":97,"up_thru":491,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6811","nonce":3208418243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6813","nonce":3208418243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6817","nonce":3208418243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3208418243},{"type":"v1","addr":"172.21.15.3:6815","nonce":3208418243}]},"public_addr":"172.21.15.3:6811/3208418243","cluster_addr":"172.21.15.3:6813/3208418243","heartbeat_back_addr":"172.21.15.3:6817/3208418243","heartbeat_front_addr":"172.21.15.3:6815/3208418243","state":["exists","up"]},{"osd":2,"uuid":"676ec1e5-071c-47ba-86ae-2193f448fe8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":107,"up_from":113,"up_thru":504,"down_at":108,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6819","nonce":1260576691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6821","nonce":1260576691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6825","nonce":1260576691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1260576691},{"type":"v1","addr":"172.21.15.3:6823","nonce":1260576691}]},"public_addr":"172.21.15.3:6819/1260576691","cluster_addr":"172.21.15.3:6821/1260576691","heartbeat_back_addr":"172.21.15.3:6825/1260576691","heartbeat_front_addr":"172.21.15.3:6823/1260576691","state":["exists","up"]},{"osd":3,"uuid":"7ec0ad16-0df3-4319-b475-72cfaaa5edae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":133,"up_from":137,"up_thru":486,"down_at":134,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6827","nonce":1196691419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6829","nonce":1196691419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6833","nonce":1196691419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1196691419},{"type":"v1","addr":"172.21.15.3:6831","nonce":1196691419}]},"public_addr":"172.21.15.3:6827/1196691419","cluster_addr":"172.21.15.3:6829/1196691419","heartbeat_back_addr":"172.21.15.3:6833/1196691419","heartbeat_front_addr":"172.21.15.3:6831/1196691419","state":["exists","up"]},{"osd":4,"uuid":"0ef54d09-4000-4695-942b-1b1cf9659983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6801","nonce":4204660131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6803","nonce":4204660131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6807","nonce":4204660131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4204660131},{"type":"v1","addr":"172.21.15.19:6805","nonce":4204660131}]},"public_addr":"172.21.15.19:6801/4204660131","cluster_addr":"172.21.15.19:6803/4204660131","heartbeat_back_addr":"172.21.15.19:6807/4204660131","heartbeat_front_addr":"172.21.15.19:6805/4204660131","state":["exists","up"]},{"osd":5,"uuid":"6dfd8fce-4d3d-4a8b-90ac-556447045378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":500,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6809","nonce":3591244104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6811","nonce":3591244104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6815","nonce":3591244104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3591244104},{"type":"v1","addr":"172.21.15.19:6813","nonce":3591244104}]},"public_addr":"172.21.15.19:6809/3591244104","cluster_addr":"172.21.15.19:6811/3591244104","heartbeat_back_addr":"172.21.15.19:6815/3591244104","heartbeat_front_addr":"172.21.15.19:6813/3591244104","state":["exists","up"]},{"osd":6,"uuid":"17e27318-c0ce-413d-b660-d756731ea743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6817","nonce":2162145215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6819","nonce":2162145215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6823","nonce":2162145215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2162145215},{"type":"v1","addr":"172.21.15.19:6821","nonce":2162145215}]},"public_addr":"172.21.15.19:6817/2162145215","cluster_addr":"172.21.15.19:6819/2162145215","heartbeat_back_addr":"172.21.15.19:6823/2162145215","heartbeat_front_addr":"172.21.15.19:6821/2162145215","state":["exists","up"]},{"osd":7,"uuid":"1edab2d1-dad5-45c1-977c-6911541c282e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6825","nonce":2814206175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6827","nonce":2814206175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6831","nonce":2814206175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2814206175},{"type":"v1","addr":"172.21.15.19:6829","nonce":2814206175}]},"public_addr":"172.21.15.19:6825/2814206175","cluster_addr":"172.21.15.19:6827/2814206175","heartbeat_back_addr":"172.21.15.19:6831/2814206175","heartbeat_front_addr":"172.21.15.19:6829/2814206175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:11.323986+0000","dead_epoch":79},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:34.429745+0000","dead_epoch":93},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:59.523592+0000","dead_epoch":107},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:24.423713+0000","dead_epoch":133},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:18:46.452261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:10.483064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:34.315912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:19:58.203878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":3,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/2886196962":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1330279387":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/2143031132":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6801/3309089360":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1361669764":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/738145194":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2893689034":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/1118901319":"2024-09-14T14:24:43.914607+0000","172.21.15.3:6800/3309089360":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6801/70918773":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/1087248546":"2024-09-14T14:27:27.242023+0000","172.21.15.3:0/1863609487":"2024-09-14T14:24:43.914607+0000","172.21.15.19:0/211301439":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/1018276975":"2024-09-14T14:20:33.329796+0000","172.21.15.19:6834/2089501540":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/253916983":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/3529413292":"2024-09-14T14:24:43.914607+0000","172.21.15.3:0/3505363388":"2024-09-14T14:15:28.360446+0000","172.21.15.19:6835/2089501540":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6800/70918773":"2024-09-14T14:24:43.914607+0000","172.21.15.19:0/3762108460":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6801/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.19:0/255629335":"2024-09-14T14:26:31.587235+0000","172.21.15.3:0/2346548909":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/3615653824":"2024-09-14T14:20:33.329796+0000","172.21.15.19:0/1270823610":"2024-09-14T14:26:31.587235+0000","172.21.15.3:6800/4142617802":"2024-09-14T14:15:52.081305+0000","172.21.15.3:6801/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:6800/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:6801/931806668":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/2723658197":"2024-09-14T14:15:52.081305+0000","172.21.15.3:0/1688297948":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/416475212":"2024-09-14T14:27:27.242023+0000","172.21.15.3:6800/2010213868":"2024-09-14T14:20:33.329796+0000","172.21.15.3:0/2497820386":"2024-09-14T14:15:28.360446+0000","172.21.15.3:0/3631040762":"2024-09-14T14:15:52.081305+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:11:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:41 smithi019 ceph-mon[91296]: pgmap v1758: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:41.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:41 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2989803826' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:11:41.682 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:41 smithi003 ceph-mon[107278]: pgmap v1758: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:41.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:41 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2989803826' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:11:41.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:41 smithi003 ceph-mon[104007]: pgmap v1758: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:41.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:41 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2989803826' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:11:42.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T15:11:42.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T15:11:42.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T15:11:42.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T15:11:42.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T15:11:42.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T15:11:42.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T15:11:42.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T15:11:42.524 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:42.532 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:42.551 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:42.561 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:42.562 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:42.582 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:43.604 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:43 smithi003 ceph-mon[107278]: pgmap v1759: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:43.604 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:43 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:43.605 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:43 smithi003 ceph-mon[104007]: pgmap v1759: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:43.605 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:43 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:43 smithi019 ceph-mon[91296]: pgmap v1759: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:43.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:43 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:44.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:11:44 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:11:44] "GET /metrics HTTP/1.1" 200 34115 "" "Prometheus/2.51.0" 2024-09-13T15:11:45.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:45 smithi019 ceph-mon[91296]: pgmap v1760: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:45.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:45 smithi003 ceph-mon[104007]: pgmap v1760: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:45.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:45 smithi003 ceph-mon[107278]: pgmap v1760: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:47.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:47 smithi019 ceph-mon[91296]: pgmap v1761: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:47.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:47 smithi003 ceph-mon[104007]: pgmap v1761: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:47.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:47 smithi003 ceph-mon[107278]: pgmap v1761: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:49.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:49 smithi019 ceph-mon[91296]: pgmap v1762: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:49.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:49 smithi003 ceph-mon[104007]: pgmap v1762: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:49.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:49 smithi003 ceph-mon[107278]: pgmap v1762: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:50.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:11:50 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:11:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:11:51.141 INFO:teuthology.orchestra.run.smithi003.stdout:188978561656 2024-09-13T15:11:51.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T15:11:51.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:51 smithi019 ceph-mon[91296]: pgmap v1763: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:51.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:51 smithi003 ceph-mon[107278]: pgmap v1763: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:51.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:51 smithi003 ceph-mon[104007]: pgmap v1763: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:53.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:53 smithi003 ceph-mon[107278]: pgmap v1764: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:53.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:53 smithi003 ceph-mon[104007]: pgmap v1764: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:53.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:53 smithi019 ceph-mon[91296]: pgmap v1764: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:54.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:11:54 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:11:54] "GET /metrics HTTP/1.1" 200 34115 "" "Prometheus/2.51.0" 2024-09-13T15:11:55.322 INFO:teuthology.orchestra.run.smithi003.stdout:133143986816 2024-09-13T15:11:55.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T15:11:55.610 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:55 smithi003 ceph-mon[107278]: pgmap v1765: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:55.611 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:55 smithi003 ceph-mon[104007]: pgmap v1765: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:55.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:55 smithi019 ceph-mon[91296]: pgmap v1765: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:57.164 INFO:teuthology.orchestra.run.smithi003.stdout:356482286056 2024-09-13T15:11:57.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T15:11:57.417 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:57 smithi003 ceph-mon[107278]: pgmap v1766: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:57.417 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:57 smithi003 ceph-mon[104007]: pgmap v1766: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:57.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:57 smithi019 ceph-mon[91296]: pgmap v1766: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:58.246 INFO:teuthology.orchestra.run.smithi003.stdout:416611828191 2024-09-13T15:11:58.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T15:11:58.627 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:58 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:58.628 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:58 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:58.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:58 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:58.961 INFO:teuthology.orchestra.run.smithi003.stdout:214748365427 2024-09-13T15:11:58.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T15:11:59.390 INFO:teuthology.orchestra.run.smithi003.stdout:485331304888 2024-09-13T15:11:59.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T15:11:59.615 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:11:59 smithi003 ceph-mon[107278]: pgmap v1767: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:59.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:11:59 smithi003 ceph-mon[104007]: pgmap v1767: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:11:59.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:11:59 smithi019 ceph-mon[91296]: pgmap v1767: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:00.341 INFO:teuthology.orchestra.run.smithi003.stdout:588410519973 2024-09-13T15:12:00.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T15:12:00.564 INFO:teuthology.orchestra.run.smithi003.stdout:158913790588 2024-09-13T15:12:00.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T15:12:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:00.598 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:00.598 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:00.622 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:00.626 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:00.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:00 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:12:00.872 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:01.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:01 smithi019 ceph-mon[91296]: pgmap v1768: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:01.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:01 smithi003 ceph-mon[107278]: pgmap v1768: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:01.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:01 smithi003 ceph-mon[104007]: pgmap v1768: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:03.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:03 smithi019 ceph-mon[91296]: pgmap v1769: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:03.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:03 smithi003 ceph-mon[107278]: pgmap v1769: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:03.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:03 smithi003 ceph-mon[104007]: pgmap v1769: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:04.430 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:04 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:12:04] "GET /metrics HTTP/1.1" 200 34112 "" "Prometheus/2.51.0" 2024-09-13T15:12:05.553 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:05 smithi003 ceph-mon[107278]: pgmap v1770: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:05.553 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:05 smithi003 ceph-mon[104007]: pgmap v1770: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:05.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:05 smithi019 ceph-mon[91296]: pgmap v1770: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout:416611828193 2024-09-13T15:12:07.347 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:07.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:07 smithi003 ceph-mon[107278]: pgmap v1771: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:07.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:07 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/4034017464' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:12:07.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:07 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2558146108' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:12:07.648 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:07 smithi003 ceph-mon[104007]: pgmap v1771: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:07.648 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:07 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/4034017464' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:12:07.648 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:07 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2558146108' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:12:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:07 smithi019 ceph-mon[91296]: pgmap v1771: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:07 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/4034017464' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:12:07.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:07 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2558146108' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:12:08.004 INFO:teuthology.orchestra.run.smithi003.stdout:133143986819 2024-09-13T15:12:08.570 INFO:teuthology.orchestra.run.smithi003.stdout:214748365429 2024-09-13T15:12:09.397 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365427 got 214748365429 for osd.7 2024-09-13T15:12:09.397 DEBUG:teuthology.parallel:result is None 2024-09-13T15:12:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:09 smithi019 ceph-mon[91296]: pgmap v1772: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/692751746' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:12:09.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:09 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/262801071' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:12:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:09 smithi003 ceph-mon[107278]: pgmap v1772: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/692751746' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:12:09.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:09 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/262801071' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:12:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:09 smithi003 ceph-mon[104007]: pgmap v1772: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/692751746' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:12:09.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:09 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/262801071' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:12:09.852 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986816 got 133143986819 for osd.4 2024-09-13T15:12:09.853 DEBUG:teuthology.parallel:result is None 2024-09-13T15:12:10.260 INFO:teuthology.orchestra.run.smithi003.stdout:188978561659 2024-09-13T15:12:10.583 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:12:11.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:11 smithi019 ceph-mon[91296]: pgmap v1773: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:11.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:11 smithi003 ceph-mon[107278]: pgmap v1773: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:11.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:11 smithi003 ceph-mon[104007]: pgmap v1773: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:11.803 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:13 smithi019 ceph-mon[91296]: pgmap v1774: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:13.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:13 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:13 smithi003 ceph-mon[107278]: pgmap v1774: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:13.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:13 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:13.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:13 smithi003 ceph-mon[104007]: pgmap v1774: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:13.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:13 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:14.073 INFO:teuthology.orchestra.run.smithi003.stdout:356482286059 2024-09-13T15:12:14.407 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:14 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:12:14] "GET /metrics HTTP/1.1" 200 34115 "" "Prometheus/2.51.0" 2024-09-13T15:12:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:14 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1315416033' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:12:14.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:14 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:12:14.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:14 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1315416033' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:12:14.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:14 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:12:14.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:14 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1315416033' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:12:14.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:14 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:12:14.729 INFO:teuthology.orchestra.run.smithi003.stdout:158913790593 2024-09-13T15:12:15.471 INFO:tasks.cephadm.ceph_manager.ceph:need seq 416611828191 got 416611828193 for osd.1 2024-09-13T15:12:15.471 DEBUG:teuthology.parallel:result is None 2024-09-13T15:12:15.578 INFO:teuthology.orchestra.run.smithi003.stdout:485331304892 2024-09-13T15:12:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:15 smithi019 ceph-mon[91296]: pgmap v1775: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/1570604737' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:12:15.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:15 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2319613078' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:12:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:15 smithi003 ceph-mon[104007]: pgmap v1775: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/1570604737' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:12:15.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:15 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2319613078' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:12:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:15 smithi003 ceph-mon[107278]: pgmap v1775: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/1570604737' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:12:15.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:15 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2319613078' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:12:16.322 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561656 got 188978561659 for osd.6 2024-09-13T15:12:16.323 DEBUG:teuthology.parallel:result is None 2024-09-13T15:12:16.683 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482286056 got 356482286059 for osd.0 2024-09-13T15:12:16.683 DEBUG:teuthology.parallel:result is None 2024-09-13T15:12:17.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:17 smithi003 ceph-mon[104007]: pgmap v1776: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:17.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:17 smithi003 ceph-mon[107278]: pgmap v1776: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:17.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:17 smithi019 ceph-mon[91296]: pgmap v1776: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:17.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790588 got 158913790593 for osd.5 2024-09-13T15:12:17.690 DEBUG:teuthology.parallel:result is None 2024-09-13T15:12:18.311 INFO:teuthology.orchestra.run.smithi003.stdout:588410519976 2024-09-13T15:12:18.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:18 smithi003 ceph-mon[107278]: from='client.? 172.21.15.3:0/2158150512' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:12:18.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:18 smithi003 ceph-mon[104007]: from='client.? 172.21.15.3:0/2158150512' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:12:18.681 INFO:tasks.cephadm.ceph_manager.ceph:need seq 485331304888 got 485331304892 for osd.2 2024-09-13T15:12:18.682 DEBUG:teuthology.parallel:result is None 2024-09-13T15:12:18.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:18 smithi019 ceph-mon[91296]: from='client.? 172.21.15.3:0/2158150512' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:12:19.537 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:19 smithi003 ceph-mon[107278]: pgmap v1777: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:19.537 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:19 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:12:19.537 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:19 smithi003 ceph-mon[107278]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:12:19.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:19 smithi003 ceph-mon[104007]: pgmap v1777: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:19.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:19 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:12:19.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:19 smithi003 ceph-mon[104007]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:12:19.562 INFO:tasks.cephadm.ceph_manager.ceph:need seq 588410519973 got 588410519976 for osd.3 2024-09-13T15:12:19.562 DEBUG:teuthology.parallel:result is None 2024-09-13T15:12:19.562 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T15:12:19.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:12:19.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:19 smithi019 ceph-mon[91296]: pgmap v1777: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:19.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:19 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:12:19.933 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:19 smithi019 ceph-mon[91296]: from='mgr.34107 172.21.15.3:0/4042036975' entity='mgr.y' 2024-09-13T15:12:20.062 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:20.682 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:12:21.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:21 smithi003 ceph-mon[104007]: pgmap v1778: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:21.559 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:21 smithi003 ceph-mon[107278]: pgmap v1778: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:21.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:21 smithi019 ceph-mon[91296]: pgmap v1778: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:22.018 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:12:22.019 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:12:23.080 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1778,"stamp":"2024-09-13T15:12:20.309844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":1778609,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3159,"ondisk_log_size":3159,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11238796,"kb_used_data":56356,"kb_used_omap":29,"kb_used_meta":11182306,"kb_avail":738722420,"statfs":{"total":767960285184,"available":756451758080,"internally_reserved":0,"allocated":57708544,"data_stored":34125392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":11450681870},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003128"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":280,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642471+0000","last_change":"2024-09-13T15:10:54.352401+0000","last_active":"2024-09-13T15:11:35.642471+0000","last_peered":"2024-09-13T15:11:35.642471+0000","last_clean":"2024-09-13T15:11:35.642471+0000","last_became_active":"2024-09-13T15:10:54.352024+0000","last_became_peered":"2024-09-13T15:10:54.352024+0000","last_unstale":"2024-09-13T15:11:35.642471+0000","last_undegraded":"2024-09-13T15:11:35.642471+0000","last_fullsized":"2024-09-13T15:11:35.642471+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:48.428315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:48.428315+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:30.735223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":310,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224655+0000","last_change":"2024-09-13T15:10:38.596993+0000","last_active":"2024-09-13T15:11:36.224655+0000","last_peered":"2024-09-13T15:11:36.224655+0000","last_clean":"2024-09-13T15:11:36.224655+0000","last_became_active":"2024-09-13T15:10:38.586876+0000","last_became_peered":"2024-09-13T15:10:38.586876+0000","last_unstale":"2024-09-13T15:11:36.224655+0000","last_undegraded":"2024-09-13T15:11:36.224655+0000","last_fullsized":"2024-09-13T15:11:36.224655+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:30.004190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":111,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609677+0000","last_change":"2024-09-13T15:10:33.185195+0000","last_active":"2024-09-13T15:11:33.609677+0000","last_peered":"2024-09-13T15:11:33.609677+0000","last_clean":"2024-09-13T15:11:33.609677+0000","last_became_active":"2024-09-13T15:10:33.184860+0000","last_became_peered":"2024-09-13T15:10:33.184860+0000","last_unstale":"2024-09-13T15:11:33.609677+0000","last_undegraded":"2024-09-13T15:11:33.609677+0000","last_fullsized":"2024-09-13T15:11:33.609677+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:19.510335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":225,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624527+0000","last_change":"2024-09-13T15:10:33.185232+0000","last_active":"2024-09-13T15:11:32.624527+0000","last_peered":"2024-09-13T15:11:32.624527+0000","last_clean":"2024-09-13T15:11:32.624527+0000","last_became_active":"2024-09-13T15:10:33.184875+0000","last_became_peered":"2024-09-13T15:10:33.184875+0000","last_unstale":"2024-09-13T15:11:32.624527+0000","last_undegraded":"2024-09-13T15:11:32.624527+0000","last_fullsized":"2024-09-13T15:11:32.624527+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:36.667464+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":271,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.911076+0000","last_change":"2024-09-13T15:10:38.591395+0000","last_active":"2024-09-13T15:11:36.911076+0000","last_peered":"2024-09-13T15:11:36.911076+0000","last_clean":"2024-09-13T15:11:36.911076+0000","last_became_active":"2024-09-13T15:10:38.591019+0000","last_became_peered":"2024-09-13T15:10:38.591019+0000","last_unstale":"2024-09-13T15:11:36.911076+0000","last_undegraded":"2024-09-13T15:11:36.911076+0000","last_fullsized":"2024-09-13T15:11:36.911076+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:52.602086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":263,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910493+0000","last_change":"2024-09-13T15:10:38.586543+0000","last_active":"2024-09-13T15:11:36.910493+0000","last_peered":"2024-09-13T15:11:36.910493+0000","last_clean":"2024-09-13T15:11:36.910493+0000","last_became_active":"2024-09-13T15:10:38.586092+0000","last_became_peered":"2024-09-13T15:10:38.586092+0000","last_unstale":"2024-09-13T15:11:36.910493+0000","last_undegraded":"2024-09-13T15:11:36.910493+0000","last_fullsized":"2024-09-13T15:11:36.910493+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:52.125514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":40,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610409+0000","last_change":"2024-09-13T15:11:23.586584+0000","last_active":"2024-09-13T15:11:33.610409+0000","last_peered":"2024-09-13T15:11:33.610409+0000","last_clean":"2024-09-13T15:11:33.610409+0000","last_became_active":"2024-09-13T15:11:23.585832+0000","last_became_peered":"2024-09-13T15:11:23.585832+0000","last_unstale":"2024-09-13T15:11:33.610409+0000","last_undegraded":"2024-09-13T15:11:33.610409+0000","last_fullsized":"2024-09-13T15:11:33.610409+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:50.699792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":81,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610262+0000","last_change":"2024-09-13T15:07:34.580859+0000","last_active":"2024-09-13T15:11:33.610262+0000","last_peered":"2024-09-13T15:11:33.610262+0000","last_clean":"2024-09-13T15:11:33.610262+0000","last_became_active":"2024-09-13T15:07:34.580521+0000","last_became_peered":"2024-09-13T15:07:34.580521+0000","last_unstale":"2024-09-13T15:11:33.610262+0000","last_undegraded":"2024-09-13T15:11:33.610262+0000","last_fullsized":"2024-09-13T15:11:33.610262+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:01.252401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":258,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.911039+0000","last_change":"2024-09-13T15:03:47.469254+0000","last_active":"2024-09-13T15:11:36.911039+0000","last_peered":"2024-09-13T15:11:36.911039+0000","last_clean":"2024-09-13T15:11:36.911039+0000","last_became_active":"2024-09-13T15:03:46.854242+0000","last_became_peered":"2024-09-13T15:03:46.854242+0000","last_unstale":"2024-09-13T15:11:36.911039+0000","last_undegraded":"2024-09-13T15:11:36.911039+0000","last_fullsized":"2024-09-13T15:11:36.911039+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:47.469211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:47.469211+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:59.563316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":40,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610320+0000","last_change":"2024-09-13T15:11:23.587381+0000","last_active":"2024-09-13T15:11:33.610320+0000","last_peered":"2024-09-13T15:11:33.610320+0000","last_clean":"2024-09-13T15:11:33.610320+0000","last_became_active":"2024-09-13T15:11:23.586841+0000","last_became_peered":"2024-09-13T15:11:23.586841+0000","last_unstale":"2024-09-13T15:11:33.610320+0000","last_undegraded":"2024-09-13T15:11:33.610320+0000","last_fullsized":"2024-09-13T15:11:33.610320+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:46.983890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":101,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.611015+0000","last_change":"2024-09-13T15:10:33.187680+0000","last_active":"2024-09-13T15:11:33.611015+0000","last_peered":"2024-09-13T15:11:33.611015+0000","last_clean":"2024-09-13T15:11:33.611015+0000","last_became_active":"2024-09-13T15:10:33.187300+0000","last_became_peered":"2024-09-13T15:10:33.187300+0000","last_unstale":"2024-09-13T15:11:33.611015+0000","last_undegraded":"2024-09-13T15:11:33.611015+0000","last_fullsized":"2024-09-13T15:11:33.611015+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:41.339234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":249,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910516+0000","last_change":"2024-09-13T15:03:46.864596+0000","last_active":"2024-09-13T15:11:36.910516+0000","last_peered":"2024-09-13T15:11:36.910516+0000","last_clean":"2024-09-13T15:11:36.910516+0000","last_became_active":"2024-09-13T15:03:46.863107+0000","last_became_peered":"2024-09-13T15:03:46.863107+0000","last_unstale":"2024-09-13T15:11:36.910516+0000","last_undegraded":"2024-09-13T15:11:36.910516+0000","last_fullsized":"2024-09-13T15:11:36.910516+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:07.821841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":298,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224473+0000","last_change":"2024-09-13T15:07:34.489594+0000","last_active":"2024-09-13T15:11:36.224473+0000","last_peered":"2024-09-13T15:11:36.224473+0000","last_clean":"2024-09-13T15:11:36.224473+0000","last_became_active":"2024-09-13T15:07:34.488980+0000","last_became_peered":"2024-09-13T15:07:34.488980+0000","last_unstale":"2024-09-13T15:11:36.224473+0000","last_undegraded":"2024-09-13T15:11:36.224473+0000","last_fullsized":"2024-09-13T15:11:36.224473+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:23.320438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":52,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642760+0000","last_change":"2024-09-13T15:10:38.593385+0000","last_active":"2024-09-13T15:11:35.642760+0000","last_peered":"2024-09-13T15:11:35.642760+0000","last_clean":"2024-09-13T15:11:35.642760+0000","last_became_active":"2024-09-13T15:10:38.592989+0000","last_became_peered":"2024-09-13T15:10:38.592989+0000","last_unstale":"2024-09-13T15:11:35.642760+0000","last_undegraded":"2024-09-13T15:11:35.642760+0000","last_fullsized":"2024-09-13T15:11:35.642760+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:53.498045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":244,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609815+0000","last_change":"2024-09-13T15:10:38.608518+0000","last_active":"2024-09-13T15:11:33.609815+0000","last_peered":"2024-09-13T15:11:33.609815+0000","last_clean":"2024-09-13T15:11:33.609815+0000","last_became_active":"2024-09-13T15:10:38.597305+0000","last_became_peered":"2024-09-13T15:10:38.597305+0000","last_unstale":"2024-09-13T15:11:33.609815+0000","last_undegraded":"2024-09-13T15:11:33.609815+0000","last_fullsized":"2024-09-13T15:11:33.609815+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:13.518298+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":111,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610177+0000","last_change":"2024-09-13T15:10:38.596123+0000","last_active":"2024-09-13T15:11:33.610177+0000","last_peered":"2024-09-13T15:11:33.610177+0000","last_clean":"2024-09-13T15:11:33.610177+0000","last_became_active":"2024-09-13T15:10:38.595744+0000","last_became_peered":"2024-09-13T15:10:38.595744+0000","last_unstale":"2024-09-13T15:11:33.610177+0000","last_undegraded":"2024-09-13T15:11:33.610177+0000","last_fullsized":"2024-09-13T15:11:33.610177+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:36.123362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":277,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910825+0000","last_change":"2024-09-13T15:10:33.190284+0000","last_active":"2024-09-13T15:11:36.910825+0000","last_peered":"2024-09-13T15:11:36.910825+0000","last_clean":"2024-09-13T15:11:36.910825+0000","last_became_active":"2024-09-13T15:10:33.189954+0000","last_became_peered":"2024-09-13T15:10:33.189954+0000","last_unstale":"2024-09-13T15:11:36.910825+0000","last_undegraded":"2024-09-13T15:11:36.910825+0000","last_fullsized":"2024-09-13T15:11:36.910825+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:56.658366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":238,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610325+0000","last_change":"2024-09-13T15:07:34.492713+0000","last_active":"2024-09-13T15:11:33.610325+0000","last_peered":"2024-09-13T15:11:33.610325+0000","last_clean":"2024-09-13T15:11:33.610325+0000","last_became_active":"2024-09-13T15:07:34.492318+0000","last_became_peered":"2024-09-13T15:07:34.492318+0000","last_unstale":"2024-09-13T15:11:33.610325+0000","last_undegraded":"2024-09-13T15:11:33.610325+0000","last_fullsized":"2024-09-13T15:11:33.610325+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:02.258576+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":56,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642355+0000","last_change":"2024-09-13T15:10:33.182698+0000","last_active":"2024-09-13T15:11:35.642355+0000","last_peered":"2024-09-13T15:11:35.642355+0000","last_clean":"2024-09-13T15:11:35.642355+0000","last_became_active":"2024-09-13T15:10:33.182378+0000","last_became_peered":"2024-09-13T15:10:33.182378+0000","last_unstale":"2024-09-13T15:11:35.642355+0000","last_undegraded":"2024-09-13T15:11:35.642355+0000","last_fullsized":"2024-09-13T15:11:35.642355+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:30.559273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":292,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224418+0000","last_change":"2024-09-13T15:03:46.864512+0000","last_active":"2024-09-13T15:11:36.224418+0000","last_peered":"2024-09-13T15:11:36.224418+0000","last_clean":"2024-09-13T15:11:36.224418+0000","last_became_active":"2024-09-13T15:03:46.864120+0000","last_became_peered":"2024-09-13T15:03:46.864120+0000","last_unstale":"2024-09-13T15:11:36.224418+0000","last_undegraded":"2024-09-13T15:11:36.224418+0000","last_fullsized":"2024-09-13T15:11:36.224418+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:36.450217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610073+0000","last_change":"2024-09-13T15:10:33.187979+0000","last_active":"2024-09-13T15:11:33.610073+0000","last_peered":"2024-09-13T15:11:33.610073+0000","last_clean":"2024-09-13T15:11:33.610073+0000","last_became_active":"2024-09-13T15:10:33.187401+0000","last_became_peered":"2024-09-13T15:10:33.187401+0000","last_unstale":"2024-09-13T15:11:33.610073+0000","last_undegraded":"2024-09-13T15:11:33.610073+0000","last_fullsized":"2024-09-13T15:11:33.610073+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:06:47.046963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:06:47.046963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:26.908846+0000","scrub_duration":0.018211127000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":270,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910171+0000","last_change":"2024-09-13T15:10:33.186105+0000","last_active":"2024-09-13T15:11:36.910171+0000","last_peered":"2024-09-13T15:11:36.910171+0000","last_clean":"2024-09-13T15:11:36.910171+0000","last_became_active":"2024-09-13T15:10:33.184962+0000","last_became_peered":"2024-09-13T15:10:33.184962+0000","last_unstale":"2024-09-13T15:11:36.910171+0000","last_undegraded":"2024-09-13T15:11:36.910171+0000","last_fullsized":"2024-09-13T15:11:36.910171+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:04.267837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":509,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016157+0000","last_change":"2024-09-13T15:10:33.176387+0000","last_active":"2024-09-13T15:11:33.016157+0000","last_peered":"2024-09-13T15:11:33.016157+0000","last_clean":"2024-09-13T15:11:33.016157+0000","last_became_active":"2024-09-13T15:10:33.176035+0000","last_became_peered":"2024-09-13T15:10:33.176035+0000","last_unstale":"2024-09-13T15:11:33.016157+0000","last_undegraded":"2024-09-13T15:11:33.016157+0000","last_fullsized":"2024-09-13T15:11:33.016157+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:07.953512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":251,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624433+0000","last_change":"2024-09-13T15:10:33.179766+0000","last_active":"2024-09-13T15:11:32.624433+0000","last_peered":"2024-09-13T15:11:32.624433+0000","last_clean":"2024-09-13T15:11:32.624433+0000","last_became_active":"2024-09-13T15:10:33.179400+0000","last_became_peered":"2024-09-13T15:10:33.179400+0000","last_unstale":"2024-09-13T15:11:32.624433+0000","last_undegraded":"2024-09-13T15:11:32.624433+0000","last_fullsized":"2024-09-13T15:11:32.624433+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:47.162626+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":270,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910294+0000","last_change":"2024-09-13T15:10:33.186828+0000","last_active":"2024-09-13T15:11:36.910294+0000","last_peered":"2024-09-13T15:11:36.910294+0000","last_clean":"2024-09-13T15:11:36.910294+0000","last_became_active":"2024-09-13T15:10:33.185398+0000","last_became_peered":"2024-09-13T15:10:33.185398+0000","last_unstale":"2024-09-13T15:11:36.910294+0000","last_undegraded":"2024-09-13T15:11:36.910294+0000","last_fullsized":"2024-09-13T15:11:36.910294+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:18:55.472316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":285,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224697+0000","last_change":"2024-09-13T15:03:46.871342+0000","last_active":"2024-09-13T15:11:36.224697+0000","last_peered":"2024-09-13T15:11:36.224697+0000","last_clean":"2024-09-13T15:11:36.224697+0000","last_became_active":"2024-09-13T15:03:46.863406+0000","last_became_peered":"2024-09-13T15:03:46.863406+0000","last_unstale":"2024-09-13T15:11:36.224697+0000","last_undegraded":"2024-09-13T15:11:36.224697+0000","last_fullsized":"2024-09-13T15:11:36.224697+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:53.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":249,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910383+0000","last_change":"2024-09-13T15:03:46.859350+0000","last_active":"2024-09-13T15:11:36.910383+0000","last_peered":"2024-09-13T15:11:36.910383+0000","last_clean":"2024-09-13T15:11:36.910383+0000","last_became_active":"2024-09-13T15:03:46.853202+0000","last_became_peered":"2024-09-13T15:03:46.853202+0000","last_unstale":"2024-09-13T15:11:36.910383+0000","last_undegraded":"2024-09-13T15:11:36.910383+0000","last_fullsized":"2024-09-13T15:11:36.910383+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:42.594537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":208,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610195+0000","last_change":"2024-09-13T15:10:33.186077+0000","last_active":"2024-09-13T15:11:33.610195+0000","last_peered":"2024-09-13T15:11:33.610195+0000","last_clean":"2024-09-13T15:11:33.610195+0000","last_became_active":"2024-09-13T15:10:33.185745+0000","last_became_peered":"2024-09-13T15:10:33.185745+0000","last_unstale":"2024-09-13T15:11:33.610195+0000","last_undegraded":"2024-09-13T15:11:33.610195+0000","last_fullsized":"2024-09-13T15:11:33.610195+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:39.042475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":52,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642159+0000","last_change":"2024-09-13T15:10:38.595338+0000","last_active":"2024-09-13T15:11:35.642159+0000","last_peered":"2024-09-13T15:11:35.642159+0000","last_clean":"2024-09-13T15:11:35.642159+0000","last_became_active":"2024-09-13T15:10:38.594739+0000","last_became_peered":"2024-09-13T15:10:38.594739+0000","last_unstale":"2024-09-13T15:11:35.642159+0000","last_undegraded":"2024-09-13T15:11:35.642159+0000","last_fullsized":"2024-09-13T15:11:35.642159+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:47.044170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":208,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610529+0000","last_change":"2024-09-13T15:02:19.406264+0000","last_active":"2024-09-13T15:11:33.610529+0000","last_peered":"2024-09-13T15:11:33.610529+0000","last_clean":"2024-09-13T15:11:33.610529+0000","last_became_active":"2024-09-13T15:02:19.405876+0000","last_became_peered":"2024-09-13T15:02:19.405876+0000","last_unstale":"2024-09-13T15:11:33.610529+0000","last_undegraded":"2024-09-13T15:11:33.610529+0000","last_fullsized":"2024-09-13T15:11:33.610529+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:19.208307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":334,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624133+0000","last_change":"2024-09-13T15:07:34.622674+0000","last_active":"2024-09-13T15:11:32.624133+0000","last_peered":"2024-09-13T15:11:32.624133+0000","last_clean":"2024-09-13T15:11:32.624133+0000","last_became_active":"2024-09-13T15:07:34.474941+0000","last_became_peered":"2024-09-13T15:07:34.474941+0000","last_unstale":"2024-09-13T15:11:32.624133+0000","last_undegraded":"2024-09-13T15:11:32.624133+0000","last_fullsized":"2024-09-13T15:11:32.624133+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:14.657841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":462,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642964+0000","last_change":"2024-09-13T15:10:54.563186+0000","last_active":"2024-09-13T15:11:35.642964+0000","last_peered":"2024-09-13T15:11:35.642964+0000","last_clean":"2024-09-13T15:11:35.642964+0000","last_became_active":"2024-09-13T15:10:54.562641+0000","last_became_peered":"2024-09-13T15:10:54.562641+0000","last_unstale":"2024-09-13T15:11:35.642964+0000","last_undegraded":"2024-09-13T15:11:35.642964+0000","last_fullsized":"2024-09-13T15:11:35.642964+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:55.430836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":56,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.643196+0000","last_change":"2024-09-13T15:10:33.178610+0000","last_active":"2024-09-13T15:11:35.643196+0000","last_peered":"2024-09-13T15:11:35.643196+0000","last_clean":"2024-09-13T15:11:35.643196+0000","last_became_active":"2024-09-13T15:10:33.178262+0000","last_became_peered":"2024-09-13T15:10:33.178262+0000","last_unstale":"2024-09-13T15:11:35.643196+0000","last_undegraded":"2024-09-13T15:11:35.643196+0000","last_fullsized":"2024-09-13T15:11:35.643196+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:54.033379+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610263+0000","last_change":"2024-09-13T15:07:34.603715+0000","last_active":"2024-09-13T15:11:33.610263+0000","last_peered":"2024-09-13T15:11:33.610263+0000","last_clean":"2024-09-13T15:11:33.610263+0000","last_became_active":"2024-09-13T15:07:34.476556+0000","last_became_peered":"2024-09-13T15:07:34.476556+0000","last_unstale":"2024-09-13T15:11:33.610263+0000","last_undegraded":"2024-09-13T15:11:33.610263+0000","last_fullsized":"2024-09-13T15:11:33.610263+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:49:06.015543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610940+0000","last_change":"2024-09-13T15:10:33.186688+0000","last_active":"2024-09-13T15:11:33.610940+0000","last_peered":"2024-09-13T15:11:33.610940+0000","last_clean":"2024-09-13T15:11:33.610940+0000","last_became_active":"2024-09-13T15:10:33.186244+0000","last_became_peered":"2024-09-13T15:10:33.186244+0000","last_unstale":"2024-09-13T15:11:33.610940+0000","last_undegraded":"2024-09-13T15:11:33.610940+0000","last_fullsized":"2024-09-13T15:11:33.610940+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:48.008827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":81,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609809+0000","last_change":"2024-09-13T15:07:34.470575+0000","last_active":"2024-09-13T15:11:33.609809+0000","last_peered":"2024-09-13T15:11:33.609809+0000","last_clean":"2024-09-13T15:11:33.609809+0000","last_became_active":"2024-09-13T15:07:34.470231+0000","last_became_peered":"2024-09-13T15:07:34.470231+0000","last_unstale":"2024-09-13T15:11:33.609809+0000","last_undegraded":"2024-09-13T15:11:33.609809+0000","last_fullsized":"2024-09-13T15:11:33.609809+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:34.618632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":522,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016181+0000","last_change":"2024-09-13T15:10:38.587982+0000","last_active":"2024-09-13T15:11:33.016181+0000","last_peered":"2024-09-13T15:11:33.016181+0000","last_clean":"2024-09-13T15:11:33.016181+0000","last_became_active":"2024-09-13T15:10:38.587616+0000","last_became_peered":"2024-09-13T15:10:38.587616+0000","last_unstale":"2024-09-13T15:11:33.016181+0000","last_undegraded":"2024-09-13T15:11:33.016181+0000","last_fullsized":"2024-09-13T15:11:33.016181+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:05.658213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":326,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224592+0000","last_change":"2024-09-13T15:10:33.194943+0000","last_active":"2024-09-13T15:11:36.224592+0000","last_peered":"2024-09-13T15:11:36.224592+0000","last_clean":"2024-09-13T15:11:36.224592+0000","last_became_active":"2024-09-13T15:10:33.179540+0000","last_became_peered":"2024-09-13T15:10:33.179540+0000","last_unstale":"2024-09-13T15:11:36.224592+0000","last_undegraded":"2024-09-13T15:11:36.224592+0000","last_fullsized":"2024-09-13T15:11:36.224592+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:09.321347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":250,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910696+0000","last_change":"2024-09-13T15:03:46.868597+0000","last_active":"2024-09-13T15:11:36.910696+0000","last_peered":"2024-09-13T15:11:36.910696+0000","last_clean":"2024-09-13T15:11:36.910696+0000","last_became_active":"2024-09-13T15:03:46.866939+0000","last_became_peered":"2024-09-13T15:03:46.866939+0000","last_unstale":"2024-09-13T15:11:36.910696+0000","last_undegraded":"2024-09-13T15:11:36.910696+0000","last_fullsized":"2024-09-13T15:11:36.910696+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.060483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":250,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910191+0000","last_change":"2024-09-13T15:03:46.865877+0000","last_active":"2024-09-13T15:11:36.910191+0000","last_peered":"2024-09-13T15:11:36.910191+0000","last_clean":"2024-09-13T15:11:36.910191+0000","last_became_active":"2024-09-13T15:03:46.865029+0000","last_became_peered":"2024-09-13T15:03:46.865029+0000","last_unstale":"2024-09-13T15:11:36.910191+0000","last_undegraded":"2024-09-13T15:11:36.910191+0000","last_fullsized":"2024-09-13T15:11:36.910191+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:28.871714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":395,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642827+0000","last_change":"2024-09-13T15:10:54.350796+0000","last_active":"2024-09-13T15:11:35.642827+0000","last_peered":"2024-09-13T15:11:35.642827+0000","last_clean":"2024-09-13T15:11:35.642827+0000","last_became_active":"2024-09-13T15:10:54.350535+0000","last_became_peered":"2024-09-13T15:10:54.350535+0000","last_unstale":"2024-09-13T15:11:35.642827+0000","last_undegraded":"2024-09-13T15:11:35.642827+0000","last_fullsized":"2024-09-13T15:11:35.642827+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:58.233067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":253,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.911003+0000","last_change":"2024-09-13T15:03:46.879528+0000","last_active":"2024-09-13T15:11:36.911003+0000","last_peered":"2024-09-13T15:11:36.911003+0000","last_clean":"2024-09-13T15:11:36.911003+0000","last_became_active":"2024-09-13T15:03:46.860753+0000","last_became_peered":"2024-09-13T15:03:46.860753+0000","last_unstale":"2024-09-13T15:11:36.911003+0000","last_undegraded":"2024-09-13T15:11:36.911003+0000","last_fullsized":"2024-09-13T15:11:36.911003+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:28.561501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":101,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.611070+0000","last_change":"2024-09-13T15:10:38.597845+0000","last_active":"2024-09-13T15:11:33.611070+0000","last_peered":"2024-09-13T15:11:33.611070+0000","last_clean":"2024-09-13T15:11:33.611070+0000","last_became_active":"2024-09-13T15:10:38.597521+0000","last_became_peered":"2024-09-13T15:10:38.597521+0000","last_unstale":"2024-09-13T15:11:33.611070+0000","last_undegraded":"2024-09-13T15:11:33.611070+0000","last_fullsized":"2024-09-13T15:11:33.611070+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:06:46.602703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:06:46.602703+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:35.721654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":278,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224534+0000","last_change":"2024-09-13T15:02:19.423088+0000","last_active":"2024-09-13T15:11:36.224534+0000","last_peered":"2024-09-13T15:11:36.224534+0000","last_clean":"2024-09-13T15:11:36.224534+0000","last_became_active":"2024-09-13T15:02:19.422675+0000","last_became_peered":"2024-09-13T15:02:19.422675+0000","last_unstale":"2024-09-13T15:11:36.224534+0000","last_undegraded":"2024-09-13T15:11:36.224534+0000","last_fullsized":"2024-09-13T15:11:36.224534+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:58.764834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":532,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624502+0000","last_change":"2024-09-13T15:07:34.516311+0000","last_active":"2024-09-13T15:11:32.624502+0000","last_peered":"2024-09-13T15:11:32.624502+0000","last_clean":"2024-09-13T15:11:32.624502+0000","last_became_active":"2024-09-13T15:07:34.516030+0000","last_became_peered":"2024-09-13T15:07:34.516030+0000","last_unstale":"2024-09-13T15:11:32.624502+0000","last_undegraded":"2024-09-13T15:11:32.624502+0000","last_fullsized":"2024-09-13T15:11:32.624502+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:28.151366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":762,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642520+0000","last_change":"2024-09-13T15:10:54.562525+0000","last_active":"2024-09-13T15:11:35.642520+0000","last_peered":"2024-09-13T15:11:35.642520+0000","last_clean":"2024-09-13T15:11:35.642520+0000","last_became_active":"2024-09-13T15:10:54.562151+0000","last_became_peered":"2024-09-13T15:10:54.562151+0000","last_unstale":"2024-09-13T15:11:35.642520+0000","last_undegraded":"2024-09-13T15:11:35.642520+0000","last_fullsized":"2024-09-13T15:11:35.642520+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:56.010672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":503,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.015928+0000","last_change":"2024-09-13T15:03:46.869628+0000","last_active":"2024-09-13T15:11:33.015928+0000","last_peered":"2024-09-13T15:11:33.015928+0000","last_clean":"2024-09-13T15:11:33.015928+0000","last_became_active":"2024-09-13T15:03:46.869274+0000","last_became_peered":"2024-09-13T15:03:46.869274+0000","last_unstale":"2024-09-13T15:11:33.015928+0000","last_undegraded":"2024-09-13T15:11:33.015928+0000","last_fullsized":"2024-09-13T15:11:33.015928+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:21.999130+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642152+0000","last_change":"2024-09-13T15:11:31.937236+0000","last_active":"2024-09-13T15:11:35.642152+0000","last_peered":"2024-09-13T15:11:35.642152+0000","last_clean":"2024-09-13T15:11:35.642152+0000","last_became_active":"2024-09-13T15:11:31.936836+0000","last_became_peered":"2024-09-13T15:11:31.936836+0000","last_unstale":"2024-09-13T15:11:35.642152+0000","last_undegraded":"2024-09-13T15:11:35.642152+0000","last_fullsized":"2024-09-13T15:11:35.642152+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:47.852070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":287,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642726+0000","last_change":"2024-09-13T15:10:54.562225+0000","last_active":"2024-09-13T15:11:35.642726+0000","last_peered":"2024-09-13T15:11:35.642726+0000","last_clean":"2024-09-13T15:11:35.642726+0000","last_became_active":"2024-09-13T15:10:54.561720+0000","last_became_peered":"2024-09-13T15:10:54.561720+0000","last_unstale":"2024-09-13T15:11:35.642726+0000","last_undegraded":"2024-09-13T15:11:35.642726+0000","last_fullsized":"2024-09-13T15:11:35.642726+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:56.605755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":228,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610125+0000","last_change":"2024-09-13T15:10:38.598292+0000","last_active":"2024-09-13T15:11:33.610125+0000","last_peered":"2024-09-13T15:11:33.610125+0000","last_clean":"2024-09-13T15:11:33.610125+0000","last_became_active":"2024-09-13T15:10:38.597953+0000","last_became_peered":"2024-09-13T15:10:38.597953+0000","last_unstale":"2024-09-13T15:11:33.610125+0000","last_undegraded":"2024-09-13T15:11:33.610125+0000","last_fullsized":"2024-09-13T15:11:33.610125+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:32.999171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":503,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016079+0000","last_change":"2024-09-13T15:03:46.851975+0000","last_active":"2024-09-13T15:11:33.016079+0000","last_peered":"2024-09-13T15:11:33.016079+0000","last_clean":"2024-09-13T15:11:33.016079+0000","last_became_active":"2024-09-13T15:03:46.851616+0000","last_became_peered":"2024-09-13T15:03:46.851616+0000","last_unstale":"2024-09-13T15:11:33.016079+0000","last_undegraded":"2024-09-13T15:11:33.016079+0000","last_fullsized":"2024-09-13T15:11:33.016079+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:01.819924+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":672,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642295+0000","last_change":"2024-09-13T15:10:54.562476+0000","last_active":"2024-09-13T15:11:35.642295+0000","last_peered":"2024-09-13T15:11:35.642295+0000","last_clean":"2024-09-13T15:11:35.642295+0000","last_became_active":"2024-09-13T15:10:54.561878+0000","last_became_peered":"2024-09-13T15:10:54.561878+0000","last_unstale":"2024-09-13T15:11:35.642295+0000","last_undegraded":"2024-09-13T15:11:35.642295+0000","last_fullsized":"2024-09-13T15:11:35.642295+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:57.482937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":81,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610857+0000","last_change":"2024-09-13T15:07:34.598991+0000","last_active":"2024-09-13T15:11:33.610857+0000","last_peered":"2024-09-13T15:11:33.610857+0000","last_clean":"2024-09-13T15:11:33.610857+0000","last_became_active":"2024-09-13T15:07:34.598343+0000","last_became_peered":"2024-09-13T15:07:34.598343+0000","last_unstale":"2024-09-13T15:11:33.610857+0000","last_undegraded":"2024-09-13T15:11:33.610857+0000","last_fullsized":"2024-09-13T15:11:33.610857+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:31.371179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":40,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609849+0000","last_change":"2024-09-13T15:11:23.585996+0000","last_active":"2024-09-13T15:11:33.609849+0000","last_peered":"2024-09-13T15:11:33.609849+0000","last_clean":"2024-09-13T15:11:33.609849+0000","last_became_active":"2024-09-13T15:11:23.585520+0000","last_became_peered":"2024-09-13T15:11:23.585520+0000","last_unstale":"2024-09-13T15:11:33.609849+0000","last_undegraded":"2024-09-13T15:11:33.609849+0000","last_fullsized":"2024-09-13T15:11:33.609849+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:38.302040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":46,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609663+0000","last_change":"2024-09-13T15:11:23.587475+0000","last_active":"2024-09-13T15:11:33.609663+0000","last_peered":"2024-09-13T15:11:33.609663+0000","last_clean":"2024-09-13T15:11:33.609663+0000","last_became_active":"2024-09-13T15:11:23.587067+0000","last_became_peered":"2024-09-13T15:11:23.587067+0000","last_unstale":"2024-09-13T15:11:33.609663+0000","last_undegraded":"2024-09-13T15:11:33.609663+0000","last_fullsized":"2024-09-13T15:11:33.609663+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:17.742735+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642509+0000","last_change":"2024-09-13T15:11:31.937168+0000","last_active":"2024-09-13T15:11:35.642509+0000","last_peered":"2024-09-13T15:11:35.642509+0000","last_clean":"2024-09-13T15:11:35.642509+0000","last_became_active":"2024-09-13T15:11:31.936814+0000","last_became_peered":"2024-09-13T15:11:31.936814+0000","last_unstale":"2024-09-13T15:11:35.642509+0000","last_undegraded":"2024-09-13T15:11:35.642509+0000","last_fullsized":"2024-09-13T15:11:35.642509+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:11.347185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":600,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642448+0000","last_change":"2024-09-13T15:10:54.563043+0000","last_active":"2024-09-13T15:11:35.642448+0000","last_peered":"2024-09-13T15:11:35.642448+0000","last_clean":"2024-09-13T15:11:35.642448+0000","last_became_active":"2024-09-13T15:10:54.561916+0000","last_became_peered":"2024-09-13T15:10:54.561916+0000","last_unstale":"2024-09-13T15:11:35.642448+0000","last_undegraded":"2024-09-13T15:11:35.642448+0000","last_fullsized":"2024-09-13T15:11:35.642448+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:55.603207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642180+0000","last_change":"2024-09-13T15:10:38.586172+0000","last_active":"2024-09-13T15:11:35.642180+0000","last_peered":"2024-09-13T15:11:35.642180+0000","last_clean":"2024-09-13T15:11:35.642180+0000","last_became_active":"2024-09-13T15:10:38.585821+0000","last_became_peered":"2024-09-13T15:10:38.585821+0000","last_unstale":"2024-09-13T15:11:35.642180+0000","last_undegraded":"2024-09-13T15:11:35.642180+0000","last_fullsized":"2024-09-13T15:11:35.642180+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:49.566977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":304,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624166+0000","last_change":"2024-09-13T15:07:34.464473+0000","last_active":"2024-09-13T15:11:32.624166+0000","last_peered":"2024-09-13T15:11:32.624166+0000","last_clean":"2024-09-13T15:11:32.624166+0000","last_became_active":"2024-09-13T15:07:34.464122+0000","last_became_peered":"2024-09-13T15:07:34.464122+0000","last_unstale":"2024-09-13T15:11:32.624166+0000","last_undegraded":"2024-09-13T15:11:32.624166+0000","last_fullsized":"2024-09-13T15:11:32.624166+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:55.352605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":306,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224458+0000","last_change":"2024-09-13T15:10:38.589163+0000","last_active":"2024-09-13T15:11:36.224458+0000","last_peered":"2024-09-13T15:11:36.224458+0000","last_clean":"2024-09-13T15:11:36.224458+0000","last_became_active":"2024-09-13T15:10:38.588847+0000","last_became_peered":"2024-09-13T15:10:38.588847+0000","last_unstale":"2024-09-13T15:11:36.224458+0000","last_undegraded":"2024-09-13T15:11:36.224458+0000","last_fullsized":"2024-09-13T15:11:36.224458+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:50.632833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":306,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224070+0000","last_change":"2024-09-13T15:07:34.460652+0000","last_active":"2024-09-13T15:11:36.224070+0000","last_peered":"2024-09-13T15:11:36.224070+0000","last_clean":"2024-09-13T15:11:36.224070+0000","last_became_active":"2024-09-13T15:07:34.458909+0000","last_became_peered":"2024-09-13T15:07:34.458909+0000","last_unstale":"2024-09-13T15:11:36.224070+0000","last_undegraded":"2024-09-13T15:11:36.224070+0000","last_fullsized":"2024-09-13T15:11:36.224070+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:05.444737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":559,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.623977+0000","last_change":"2024-09-13T15:10:33.176495+0000","last_active":"2024-09-13T15:11:32.623977+0000","last_peered":"2024-09-13T15:11:32.623977+0000","last_clean":"2024-09-13T15:11:32.623977+0000","last_became_active":"2024-09-13T15:10:33.176072+0000","last_became_peered":"2024-09-13T15:10:33.176072+0000","last_unstale":"2024-09-13T15:11:32.623977+0000","last_undegraded":"2024-09-13T15:11:32.623977+0000","last_fullsized":"2024-09-13T15:11:32.623977+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:01.922196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":495,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.643056+0000","last_change":"2024-09-13T15:10:54.350647+0000","last_active":"2024-09-13T15:11:35.643056+0000","last_peered":"2024-09-13T15:11:35.643056+0000","last_clean":"2024-09-13T15:11:35.643056+0000","last_became_active":"2024-09-13T15:10:54.350313+0000","last_became_peered":"2024-09-13T15:10:54.350313+0000","last_unstale":"2024-09-13T15:11:35.643056+0000","last_undegraded":"2024-09-13T15:11:35.643056+0000","last_fullsized":"2024-09-13T15:11:35.643056+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:37.114426+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":472,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016009+0000","last_change":"2024-09-13T15:07:34.462911+0000","last_active":"2024-09-13T15:11:33.016009+0000","last_peered":"2024-09-13T15:11:33.016009+0000","last_clean":"2024-09-13T15:11:33.016009+0000","last_became_active":"2024-09-13T15:07:34.462577+0000","last_became_peered":"2024-09-13T15:07:34.462577+0000","last_unstale":"2024-09-13T15:11:33.016009+0000","last_undegraded":"2024-09-13T15:11:33.016009+0000","last_fullsized":"2024-09-13T15:11:33.016009+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:51.431494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":250,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910491+0000","last_change":"2024-09-13T15:03:46.875410+0000","last_active":"2024-09-13T15:11:36.910491+0000","last_peered":"2024-09-13T15:11:36.910491+0000","last_clean":"2024-09-13T15:11:36.910491+0000","last_became_active":"2024-09-13T15:03:46.875103+0000","last_became_peered":"2024-09-13T15:03:46.875103+0000","last_unstale":"2024-09-13T15:11:36.910491+0000","last_undegraded":"2024-09-13T15:11:36.910491+0000","last_fullsized":"2024-09-13T15:11:36.910491+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:53.941854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":220,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610231+0000","last_change":"2024-09-13T15:10:38.605256+0000","last_active":"2024-09-13T15:11:33.610231+0000","last_peered":"2024-09-13T15:11:33.610231+0000","last_clean":"2024-09-13T15:11:33.610231+0000","last_became_active":"2024-09-13T15:10:38.596513+0000","last_became_peered":"2024-09-13T15:10:38.596513+0000","last_unstale":"2024-09-13T15:11:33.610231+0000","last_undegraded":"2024-09-13T15:11:33.610231+0000","last_fullsized":"2024-09-13T15:11:33.610231+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:30.444160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":284,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.623794+0000","last_change":"2024-09-13T15:02:19.407022+0000","last_active":"2024-09-13T15:11:32.623794+0000","last_peered":"2024-09-13T15:11:32.623794+0000","last_clean":"2024-09-13T15:11:32.623794+0000","last_became_active":"2024-09-13T15:02:19.406620+0000","last_became_peered":"2024-09-13T15:02:19.406620+0000","last_unstale":"2024-09-13T15:11:32.623794+0000","last_undegraded":"2024-09-13T15:11:32.623794+0000","last_fullsized":"2024-09-13T15:11:32.623794+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:37.100188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":56,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.643309+0000","last_change":"2024-09-13T15:10:33.178505+0000","last_active":"2024-09-13T15:11:35.643309+0000","last_peered":"2024-09-13T15:11:35.643309+0000","last_clean":"2024-09-13T15:11:35.643309+0000","last_became_active":"2024-09-13T15:10:33.176902+0000","last_became_peered":"2024-09-13T15:10:33.176902+0000","last_unstale":"2024-09-13T15:11:35.643309+0000","last_undegraded":"2024-09-13T15:11:35.643309+0000","last_fullsized":"2024-09-13T15:11:35.643309+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:23.999225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":270,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910275+0000","last_change":"2024-09-13T15:10:33.186515+0000","last_active":"2024-09-13T15:11:36.910275+0000","last_peered":"2024-09-13T15:11:36.910275+0000","last_clean":"2024-09-13T15:11:36.910275+0000","last_became_active":"2024-09-13T15:10:33.185795+0000","last_became_peered":"2024-09-13T15:10:33.185795+0000","last_unstale":"2024-09-13T15:11:36.910275+0000","last_undegraded":"2024-09-13T15:11:36.910275+0000","last_fullsized":"2024-09-13T15:11:36.910275+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:57.771790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":36,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610266+0000","last_change":"2024-09-13T15:11:23.586471+0000","last_active":"2024-09-13T15:11:33.610266+0000","last_peered":"2024-09-13T15:11:33.610266+0000","last_clean":"2024-09-13T15:11:33.610266+0000","last_became_active":"2024-09-13T15:11:23.585360+0000","last_became_peered":"2024-09-13T15:11:23.585360+0000","last_unstale":"2024-09-13T15:11:33.610266+0000","last_undegraded":"2024-09-13T15:11:33.610266+0000","last_fullsized":"2024-09-13T15:11:33.610266+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:10:38.865945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:10:38.865945+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:09.360128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610734+0000","last_change":"2024-09-13T15:10:33.186094+0000","last_active":"2024-09-13T15:11:33.610734+0000","last_peered":"2024-09-13T15:11:33.610734+0000","last_clean":"2024-09-13T15:11:33.610734+0000","last_became_active":"2024-09-13T15:10:33.185738+0000","last_became_peered":"2024-09-13T15:10:33.185738+0000","last_unstale":"2024-09-13T15:11:33.610734+0000","last_undegraded":"2024-09-13T15:11:33.610734+0000","last_fullsized":"2024-09-13T15:11:33.610734+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:36.389856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":111,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609849+0000","last_change":"2024-09-13T15:10:38.596092+0000","last_active":"2024-09-13T15:11:33.609849+0000","last_peered":"2024-09-13T15:11:33.609849+0000","last_clean":"2024-09-13T15:11:33.609849+0000","last_became_active":"2024-09-13T15:10:38.595779+0000","last_became_peered":"2024-09-13T15:10:38.595779+0000","last_unstale":"2024-09-13T15:11:33.609849+0000","last_undegraded":"2024-09-13T15:11:33.609849+0000","last_fullsized":"2024-09-13T15:11:33.609849+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:50.275843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":304,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642996+0000","last_change":"2024-09-13T15:10:54.348913+0000","last_active":"2024-09-13T15:11:35.642996+0000","last_peered":"2024-09-13T15:11:35.642996+0000","last_clean":"2024-09-13T15:11:35.642996+0000","last_became_active":"2024-09-13T15:10:54.348545+0000","last_became_peered":"2024-09-13T15:10:54.348545+0000","last_unstale":"2024-09-13T15:11:35.642996+0000","last_undegraded":"2024-09-13T15:11:35.642996+0000","last_fullsized":"2024-09-13T15:11:35.642996+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:02:36.128767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:02:36.128767+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:10.554097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":111,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.611029+0000","last_change":"2024-09-13T15:10:38.597354+0000","last_active":"2024-09-13T15:11:33.611029+0000","last_peered":"2024-09-13T15:11:33.611029+0000","last_clean":"2024-09-13T15:11:33.611029+0000","last_became_active":"2024-09-13T15:10:38.597069+0000","last_became_peered":"2024-09-13T15:10:38.597069+0000","last_unstale":"2024-09-13T15:11:33.611029+0000","last_undegraded":"2024-09-13T15:11:33.611029+0000","last_fullsized":"2024-09-13T15:11:33.611029+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:58:16.938993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":530,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.015740+0000","last_change":"2024-09-13T15:10:33.179554+0000","last_active":"2024-09-13T15:11:33.015740+0000","last_peered":"2024-09-13T15:11:33.015740+0000","last_clean":"2024-09-13T15:11:33.015740+0000","last_became_active":"2024-09-13T15:10:33.179245+0000","last_became_peered":"2024-09-13T15:10:33.179245+0000","last_unstale":"2024-09-13T15:11:33.015740+0000","last_undegraded":"2024-09-13T15:11:33.015740+0000","last_fullsized":"2024-09-13T15:11:33.015740+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:13.309940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":36,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609721+0000","last_change":"2024-09-13T15:11:23.587205+0000","last_active":"2024-09-13T15:11:33.609721+0000","last_peered":"2024-09-13T15:11:33.609721+0000","last_clean":"2024-09-13T15:11:33.609721+0000","last_became_active":"2024-09-13T15:11:23.586754+0000","last_became_peered":"2024-09-13T15:11:23.586754+0000","last_unstale":"2024-09-13T15:11:33.609721+0000","last_undegraded":"2024-09-13T15:11:33.609721+0000","last_fullsized":"2024-09-13T15:11:33.609721+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:53.982822+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":81,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610479+0000","last_change":"2024-09-13T15:07:34.473797+0000","last_active":"2024-09-13T15:11:33.610479+0000","last_peered":"2024-09-13T15:11:33.610479+0000","last_clean":"2024-09-13T15:11:33.610479+0000","last_became_active":"2024-09-13T15:07:34.473444+0000","last_became_peered":"2024-09-13T15:07:34.473444+0000","last_unstale":"2024-09-13T15:11:33.610479+0000","last_undegraded":"2024-09-13T15:11:33.610479+0000","last_fullsized":"2024-09-13T15:11:33.610479+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:04.278156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":286,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910576+0000","last_change":"2024-09-13T15:07:34.609901+0000","last_active":"2024-09-13T15:11:36.910576+0000","last_peered":"2024-09-13T15:11:36.910576+0000","last_clean":"2024-09-13T15:11:36.910576+0000","last_became_active":"2024-09-13T15:07:34.490242+0000","last_became_peered":"2024-09-13T15:07:34.490242+0000","last_unstale":"2024-09-13T15:11:36.910576+0000","last_undegraded":"2024-09-13T15:11:36.910576+0000","last_fullsized":"2024-09-13T15:11:36.910576+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:23.303360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":350,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.623796+0000","last_change":"2024-09-13T15:10:38.596913+0000","last_active":"2024-09-13T15:11:32.623796+0000","last_peered":"2024-09-13T15:11:32.623796+0000","last_clean":"2024-09-13T15:11:32.623796+0000","last_became_active":"2024-09-13T15:10:38.596491+0000","last_became_peered":"2024-09-13T15:10:38.596491+0000","last_unstale":"2024-09-13T15:11:32.623796+0000","last_undegraded":"2024-09-13T15:11:32.623796+0000","last_fullsized":"2024-09-13T15:11:32.623796+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:55.706139+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":313,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224414+0000","last_change":"2024-09-13T15:11:31.878256+0000","last_active":"2024-09-13T15:11:36.224414+0000","last_peered":"2024-09-13T15:11:36.224414+0000","last_clean":"2024-09-13T15:11:36.224414+0000","last_became_active":"2024-09-13T15:11:31.877964+0000","last_became_peered":"2024-09-13T15:11:31.877964+0000","last_unstale":"2024-09-13T15:11:36.224414+0000","last_undegraded":"2024-09-13T15:11:36.224414+0000","last_fullsized":"2024-09-13T15:11:36.224414+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:01.878232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":132,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610471+0000","last_change":"2024-09-13T15:11:23.586387+0000","last_active":"2024-09-13T15:11:33.610471+0000","last_peered":"2024-09-13T15:11:33.610471+0000","last_clean":"2024-09-13T15:11:33.610471+0000","last_became_active":"2024-09-13T15:11:23.586071+0000","last_became_peered":"2024-09-13T15:11:23.586071+0000","last_unstale":"2024-09-13T15:11:33.610471+0000","last_undegraded":"2024-09-13T15:11:33.610471+0000","last_fullsized":"2024-09-13T15:11:33.610471+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:48.803141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610804+0000","last_change":"2024-09-13T15:08:59.871028+0000","last_active":"2024-09-13T15:11:33.610804+0000","last_peered":"2024-09-13T15:11:33.610804+0000","last_clean":"2024-09-13T15:11:33.610804+0000","last_became_active":"2024-09-13T15:07:34.478006+0000","last_became_peered":"2024-09-13T15:07:34.478006+0000","last_unstale":"2024-09-13T15:11:33.610804+0000","last_undegraded":"2024-09-13T15:11:33.610804+0000","last_fullsized":"2024-09-13T15:11:33.610804+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:08:59.870786+0000","last_deep_scrub":"79'100","last_deep_scrub_stamp":"2024-09-13T15:08:59.870786+0000","last_clean_scrub_stamp":"2024-09-13T15:08:59.870786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:43:55.603202+0000","scrub_duration":0.010767947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":512,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.015600+0000","last_change":"2024-09-13T15:10:33.177131+0000","last_active":"2024-09-13T15:11:33.015600+0000","last_peered":"2024-09-13T15:11:33.015600+0000","last_clean":"2024-09-13T15:11:33.015600+0000","last_became_active":"2024-09-13T15:10:33.176765+0000","last_became_peered":"2024-09-13T15:10:33.176765+0000","last_unstale":"2024-09-13T15:11:33.015600+0000","last_undegraded":"2024-09-13T15:11:33.015600+0000","last_fullsized":"2024-09-13T15:11:33.015600+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:53.991570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":511,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624392+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:11:32.624392+0000","last_peered":"2024-09-13T15:11:32.624392+0000","last_clean":"2024-09-13T15:11:32.624392+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:11:32.624392+0000","last_undegraded":"2024-09-13T15:11:32.624392+0000","last_fullsized":"2024-09-13T15:11:32.624392+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":483,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016085+0000","last_change":"2024-09-13T15:07:34.463017+0000","last_active":"2024-09-13T15:11:33.016085+0000","last_peered":"2024-09-13T15:11:33.016085+0000","last_clean":"2024-09-13T15:11:33.016085+0000","last_became_active":"2024-09-13T15:07:34.462570+0000","last_became_peered":"2024-09-13T15:07:34.462570+0000","last_unstale":"2024-09-13T15:11:33.016085+0000","last_undegraded":"2024-09-13T15:11:33.016085+0000","last_fullsized":"2024-09-13T15:11:33.016085+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:56.696358+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609956+0000","last_change":"2024-09-13T15:10:33.190304+0000","last_active":"2024-09-13T15:11:33.609956+0000","last_peered":"2024-09-13T15:11:33.609956+0000","last_clean":"2024-09-13T15:11:33.609956+0000","last_became_active":"2024-09-13T15:10:33.187803+0000","last_became_peered":"2024-09-13T15:10:33.187803+0000","last_unstale":"2024-09-13T15:11:33.609956+0000","last_undegraded":"2024-09-13T15:11:33.609956+0000","last_fullsized":"2024-09-13T15:11:33.609956+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:21.100709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":529,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624514+0000","last_change":"2024-09-13T15:10:33.186232+0000","last_active":"2024-09-13T15:11:32.624514+0000","last_peered":"2024-09-13T15:11:32.624514+0000","last_clean":"2024-09-13T15:11:32.624514+0000","last_became_active":"2024-09-13T15:10:33.173156+0000","last_became_peered":"2024-09-13T15:10:33.173156+0000","last_unstale":"2024-09-13T15:11:32.624514+0000","last_undegraded":"2024-09-13T15:11:32.624514+0000","last_fullsized":"2024-09-13T15:11:32.624514+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:25.582656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":253,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910733+0000","last_change":"2024-09-13T15:03:46.882026+0000","last_active":"2024-09-13T15:11:36.910733+0000","last_peered":"2024-09-13T15:11:36.910733+0000","last_clean":"2024-09-13T15:11:36.910733+0000","last_became_active":"2024-09-13T15:03:46.859557+0000","last_became_peered":"2024-09-13T15:03:46.859557+0000","last_unstale":"2024-09-13T15:11:36.910733+0000","last_undegraded":"2024-09-13T15:11:36.910733+0000","last_fullsized":"2024-09-13T15:11:36.910733+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:00.933755+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":188,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609716+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:11:33.609716+0000","last_peered":"2024-09-13T15:11:33.609716+0000","last_clean":"2024-09-13T15:11:33.609716+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:11:33.609716+0000","last_undegraded":"2024-09-13T15:11:33.609716+0000","last_fullsized":"2024-09-13T15:11:33.609716+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":338,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624526+0000","last_change":"2024-09-13T15:07:34.635697+0000","last_active":"2024-09-13T15:11:32.624526+0000","last_peered":"2024-09-13T15:11:32.624526+0000","last_clean":"2024-09-13T15:11:32.624526+0000","last_became_active":"2024-09-13T15:07:34.568238+0000","last_became_peered":"2024-09-13T15:07:34.568238+0000","last_unstale":"2024-09-13T15:11:32.624526+0000","last_undegraded":"2024-09-13T15:11:32.624526+0000","last_fullsized":"2024-09-13T15:11:32.624526+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:55.995331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":344,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624253+0000","last_change":"2024-09-13T15:10:33.181566+0000","last_active":"2024-09-13T15:11:32.624253+0000","last_peered":"2024-09-13T15:11:32.624253+0000","last_clean":"2024-09-13T15:11:32.624253+0000","last_became_active":"2024-09-13T15:10:33.181146+0000","last_became_peered":"2024-09-13T15:10:33.181146+0000","last_unstale":"2024-09-13T15:11:32.624253+0000","last_undegraded":"2024-09-13T15:11:32.624253+0000","last_fullsized":"2024-09-13T15:11:32.624253+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:06.002058+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":598,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.015689+0000","last_change":"2024-09-13T15:02:19.426511+0000","last_active":"2024-09-13T15:11:33.015689+0000","last_peered":"2024-09-13T15:11:33.015689+0000","last_clean":"2024-09-13T15:11:33.015689+0000","last_became_active":"2024-09-13T15:02:19.391448+0000","last_became_peered":"2024-09-13T15:02:19.391448+0000","last_unstale":"2024-09-13T15:11:33.015689+0000","last_undegraded":"2024-09-13T15:11:33.015689+0000","last_fullsized":"2024-09-13T15:11:33.015689+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:40.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":263,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.911084+0000","last_change":"2024-09-13T15:10:33.185456+0000","last_active":"2024-09-13T15:11:36.911084+0000","last_peered":"2024-09-13T15:11:36.911084+0000","last_clean":"2024-09-13T15:11:36.911084+0000","last_became_active":"2024-09-13T15:10:33.185108+0000","last_became_peered":"2024-09-13T15:10:33.185108+0000","last_unstale":"2024-09-13T15:11:36.911084+0000","last_undegraded":"2024-09-13T15:11:36.911084+0000","last_fullsized":"2024-09-13T15:11:36.911084+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:44.754388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":208,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610093+0000","last_change":"2024-09-13T15:10:33.185313+0000","last_active":"2024-09-13T15:11:33.610093+0000","last_peered":"2024-09-13T15:11:33.610093+0000","last_clean":"2024-09-13T15:11:33.610093+0000","last_became_active":"2024-09-13T15:10:33.184950+0000","last_became_peered":"2024-09-13T15:10:33.184950+0000","last_unstale":"2024-09-13T15:11:33.610093+0000","last_undegraded":"2024-09-13T15:11:33.610093+0000","last_fullsized":"2024-09-13T15:11:33.610093+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:05.151863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":494,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624016+0000","last_change":"2024-09-13T15:07:34.466871+0000","last_active":"2024-09-13T15:11:32.624016+0000","last_peered":"2024-09-13T15:11:32.624016+0000","last_clean":"2024-09-13T15:11:32.624016+0000","last_became_active":"2024-09-13T15:07:34.466559+0000","last_became_peered":"2024-09-13T15:07:34.466559+0000","last_unstale":"2024-09-13T15:11:32.624016+0000","last_undegraded":"2024-09-13T15:11:32.624016+0000","last_fullsized":"2024-09-13T15:11:32.624016+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:48:24.377353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":292,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224604+0000","last_change":"2024-09-13T15:03:46.864291+0000","last_active":"2024-09-13T15:11:36.224604+0000","last_peered":"2024-09-13T15:11:36.224604+0000","last_clean":"2024-09-13T15:11:36.224604+0000","last_became_active":"2024-09-13T15:03:46.863671+0000","last_became_peered":"2024-09-13T15:03:46.863671+0000","last_unstale":"2024-09-13T15:11:36.224604+0000","last_undegraded":"2024-09-13T15:11:36.224604+0000","last_fullsized":"2024-09-13T15:11:36.224604+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:28.672041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":265,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642886+0000","last_change":"2024-09-13T15:10:54.351461+0000","last_active":"2024-09-13T15:11:35.642886+0000","last_peered":"2024-09-13T15:11:35.642886+0000","last_clean":"2024-09-13T15:11:35.642886+0000","last_became_active":"2024-09-13T15:10:54.351127+0000","last_became_peered":"2024-09-13T15:10:54.351127+0000","last_unstale":"2024-09-13T15:11:35.642886+0000","last_undegraded":"2024-09-13T15:11:35.642886+0000","last_fullsized":"2024-09-13T15:11:35.642886+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:37.993717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":292,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224294+0000","last_change":"2024-09-13T15:03:46.863848+0000","last_active":"2024-09-13T15:11:36.224294+0000","last_peered":"2024-09-13T15:11:36.224294+0000","last_clean":"2024-09-13T15:11:36.224294+0000","last_became_active":"2024-09-13T15:03:46.863523+0000","last_became_peered":"2024-09-13T15:03:46.863523+0000","last_unstale":"2024-09-13T15:11:36.224294+0000","last_undegraded":"2024-09-13T15:11:36.224294+0000","last_fullsized":"2024-09-13T15:11:36.224294+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:39.303427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":278,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610314+0000","last_change":"2024-09-13T15:11:23.586832+0000","last_active":"2024-09-13T15:11:33.610314+0000","last_peered":"2024-09-13T15:11:33.610314+0000","last_clean":"2024-09-13T15:11:33.610314+0000","last_became_active":"2024-09-13T15:11:23.586420+0000","last_became_peered":"2024-09-13T15:11:23.586420+0000","last_unstale":"2024-09-13T15:11:33.610314+0000","last_undegraded":"2024-09-13T15:11:33.610314+0000","last_fullsized":"2024-09-13T15:11:33.610314+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:13.491913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":36,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609615+0000","last_change":"2024-09-13T15:11:23.586214+0000","last_active":"2024-09-13T15:11:33.609615+0000","last_peered":"2024-09-13T15:11:33.609615+0000","last_clean":"2024-09-13T15:11:33.609615+0000","last_became_active":"2024-09-13T15:11:23.584762+0000","last_became_peered":"2024-09-13T15:11:23.584762+0000","last_unstale":"2024-09-13T15:11:33.609615+0000","last_undegraded":"2024-09-13T15:11:33.609615+0000","last_fullsized":"2024-09-13T15:11:33.609615+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:38.709689+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":553,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624137+0000","last_change":"2024-09-13T15:10:38.606150+0000","last_active":"2024-09-13T15:11:32.624137+0000","last_peered":"2024-09-13T15:11:32.624137+0000","last_clean":"2024-09-13T15:11:32.624137+0000","last_became_active":"2024-09-13T15:10:38.598310+0000","last_became_peered":"2024-09-13T15:10:38.598310+0000","last_unstale":"2024-09-13T15:11:32.624137+0000","last_undegraded":"2024-09-13T15:11:32.624137+0000","last_fullsized":"2024-09-13T15:11:32.624137+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:33.450888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":170,"num_keys_recovered":50,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":270,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910783+0000","last_change":"2024-09-13T15:10:33.188502+0000","last_active":"2024-09-13T15:11:36.910783+0000","last_peered":"2024-09-13T15:11:36.910783+0000","last_clean":"2024-09-13T15:11:36.910783+0000","last_became_active":"2024-09-13T15:10:33.187700+0000","last_became_peered":"2024-09-13T15:10:33.187700+0000","last_unstale":"2024-09-13T15:11:36.910783+0000","last_undegraded":"2024-09-13T15:11:36.910783+0000","last_fullsized":"2024-09-13T15:11:36.910783+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:15.107002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":292,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224423+0000","last_change":"2024-09-13T15:03:46.864604+0000","last_active":"2024-09-13T15:11:36.224423+0000","last_peered":"2024-09-13T15:11:36.224423+0000","last_clean":"2024-09-13T15:11:36.224423+0000","last_became_active":"2024-09-13T15:03:46.864085+0000","last_became_peered":"2024-09-13T15:03:46.864085+0000","last_unstale":"2024-09-13T15:11:36.224423+0000","last_undegraded":"2024-09-13T15:11:36.224423+0000","last_fullsized":"2024-09-13T15:11:36.224423+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:32.524052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7486,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365432,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1520584,"kb_used_data":7100,"kb_used_omap":1,"kb_used_meta":1513470,"kb_avail":92224568,"statfs":{"total":95995035648,"available":94437957632,"internally_reserved":0,"allocated":7270400,"data_stored":4309878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1549793624},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561662,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499388,"kb_used_data":7152,"kb_used_omap":1,"kb_used_meta":1492222,"kb_avail":92245764,"statfs":{"total":95995035648,"available":94459662336,"internally_reserved":0,"allocated":7323648,"data_stored":4374693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1528035672},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790594,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497652,"kb_used_data":7080,"kb_used_omap":1,"kb_used_meta":1490558,"kb_avail":92247500,"statfs":{"total":95995035648,"available":94461440000,"internally_reserved":0,"allocated":7249920,"data_stored":4305275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1526331736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986822,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509128,"kb_used_data":6840,"kb_used_omap":1,"kb_used_meta":1502270,"kb_avail":92236024,"statfs":{"total":95995035648,"available":94449688576,"internally_reserved":0,"allocated":7004160,"data_stored":4047385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1538324824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519977,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267120,"kb_used_data":7140,"kb_used_omap":4,"kb_used_meta":1259963,"kb_avail":92478032,"statfs":{"total":95995035648,"available":94697504768,"internally_reserved":0,"allocated":7311360,"data_stored":4370277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":1290202560},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304893,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1265276,"kb_used_data":7084,"kb_used_omap":6,"kb_used_meta":1258169,"kb_avail":92479876,"statfs":{"total":95995035648,"available":94699393024,"internally_reserved":0,"allocated":7254016,"data_stored":4305799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":1288365511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828196,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1423152,"kb_used_data":6820,"kb_used_omap":5,"kb_used_meta":1416314,"kb_avail":92322000,"statfs":{"total":95995035648,"available":94537728000,"internally_reserved":0,"allocated":6983680,"data_stored":4042534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1450305772},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482286061,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1256496,"kb_used_data":7140,"kb_used_omap":5,"kb_used_meta":1249338,"kb_avail":92488656,"statfs":{"total":95995035648,"available":94708383744,"internally_reserved":0,"allocated":7311360,"data_stored":4369551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":1279322171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:12:23.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 77a3f016-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:12:23.457 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:23.764 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:23 smithi003 ceph-mon[107278]: from='client.47798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:12:23.764 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:23 smithi003 ceph-mon[107278]: pgmap v1779: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:23.764 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:23 smithi003 ceph-mon[104007]: from='client.47798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:12:23.764 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:23 smithi003 ceph-mon[104007]: pgmap v1779: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:23.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:23 smithi019 ceph-mon[91296]: from='client.47798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:12:23.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:23 smithi019 ceph-mon[91296]: pgmap v1779: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:24.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:24 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: ::ffff:172.21.15.19 - - [13/Sep/2024:15:12:24] "GET /metrics HTTP/1.1" 200 34115 "" "Prometheus/2.51.0" 2024-09-13T15:12:25.681 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:25 smithi003 ceph-mon[104007]: pgmap v1780: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:25.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:25 smithi003 ceph-mon[107278]: pgmap v1780: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:25.932 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:25 smithi019 ceph-mon[91296]: pgmap v1780: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:26.096 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-13T15:12:26.097 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-13T15:12:26.868 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1780,"stamp":"2024-09-13T15:12:24.311083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":1778609,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3159,"ondisk_log_size":3159,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11238796,"kb_used_data":56356,"kb_used_omap":29,"kb_used_meta":11182306,"kb_avail":738722420,"statfs":{"total":767960285184,"available":756451758080,"internally_reserved":0,"allocated":57708544,"data_stored":34125392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30194,"internal_metadata":11450681870},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003149"},"pg_stats":[{"pgid":"1.1d","version":"79'100","reported_seq":280,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642471+0000","last_change":"2024-09-13T15:10:54.352401+0000","last_active":"2024-09-13T15:11:35.642471+0000","last_peered":"2024-09-13T15:11:35.642471+0000","last_clean":"2024-09-13T15:11:35.642471+0000","last_became_active":"2024-09-13T15:10:54.352024+0000","last_became_peered":"2024-09-13T15:10:54.352024+0000","last_unstale":"2024-09-13T15:11:35.642471+0000","last_undegraded":"2024-09-13T15:11:35.642471+0000","last_fullsized":"2024-09-13T15:11:35.642471+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:48.428315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:48.428315+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:30.735223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4","reported_seq":310,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224655+0000","last_change":"2024-09-13T15:10:38.596993+0000","last_active":"2024-09-13T15:11:36.224655+0000","last_peered":"2024-09-13T15:11:36.224655+0000","last_clean":"2024-09-13T15:11:36.224655+0000","last_became_active":"2024-09-13T15:10:38.586876+0000","last_became_peered":"2024-09-13T15:10:38.586876+0000","last_unstale":"2024-09-13T15:11:36.224655+0000","last_undegraded":"2024-09-13T15:11:36.224655+0000","last_fullsized":"2024-09-13T15:11:36.224655+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:30.004190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":111,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609677+0000","last_change":"2024-09-13T15:10:33.185195+0000","last_active":"2024-09-13T15:11:33.609677+0000","last_peered":"2024-09-13T15:11:33.609677+0000","last_clean":"2024-09-13T15:11:33.609677+0000","last_became_active":"2024-09-13T15:10:33.184860+0000","last_became_peered":"2024-09-13T15:10:33.184860+0000","last_unstale":"2024-09-13T15:11:33.609677+0000","last_undegraded":"2024-09-13T15:11:33.609677+0000","last_fullsized":"2024-09-13T15:11:33.609677+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:19.510335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"79'100","reported_seq":225,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624527+0000","last_change":"2024-09-13T15:10:33.185232+0000","last_active":"2024-09-13T15:11:32.624527+0000","last_peered":"2024-09-13T15:11:32.624527+0000","last_clean":"2024-09-13T15:11:32.624527+0000","last_became_active":"2024-09-13T15:10:33.184875+0000","last_became_peered":"2024-09-13T15:10:33.184875+0000","last_unstale":"2024-09-13T15:11:32.624527+0000","last_undegraded":"2024-09-13T15:11:32.624527+0000","last_fullsized":"2024-09-13T15:11:32.624527+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:58.200349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:58.200349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:36.667464+0000","scrub_duration":0.001707194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"65'1","reported_seq":271,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.911076+0000","last_change":"2024-09-13T15:10:38.591395+0000","last_active":"2024-09-13T15:11:36.911076+0000","last_peered":"2024-09-13T15:11:36.911076+0000","last_clean":"2024-09-13T15:11:36.911076+0000","last_became_active":"2024-09-13T15:10:38.591019+0000","last_became_peered":"2024-09-13T15:10:38.591019+0000","last_unstale":"2024-09-13T15:11:36.911076+0000","last_undegraded":"2024-09-13T15:11:36.911076+0000","last_fullsized":"2024-09-13T15:11:36.911076+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:52.602086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":263,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910493+0000","last_change":"2024-09-13T15:10:38.586543+0000","last_active":"2024-09-13T15:11:36.910493+0000","last_peered":"2024-09-13T15:11:36.910493+0000","last_clean":"2024-09-13T15:11:36.910493+0000","last_became_active":"2024-09-13T15:10:38.586092+0000","last_became_peered":"2024-09-13T15:10:38.586092+0000","last_unstale":"2024-09-13T15:11:36.910493+0000","last_undegraded":"2024-09-13T15:11:36.910493+0000","last_fullsized":"2024-09-13T15:11:36.910493+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:52.125514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"65'2","reported_seq":40,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610409+0000","last_change":"2024-09-13T15:11:23.586584+0000","last_active":"2024-09-13T15:11:33.610409+0000","last_peered":"2024-09-13T15:11:33.610409+0000","last_clean":"2024-09-13T15:11:33.610409+0000","last_became_active":"2024-09-13T15:11:23.585832+0000","last_became_peered":"2024-09-13T15:11:23.585832+0000","last_unstale":"2024-09-13T15:11:33.610409+0000","last_undegraded":"2024-09-13T15:11:33.610409+0000","last_fullsized":"2024-09-13T15:11:33.610409+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:50.699792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":81,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610262+0000","last_change":"2024-09-13T15:07:34.580859+0000","last_active":"2024-09-13T15:11:33.610262+0000","last_peered":"2024-09-13T15:11:33.610262+0000","last_clean":"2024-09-13T15:11:33.610262+0000","last_became_active":"2024-09-13T15:07:34.580521+0000","last_became_peered":"2024-09-13T15:07:34.580521+0000","last_unstale":"2024-09-13T15:11:33.610262+0000","last_undegraded":"2024-09-13T15:11:33.610262+0000","last_fullsized":"2024-09-13T15:11:33.610262+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:01.252401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"79'100","reported_seq":258,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.911039+0000","last_change":"2024-09-13T15:03:47.469254+0000","last_active":"2024-09-13T15:11:36.911039+0000","last_peered":"2024-09-13T15:11:36.911039+0000","last_clean":"2024-09-13T15:11:36.911039+0000","last_became_active":"2024-09-13T15:03:46.854242+0000","last_became_peered":"2024-09-13T15:03:46.854242+0000","last_unstale":"2024-09-13T15:11:36.911039+0000","last_undegraded":"2024-09-13T15:11:36.911039+0000","last_fullsized":"2024-09-13T15:11:36.911039+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:03:47.469211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:03:47.469211+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:59.563316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"65'2","reported_seq":40,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610320+0000","last_change":"2024-09-13T15:11:23.587381+0000","last_active":"2024-09-13T15:11:33.610320+0000","last_peered":"2024-09-13T15:11:33.610320+0000","last_clean":"2024-09-13T15:11:33.610320+0000","last_became_active":"2024-09-13T15:11:23.586841+0000","last_became_peered":"2024-09-13T15:11:23.586841+0000","last_unstale":"2024-09-13T15:11:33.610320+0000","last_undegraded":"2024-09-13T15:11:33.610320+0000","last_fullsized":"2024-09-13T15:11:33.610320+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:46.983890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":101,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.611015+0000","last_change":"2024-09-13T15:10:33.187680+0000","last_active":"2024-09-13T15:11:33.611015+0000","last_peered":"2024-09-13T15:11:33.611015+0000","last_clean":"2024-09-13T15:11:33.611015+0000","last_became_active":"2024-09-13T15:10:33.187300+0000","last_became_peered":"2024-09-13T15:10:33.187300+0000","last_unstale":"2024-09-13T15:11:33.611015+0000","last_undegraded":"2024-09-13T15:11:33.611015+0000","last_fullsized":"2024-09-13T15:11:33.611015+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:41.339234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"79'100","reported_seq":249,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910516+0000","last_change":"2024-09-13T15:03:46.864596+0000","last_active":"2024-09-13T15:11:36.910516+0000","last_peered":"2024-09-13T15:11:36.910516+0000","last_clean":"2024-09-13T15:11:36.910516+0000","last_became_active":"2024-09-13T15:03:46.863107+0000","last_became_peered":"2024-09-13T15:03:46.863107+0000","last_unstale":"2024-09-13T15:11:36.910516+0000","last_undegraded":"2024-09-13T15:11:36.910516+0000","last_fullsized":"2024-09-13T15:11:36.910516+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:44.676745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.676745+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:07.821841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'1","reported_seq":299,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224523+0000","last_change":"2024-09-13T15:07:34.489594+0000","last_active":"2024-09-13T15:11:36.224523+0000","last_peered":"2024-09-13T15:11:36.224523+0000","last_clean":"2024-09-13T15:11:36.224523+0000","last_became_active":"2024-09-13T15:07:34.488980+0000","last_became_peered":"2024-09-13T15:07:34.488980+0000","last_unstale":"2024-09-13T15:11:36.224523+0000","last_undegraded":"2024-09-13T15:11:36.224523+0000","last_fullsized":"2024-09-13T15:11:36.224523+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:23.320438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":52,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642760+0000","last_change":"2024-09-13T15:10:38.593385+0000","last_active":"2024-09-13T15:11:35.642760+0000","last_peered":"2024-09-13T15:11:35.642760+0000","last_clean":"2024-09-13T15:11:35.642760+0000","last_became_active":"2024-09-13T15:10:38.592989+0000","last_became_peered":"2024-09-13T15:10:38.592989+0000","last_unstale":"2024-09-13T15:11:35.642760+0000","last_undegraded":"2024-09-13T15:11:35.642760+0000","last_fullsized":"2024-09-13T15:11:35.642760+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:53.498045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"79'100","reported_seq":244,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609815+0000","last_change":"2024-09-13T15:10:38.608518+0000","last_active":"2024-09-13T15:11:33.609815+0000","last_peered":"2024-09-13T15:11:33.609815+0000","last_clean":"2024-09-13T15:11:33.609815+0000","last_became_active":"2024-09-13T15:10:38.597305+0000","last_became_peered":"2024-09-13T15:10:38.597305+0000","last_unstale":"2024-09-13T15:11:33.609815+0000","last_undegraded":"2024-09-13T15:11:33.609815+0000","last_fullsized":"2024-09-13T15:11:33.609815+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:09.011870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:09.011870+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:13.518298+0000","scrub_duration":0.0098790500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'1","reported_seq":111,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610177+0000","last_change":"2024-09-13T15:10:38.596123+0000","last_active":"2024-09-13T15:11:33.610177+0000","last_peered":"2024-09-13T15:11:33.610177+0000","last_clean":"2024-09-13T15:11:33.610177+0000","last_became_active":"2024-09-13T15:10:38.595744+0000","last_became_peered":"2024-09-13T15:10:38.595744+0000","last_unstale":"2024-09-13T15:11:33.610177+0000","last_undegraded":"2024-09-13T15:11:33.610177+0000","last_fullsized":"2024-09-13T15:11:33.610177+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:36.123362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":277,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910825+0000","last_change":"2024-09-13T15:10:33.190284+0000","last_active":"2024-09-13T15:11:36.910825+0000","last_peered":"2024-09-13T15:11:36.910825+0000","last_clean":"2024-09-13T15:11:36.910825+0000","last_became_active":"2024-09-13T15:10:33.189954+0000","last_became_peered":"2024-09-13T15:10:33.189954+0000","last_unstale":"2024-09-13T15:11:36.910825+0000","last_undegraded":"2024-09-13T15:11:36.910825+0000","last_fullsized":"2024-09-13T15:11:36.910825+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:56.658366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"79'100","reported_seq":238,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610325+0000","last_change":"2024-09-13T15:07:34.492713+0000","last_active":"2024-09-13T15:11:33.610325+0000","last_peered":"2024-09-13T15:11:33.610325+0000","last_clean":"2024-09-13T15:11:33.610325+0000","last_became_active":"2024-09-13T15:07:34.492318+0000","last_became_peered":"2024-09-13T15:07:34.492318+0000","last_unstale":"2024-09-13T15:11:33.610325+0000","last_undegraded":"2024-09-13T15:11:33.610325+0000","last_fullsized":"2024-09-13T15:11:33.610325+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.066377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.066377+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:02.258576+0000","scrub_duration":0.0028187949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":56,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642355+0000","last_change":"2024-09-13T15:10:33.182698+0000","last_active":"2024-09-13T15:11:35.642355+0000","last_peered":"2024-09-13T15:11:35.642355+0000","last_clean":"2024-09-13T15:11:35.642355+0000","last_became_active":"2024-09-13T15:10:33.182378+0000","last_became_peered":"2024-09-13T15:10:33.182378+0000","last_unstale":"2024-09-13T15:11:35.642355+0000","last_undegraded":"2024-09-13T15:11:35.642355+0000","last_fullsized":"2024-09-13T15:11:35.642355+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:30.559273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":293,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224482+0000","last_change":"2024-09-13T15:03:46.864512+0000","last_active":"2024-09-13T15:11:36.224482+0000","last_peered":"2024-09-13T15:11:36.224482+0000","last_clean":"2024-09-13T15:11:36.224482+0000","last_became_active":"2024-09-13T15:03:46.864120+0000","last_became_peered":"2024-09-13T15:03:46.864120+0000","last_unstale":"2024-09-13T15:11:36.224482+0000","last_undegraded":"2024-09-13T15:11:36.224482+0000","last_fullsized":"2024-09-13T15:11:36.224482+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:36.450217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"79'100","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610073+0000","last_change":"2024-09-13T15:10:33.187979+0000","last_active":"2024-09-13T15:11:33.610073+0000","last_peered":"2024-09-13T15:11:33.610073+0000","last_clean":"2024-09-13T15:11:33.610073+0000","last_became_active":"2024-09-13T15:10:33.187401+0000","last_became_peered":"2024-09-13T15:10:33.187401+0000","last_unstale":"2024-09-13T15:11:33.610073+0000","last_undegraded":"2024-09-13T15:11:33.610073+0000","last_fullsized":"2024-09-13T15:11:33.610073+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:06:47.046963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:06:47.046963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:26.908846+0000","scrub_duration":0.018211127000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":270,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910171+0000","last_change":"2024-09-13T15:10:33.186105+0000","last_active":"2024-09-13T15:11:36.910171+0000","last_peered":"2024-09-13T15:11:36.910171+0000","last_clean":"2024-09-13T15:11:36.910171+0000","last_became_active":"2024-09-13T15:10:33.184962+0000","last_became_peered":"2024-09-13T15:10:33.184962+0000","last_unstale":"2024-09-13T15:11:36.910171+0000","last_undegraded":"2024-09-13T15:11:36.910171+0000","last_fullsized":"2024-09-13T15:11:36.910171+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:04.267837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":509,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016157+0000","last_change":"2024-09-13T15:10:33.176387+0000","last_active":"2024-09-13T15:11:33.016157+0000","last_peered":"2024-09-13T15:11:33.016157+0000","last_clean":"2024-09-13T15:11:33.016157+0000","last_became_active":"2024-09-13T15:10:33.176035+0000","last_became_peered":"2024-09-13T15:10:33.176035+0000","last_unstale":"2024-09-13T15:11:33.016157+0000","last_undegraded":"2024-09-13T15:11:33.016157+0000","last_fullsized":"2024-09-13T15:11:33.016157+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:07.953512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"79'100","reported_seq":251,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624433+0000","last_change":"2024-09-13T15:10:33.179766+0000","last_active":"2024-09-13T15:11:32.624433+0000","last_peered":"2024-09-13T15:11:32.624433+0000","last_clean":"2024-09-13T15:11:32.624433+0000","last_became_active":"2024-09-13T15:10:33.179400+0000","last_became_peered":"2024-09-13T15:10:33.179400+0000","last_unstale":"2024-09-13T15:11:32.624433+0000","last_undegraded":"2024-09-13T15:11:32.624433+0000","last_fullsized":"2024-09-13T15:11:32.624433+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:04.168847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:04.168847+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:47.162626+0000","scrub_duration":0.0080619750000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":270,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910294+0000","last_change":"2024-09-13T15:10:33.186828+0000","last_active":"2024-09-13T15:11:36.910294+0000","last_peered":"2024-09-13T15:11:36.910294+0000","last_clean":"2024-09-13T15:11:36.910294+0000","last_became_active":"2024-09-13T15:10:33.185398+0000","last_became_peered":"2024-09-13T15:10:33.185398+0000","last_unstale":"2024-09-13T15:11:36.910294+0000","last_undegraded":"2024-09-13T15:11:36.910294+0000","last_fullsized":"2024-09-13T15:11:36.910294+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:18:55.472316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":285,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224697+0000","last_change":"2024-09-13T15:03:46.871342+0000","last_active":"2024-09-13T15:11:36.224697+0000","last_peered":"2024-09-13T15:11:36.224697+0000","last_clean":"2024-09-13T15:11:36.224697+0000","last_became_active":"2024-09-13T15:03:46.863406+0000","last_became_peered":"2024-09-13T15:03:46.863406+0000","last_unstale":"2024-09-13T15:11:36.224697+0000","last_undegraded":"2024-09-13T15:11:36.224697+0000","last_fullsized":"2024-09-13T15:11:36.224697+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:53.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":249,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910383+0000","last_change":"2024-09-13T15:03:46.859350+0000","last_active":"2024-09-13T15:11:36.910383+0000","last_peered":"2024-09-13T15:11:36.910383+0000","last_clean":"2024-09-13T15:11:36.910383+0000","last_became_active":"2024-09-13T15:03:46.853202+0000","last_became_peered":"2024-09-13T15:03:46.853202+0000","last_unstale":"2024-09-13T15:11:36.910383+0000","last_undegraded":"2024-09-13T15:11:36.910383+0000","last_fullsized":"2024-09-13T15:11:36.910383+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:42.594537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":208,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610195+0000","last_change":"2024-09-13T15:10:33.186077+0000","last_active":"2024-09-13T15:11:33.610195+0000","last_peered":"2024-09-13T15:11:33.610195+0000","last_clean":"2024-09-13T15:11:33.610195+0000","last_became_active":"2024-09-13T15:10:33.185745+0000","last_became_peered":"2024-09-13T15:10:33.185745+0000","last_unstale":"2024-09-13T15:11:33.610195+0000","last_undegraded":"2024-09-13T15:11:33.610195+0000","last_fullsized":"2024-09-13T15:11:33.610195+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:39.042475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"79'100","reported_seq":52,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642159+0000","last_change":"2024-09-13T15:10:38.595338+0000","last_active":"2024-09-13T15:11:35.642159+0000","last_peered":"2024-09-13T15:11:35.642159+0000","last_clean":"2024-09-13T15:11:35.642159+0000","last_became_active":"2024-09-13T15:10:38.594739+0000","last_became_peered":"2024-09-13T15:10:38.594739+0000","last_unstale":"2024-09-13T15:11:35.642159+0000","last_undegraded":"2024-09-13T15:11:35.642159+0000","last_fullsized":"2024-09-13T15:11:35.642159+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:46.396394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:46.396394+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:47.044170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"79'100","reported_seq":208,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610529+0000","last_change":"2024-09-13T15:02:19.406264+0000","last_active":"2024-09-13T15:11:33.610529+0000","last_peered":"2024-09-13T15:11:33.610529+0000","last_clean":"2024-09-13T15:11:33.610529+0000","last_became_active":"2024-09-13T15:02:19.405876+0000","last_became_peered":"2024-09-13T15:02:19.405876+0000","last_unstale":"2024-09-13T15:11:33.610529+0000","last_undegraded":"2024-09-13T15:11:33.610529+0000","last_fullsized":"2024-09-13T15:11:33.610529+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.384159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.384159+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:19.208307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"65'3","reported_seq":334,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624133+0000","last_change":"2024-09-13T15:07:34.622674+0000","last_active":"2024-09-13T15:11:32.624133+0000","last_peered":"2024-09-13T15:11:32.624133+0000","last_clean":"2024-09-13T15:11:32.624133+0000","last_became_active":"2024-09-13T15:07:34.474941+0000","last_became_peered":"2024-09-13T15:07:34.474941+0000","last_unstale":"2024-09-13T15:11:32.624133+0000","last_undegraded":"2024-09-13T15:11:32.624133+0000","last_fullsized":"2024-09-13T15:11:32.624133+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:14.657841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":462,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642964+0000","last_change":"2024-09-13T15:10:54.563186+0000","last_active":"2024-09-13T15:11:35.642964+0000","last_peered":"2024-09-13T15:11:35.642964+0000","last_clean":"2024-09-13T15:11:35.642964+0000","last_became_active":"2024-09-13T15:10:54.562641+0000","last_became_peered":"2024-09-13T15:10:54.562641+0000","last_unstale":"2024-09-13T15:11:35.642964+0000","last_undegraded":"2024-09-13T15:11:35.642964+0000","last_fullsized":"2024-09-13T15:11:35.642964+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:55.430836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"79'100","reported_seq":56,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.643196+0000","last_change":"2024-09-13T15:10:33.178610+0000","last_active":"2024-09-13T15:11:35.643196+0000","last_peered":"2024-09-13T15:11:35.643196+0000","last_clean":"2024-09-13T15:11:35.643196+0000","last_became_active":"2024-09-13T15:10:33.178262+0000","last_became_peered":"2024-09-13T15:10:33.178262+0000","last_unstale":"2024-09-13T15:11:35.643196+0000","last_undegraded":"2024-09-13T15:11:35.643196+0000","last_fullsized":"2024-09-13T15:11:35.643196+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":319,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:40.932451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.932451+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:54.033379+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"79'23","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610263+0000","last_change":"2024-09-13T15:07:34.603715+0000","last_active":"2024-09-13T15:11:33.610263+0000","last_peered":"2024-09-13T15:11:33.610263+0000","last_clean":"2024-09-13T15:11:33.610263+0000","last_became_active":"2024-09-13T15:07:34.476556+0000","last_became_peered":"2024-09-13T15:07:34.476556+0000","last_unstale":"2024-09-13T15:11:33.610263+0000","last_undegraded":"2024-09-13T15:11:33.610263+0000","last_fullsized":"2024-09-13T15:11:33.610263+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:49:06.015543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610940+0000","last_change":"2024-09-13T15:10:33.186688+0000","last_active":"2024-09-13T15:11:33.610940+0000","last_peered":"2024-09-13T15:11:33.610940+0000","last_clean":"2024-09-13T15:11:33.610940+0000","last_became_active":"2024-09-13T15:10:33.186244+0000","last_became_peered":"2024-09-13T15:10:33.186244+0000","last_unstale":"2024-09-13T15:11:33.610940+0000","last_undegraded":"2024-09-13T15:11:33.610940+0000","last_fullsized":"2024-09-13T15:11:33.610940+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:48.008827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":81,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609809+0000","last_change":"2024-09-13T15:07:34.470575+0000","last_active":"2024-09-13T15:11:33.609809+0000","last_peered":"2024-09-13T15:11:33.609809+0000","last_clean":"2024-09-13T15:11:33.609809+0000","last_became_active":"2024-09-13T15:07:34.470231+0000","last_became_peered":"2024-09-13T15:07:34.470231+0000","last_unstale":"2024-09-13T15:11:33.609809+0000","last_undegraded":"2024-09-13T15:11:33.609809+0000","last_fullsized":"2024-09-13T15:11:33.609809+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:34.618632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":522,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016181+0000","last_change":"2024-09-13T15:10:38.587982+0000","last_active":"2024-09-13T15:11:33.016181+0000","last_peered":"2024-09-13T15:11:33.016181+0000","last_clean":"2024-09-13T15:11:33.016181+0000","last_became_active":"2024-09-13T15:10:38.587616+0000","last_became_peered":"2024-09-13T15:10:38.587616+0000","last_unstale":"2024-09-13T15:11:33.016181+0000","last_undegraded":"2024-09-13T15:11:33.016181+0000","last_fullsized":"2024-09-13T15:11:33.016181+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:05.658213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":326,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224592+0000","last_change":"2024-09-13T15:10:33.194943+0000","last_active":"2024-09-13T15:11:36.224592+0000","last_peered":"2024-09-13T15:11:36.224592+0000","last_clean":"2024-09-13T15:11:36.224592+0000","last_became_active":"2024-09-13T15:10:33.179540+0000","last_became_peered":"2024-09-13T15:10:33.179540+0000","last_unstale":"2024-09-13T15:11:36.224592+0000","last_undegraded":"2024-09-13T15:11:36.224592+0000","last_fullsized":"2024-09-13T15:11:36.224592+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:09.321347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'100","reported_seq":250,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910696+0000","last_change":"2024-09-13T15:03:46.868597+0000","last_active":"2024-09-13T15:11:36.910696+0000","last_peered":"2024-09-13T15:11:36.910696+0000","last_clean":"2024-09-13T15:11:36.910696+0000","last_became_active":"2024-09-13T15:03:46.866939+0000","last_became_peered":"2024-09-13T15:03:46.866939+0000","last_unstale":"2024-09-13T15:11:36.910696+0000","last_undegraded":"2024-09-13T15:11:36.910696+0000","last_fullsized":"2024-09-13T15:11:36.910696+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:45.352264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:45.352264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.060483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":250,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910191+0000","last_change":"2024-09-13T15:03:46.865877+0000","last_active":"2024-09-13T15:11:36.910191+0000","last_peered":"2024-09-13T15:11:36.910191+0000","last_clean":"2024-09-13T15:11:36.910191+0000","last_became_active":"2024-09-13T15:03:46.865029+0000","last_became_peered":"2024-09-13T15:03:46.865029+0000","last_unstale":"2024-09-13T15:11:36.910191+0000","last_undegraded":"2024-09-13T15:11:36.910191+0000","last_fullsized":"2024-09-13T15:11:36.910191+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:28.871714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":395,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642827+0000","last_change":"2024-09-13T15:10:54.350796+0000","last_active":"2024-09-13T15:11:35.642827+0000","last_peered":"2024-09-13T15:11:35.642827+0000","last_clean":"2024-09-13T15:11:35.642827+0000","last_became_active":"2024-09-13T15:10:54.350535+0000","last_became_peered":"2024-09-13T15:10:54.350535+0000","last_unstale":"2024-09-13T15:11:35.642827+0000","last_undegraded":"2024-09-13T15:11:35.642827+0000","last_fullsized":"2024-09-13T15:11:35.642827+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:58.233067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"65'2","reported_seq":253,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.911003+0000","last_change":"2024-09-13T15:03:46.879528+0000","last_active":"2024-09-13T15:11:36.911003+0000","last_peered":"2024-09-13T15:11:36.911003+0000","last_clean":"2024-09-13T15:11:36.911003+0000","last_became_active":"2024-09-13T15:03:46.860753+0000","last_became_peered":"2024-09-13T15:03:46.860753+0000","last_unstale":"2024-09-13T15:11:36.911003+0000","last_undegraded":"2024-09-13T15:11:36.911003+0000","last_fullsized":"2024-09-13T15:11:36.911003+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:28.561501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'100","reported_seq":101,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.611070+0000","last_change":"2024-09-13T15:10:38.597845+0000","last_active":"2024-09-13T15:11:33.611070+0000","last_peered":"2024-09-13T15:11:33.611070+0000","last_clean":"2024-09-13T15:11:33.611070+0000","last_became_active":"2024-09-13T15:10:38.597521+0000","last_became_peered":"2024-09-13T15:10:38.597521+0000","last_unstale":"2024-09-13T15:11:33.611070+0000","last_undegraded":"2024-09-13T15:11:33.611070+0000","last_fullsized":"2024-09-13T15:11:33.611070+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:06:46.602703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:06:46.602703+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:35.721654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":278,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224534+0000","last_change":"2024-09-13T15:02:19.423088+0000","last_active":"2024-09-13T15:11:36.224534+0000","last_peered":"2024-09-13T15:11:36.224534+0000","last_clean":"2024-09-13T15:11:36.224534+0000","last_became_active":"2024-09-13T15:02:19.422675+0000","last_became_peered":"2024-09-13T15:02:19.422675+0000","last_unstale":"2024-09-13T15:11:36.224534+0000","last_undegraded":"2024-09-13T15:11:36.224534+0000","last_fullsized":"2024-09-13T15:11:36.224534+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:58.764834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":532,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624502+0000","last_change":"2024-09-13T15:07:34.516311+0000","last_active":"2024-09-13T15:11:32.624502+0000","last_peered":"2024-09-13T15:11:32.624502+0000","last_clean":"2024-09-13T15:11:32.624502+0000","last_became_active":"2024-09-13T15:07:34.516030+0000","last_became_peered":"2024-09-13T15:07:34.516030+0000","last_unstale":"2024-09-13T15:11:32.624502+0000","last_undegraded":"2024-09-13T15:11:32.624502+0000","last_fullsized":"2024-09-13T15:11:32.624502+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:28.151366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028719999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":762,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642520+0000","last_change":"2024-09-13T15:10:54.562525+0000","last_active":"2024-09-13T15:11:35.642520+0000","last_peered":"2024-09-13T15:11:35.642520+0000","last_clean":"2024-09-13T15:11:35.642520+0000","last_became_active":"2024-09-13T15:10:54.562151+0000","last_became_peered":"2024-09-13T15:10:54.562151+0000","last_unstale":"2024-09-13T15:11:35.642520+0000","last_undegraded":"2024-09-13T15:11:35.642520+0000","last_fullsized":"2024-09-13T15:11:35.642520+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:56.010672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'100","reported_seq":503,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.015928+0000","last_change":"2024-09-13T15:03:46.869628+0000","last_active":"2024-09-13T15:11:33.015928+0000","last_peered":"2024-09-13T15:11:33.015928+0000","last_clean":"2024-09-13T15:11:33.015928+0000","last_became_active":"2024-09-13T15:03:46.869274+0000","last_became_peered":"2024-09-13T15:03:46.869274+0000","last_unstale":"2024-09-13T15:11:33.015928+0000","last_undegraded":"2024-09-13T15:11:33.015928+0000","last_fullsized":"2024-09-13T15:11:33.015928+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:06.244944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:06.244944+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:21.999130+0000","scrub_duration":0.0098850069999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642152+0000","last_change":"2024-09-13T15:11:31.937236+0000","last_active":"2024-09-13T15:11:35.642152+0000","last_peered":"2024-09-13T15:11:35.642152+0000","last_clean":"2024-09-13T15:11:35.642152+0000","last_became_active":"2024-09-13T15:11:31.936836+0000","last_became_peered":"2024-09-13T15:11:31.936836+0000","last_unstale":"2024-09-13T15:11:35.642152+0000","last_undegraded":"2024-09-13T15:11:35.642152+0000","last_fullsized":"2024-09-13T15:11:35.642152+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:47.852070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":287,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642726+0000","last_change":"2024-09-13T15:10:54.562225+0000","last_active":"2024-09-13T15:11:35.642726+0000","last_peered":"2024-09-13T15:11:35.642726+0000","last_clean":"2024-09-13T15:11:35.642726+0000","last_became_active":"2024-09-13T15:10:54.561720+0000","last_became_peered":"2024-09-13T15:10:54.561720+0000","last_unstale":"2024-09-13T15:11:35.642726+0000","last_undegraded":"2024-09-13T15:11:35.642726+0000","last_fullsized":"2024-09-13T15:11:35.642726+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:56.605755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026533699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":228,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610125+0000","last_change":"2024-09-13T15:10:38.598292+0000","last_active":"2024-09-13T15:11:33.610125+0000","last_peered":"2024-09-13T15:11:33.610125+0000","last_clean":"2024-09-13T15:11:33.610125+0000","last_became_active":"2024-09-13T15:10:38.597953+0000","last_became_peered":"2024-09-13T15:10:38.597953+0000","last_unstale":"2024-09-13T15:11:33.610125+0000","last_undegraded":"2024-09-13T15:11:33.610125+0000","last_fullsized":"2024-09-13T15:11:33.610125+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:32.999171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'100","reported_seq":503,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016079+0000","last_change":"2024-09-13T15:03:46.851975+0000","last_active":"2024-09-13T15:11:33.016079+0000","last_peered":"2024-09-13T15:11:33.016079+0000","last_clean":"2024-09-13T15:11:33.016079+0000","last_became_active":"2024-09-13T15:03:46.851616+0000","last_became_peered":"2024-09-13T15:03:46.851616+0000","last_unstale":"2024-09-13T15:11:33.016079+0000","last_undegraded":"2024-09-13T15:11:33.016079+0000","last_fullsized":"2024-09-13T15:11:33.016079+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:58:05.197666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.197666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:01.819924+0000","scrub_duration":0.0035411650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":672,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642295+0000","last_change":"2024-09-13T15:10:54.562476+0000","last_active":"2024-09-13T15:11:35.642295+0000","last_peered":"2024-09-13T15:11:35.642295+0000","last_clean":"2024-09-13T15:11:35.642295+0000","last_became_active":"2024-09-13T15:10:54.561878+0000","last_became_peered":"2024-09-13T15:10:54.561878+0000","last_unstale":"2024-09-13T15:11:35.642295+0000","last_undegraded":"2024-09-13T15:11:35.642295+0000","last_fullsized":"2024-09-13T15:11:35.642295+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:57.482937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":81,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610857+0000","last_change":"2024-09-13T15:07:34.598991+0000","last_active":"2024-09-13T15:11:33.610857+0000","last_peered":"2024-09-13T15:11:33.610857+0000","last_clean":"2024-09-13T15:11:33.610857+0000","last_became_active":"2024-09-13T15:07:34.598343+0000","last_became_peered":"2024-09-13T15:07:34.598343+0000","last_unstale":"2024-09-13T15:11:33.610857+0000","last_undegraded":"2024-09-13T15:11:33.610857+0000","last_fullsized":"2024-09-13T15:11:33.610857+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:31.371179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"59'2","reported_seq":40,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609849+0000","last_change":"2024-09-13T15:11:23.585996+0000","last_active":"2024-09-13T15:11:33.609849+0000","last_peered":"2024-09-13T15:11:33.609849+0000","last_clean":"2024-09-13T15:11:33.609849+0000","last_became_active":"2024-09-13T15:11:23.585520+0000","last_became_peered":"2024-09-13T15:11:23.585520+0000","last_unstale":"2024-09-13T15:11:33.609849+0000","last_undegraded":"2024-09-13T15:11:33.609849+0000","last_fullsized":"2024-09-13T15:11:33.609849+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:38.302040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'100","reported_seq":46,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609663+0000","last_change":"2024-09-13T15:11:23.587475+0000","last_active":"2024-09-13T15:11:33.609663+0000","last_peered":"2024-09-13T15:11:33.609663+0000","last_clean":"2024-09-13T15:11:33.609663+0000","last_became_active":"2024-09-13T15:11:23.587067+0000","last_became_peered":"2024-09-13T15:11:23.587067+0000","last_unstale":"2024-09-13T15:11:33.609663+0000","last_undegraded":"2024-09-13T15:11:33.609663+0000","last_fullsized":"2024-09-13T15:11:33.609663+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:33.277410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:33.277410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:17.742735+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642509+0000","last_change":"2024-09-13T15:11:31.937168+0000","last_active":"2024-09-13T15:11:35.642509+0000","last_peered":"2024-09-13T15:11:35.642509+0000","last_clean":"2024-09-13T15:11:35.642509+0000","last_became_active":"2024-09-13T15:11:31.936814+0000","last_became_peered":"2024-09-13T15:11:31.936814+0000","last_unstale":"2024-09-13T15:11:35.642509+0000","last_undegraded":"2024-09-13T15:11:35.642509+0000","last_fullsized":"2024-09-13T15:11:35.642509+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:11.347185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":600,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642448+0000","last_change":"2024-09-13T15:10:54.563043+0000","last_active":"2024-09-13T15:11:35.642448+0000","last_peered":"2024-09-13T15:11:35.642448+0000","last_clean":"2024-09-13T15:11:35.642448+0000","last_became_active":"2024-09-13T15:10:54.561916+0000","last_became_peered":"2024-09-13T15:10:54.561916+0000","last_unstale":"2024-09-13T15:11:35.642448+0000","last_undegraded":"2024-09-13T15:11:35.642448+0000","last_fullsized":"2024-09-13T15:11:35.642448+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:55.603207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642180+0000","last_change":"2024-09-13T15:10:38.586172+0000","last_active":"2024-09-13T15:11:35.642180+0000","last_peered":"2024-09-13T15:11:35.642180+0000","last_clean":"2024-09-13T15:11:35.642180+0000","last_became_active":"2024-09-13T15:10:38.585821+0000","last_became_peered":"2024-09-13T15:10:38.585821+0000","last_unstale":"2024-09-13T15:11:35.642180+0000","last_undegraded":"2024-09-13T15:11:35.642180+0000","last_fullsized":"2024-09-13T15:11:35.642180+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:49.566977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021839,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'100","reported_seq":304,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624166+0000","last_change":"2024-09-13T15:07:34.464473+0000","last_active":"2024-09-13T15:11:32.624166+0000","last_peered":"2024-09-13T15:11:32.624166+0000","last_clean":"2024-09-13T15:11:32.624166+0000","last_became_active":"2024-09-13T15:07:34.464122+0000","last_became_peered":"2024-09-13T15:07:34.464122+0000","last_unstale":"2024-09-13T15:11:32.624166+0000","last_undegraded":"2024-09-13T15:11:32.624166+0000","last_fullsized":"2024-09-13T15:11:32.624166+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:38.353052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:38.353052+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:55.352605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":306,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224458+0000","last_change":"2024-09-13T15:10:38.589163+0000","last_active":"2024-09-13T15:11:36.224458+0000","last_peered":"2024-09-13T15:11:36.224458+0000","last_clean":"2024-09-13T15:11:36.224458+0000","last_became_active":"2024-09-13T15:10:38.588847+0000","last_became_peered":"2024-09-13T15:10:38.588847+0000","last_unstale":"2024-09-13T15:11:36.224458+0000","last_undegraded":"2024-09-13T15:11:36.224458+0000","last_fullsized":"2024-09-13T15:11:36.224458+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:50.632833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":306,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224070+0000","last_change":"2024-09-13T15:07:34.460652+0000","last_active":"2024-09-13T15:11:36.224070+0000","last_peered":"2024-09-13T15:11:36.224070+0000","last_clean":"2024-09-13T15:11:36.224070+0000","last_became_active":"2024-09-13T15:07:34.458909+0000","last_became_peered":"2024-09-13T15:07:34.458909+0000","last_unstale":"2024-09-13T15:11:36.224070+0000","last_undegraded":"2024-09-13T15:11:36.224070+0000","last_fullsized":"2024-09-13T15:11:36.224070+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:05.444737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":559,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.623977+0000","last_change":"2024-09-13T15:10:33.176495+0000","last_active":"2024-09-13T15:11:32.623977+0000","last_peered":"2024-09-13T15:11:32.623977+0000","last_clean":"2024-09-13T15:11:32.623977+0000","last_became_active":"2024-09-13T15:10:33.176072+0000","last_became_peered":"2024-09-13T15:10:33.176072+0000","last_unstale":"2024-09-13T15:11:32.623977+0000","last_undegraded":"2024-09-13T15:11:32.623977+0000","last_fullsized":"2024-09-13T15:11:32.623977+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:01.922196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000324863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'100","reported_seq":495,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.643056+0000","last_change":"2024-09-13T15:10:54.350647+0000","last_active":"2024-09-13T15:11:35.643056+0000","last_peered":"2024-09-13T15:11:35.643056+0000","last_clean":"2024-09-13T15:11:35.643056+0000","last_became_active":"2024-09-13T15:10:54.350313+0000","last_became_peered":"2024-09-13T15:10:54.350313+0000","last_unstale":"2024-09-13T15:11:35.643056+0000","last_undegraded":"2024-09-13T15:11:35.643056+0000","last_fullsized":"2024-09-13T15:11:35.643056+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:59.729917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:59.729917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:37.114426+0000","scrub_duration":0.0080954670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":472,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016009+0000","last_change":"2024-09-13T15:07:34.462911+0000","last_active":"2024-09-13T15:11:33.016009+0000","last_peered":"2024-09-13T15:11:33.016009+0000","last_clean":"2024-09-13T15:11:33.016009+0000","last_became_active":"2024-09-13T15:07:34.462577+0000","last_became_peered":"2024-09-13T15:07:34.462577+0000","last_unstale":"2024-09-13T15:11:33.016009+0000","last_undegraded":"2024-09-13T15:11:33.016009+0000","last_fullsized":"2024-09-13T15:11:33.016009+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:51.431494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":250,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910491+0000","last_change":"2024-09-13T15:03:46.875410+0000","last_active":"2024-09-13T15:11:36.910491+0000","last_peered":"2024-09-13T15:11:36.910491+0000","last_clean":"2024-09-13T15:11:36.910491+0000","last_became_active":"2024-09-13T15:03:46.875103+0000","last_became_peered":"2024-09-13T15:03:46.875103+0000","last_unstale":"2024-09-13T15:11:36.910491+0000","last_undegraded":"2024-09-13T15:11:36.910491+0000","last_fullsized":"2024-09-13T15:11:36.910491+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:53.941854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'1","reported_seq":220,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610231+0000","last_change":"2024-09-13T15:10:38.605256+0000","last_active":"2024-09-13T15:11:33.610231+0000","last_peered":"2024-09-13T15:11:33.610231+0000","last_clean":"2024-09-13T15:11:33.610231+0000","last_became_active":"2024-09-13T15:10:38.596513+0000","last_became_peered":"2024-09-13T15:10:38.596513+0000","last_unstale":"2024-09-13T15:11:33.610231+0000","last_undegraded":"2024-09-13T15:11:33.610231+0000","last_fullsized":"2024-09-13T15:11:33.610231+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:36.342650+0000","last_clean_scrub_stamp":"2024-09-13T14:20:36.342650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:30.444160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'100","reported_seq":284,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.623794+0000","last_change":"2024-09-13T15:02:19.407022+0000","last_active":"2024-09-13T15:11:32.623794+0000","last_peered":"2024-09-13T15:11:32.623794+0000","last_clean":"2024-09-13T15:11:32.623794+0000","last_became_active":"2024-09-13T15:02:19.406620+0000","last_became_peered":"2024-09-13T15:02:19.406620+0000","last_unstale":"2024-09-13T15:11:32.623794+0000","last_undegraded":"2024-09-13T15:11:32.623794+0000","last_fullsized":"2024-09-13T15:11:32.623794+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:43.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:43.363650+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:37.100188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":56,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.643309+0000","last_change":"2024-09-13T15:10:33.178505+0000","last_active":"2024-09-13T15:11:35.643309+0000","last_peered":"2024-09-13T15:11:35.643309+0000","last_clean":"2024-09-13T15:11:35.643309+0000","last_became_active":"2024-09-13T15:10:33.176902+0000","last_became_peered":"2024-09-13T15:10:33.176902+0000","last_unstale":"2024-09-13T15:11:35.643309+0000","last_undegraded":"2024-09-13T15:11:35.643309+0000","last_fullsized":"2024-09-13T15:11:35.643309+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:23.999225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":270,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910275+0000","last_change":"2024-09-13T15:10:33.186515+0000","last_active":"2024-09-13T15:11:36.910275+0000","last_peered":"2024-09-13T15:11:36.910275+0000","last_clean":"2024-09-13T15:11:36.910275+0000","last_became_active":"2024-09-13T15:10:33.185795+0000","last_became_peered":"2024-09-13T15:10:33.185795+0000","last_unstale":"2024-09-13T15:11:36.910275+0000","last_undegraded":"2024-09-13T15:11:36.910275+0000","last_fullsized":"2024-09-13T15:11:36.910275+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:57.771790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"79'100","reported_seq":36,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610266+0000","last_change":"2024-09-13T15:11:23.586471+0000","last_active":"2024-09-13T15:11:33.610266+0000","last_peered":"2024-09-13T15:11:33.610266+0000","last_clean":"2024-09-13T15:11:33.610266+0000","last_became_active":"2024-09-13T15:11:23.585360+0000","last_became_peered":"2024-09-13T15:11:23.585360+0000","last_unstale":"2024-09-13T15:11:33.610266+0000","last_undegraded":"2024-09-13T15:11:33.610266+0000","last_fullsized":"2024-09-13T15:11:33.610266+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:10:38.865945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:10:38.865945+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:09.360128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610734+0000","last_change":"2024-09-13T15:10:33.186094+0000","last_active":"2024-09-13T15:11:33.610734+0000","last_peered":"2024-09-13T15:11:33.610734+0000","last_clean":"2024-09-13T15:11:33.610734+0000","last_became_active":"2024-09-13T15:10:33.185738+0000","last_became_peered":"2024-09-13T15:10:33.185738+0000","last_unstale":"2024-09-13T15:11:33.610734+0000","last_undegraded":"2024-09-13T15:11:33.610734+0000","last_fullsized":"2024-09-13T15:11:33.610734+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:36.389856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":111,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609849+0000","last_change":"2024-09-13T15:10:38.596092+0000","last_active":"2024-09-13T15:11:33.609849+0000","last_peered":"2024-09-13T15:11:33.609849+0000","last_clean":"2024-09-13T15:11:33.609849+0000","last_became_active":"2024-09-13T15:10:38.595779+0000","last_became_peered":"2024-09-13T15:10:38.595779+0000","last_unstale":"2024-09-13T15:11:33.609849+0000","last_undegraded":"2024-09-13T15:11:33.609849+0000","last_fullsized":"2024-09-13T15:11:33.609849+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:50.275843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"79'100","reported_seq":304,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642996+0000","last_change":"2024-09-13T15:10:54.348913+0000","last_active":"2024-09-13T15:11:35.642996+0000","last_peered":"2024-09-13T15:11:35.642996+0000","last_clean":"2024-09-13T15:11:35.642996+0000","last_became_active":"2024-09-13T15:10:54.348545+0000","last_became_peered":"2024-09-13T15:10:54.348545+0000","last_unstale":"2024-09-13T15:11:35.642996+0000","last_undegraded":"2024-09-13T15:11:35.642996+0000","last_fullsized":"2024-09-13T15:11:35.642996+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:02:36.128767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T15:02:36.128767+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:10.554097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":111,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.611029+0000","last_change":"2024-09-13T15:10:38.597354+0000","last_active":"2024-09-13T15:11:33.611029+0000","last_peered":"2024-09-13T15:11:33.611029+0000","last_clean":"2024-09-13T15:11:33.611029+0000","last_became_active":"2024-09-13T15:10:38.597069+0000","last_became_peered":"2024-09-13T15:10:38.597069+0000","last_unstale":"2024-09-13T15:11:33.611029+0000","last_undegraded":"2024-09-13T15:11:33.611029+0000","last_fullsized":"2024-09-13T15:11:33.611029+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:58:16.938993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":530,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.015740+0000","last_change":"2024-09-13T15:10:33.179554+0000","last_active":"2024-09-13T15:11:33.015740+0000","last_peered":"2024-09-13T15:11:33.015740+0000","last_clean":"2024-09-13T15:11:33.015740+0000","last_became_active":"2024-09-13T15:10:33.179245+0000","last_became_peered":"2024-09-13T15:10:33.179245+0000","last_unstale":"2024-09-13T15:11:33.015740+0000","last_undegraded":"2024-09-13T15:11:33.015740+0000","last_fullsized":"2024-09-13T15:11:33.015740+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:13.309940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"79'100","reported_seq":36,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609721+0000","last_change":"2024-09-13T15:11:23.587205+0000","last_active":"2024-09-13T15:11:33.609721+0000","last_peered":"2024-09-13T15:11:33.609721+0000","last_clean":"2024-09-13T15:11:33.609721+0000","last_became_active":"2024-09-13T15:11:23.586754+0000","last_became_peered":"2024-09-13T15:11:23.586754+0000","last_unstale":"2024-09-13T15:11:33.609721+0000","last_undegraded":"2024-09-13T15:11:33.609721+0000","last_fullsized":"2024-09-13T15:11:33.609721+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.634401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.634401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:53.982822+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":81,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610479+0000","last_change":"2024-09-13T15:07:34.473797+0000","last_active":"2024-09-13T15:11:33.610479+0000","last_peered":"2024-09-13T15:11:33.610479+0000","last_clean":"2024-09-13T15:11:33.610479+0000","last_became_active":"2024-09-13T15:07:34.473444+0000","last_became_peered":"2024-09-13T15:07:34.473444+0000","last_unstale":"2024-09-13T15:11:33.610479+0000","last_undegraded":"2024-09-13T15:11:33.610479+0000","last_fullsized":"2024-09-13T15:11:33.610479+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:04.278156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"65'2","reported_seq":286,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910576+0000","last_change":"2024-09-13T15:07:34.609901+0000","last_active":"2024-09-13T15:11:36.910576+0000","last_peered":"2024-09-13T15:11:36.910576+0000","last_clean":"2024-09-13T15:11:36.910576+0000","last_became_active":"2024-09-13T15:07:34.490242+0000","last_became_peered":"2024-09-13T15:07:34.490242+0000","last_unstale":"2024-09-13T15:11:36.910576+0000","last_undegraded":"2024-09-13T15:11:36.910576+0000","last_fullsized":"2024-09-13T15:11:36.910576+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:23.303360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'100","reported_seq":350,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.623796+0000","last_change":"2024-09-13T15:10:38.596913+0000","last_active":"2024-09-13T15:11:32.623796+0000","last_peered":"2024-09-13T15:11:32.623796+0000","last_clean":"2024-09-13T15:11:32.623796+0000","last_became_active":"2024-09-13T15:10:38.596491+0000","last_became_peered":"2024-09-13T15:10:38.596491+0000","last_unstale":"2024-09-13T15:11:32.623796+0000","last_undegraded":"2024-09-13T15:11:32.623796+0000","last_fullsized":"2024-09-13T15:11:32.623796+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:57.223620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:57.223620+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:55.706139+0000","scrub_duration":0.0051153759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":313,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224414+0000","last_change":"2024-09-13T15:11:31.878256+0000","last_active":"2024-09-13T15:11:36.224414+0000","last_peered":"2024-09-13T15:11:36.224414+0000","last_clean":"2024-09-13T15:11:36.224414+0000","last_became_active":"2024-09-13T15:11:31.877964+0000","last_became_peered":"2024-09-13T15:11:31.877964+0000","last_unstale":"2024-09-13T15:11:36.224414+0000","last_undegraded":"2024-09-13T15:11:36.224414+0000","last_fullsized":"2024-09-13T15:11:36.224414+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:01.878232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":132,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610471+0000","last_change":"2024-09-13T15:11:23.586387+0000","last_active":"2024-09-13T15:11:33.610471+0000","last_peered":"2024-09-13T15:11:33.610471+0000","last_clean":"2024-09-13T15:11:33.610471+0000","last_became_active":"2024-09-13T15:11:23.586071+0000","last_became_peered":"2024-09-13T15:11:23.586071+0000","last_unstale":"2024-09-13T15:11:33.610471+0000","last_undegraded":"2024-09-13T15:11:33.610471+0000","last_fullsized":"2024-09-13T15:11:33.610471+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:48.803141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'100","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610804+0000","last_change":"2024-09-13T15:08:59.871028+0000","last_active":"2024-09-13T15:11:33.610804+0000","last_peered":"2024-09-13T15:11:33.610804+0000","last_clean":"2024-09-13T15:11:33.610804+0000","last_became_active":"2024-09-13T15:07:34.478006+0000","last_became_peered":"2024-09-13T15:07:34.478006+0000","last_unstale":"2024-09-13T15:11:33.610804+0000","last_undegraded":"2024-09-13T15:11:33.610804+0000","last_fullsized":"2024-09-13T15:11:33.610804+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T15:08:59.870786+0000","last_deep_scrub":"79'100","last_deep_scrub_stamp":"2024-09-13T15:08:59.870786+0000","last_clean_scrub_stamp":"2024-09-13T15:08:59.870786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:43:55.603202+0000","scrub_duration":0.010767947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":512,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.015600+0000","last_change":"2024-09-13T15:10:33.177131+0000","last_active":"2024-09-13T15:11:33.015600+0000","last_peered":"2024-09-13T15:11:33.015600+0000","last_clean":"2024-09-13T15:11:33.015600+0000","last_became_active":"2024-09-13T15:10:33.176765+0000","last_became_peered":"2024-09-13T15:10:33.176765+0000","last_unstale":"2024-09-13T15:11:33.015600+0000","last_undegraded":"2024-09-13T15:11:33.015600+0000","last_fullsized":"2024-09-13T15:11:33.015600+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:53.991570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"65'1","reported_seq":511,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624392+0000","last_change":"2024-09-13T14:57:53.363137+0000","last_active":"2024-09-13T15:11:32.624392+0000","last_peered":"2024-09-13T15:11:32.624392+0000","last_clean":"2024-09-13T15:11:32.624392+0000","last_became_active":"2024-09-13T14:57:53.355432+0000","last_became_peered":"2024-09-13T14:57:53.355432+0000","last_unstale":"2024-09-13T15:11:32.624392+0000","last_undegraded":"2024-09-13T15:11:32.624392+0000","last_fullsized":"2024-09-13T15:11:32.624392+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:01.826907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"79'100","reported_seq":483,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.016085+0000","last_change":"2024-09-13T15:07:34.463017+0000","last_active":"2024-09-13T15:11:33.016085+0000","last_peered":"2024-09-13T15:11:33.016085+0000","last_clean":"2024-09-13T15:11:33.016085+0000","last_became_active":"2024-09-13T15:07:34.462570+0000","last_became_peered":"2024-09-13T15:07:34.462570+0000","last_unstale":"2024-09-13T15:11:33.016085+0000","last_undegraded":"2024-09-13T15:11:33.016085+0000","last_fullsized":"2024-09-13T15:11:33.016085+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:47:48.848161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:47:48.848161+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:56.696358+0000","scrub_duration":0.0017193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":91,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609956+0000","last_change":"2024-09-13T15:10:33.190304+0000","last_active":"2024-09-13T15:11:33.609956+0000","last_peered":"2024-09-13T15:11:33.609956+0000","last_clean":"2024-09-13T15:11:33.609956+0000","last_became_active":"2024-09-13T15:10:33.187803+0000","last_became_peered":"2024-09-13T15:10:33.187803+0000","last_unstale":"2024-09-13T15:11:33.609956+0000","last_undegraded":"2024-09-13T15:11:33.609956+0000","last_fullsized":"2024-09-13T15:11:33.609956+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:21.100709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'2","reported_seq":529,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624514+0000","last_change":"2024-09-13T15:10:33.186232+0000","last_active":"2024-09-13T15:11:32.624514+0000","last_peered":"2024-09-13T15:11:32.624514+0000","last_clean":"2024-09-13T15:11:32.624514+0000","last_became_active":"2024-09-13T15:10:33.173156+0000","last_became_peered":"2024-09-13T15:10:33.173156+0000","last_unstale":"2024-09-13T15:11:32.624514+0000","last_undegraded":"2024-09-13T15:11:32.624514+0000","last_fullsized":"2024-09-13T15:11:32.624514+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:25.582656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'100","reported_seq":253,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910733+0000","last_change":"2024-09-13T15:03:46.882026+0000","last_active":"2024-09-13T15:11:36.910733+0000","last_peered":"2024-09-13T15:11:36.910733+0000","last_clean":"2024-09-13T15:11:36.910733+0000","last_became_active":"2024-09-13T15:03:46.859557+0000","last_became_peered":"2024-09-13T15:03:46.859557+0000","last_unstale":"2024-09-13T15:11:36.910733+0000","last_undegraded":"2024-09-13T15:11:36.910733+0000","last_fullsized":"2024-09-13T15:11:36.910733+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:41.568692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.568692+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:00.933755+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":188,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609716+0000","last_change":"2024-09-13T14:57:53.358287+0000","last_active":"2024-09-13T15:11:33.609716+0000","last_peered":"2024-09-13T15:11:33.609716+0000","last_clean":"2024-09-13T15:11:33.609716+0000","last_became_active":"2024-09-13T14:57:53.357199+0000","last_became_peered":"2024-09-13T14:57:53.357199+0000","last_unstale":"2024-09-13T15:11:33.609716+0000","last_undegraded":"2024-09-13T15:11:33.609716+0000","last_fullsized":"2024-09-13T15:11:33.609716+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:41.970012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'1","reported_seq":338,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624526+0000","last_change":"2024-09-13T15:07:34.635697+0000","last_active":"2024-09-13T15:11:32.624526+0000","last_peered":"2024-09-13T15:11:32.624526+0000","last_clean":"2024-09-13T15:11:32.624526+0000","last_became_active":"2024-09-13T15:07:34.568238+0000","last_became_peered":"2024-09-13T15:07:34.568238+0000","last_unstale":"2024-09-13T15:11:32.624526+0000","last_undegraded":"2024-09-13T15:11:32.624526+0000","last_fullsized":"2024-09-13T15:11:32.624526+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:55.995331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"79'100","reported_seq":344,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624253+0000","last_change":"2024-09-13T15:10:33.181566+0000","last_active":"2024-09-13T15:11:32.624253+0000","last_peered":"2024-09-13T15:11:32.624253+0000","last_clean":"2024-09-13T15:11:32.624253+0000","last_became_active":"2024-09-13T15:10:33.181146+0000","last_became_peered":"2024-09-13T15:10:33.181146+0000","last_unstale":"2024-09-13T15:11:32.624253+0000","last_undegraded":"2024-09-13T15:11:32.624253+0000","last_fullsized":"2024-09-13T15:11:32.624253+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:57:55.207301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:57:55.207301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:06.002058+0000","scrub_duration":0.0028773779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"65'8","reported_seq":598,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.015689+0000","last_change":"2024-09-13T15:02:19.426511+0000","last_active":"2024-09-13T15:11:33.015689+0000","last_peered":"2024-09-13T15:11:33.015689+0000","last_clean":"2024-09-13T15:11:33.015689+0000","last_became_active":"2024-09-13T15:02:19.391448+0000","last_became_peered":"2024-09-13T15:02:19.391448+0000","last_unstale":"2024-09-13T15:11:33.015689+0000","last_undegraded":"2024-09-13T15:11:33.015689+0000","last_fullsized":"2024-09-13T15:11:33.015689+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:40.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":263,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.911084+0000","last_change":"2024-09-13T15:10:33.185456+0000","last_active":"2024-09-13T15:11:36.911084+0000","last_peered":"2024-09-13T15:11:36.911084+0000","last_clean":"2024-09-13T15:11:36.911084+0000","last_became_active":"2024-09-13T15:10:33.185108+0000","last_became_peered":"2024-09-13T15:10:33.185108+0000","last_unstale":"2024-09-13T15:11:36.911084+0000","last_undegraded":"2024-09-13T15:11:36.911084+0000","last_fullsized":"2024-09-13T15:11:36.911084+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:44.754388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"79'100","reported_seq":208,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610093+0000","last_change":"2024-09-13T15:10:33.185313+0000","last_active":"2024-09-13T15:11:33.610093+0000","last_peered":"2024-09-13T15:11:33.610093+0000","last_clean":"2024-09-13T15:11:33.610093+0000","last_became_active":"2024-09-13T15:10:33.184950+0000","last_became_peered":"2024-09-13T15:10:33.184950+0000","last_unstale":"2024-09-13T15:11:33.610093+0000","last_undegraded":"2024-09-13T15:11:33.610093+0000","last_fullsized":"2024-09-13T15:11:33.610093+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:39.383015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:39.383015+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:05.151863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":494,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624016+0000","last_change":"2024-09-13T15:07:34.466871+0000","last_active":"2024-09-13T15:11:32.624016+0000","last_peered":"2024-09-13T15:11:32.624016+0000","last_clean":"2024-09-13T15:11:32.624016+0000","last_became_active":"2024-09-13T15:07:34.466559+0000","last_became_peered":"2024-09-13T15:07:34.466559+0000","last_unstale":"2024-09-13T15:11:32.624016+0000","last_undegraded":"2024-09-13T15:11:32.624016+0000","last_fullsized":"2024-09-13T15:11:32.624016+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:48:24.377353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":292,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224604+0000","last_change":"2024-09-13T15:03:46.864291+0000","last_active":"2024-09-13T15:11:36.224604+0000","last_peered":"2024-09-13T15:11:36.224604+0000","last_clean":"2024-09-13T15:11:36.224604+0000","last_became_active":"2024-09-13T15:03:46.863671+0000","last_became_peered":"2024-09-13T15:03:46.863671+0000","last_unstale":"2024-09-13T15:11:36.224604+0000","last_undegraded":"2024-09-13T15:11:36.224604+0000","last_fullsized":"2024-09-13T15:11:36.224604+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:28.672041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"79'100","reported_seq":265,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:35.642886+0000","last_change":"2024-09-13T15:10:54.351461+0000","last_active":"2024-09-13T15:11:35.642886+0000","last_peered":"2024-09-13T15:11:35.642886+0000","last_clean":"2024-09-13T15:11:35.642886+0000","last_became_active":"2024-09-13T15:10:54.351127+0000","last_became_peered":"2024-09-13T15:10:54.351127+0000","last_unstale":"2024-09-13T15:11:35.642886+0000","last_undegraded":"2024-09-13T15:11:35.642886+0000","last_fullsized":"2024-09-13T15:11:35.642886+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:42.355127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:42.355127+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:37.993717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":292,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224294+0000","last_change":"2024-09-13T15:03:46.863848+0000","last_active":"2024-09-13T15:11:36.224294+0000","last_peered":"2024-09-13T15:11:36.224294+0000","last_clean":"2024-09-13T15:11:36.224294+0000","last_became_active":"2024-09-13T15:03:46.863523+0000","last_became_peered":"2024-09-13T15:03:46.863523+0000","last_unstale":"2024-09-13T15:11:36.224294+0000","last_undegraded":"2024-09-13T15:11:36.224294+0000","last_fullsized":"2024-09-13T15:11:36.224294+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:39.303427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":278,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.610314+0000","last_change":"2024-09-13T15:11:23.586832+0000","last_active":"2024-09-13T15:11:33.610314+0000","last_peered":"2024-09-13T15:11:33.610314+0000","last_clean":"2024-09-13T15:11:33.610314+0000","last_became_active":"2024-09-13T15:11:23.586420+0000","last_became_peered":"2024-09-13T15:11:23.586420+0000","last_unstale":"2024-09-13T15:11:33.610314+0000","last_undegraded":"2024-09-13T15:11:33.610314+0000","last_fullsized":"2024-09-13T15:11:33.610314+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:13.491913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"79'100","reported_seq":36,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:33.609615+0000","last_change":"2024-09-13T15:11:23.586214+0000","last_active":"2024-09-13T15:11:33.609615+0000","last_peered":"2024-09-13T15:11:33.609615+0000","last_clean":"2024-09-13T15:11:33.609615+0000","last_became_active":"2024-09-13T15:11:23.584762+0000","last_became_peered":"2024-09-13T15:11:23.584762+0000","last_unstale":"2024-09-13T15:11:33.609615+0000","last_undegraded":"2024-09-13T15:11:33.609615+0000","last_fullsized":"2024-09-13T15:11:33.609615+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:32.271263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:32.271263+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:38.709689+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'2","reported_seq":553,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-13T15:11:32.624137+0000","last_change":"2024-09-13T15:10:38.606150+0000","last_active":"2024-09-13T15:11:32.624137+0000","last_peered":"2024-09-13T15:11:32.624137+0000","last_clean":"2024-09-13T15:11:32.624137+0000","last_became_active":"2024-09-13T15:10:38.598310+0000","last_became_peered":"2024-09-13T15:10:38.598310+0000","last_unstale":"2024-09-13T15:11:32.624137+0000","last_undegraded":"2024-09-13T15:11:32.624137+0000","last_fullsized":"2024-09-13T15:11:32.624137+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:51.591244+0000","last_clean_scrub_stamp":"2024-09-13T14:21:51.591244+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:33.450888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":170,"num_keys_recovered":50,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":270,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.910783+0000","last_change":"2024-09-13T15:10:33.188502+0000","last_active":"2024-09-13T15:11:36.910783+0000","last_peered":"2024-09-13T15:11:36.910783+0000","last_clean":"2024-09-13T15:11:36.910783+0000","last_became_active":"2024-09-13T15:10:33.187700+0000","last_became_peered":"2024-09-13T15:10:33.187700+0000","last_unstale":"2024-09-13T15:11:36.910783+0000","last_undegraded":"2024-09-13T15:11:36.910783+0000","last_fullsized":"2024-09-13T15:11:36.910783+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:21:52.595660+0000","last_clean_scrub_stamp":"2024-09-13T14:21:52.595660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:15.107002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"79'100","reported_seq":292,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-13T15:11:36.224423+0000","last_change":"2024-09-13T15:03:46.864604+0000","last_active":"2024-09-13T15:11:36.224423+0000","last_peered":"2024-09-13T15:11:36.224423+0000","last_clean":"2024-09-13T15:11:36.224423+0000","last_became_active":"2024-09-13T15:03:46.864085+0000","last_became_peered":"2024-09-13T15:03:46.864085+0000","last_unstale":"2024-09-13T15:11:36.224423+0000","last_undegraded":"2024-09-13T15:11:36.224423+0000","last_fullsized":"2024-09-13T15:11:36.224423+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":213,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'100","last_scrub_stamp":"2024-09-13T14:46:36.384576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:03.133314+0000","last_clean_scrub_stamp":"2024-09-13T14:46:36.384576+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:32.524052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7486,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13714,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365433,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1520584,"kb_used_data":7100,"kb_used_omap":1,"kb_used_meta":1513470,"kb_avail":92224568,"statfs":{"total":95995035648,"available":94437957632,"internally_reserved":0,"allocated":7270400,"data_stored":4309878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1549793624},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561663,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499388,"kb_used_data":7152,"kb_used_omap":1,"kb_used_meta":1492222,"kb_avail":92245764,"statfs":{"total":95995035648,"available":94459662336,"internally_reserved":0,"allocated":7323648,"data_stored":4374693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1528035672},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790595,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497652,"kb_used_data":7080,"kb_used_omap":1,"kb_used_meta":1490558,"kb_avail":92247500,"statfs":{"total":95995035648,"available":94461440000,"internally_reserved":0,"allocated":7249920,"data_stored":4305275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1526331736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986823,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509128,"kb_used_data":6840,"kb_used_omap":1,"kb_used_meta":1502270,"kb_avail":92236024,"statfs":{"total":95995035648,"available":94449688576,"internally_reserved":0,"allocated":7004160,"data_stored":4047385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1538324824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":137,"seq":588410519978,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267120,"kb_used_data":7140,"kb_used_omap":4,"kb_used_meta":1259963,"kb_avail":92478032,"statfs":{"total":95995035648,"available":94697504768,"internally_reserved":0,"allocated":7311360,"data_stored":4370277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4672,"internal_metadata":1290202560},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":113,"seq":485331304894,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1265276,"kb_used_data":7084,"kb_used_omap":6,"kb_used_meta":1258169,"kb_avail":92479876,"statfs":{"total":95995035648,"available":94699393024,"internally_reserved":0,"allocated":7254016,"data_stored":4305799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6713,"internal_metadata":1288365511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828197,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1423152,"kb_used_data":6820,"kb_used_omap":5,"kb_used_meta":1416314,"kb_avail":92322000,"statfs":{"total":95995035648,"available":94537728000,"internally_reserved":0,"allocated":6983680,"data_stored":4042534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5908,"internal_metadata":1450305772},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":83,"seq":356482286062,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1256496,"kb_used_data":7140,"kb_used_omap":5,"kb_used_meta":1249338,"kb_avail":92488656,"statfs":{"total":95995035648,"available":94708383744,"internally_reserved":0,"allocated":7311360,"data_stored":4369551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6085,"internal_metadata":1279322171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4381,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4204,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2968,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:12:26.874 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T15:12:26.874 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-13T15:12:27.093 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=47d64360dfd044f687078225323e4dec Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-13T15:12:27.096 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-13T15:12:27.106 INFO:tasks.cephadm:Teardown begin 2024-09-13T15:12:27.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:12:27.170 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:12:27.213 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-13T15:12:27.213 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:12:27.231 DEBUG:teuthology.orchestra.run.smithi019:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:12:27.272 INFO:tasks.cephadm:Stopping all daemons... 2024-09-13T15:12:27.272 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-13T15:12:27.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a 2024-09-13T15:12:27.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:27 smithi003 systemd[1]: Stopping Ceph mon.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:12:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:27 smithi019 ceph-mon[91296]: from='client.37749 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:12:27.682 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:27 smithi019 ceph-mon[91296]: pgmap v1781: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:27.716 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:27 smithi003 ceph-mon[107278]: from='client.37749 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:12:27.717 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:27 smithi003 ceph-mon[107278]: pgmap v1781: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:27.717 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:27 smithi003 ceph-mon[104007]: from='client.37749 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:12:27.717 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:27 smithi003 ceph-mon[104007]: pgmap v1781: 103 pgs: 103 active+clean; 582 KiB data, 11 GiB used, 705 GiB / 715 GiB avail 2024-09-13T15:12:28.073 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:15:12:27] ENGINE Bus STOPPING 2024-09-13T15:12:28.073 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:12:27.713+0000 7fdb10bd5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:12:28.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:27 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a[103984]: 2024-09-13T15:12:27.713+0000 7fdb10bd5640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-13T15:12:28.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:27 smithi003 podman[372746]: 2024-09-13 15:12:27.765498561 +0000 UTC m=+0.180378040 container died 265013e6136c03d49868cb6aaa0cd49f1ea1b7e16800a371517e0d910d676bca (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=) 2024-09-13T15:12:28.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:28 smithi003 podman[372746]: 2024-09-13 15:12:28.254155795 +0000 UTC m=+0.669035317 container cleanup 265013e6136c03d49868cb6aaa0cd49f1ea1b7e16800a371517e0d910d676bca (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:12:28.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 13 15:12:28 smithi003 bash[372746]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-a 2024-09-13T15:12:28.434 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:15:12:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T15:12:28.434 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:15:12:28] ENGINE Bus STOPPED 2024-09-13T15:12:28.434 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:15:12:28] ENGINE Bus STARTING 2024-09-13T15:12:28.434 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:15:12:28] ENGINE Serving on http://:::9283 2024-09-13T15:12:28.435 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:28 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: [13/Sep/2024:15:12:28] ENGINE Bus STARTED 2024-09-13T15:12:28.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.a.service' 2024-09-13T15:12:28.803 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:12:28.804 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-13T15:12:28.804 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-13T15:12:28.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.c 2024-09-13T15:12:29.112 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:28 smithi003 systemd[1]: Stopping Ceph mon.c for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:12:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c[107257]: 2024-09-13T15:12:29.110+0000 7fc612e9e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:12:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:29 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c[107257]: 2024-09-13T15:12:29.110+0000 7fc612e9e640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-13T15:12:29.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:29 smithi003 podman[372920]: 2024-09-13 15:12:29.152069881 +0000 UTC m=+0.166910703 container died 8539c77bb8fb82cfe0bf98a107e5865f14820f407d5498e59675b8c33498d798 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-13T15:12:29.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:29 smithi003 podman[372920]: 2024-09-13 15:12:29.575492873 +0000 UTC m=+0.590333683 container cleanup 8539c77bb8fb82cfe0bf98a107e5865f14820f407d5498e59675b8c33498d798 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:12:29.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:29 smithi003 bash[372920]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c 2024-09-13T15:12:30.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Sep 13 15:12:29 smithi003 podman[372937]: 2024-09-13 15:12:29.822935438 +0000 UTC m=+0.669383176 container remove 8539c77bb8fb82cfe0bf98a107e5865f14820f407d5498e59675b8c33498d798 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-13T15:12:30.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.c.service' 2024-09-13T15:12:30.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:12:30.127 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-13T15:12:30.127 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-13T15:12:30.128 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.b 2024-09-13T15:12:30.432 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:30 smithi019 systemd[1]: Stopping Ceph mon.b for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:12:30.433 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"77a3f016-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:12:30.898 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b[91272]: 2024-09-13T15:12:30.462+0000 7f621707a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:12:30.898 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b[91272]: 2024-09-13T15:12:30.462+0000 7f621707a640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-13T15:12:30.899 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:30 smithi019 podman[243866]: 2024-09-13 15:12:30.586020195 +0000 UTC m=+0.247474417 container died 301a92c4f814e9b5af3e5b323e79f4dcfda970d2191a32a6e53ad69098662246 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T15:12:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:31 smithi019 podman[243866]: 2024-09-13 15:12:31.061468244 +0000 UTC m=+0.722922501 container cleanup 301a92c4f814e9b5af3e5b323e79f4dcfda970d2191a32a6e53ad69098662246 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-13T15:12:31.182 INFO:journalctl@ceph.mon.b.smithi019.stdout:Sep 13 15:12:31 smithi019 bash[243866]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mon-b 2024-09-13T15:12:31.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mon.b.service' 2024-09-13T15:12:31.548 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:12:31.549 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-13T15:12:31.549 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-13T15:12:31.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y 2024-09-13T15:12:31.842 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:31 smithi003 systemd[1]: Stopping Ceph mgr.y for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:12:32.120 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T15:12:31.839+0000 7f0fdc6c0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:12:32.120 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:31 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y[94389]: 2024-09-13T15:12:31.839+0000 7f0fdc6c0640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-13T15:12:32.120 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:31 smithi003 podman[373052]: 2024-09-13 15:12:31.882755324 +0000 UTC m=+0.139072140 container died e8e7986695ad32198064f64e4dc22a78705e99a8631ebb404845d6b5a373358d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:12:32.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:32 smithi003 podman[373052]: 2024-09-13 15:12:32.280533767 +0000 UTC m=+0.536850575 container cleanup e8e7986695ad32198064f64e4dc22a78705e99a8631ebb404845d6b5a373358d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T15:12:32.433 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Sep 13 15:12:32 smithi003 bash[373052]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-y 2024-09-13T15:12:32.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-13T15:12:32.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:12:32.799 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-13T15:12:32.799 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-13T15:12:32.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x 2024-09-13T15:12:33.175 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 15:12:32 smithi019 systemd[1]: Stopping Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:12:33.432 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 15:12:33 smithi019 podman[243998]: 2024-09-13 15:12:33.174243174 +0000 UTC m=+0.167702918 container died f0b9ab84e7f361356fa5fe2e63d65f9e2d9d29176535f9aa00e50077e3e47fe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-13T15:12:33.860 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 15:12:33 smithi019 podman[243998]: 2024-09-13 15:12:33.607914967 +0000 UTC m=+0.601374708 container cleanup f0b9ab84e7f361356fa5fe2e63d65f9e2d9d29176535f9aa00e50077e3e47fe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:12:33.861 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 15:12:33 smithi019 bash[243998]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x 2024-09-13T15:12:34.089 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-13T15:12:34.111 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 15:12:33 smithi019 podman[244017]: 2024-09-13 15:12:33.85948914 +0000 UTC m=+0.674961446 container remove f0b9ab84e7f361356fa5fe2e63d65f9e2d9d29176535f9aa00e50077e3e47fe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:12:34.111 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 15:12:33 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:12:34.111 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 15:12:34 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-13T15:12:34.112 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 15:12:34 smithi019 systemd[1]: Stopped Ceph mgr.x for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T15:12:34.112 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Sep 13 15:12:34 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@mgr.x.service: Consumed 17.784s CPU time. 2024-09-13T15:12:34.125 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:12:34.125 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-13T15:12:34.125 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-13T15:12:34.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.0 2024-09-13T15:12:34.433 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:34 smithi003 systemd[1]: Stopping Ceph osd.0 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:12:34.933 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[116698]: 2024-09-13T15:12:34.436+0000 7f9dc6424640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:12:34.933 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[116698]: 2024-09-13T15:12:34.437+0000 7f9dc6424640 -1 osd.0 507 *** Got signal Terminated *** 2024-09-13T15:12:34.933 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:34 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0[116698]: 2024-09-13T15:12:34.437+0000 7f9dc6424640 -1 osd.0 507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:12:40.238 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:39 smithi003 podman[373194]: 2024-09-13 15:12:39.940647621 +0000 UTC m=+5.617715836 container died 8e2479752fc34e051b9551a21a573d547f13b87644b6a02be72a3e2f04ac8d79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:12:40.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:40 smithi003 podman[373194]: 2024-09-13 15:12:40.43496613 +0000 UTC m=+6.112034345 container cleanup 8e2479752fc34e051b9551a21a573d547f13b87644b6a02be72a3e2f04ac8d79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-13T15:12:40.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:40 smithi003 bash[373194]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0 2024-09-13T15:12:41.050 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:40 smithi003 podman[373214]: 2024-09-13 15:12:40.728656799 +0000 UTC m=+0.776118016 container remove 8e2479752fc34e051b9551a21a573d547f13b87644b6a02be72a3e2f04ac8d79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:12:41.308 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:41 smithi003 podman[373305]: 2024-09-13 15:12:40.952614449 +0000 UTC m=+0.016497924 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:12:41.308 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:41 smithi003 podman[373305]: 2024-09-13 15:12:41.054413994 +0000 UTC m=+0.118297456 container create d6c5a55bb655a87efa82ea74c67775679cf8f98c243b70765a4e87615d794e84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-13T15:12:41.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:41 smithi003 podman[373305]: 2024-09-13 15:12:41.30712748 +0000 UTC m=+0.371010945 container init d6c5a55bb655a87efa82ea74c67775679cf8f98c243b70765a4e87615d794e84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-13T15:12:41.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:41 smithi003 podman[373305]: 2024-09-13 15:12:41.312071137 +0000 UTC m=+0.375954618 container start d6c5a55bb655a87efa82ea74c67775679cf8f98c243b70765a4e87615d794e84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T15:12:41.683 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:41 smithi003 podman[373305]: 2024-09-13 15:12:41.436180992 +0000 UTC m=+0.500064467 container attach d6c5a55bb655a87efa82ea74c67775679cf8f98c243b70765a4e87615d794e84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-13T15:12:42.603 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 13 15:12:42 smithi003 podman[373305]: 2024-09-13 15:12:42.294944519 +0000 UTC m=+1.358827990 container died d6c5a55bb655a87efa82ea74c67775679cf8f98c243b70765a4e87615d794e84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=) 2024-09-13T15:12:43.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.0.service' 2024-09-13T15:12:43.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:12:43.066 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-13T15:12:43.066 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-13T15:12:43.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.1 2024-09-13T15:12:43.417 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:43 smithi003 systemd[1]: Stopping Ceph osd.1 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:12:43.683 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:43 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[123268]: 2024-09-13T15:12:43.415+0000 7f59bd63a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:12:43.683 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:43 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[123268]: 2024-09-13T15:12:43.415+0000 7f59bd63a640 -1 osd.1 507 *** Got signal Terminated *** 2024-09-13T15:12:43.684 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:43 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1[123268]: 2024-09-13T15:12:43.415+0000 7f59bd63a640 -1 osd.1 507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:12:49.308 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:49 smithi003 podman[373435]: 2024-09-13 15:12:49.045414165 +0000 UTC m=+5.738786874 container died 3126fff5816571e7e11301859cc4536af2765da1cf451a39bf3254ac08c5c207 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-13T15:12:49.683 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:49 smithi003 podman[373435]: 2024-09-13 15:12:49.529956972 +0000 UTC m=+6.223329687 container cleanup 3126fff5816571e7e11301859cc4536af2765da1cf451a39bf3254ac08c5c207 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:12:49.683 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:49 smithi003 bash[373435]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1 2024-09-13T15:12:50.158 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:49 smithi003 podman[373455]: 2024-09-13 15:12:49.798119337 +0000 UTC m=+0.746655702 container remove 3126fff5816571e7e11301859cc4536af2765da1cf451a39bf3254ac08c5c207 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-13T15:12:50.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:50 smithi003 podman[373550]: 2024-09-13 15:12:50.157072707 +0000 UTC m=+0.111985964 container create 6aa799884105409042bb940b9eafef6c4fbdef3e0c496ac1d41a519a0c400a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T15:12:50.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:50 smithi003 podman[373550]: 2024-09-13 15:12:50.061594677 +0000 UTC m=+0.016507934 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:12:50.433 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:50 smithi003 podman[373550]: 2024-09-13 15:12:50.384927913 +0000 UTC m=+0.339841171 container init 6aa799884105409042bb940b9eafef6c4fbdef3e0c496ac1d41a519a0c400a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:12:50.434 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:50 smithi003 podman[373550]: 2024-09-13 15:12:50.389812421 +0000 UTC m=+0.344725683 container start 6aa799884105409042bb940b9eafef6c4fbdef3e0c496ac1d41a519a0c400a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T15:12:50.872 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:50 smithi003 podman[373550]: 2024-09-13 15:12:50.476489084 +0000 UTC m=+0.431402349 container attach 6aa799884105409042bb940b9eafef6c4fbdef3e0c496ac1d41a519a0c400a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:12:50.872 INFO:journalctl@ceph.osd.1.smithi003.stdout:Sep 13 15:12:50 smithi003 podman[373550]: 2024-09-13 15:12:50.581894843 +0000 UTC m=+0.536808101 container died 6aa799884105409042bb940b9eafef6c4fbdef3e0c496ac1d41a519a0c400a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T15:12:51.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.1.service' 2024-09-13T15:12:51.292 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:12:51.292 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-13T15:12:51.293 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-13T15:12:51.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.2 2024-09-13T15:12:51.620 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:51 smithi003 systemd[1]: Stopping Ceph osd.2 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:12:51.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[133375]: 2024-09-13T15:12:51.617+0000 7efecb40a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:12:51.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[133375]: 2024-09-13T15:12:51.617+0000 7efecb40a640 -1 osd.2 507 *** Got signal Terminated *** 2024-09-13T15:12:51.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:51 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2[133375]: 2024-09-13T15:12:51.617+0000 7efecb40a640 -1 osd.2 507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:12:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:54.827Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-09-13T15:12:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:54.827Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-09-13T15:12:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:54.827Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-09-13T15:12:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:54.827Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-09-13T15:12:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:54.827Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-09-13T15:12:55.183 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:12:54 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:12:54.827Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-09-13T15:12:57.550 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:57 smithi003 podman[373678]: 2024-09-13 15:12:57.259452447 +0000 UTC m=+5.745582555 container died 357adcd35fd784ab04273b17f6138d403b09e5ca9d33f115b977c117e5021ca9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-13T15:12:57.900 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:57 smithi003 podman[373678]: 2024-09-13 15:12:57.698439413 +0000 UTC m=+6.184569522 container cleanup 357adcd35fd784ab04273b17f6138d403b09e5ca9d33f115b977c117e5021ca9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:12:57.901 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:57 smithi003 bash[373678]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2 2024-09-13T15:12:58.183 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:57 smithi003 podman[373698]: 2024-09-13 15:12:57.899804036 +0000 UTC m=+0.643101281 container remove 357adcd35fd784ab04273b17f6138d403b09e5ca9d33f115b977c117e5021ca9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T15:12:58.637 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:58 smithi003 podman[373795]: 2024-09-13 15:12:58.308809922 +0000 UTC m=+0.100665338 container create 35c9c60e04ea934f1a3305465a7053b92495d3dec836fb846a87104ae6ce9cba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-13T15:12:58.638 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:58 smithi003 podman[373795]: 2024-09-13 15:12:58.224507109 +0000 UTC m=+0.016362523 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:12:58.638 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:58 smithi003 podman[373795]: 2024-09-13 15:12:58.552980183 +0000 UTC m=+0.344835595 container init 35c9c60e04ea934f1a3305465a7053b92495d3dec836fb846a87104ae6ce9cba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-13T15:12:58.638 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:58 smithi003 podman[373795]: 2024-09-13 15:12:58.557906898 +0000 UTC m=+0.349762311 container start 35c9c60e04ea934f1a3305465a7053b92495d3dec836fb846a87104ae6ce9cba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-13T15:12:58.638 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:58 smithi003 podman[373795]: 2024-09-13 15:12:58.63601461 +0000 UTC m=+0.427870026 container attach 35c9c60e04ea934f1a3305465a7053b92495d3dec836fb846a87104ae6ce9cba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-13T15:12:58.933 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:58 smithi003 podman[373795]: 2024-09-13 15:12:58.753481464 +0000 UTC m=+0.545336883 container died 35c9c60e04ea934f1a3305465a7053b92495d3dec836fb846a87104ae6ce9cba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-13T15:12:59.293 INFO:journalctl@ceph.osd.2.smithi003.stdout:Sep 13 15:12:59 smithi003 podman[373838]: 2024-09-13 15:12:59.289322994 +0000 UTC m=+0.529335929 container remove 35c9c60e04ea934f1a3305465a7053b92495d3dec836fb846a87104ae6ce9cba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-13T15:12:59.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.2.service' 2024-09-13T15:12:59.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:12:59.351 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-13T15:12:59.351 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-13T15:12:59.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.3 2024-09-13T15:12:59.683 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:12:59 smithi003 systemd[1]: Stopping Ceph osd.3 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:00.183 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:12:59 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T15:12:59.710+0000 7f3e818be640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:13:00.183 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:12:59 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T15:12:59.710+0000 7f3e818be640 -1 osd.3 507 *** Got signal Terminated *** 2024-09-13T15:13:00.183 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:12:59 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T15:12:59.710+0000 7f3e818be640 -1 osd.3 507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:13:01.084 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:00 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T15:13:00.752+0000 7f3e7ded3640 -1 osd.3 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:36.484228+0000 front 2024-09-13T15:12:36.484257+0000 (oldest deadline 2024-09-13T15:13:00.584328+0000) 2024-09-13T15:13:01.932 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:01.657+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:02.183 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:01 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T15:13:01.761+0000 7f3e7ded3640 -1 osd.3 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:36.484228+0000 front 2024-09-13T15:12:36.484257+0000 (oldest deadline 2024-09-13T15:13:00.584328+0000) 2024-09-13T15:13:02.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:01 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:01.944+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:02.927 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:02.741+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:02.927 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:02.653+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:03.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:02 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:02.925+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:03.183 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:02 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T15:13:02.782+0000 7f3e7ded3640 -1 osd.3 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:36.484228+0000 front 2024-09-13T15:12:36.484257+0000 (oldest deadline 2024-09-13T15:13:00.584328+0000) 2024-09-13T15:13:03.932 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:03 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:03.749+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:03.932 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:03 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:03.664+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:04.183 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:03 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3[141454]: 2024-09-13T15:13:03.802+0000 7f3e7ded3640 -1 osd.3 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:36.484228+0000 front 2024-09-13T15:12:36.484257+0000 (oldest deadline 2024-09-13T15:13:00.584328+0000) 2024-09-13T15:13:04.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:03 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:03.950+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:04.932 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:04 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:04.718+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:04.932 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:04 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:04.656+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:04.933 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:04 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:04.791+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:33.632329+0000 front 2024-09-13T15:12:33.632482+0000 (oldest deadline 2024-09-13T15:13:04.232980+0000) 2024-09-13T15:13:05.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:04 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:04.993+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:05.483 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:05 smithi003 podman[373921]: 2024-09-13 15:13:05.223344877 +0000 UTC m=+5.637362911 container died c85652aac1b4ceb4fe56e8e8255e4b021ea3aef4b343a4bd711dfd45d4c796ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T15:13:05.932 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:05.790+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:33.632329+0000 front 2024-09-13T15:12:33.632482+0000 (oldest deadline 2024-09-13T15:13:04.232980+0000) 2024-09-13T15:13:05.933 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:05.672+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:05.933 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:05.723+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:05.933 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:05 smithi003 podman[373921]: 2024-09-13 15:13:05.708702231 +0000 UTC m=+6.122720264 container cleanup c85652aac1b4ceb4fe56e8e8255e4b021ea3aef4b343a4bd711dfd45d4c796ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T15:13:05.933 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:05 smithi003 bash[373921]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3 2024-09-13T15:13:06.283 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:05 smithi003 podman[373955]: 2024-09-13 15:13:05.951590327 +0000 UTC m=+0.724776141 container remove c85652aac1b4ceb4fe56e8e8255e4b021ea3aef4b343a4bd711dfd45d4c796ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-13T15:13:06.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:05 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:05.963+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:06.588 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:06 smithi003 podman[374046]: 2024-09-13 15:13:06.185410818 +0000 UTC m=+0.016322828 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:13:06.589 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:06 smithi003 podman[374046]: 2024-09-13 15:13:06.29389585 +0000 UTC m=+0.124807850 container create d72cbb81ba8e8d2d17f6a9de431b891222557e4e95302a4eb959b53aa20b1bec (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:13:06.589 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:06 smithi003 podman[374046]: 2024-09-13 15:13:06.504588135 +0000 UTC m=+0.335500136 container init d72cbb81ba8e8d2d17f6a9de431b891222557e4e95302a4eb959b53aa20b1bec (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-13T15:13:06.589 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:06 smithi003 podman[374046]: 2024-09-13 15:13:06.50961366 +0000 UTC m=+0.340525667 container start d72cbb81ba8e8d2d17f6a9de431b891222557e4e95302a4eb959b53aa20b1bec (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-13T15:13:06.933 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:06 smithi003 podman[374046]: 2024-09-13 15:13:06.587528175 +0000 UTC m=+0.418440188 container attach d72cbb81ba8e8d2d17f6a9de431b891222557e4e95302a4eb959b53aa20b1bec (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:13:06.933 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 13 15:13:06 smithi003 podman[374046]: 2024-09-13 15:13:06.70389478 +0000 UTC m=+0.534806785 container died d72cbb81ba8e8d2d17f6a9de431b891222557e4e95302a4eb959b53aa20b1bec (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:13:06.964 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:06 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:06.680+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:06.965 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:06 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:06.776+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:33.632329+0000 front 2024-09-13T15:12:33.632482+0000 (oldest deadline 2024-09-13T15:13:04.232980+0000) 2024-09-13T15:13:06.965 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:06 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:06.759+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:07.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.3.service' 2024-09-13T15:13:07.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:06 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:06.963+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:07.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:13:07.440 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-13T15:13:07.440 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-13T15:13:07.440 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.4 2024-09-13T15:13:07.723 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:07 smithi019 systemd[1]: Stopping Ceph osd.4 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:07.997 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:07 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:07.720+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:07.997 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:07 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:07.726+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:07.998 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:07 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:07.746+0000 7f0c7b280640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:13:07.998 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:07 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:07.746+0000 7f0c7b280640 -1 osd.4 507 *** Got signal Terminated *** 2024-09-13T15:13:07.998 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:07 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:07.746+0000 7f0c7b280640 -1 osd.4 507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:13:07.998 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:07 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:07.760+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:33.632329+0000 front 2024-09-13T15:12:33.632482+0000 (oldest deadline 2024-09-13T15:13:04.232980+0000) 2024-09-13T15:13:08.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:07 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:07.995+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:08.971 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:08 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:08.696+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:08.972 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:08 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:08.742+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:33.632329+0000 front 2024-09-13T15:12:33.632482+0000 (oldest deadline 2024-09-13T15:13:04.232980+0000) 2024-09-13T15:13:08.972 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:08 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:08.757+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:08.973 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:08 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:08.970+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:09.932 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:09 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:09.744+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:33.632329+0000 front 2024-09-13T15:12:33.632482+0000 (oldest deadline 2024-09-13T15:13:04.232980+0000) 2024-09-13T15:13:09.933 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:09 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:09.660+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:09.933 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:09 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:09.789+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:09.933 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:09 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:09.789+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:10.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:10.006+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:10.932 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:10.669+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:10.932 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:10.704+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:33.632329+0000 front 2024-09-13T15:12:33.632482+0000 (oldest deadline 2024-09-13T15:13:04.232980+0000) 2024-09-13T15:13:10.933 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:10.704+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.834701+0000 front 2024-09-13T15:12:45.834686+0000 (oldest deadline 2024-09-13T15:13:10.534608+0000) 2024-09-13T15:13:10.933 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:10.762+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:10.933 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:10 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:10.762+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:11.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:11.005+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:11.932 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:11.634+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:11.932 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:11.675+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:33.632329+0000 front 2024-09-13T15:12:33.632482+0000 (oldest deadline 2024-09-13T15:13:04.232980+0000) 2024-09-13T15:13:11.933 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:11.675+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.834701+0000 front 2024-09-13T15:12:45.834686+0000 (oldest deadline 2024-09-13T15:13:10.534608+0000) 2024-09-13T15:13:11.933 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:11.714+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:11.933 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:11.714+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:12.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:11 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:11.974+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:12.696 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:12 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:12.633+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:33.632329+0000 front 2024-09-13T15:12:33.632482+0000 (oldest deadline 2024-09-13T15:13:04.232980+0000) 2024-09-13T15:13:12.696 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:12 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4[38316]: 2024-09-13T15:13:12.633+0000 7f0c76896640 -1 osd.4 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.834701+0000 front 2024-09-13T15:12:45.834686+0000 (oldest deadline 2024-09-13T15:13:10.534608+0000) 2024-09-13T15:13:12.696 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:12 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:12.673+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:12.696 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:12 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:12.673+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:13.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:12 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:12.694+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:13.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:12 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:12.694+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:13.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:12 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:12.925+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:13.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:12 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:12.925+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:13.631 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:13 smithi019 podman[244143]: 2024-09-13 15:13:13.37746362 +0000 UTC m=+5.726917368 container died e50c61f762b07da7c739106b3cdce783713f959042844e7abad84945adcab553 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-13T15:13:13.906 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:13 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:13.629+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:13.906 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:13 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:13.629+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:13.907 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:13 smithi019 podman[244143]: 2024-09-13 15:13:13.777566742 +0000 UTC m=+6.127020493 container cleanup e50c61f762b07da7c739106b3cdce783713f959042844e7abad84945adcab553 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa) 2024-09-13T15:13:13.907 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:13 smithi019 bash[244143]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4 2024-09-13T15:13:13.907 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:13 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:13.680+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:13.907 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:13 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:13.680+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:14.182 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:14 smithi019 podman[244168]: 2024-09-13 15:13:14.021323124 +0000 UTC m=+0.637227951 container remove e50c61f762b07da7c739106b3cdce783713f959042844e7abad84945adcab553 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:13:14.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:13 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:13.904+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:14.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:13 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:13.904+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:14.722 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:14.582+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:14.722 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:14.582+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:14.723 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:14 smithi019 podman[244262]: 2024-09-13 15:13:14.337354968 +0000 UTC m=+0.016617141 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be 2024-09-13T15:13:14.723 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:14 smithi019 podman[244262]: 2024-09-13 15:13:14.463966459 +0000 UTC m=+0.143228666 container create 0829634f7cfe5d81ee5da143725751f009cfe9f953e0b9358285b7a818dedf85 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-13T15:13:14.723 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:14 smithi019 podman[244262]: 2024-09-13 15:13:14.674731801 +0000 UTC m=+0.353993965 container init 0829634f7cfe5d81ee5da143725751f009cfe9f953e0b9358285b7a818dedf85 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T15:13:14.723 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:14 smithi019 podman[244262]: 2024-09-13 15:13:14.679685022 +0000 UTC m=+0.358947179 container start 0829634f7cfe5d81ee5da143725751f009cfe9f953e0b9358285b7a818dedf85 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-13T15:13:15.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:14.939+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:15.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:14.939+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:15.182 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:14 smithi019 podman[244262]: 2024-09-13 15:13:14.766400651 +0000 UTC m=+0.445662813 container attach 0829634f7cfe5d81ee5da143725751f009cfe9f953e0b9358285b7a818dedf85 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-13T15:13:15.183 INFO:journalctl@ceph.osd.4.smithi019.stdout:Sep 13 15:13:14 smithi019 podman[244262]: 2024-09-13 15:13:14.917006524 +0000 UTC m=+0.596268701 container died 0829634f7cfe5d81ee5da143725751f009cfe9f953e0b9358285b7a818dedf85 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:13:15.183 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:14.720+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:15.183 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:14 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:14.720+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:15.533 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.4.service' 2024-09-13T15:13:15.575 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:13:15.576 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-13T15:13:15.576 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-13T15:13:15.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.5 2024-09-13T15:13:15.910 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:15 smithi019 systemd[1]: Stopping Ceph osd.5 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:15.910 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:15.713+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:15.911 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:15.713+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:15.911 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:15.596+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:15.911 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:15.596+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:15.911 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:15.908+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:15.912 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:15.908+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:16.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:15.944+0000 7f51a459c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:13:16.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:15.944+0000 7f51a459c640 -1 osd.5 507 *** Got signal Terminated *** 2024-09-13T15:13:16.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:15 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:15.944+0000 7f51a459c640 -1 osd.5 507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:13:16.892 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:16.702+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:16.892 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:16.702+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:16.892 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:16.608+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:16.893 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:16.608+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:17.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:16.890+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:17.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:16 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:16.890+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:17.886 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:17 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:17.655+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:17.886 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:17 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:17.655+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:17.886 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:17 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:17.629+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:17.886 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:17 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:17.629+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:18.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:17 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:17.884+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:18.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:17 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:17.884+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:18.898 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:18 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:18.596+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:18.899 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:18 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:18.596+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:18.899 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:18 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:18.632+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:18.899 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:18 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:18.632+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:19.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:18 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:18.896+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:19.209 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:18 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:18.897+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:19.209 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:18 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:18.897+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:19.884 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:19 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:19.675+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:19.884 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:19 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:19.675+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:19.885 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:19 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:19.573+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:19.885 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:19 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:19.573+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:19.885 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:19 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:19.573+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:20.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:19 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:19.882+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:20.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:19 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:19.882+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:20.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:19 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:19.882+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:20.877 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:20.720+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:35.874411+0000 front 2024-09-13T15:12:35.874486+0000 (oldest deadline 2024-09-13T15:13:01.774498+0000) 2024-09-13T15:13:20.878 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:20.720+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:45.576665+0000 front 2024-09-13T15:12:45.576630+0000 (oldest deadline 2024-09-13T15:13:09.676552+0000) 2024-09-13T15:13:20.878 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5[44436]: 2024-09-13T15:13:20.720+0000 7f519dbae640 -1 osd.5 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:54.878217+0000 front 2024-09-13T15:12:54.878365+0000 (oldest deadline 2024-09-13T15:13:20.178267+0000) 2024-09-13T15:13:20.878 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:20.609+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:20.878 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:20.609+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:20.878 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:20.609+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:21.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:20.876+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:21.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:20.876+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:21.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:20 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:20.876+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:21.844 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:21 smithi019 podman[244391]: 2024-09-13 15:13:21.66649122 +0000 UTC m=+5.856853466 container died 1cd389248627e508c56079f8f32944fbc5208891ed310d09cf79b33bda6c0526 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T15:13:21.844 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:21 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:21.568+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:21.844 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:21 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:21.568+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:21.845 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:21 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:21.568+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:22.099 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:21 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:21.842+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:22.099 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:21 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:21.842+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:22.099 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:21 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:21.842+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:22.432 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:22 smithi019 podman[244391]: 2024-09-13 15:13:22.09839137 +0000 UTC m=+6.288753649 container cleanup 1cd389248627e508c56079f8f32944fbc5208891ed310d09cf79b33bda6c0526 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:13:22.432 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:22 smithi019 bash[244391]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5 2024-09-13T15:13:22.432 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:22 smithi019 podman[244413]: 2024-09-13 15:13:22.324927868 +0000 UTC m=+0.660088849 container remove 1cd389248627e508c56079f8f32944fbc5208891ed310d09cf79b33bda6c0526 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-13T15:13:22.890 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:22 smithi019 podman[244503]: 2024-09-13 15:13:22.672852913 +0000 UTC m=+0.016768821 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be 2024-09-13T15:13:22.890 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:22 smithi019 podman[244503]: 2024-09-13 15:13:22.775751308 +0000 UTC m=+0.119667207 container create a2c6eeaafb25941a8d4417486e81531685f9dbcffb11bbb458c8391292244598 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T15:13:22.890 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:22 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:22.813+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:22.890 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:22 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:22.813+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:22.891 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:22 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:22.813+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:22.891 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:22 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:22.571+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:22.891 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:22 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:22.571+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:22.891 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:22 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:22.571+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:23.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:22 smithi019 podman[244503]: 2024-09-13 15:13:22.995027832 +0000 UTC m=+0.338943729 container init a2c6eeaafb25941a8d4417486e81531685f9dbcffb11bbb458c8391292244598 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T15:13:23.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:23 smithi019 podman[244503]: 2024-09-13 15:13:23.00004007 +0000 UTC m=+0.343955977 container start a2c6eeaafb25941a8d4417486e81531685f9dbcffb11bbb458c8391292244598 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=quincy-bbdadfa, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T15:13:23.182 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:23 smithi019 podman[244503]: 2024-09-13 15:13:23.086473155 +0000 UTC m=+0.430389066 container attach a2c6eeaafb25941a8d4417486e81531685f9dbcffb11bbb458c8391292244598 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-13T15:13:23.537 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:23 smithi019 podman[244503]: 2024-09-13 15:13:23.183665963 +0000 UTC m=+0.527581870 container died a2c6eeaafb25941a8d4417486e81531685f9dbcffb11bbb458c8391292244598 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-13T15:13:23.805 INFO:journalctl@ceph.osd.5.smithi019.stdout:Sep 13 15:13:23 smithi019 podman[244546]: 2024-09-13 15:13:23.781979659 +0000 UTC m=+0.587814934 container remove a2c6eeaafb25941a8d4417486e81531685f9dbcffb11bbb458c8391292244598 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:13:23.805 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:23 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:23.535+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:23.805 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:23 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:23.535+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:23.805 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:23 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:23.535+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:23.811 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.5.service' 2024-09-13T15:13:23.854 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:13:23.854 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-13T15:13:23.854 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-13T15:13:23.854 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.6 2024-09-13T15:13:24.179 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:23 smithi019 systemd[1]: Stopping Ceph osd.6 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:24.180 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:23 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:23.839+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:24.180 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:23 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:23.839+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:24.180 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:23 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:23.839+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:24.432 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:24.178+0000 7fd9564bd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:13:24.432 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:24.178+0000 7fd9564bd640 -1 osd.6 507 *** Got signal Terminated *** 2024-09-13T15:13:24.432 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:24.178+0000 7fd9564bd640 -1 osd.6 507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:13:24.813 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:24.534+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:24.813 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:24.534+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:24.814 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:24.534+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:25.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:24.811+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:25.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:24.811+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:25.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:24 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:24.811+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:25.841 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:25.527+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:25.841 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:25.527+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:25.842 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:25.527+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:26.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:25.840+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:26.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:25.840+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:26.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:25 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:25.840+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:26.892 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:26.552+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:26.892 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:26.552+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:26.892 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:26.552+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:26.892 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:26.552+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:00.029981+0000 front 2024-09-13T15:13:00.029964+0000 (oldest deadline 2024-09-13T15:13:25.929750+0000) 2024-09-13T15:13:27.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:26.890+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:27.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:26.890+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:27.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:26 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:26.890+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:27.897 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:27.529+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:27.897 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:27.529+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:27.897 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:27.529+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:27.897 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:27.529+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:00.029981+0000 front 2024-09-13T15:13:00.029964+0000 (oldest deadline 2024-09-13T15:13:25.929750+0000) 2024-09-13T15:13:28.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:27.895+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:28.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:27.895+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:28.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:27 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:27.895+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:28.929 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:28.579+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:34.525492+0000 front 2024-09-13T15:12:34.525478+0000 (oldest deadline 2024-09-13T15:13:00.926104+0000) 2024-09-13T15:13:28.930 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:28.579+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:47.927638+0000 front 2024-09-13T15:12:47.927541+0000 (oldest deadline 2024-09-13T15:13:12.027478+0000) 2024-09-13T15:13:28.930 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:28.579+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:52.027978+0000 front 2024-09-13T15:12:52.027965+0000 (oldest deadline 2024-09-13T15:13:19.528936+0000) 2024-09-13T15:13:28.930 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6[50370]: 2024-09-13T15:13:28.579+0000 7fd94facf640 -1 osd.6 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:00.029981+0000 front 2024-09-13T15:13:00.029964+0000 (oldest deadline 2024-09-13T15:13:25.929750+0000) 2024-09-13T15:13:29.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:28.928+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:29.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:28.928+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:29.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:28.928+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:29.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:28 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:28.928+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:04.456049+0000 front 2024-09-13T15:13:04.455998+0000 (oldest deadline 2024-09-13T15:13:28.555968+0000) 2024-09-13T15:13:30.104 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:29 smithi019 podman[244630]: 2024-09-13 15:13:29.775360056 +0000 UTC m=+5.694192012 container died aa95f3146e471922f8614927b5b295ea3fed66ceead0ea62909acfaecd0511a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-13T15:13:30.104 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:29 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:29.894+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:30.104 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:29 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:29.894+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:30.105 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:29 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:29.894+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:30.105 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:29 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:29.894+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:04.456049+0000 front 2024-09-13T15:13:04.455998+0000 (oldest deadline 2024-09-13T15:13:28.555968+0000) 2024-09-13T15:13:30.432 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:30 smithi019 podman[244630]: 2024-09-13 15:13:30.276578296 +0000 UTC m=+6.195410252 container cleanup aa95f3146e471922f8614927b5b295ea3fed66ceead0ea62909acfaecd0511a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-13T15:13:30.432 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:30 smithi019 bash[244630]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6 2024-09-13T15:13:30.852 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:30 smithi019 podman[244653]: 2024-09-13 15:13:30.53662729 +0000 UTC m=+0.759374257 container remove aa95f3146e471922f8614927b5b295ea3fed66ceead0ea62909acfaecd0511a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-09-13T15:13:31.149 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:30.871+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:31.149 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:30.871+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:31.149 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:30.871+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:31.149 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:30 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:30.871+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:04.456049+0000 front 2024-09-13T15:13:04.455998+0000 (oldest deadline 2024-09-13T15:13:28.555968+0000) 2024-09-13T15:13:31.150 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:30 smithi019 podman[244746]: 2024-09-13 15:13:30.75471842 +0000 UTC m=+0.016210799 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be 2024-09-13T15:13:31.150 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:30 smithi019 podman[244746]: 2024-09-13 15:13:30.862623252 +0000 UTC m=+0.124115661 container create ed54e8c551fbcc827b1271d3f10b3a0cc2413da9a95ecfbc0c37ec6458e60147 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-13T15:13:31.150 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:31 smithi019 podman[244746]: 2024-09-13 15:13:31.073266596 +0000 UTC m=+0.334758960 container init ed54e8c551fbcc827b1271d3f10b3a0cc2413da9a95ecfbc0c37ec6458e60147 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:13:31.150 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:31 smithi019 podman[244746]: 2024-09-13 15:13:31.078260958 +0000 UTC m=+0.339753322 container start ed54e8c551fbcc827b1271d3f10b3a0cc2413da9a95ecfbc0c37ec6458e60147 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux ) 2024-09-13T15:13:31.150 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:31 smithi019 podman[244746]: 2024-09-13 15:13:31.148150166 +0000 UTC m=+0.409642543 container attach ed54e8c551fbcc827b1271d3f10b3a0cc2413da9a95ecfbc0c37ec6458e60147 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-13T15:13:31.432 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:31 smithi019 podman[244746]: 2024-09-13 15:13:31.259026986 +0000 UTC m=+0.520519360 container died ed54e8c551fbcc827b1271d3f10b3a0cc2413da9a95ecfbc0c37ec6458e60147 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-09-13T15:13:31.794 INFO:journalctl@ceph.osd.6.smithi019.stdout:Sep 13 15:13:31 smithi019 podman[244790]: 2024-09-13 15:13:31.776647809 +0000 UTC m=+0.506277980 container remove ed54e8c551fbcc827b1271d3f10b3a0cc2413da9a95ecfbc0c37ec6458e60147 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-13T15:13:31.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.6.service' 2024-09-13T15:13:31.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:13:31.842 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-13T15:13:31.842 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-13T15:13:31.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.7 2024-09-13T15:13:32.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:31.897+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:32.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:31.897+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:32.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:31.897+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:32.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:31 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:31.897+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:04.456049+0000 front 2024-09-13T15:13:04.455998+0000 (oldest deadline 2024-09-13T15:13:28.555968+0000) 2024-09-13T15:13:32.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:31 smithi019 systemd[1]: Stopping Ceph osd.7 for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:32.682 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:32.189+0000 7fd823ce2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:13:32.682 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:32.189+0000 7fd823ce2640 -1 osd.7 507 *** Got signal Terminated *** 2024-09-13T15:13:32.683 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:32.189+0000 7fd823ce2640 -1 osd.7 507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:13:33.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:32.881+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:33.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:32.881+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:33.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:32.881+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:33.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:32 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:32.881+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:04.456049+0000 front 2024-09-13T15:13:04.455998+0000 (oldest deadline 2024-09-13T15:13:28.555968+0000) 2024-09-13T15:13:34.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:33.914+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:34.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:33.914+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:34.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:33.914+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:34.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:33.914+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:04.456049+0000 front 2024-09-13T15:13:04.455998+0000 (oldest deadline 2024-09-13T15:13:28.555968+0000) 2024-09-13T15:13:34.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:33 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:33.914+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.19:6804 osd.4 since back 2024-09-13T15:13:08.556467+0000 front 2024-09-13T15:13:08.556521+0000 (oldest deadline 2024-09-13T15:13:33.856486+0000) 2024-09-13T15:13:35.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:34.865+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:35.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:34.865+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:35.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:34.865+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:35.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:34.865+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:04.456049+0000 front 2024-09-13T15:13:04.455998+0000 (oldest deadline 2024-09-13T15:13:28.555968+0000) 2024-09-13T15:13:35.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:34 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:34.865+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.19:6804 osd.4 since back 2024-09-13T15:13:08.556467+0000 front 2024-09-13T15:13:08.556521+0000 (oldest deadline 2024-09-13T15:13:33.856486+0000) 2024-09-13T15:13:36.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:35.884+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:36.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:35.884+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:36.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:35.884+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:36.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:35.884+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:04.456049+0000 front 2024-09-13T15:13:04.455998+0000 (oldest deadline 2024-09-13T15:13:28.555968+0000) 2024-09-13T15:13:36.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:35 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:35.884+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.19:6804 osd.4 since back 2024-09-13T15:13:08.556467+0000 front 2024-09-13T15:13:08.556521+0000 (oldest deadline 2024-09-13T15:13:33.856486+0000) 2024-09-13T15:13:37.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:36.925+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-09-13T15:12:37.450007+0000 front 2024-09-13T15:12:37.450129+0000 (oldest deadline 2024-09-13T15:13:01.550090+0000) 2024-09-13T15:13:37.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:36.925+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-09-13T15:12:48.352931+0000 front 2024-09-13T15:12:48.352859+0000 (oldest deadline 2024-09-13T15:13:12.452742+0000) 2024-09-13T15:13:37.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:36.925+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-09-13T15:12:55.854288+0000 front 2024-09-13T15:12:55.854269+0000 (oldest deadline 2024-09-13T15:13:18.154259+0000) 2024-09-13T15:13:37.182 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:36.925+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-09-13T15:13:04.456049+0000 front 2024-09-13T15:13:04.455998+0000 (oldest deadline 2024-09-13T15:13:28.555968+0000) 2024-09-13T15:13:37.183 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:36 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7[56450]: 2024-09-13T15:13:36.925+0000 7fd81f2f8640 -1 osd.7 507 heartbeat_check: no reply from 172.21.15.19:6804 osd.4 since back 2024-09-13T15:13:08.556467+0000 front 2024-09-13T15:13:08.556521+0000 (oldest deadline 2024-09-13T15:13:33.856486+0000) 2024-09-13T15:13:37.983 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:37 smithi019 podman[244877]: 2024-09-13 15:13:37.704381149 +0000 UTC m=+5.634772511 container died 118a8cf6eadcc7bc36e4772ad94899fc47c2663ea4b3148826714f9e81c966eb (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-13T15:13:38.431 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:38 smithi019 podman[244877]: 2024-09-13 15:13:38.154166469 +0000 UTC m=+6.084557815 container cleanup 118a8cf6eadcc7bc36e4772ad94899fc47c2663ea4b3148826714f9e81c966eb (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T15:13:38.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:38 smithi019 bash[244877]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7 2024-09-13T15:13:38.682 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:38 smithi019 podman[244900]: 2024-09-13 15:13:38.430715395 +0000 UTC m=+0.727215123 container remove 118a8cf6eadcc7bc36e4772ad94899fc47c2663ea4b3148826714f9e81c966eb (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-13T15:13:39.043 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:38 smithi019 podman[244992]: 2024-09-13 15:13:38.693720401 +0000 UTC m=+0.016818369 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be 2024-09-13T15:13:39.043 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:38 smithi019 podman[244992]: 2024-09-13 15:13:38.806403169 +0000 UTC m=+0.129501123 container create 9dfc4eaf5146834647a2ecafcd0bfe36751401254f7c272232ff3abb66af1d3e (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:13:39.043 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:39 smithi019 podman[244992]: 2024-09-13 15:13:39.042376123 +0000 UTC m=+0.365474077 container init 9dfc4eaf5146834647a2ecafcd0bfe36751401254f7c272232ff3abb66af1d3e (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:13:39.432 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:39 smithi019 podman[244992]: 2024-09-13 15:13:39.047556532 +0000 UTC m=+0.370654486 container start 9dfc4eaf5146834647a2ecafcd0bfe36751401254f7c272232ff3abb66af1d3e (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, RELEASE=quincy-bbdadfa, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-13T15:13:39.433 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:39 smithi019 podman[244992]: 2024-09-13 15:13:39.133725262 +0000 UTC m=+0.456823229 container attach 9dfc4eaf5146834647a2ecafcd0bfe36751401254f7c272232ff3abb66af1d3e (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-13T15:13:39.433 INFO:journalctl@ceph.osd.7.smithi019.stdout:Sep 13 15:13:39 smithi019 podman[244992]: 2024-09-13 15:13:39.229291761 +0000 UTC m=+0.552389725 container died 9dfc4eaf5146834647a2ecafcd0bfe36751401254f7c272232ff3abb66af1d3e (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-13T15:13:39.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@osd.7.service' 2024-09-13T15:13:39.950 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:13:39.950 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-13T15:13:39.950 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-13T15:13:39.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a 2024-09-13T15:13:40.182 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 systemd[1]: Stopping Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:40.563 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.292Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T15:13:40.563 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.293Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-13T15:13:40.564 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.293Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-13T15:13:40.564 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.293Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T15:13:40.564 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.293Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-13T15:13:40.564 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.293Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-13T15:13:40.564 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.293Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T15:13:40.564 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.293Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-13T15:13:40.565 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.293Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-13T15:13:40.565 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.296Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T15:13:40.565 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.296Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-13T15:13:40.565 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a[81170]: ts=2024-09-13T15:13:40.296Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-13T15:13:40.565 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 podman[245119]: 2024-09-13 15:13:40.303097491 +0000 UTC m=+0.110622491 container died 0b10348ed2daad9e85f8fd1b82edaf6e6be981522e1b701656de477c9d594541 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T15:13:40.842 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 podman[245119]: 2024-09-13 15:13:40.648348061 +0000 UTC m=+0.455873060 container cleanup 0b10348ed2daad9e85f8fd1b82edaf6e6be981522e1b701656de477c9d594541 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T15:13:40.842 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 bash[245119]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T15:13:41.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-13T15:13:41.093 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:40 smithi019 podman[245136]: 2024-09-13 15:13:40.840902127 +0000 UTC m=+0.533174125 container remove 0b10348ed2daad9e85f8fd1b82edaf6e6be981522e1b701656de477c9d594541 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T15:13:41.093 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:41 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T15:13:41.093 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:41 smithi019 systemd[1]: Stopped Ceph prometheus.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T15:13:41.093 INFO:journalctl@ceph.prometheus.a.smithi019.stdout:Sep 13 15:13:41 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 10.446s CPU time. 2024-09-13T15:13:41.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:13:41.105 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-13T15:13:41.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 77a3f016-71da-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T15:13:42.933 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:42 smithi003 systemd[1]: Stopping Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:43.353 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:43 smithi003 ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a[92165]: ts=2024-09-13T15:13:43.043Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T15:13:43.353 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:43 smithi003 podman[374474]: 2024-09-13 15:13:43.094836044 +0000 UTC m=+0.157687016 container died b74b601869dcd87580033730e1508b6ca64775fa6e77d47d379bf70a1208a369 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:13:43.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:43 smithi003 podman[374474]: 2024-09-13 15:13:43.557713131 +0000 UTC m=+0.620564090 container cleanup b74b601869dcd87580033730e1508b6ca64775fa6e77d47d379bf70a1208a369 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:13:43.683 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:43 smithi003 bash[374474]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T15:13:44.128 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:43 smithi003 podman[374491]: 2024-09-13 15:13:43.850547259 +0000 UTC m=+0.754735580 container remove b74b601869dcd87580033730e1508b6ca64775fa6e77d47d379bf70a1208a369 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:13:44.128 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:43 smithi003 podman[374491]: 2024-09-13 15:13:43.951615641 +0000 UTC m=+0.855803916 volume remove 55a07eca77a17f394407a9f194e29610997633f60b685fd897382190c8136a63 2024-09-13T15:13:44.433 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 15:13:44 smithi003 systemd[1]: Stopping Ceph node-exporter.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:44.433 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:44 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T15:13:44.433 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:44 smithi003 systemd[1]: Stopped Ceph alertmanager.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T15:13:44.434 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Sep 13 15:13:44 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 4.247s CPU time. 2024-09-13T15:13:44.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 15:13:44 smithi003 podman[374606]: 2024-09-13 15:13:44.600489554 +0000 UTC m=+0.117631073 container died 30d257fdd98b6b331315138d554f432b8af83eca585f5afb1ff8520faf859de3 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:13:45.324 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 15:13:45 smithi003 podman[374606]: 2024-09-13 15:13:45.01263033 +0000 UTC m=+0.529771891 container cleanup 30d257fdd98b6b331315138d554f432b8af83eca585f5afb1ff8520faf859de3 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:13:45.324 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 15:13:45 smithi003 bash[374606]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T15:13:45.581 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 15:13:45 smithi003 podman[374625]: 2024-09-13 15:13:45.322384305 +0000 UTC m=+0.721431682 container remove 30d257fdd98b6b331315138d554f432b8af83eca585f5afb1ff8520faf859de3 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:13:45.581 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 15:13:45 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:13:45.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 15:13:45 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T15:13:45.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 15:13:45 smithi003 systemd[1]: Stopped Ceph node-exporter.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T15:13:45.933 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Sep 13 15:13:45 smithi003 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 10.245s CPU time. 2024-09-13T15:13:52.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 77a3f016-71da-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T15:13:54.410 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:13:54 smithi019 systemd[1]: Stopping Ceph grafana.a for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:54.680 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:13:54 smithi019 podman[245545]: 2024-09-13 15:13:54.448704505 +0000 UTC m=+0.120642797 container died 1d1343f3e5db9bde828ff653b38ea57003ddf791409dab273613d1e4141f00a4 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, io.buildah.version=1.32.0, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, description=Ceph Grafana Container, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, name=ubi8, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897) 2024-09-13T15:13:54.932 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:13:54 smithi019 podman[245545]: 2024-09-13 15:13:54.794063205 +0000 UTC m=+0.466001541 container cleanup 1d1343f3e5db9bde828ff653b38ea57003ddf791409dab273613d1e4141f00a4 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, io.openshift.expose-services=, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.buildah.version=1.32.0, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., vcs-type=git, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, name=ubi8, version=8.5, description=Ceph Grafana Container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-13T15:13:54.932 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:13:54 smithi019 bash[245545]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a 2024-09-13T15:13:55.422 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:13:55 smithi019 podman[245563]: 2024-09-13 15:13:55.103939397 +0000 UTC m=+0.656246469 container remove 1d1343f3e5db9bde828ff653b38ea57003ddf791409dab273613d1e4141f00a4 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-grafana-a, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, com.redhat.component=ubi8-container, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vendor=Red Hat, Inc., name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0) 2024-09-13T15:13:55.423 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:13:55 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:13:55.423 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:13:55 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-13T15:13:55.423 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:13:55 smithi019 systemd[1]: Stopped Ceph grafana.a for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T15:13:55.423 INFO:journalctl@ceph.grafana.a.smithi019.stdout:Sep 13 15:13:55 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@grafana.a.service: Consumed 33.761s CPU time. 2024-09-13T15:13:55.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 15:13:55 smithi019 systemd[1]: Stopping Ceph node-exporter.b for 77a3f016-71da-11ef-bceb-c7b262605968... 2024-09-13T15:13:55.950 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 15:13:55 smithi019 podman[245676]: 2024-09-13 15:13:55.686568151 +0000 UTC m=+0.117934904 container died 738f7b07d0d84a6f3f13ce1d070ad599b7aef1ab956ffea3cf6abfb5a4f47c31 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:13:56.417 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 15:13:56 smithi019 podman[245676]: 2024-09-13 15:13:56.140245408 +0000 UTC m=+0.571612206 container cleanup 738f7b07d0d84a6f3f13ce1d070ad599b7aef1ab956ffea3cf6abfb5a4f47c31 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:13:56.417 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 15:13:56 smithi019 bash[245676]: ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T15:13:56.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 15:13:56 smithi019 podman[245694]: 2024-09-13 15:13:56.416458115 +0000 UTC m=+0.731681768 container remove 738f7b07d0d84a6f3f13ce1d070ad599b7aef1ab956ffea3cf6abfb5a4f47c31 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-77a3f016-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:13:56.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 15:13:56 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:13:56.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 15:13:56 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T15:13:56.682 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 15:13:56 smithi019 systemd[1]: Stopped Ceph node-exporter.b for 77a3f016-71da-11ef-bceb-c7b262605968. 2024-09-13T15:13:56.683 INFO:journalctl@ceph.node-exporter.b.smithi019.stdout:Sep 13 15:13:56 smithi019 systemd[1]: ceph-77a3f016-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 9.442s CPU time. 2024-09-13T15:13:59.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:13:59.821 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:13:59.850 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-13T15:13:59.852 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903558/remote/smithi003/crash 2024-09-13T15:13:59.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/crash -- . 2024-09-13T15:13:59.899 INFO:teuthology.orchestra.run.smithi003.stderr:tar: /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T15:13:59.899 INFO:teuthology.orchestra.run.smithi003.stderr:tar: Error is not recoverable: exiting now 2024-09-13T15:13:59.901 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903558/remote/smithi019/crash 2024-09-13T15:13:59.902 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/crash -- . 2024-09-13T15:13:59.949 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/77a3f016-71da-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T15:13:59.950 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now 2024-09-13T15:13:59.951 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-13T15:13:59.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v OSD_DOWN | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v OSDMAP_FLAGS | egrep -v PG_AVAILABILITY | head -n 1 2024-09-13T15:14:00.038 INFO:teuthology.orchestra.run.smithi003.stdout:2024-09-13T14:40:00.000153+0000 mon.a (mon.0) 775 : cluster [WRN] pg 8.0 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [1,5] 2024-09-13T15:14:00.039 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-13T15:14:00.039 INFO:tasks.cephadm:Compressing logs... 2024-09-13T15:14:00.040 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T15:14:00.042 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T15:14:00.078 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T15:14:00.079 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-13T15:14:00.079 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi019.ydtwbg.log 2024-09-13T15:14:00.079 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T15:14:00.079 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-13T15:14:00.079 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi019.ydtwbg.log: /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-volume.log: 62.6% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi019.ydtwbg.log.gz 2024-09-13T15:14:00.085 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T15:14:00.085 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T15:14:00.085 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-13T15:14:00.086 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-13T15:14:00.086 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.log 2024-09-13T15:14:00.086 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-13T15:14:00.086 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-13T15:14:00.086 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T15:14:00.087 INFO:teuthology.orchestra.run.smithi003.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T15:14:00.087 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-13T15:14:00.087 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-13T15:14:00.087 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T15:14:00.087 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.0.log: /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T15:14:00.087 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-13T15:14:00.088 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T15:14:00.088 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-13T15:14:00.088 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi003.zmhiel.log 2024-09-13T15:14:00.088 INFO:teuthology.orchestra.run.smithi003.stderr:gzip/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-13T15:14:00.088 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi003.zmhiel.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-13T15:14:00.088 INFO:teuthology.orchestra.run.smithi003.stderr: 81.9% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi003.zmhiel.log.gz 2024-09-13T15:14:00.088 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.2.log: -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.log 2024-09-13T15:14:00.089 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-13T15:14:00.089 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T15:14:00.096 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 92.8% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T15:14:00.231 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 84.5% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T15:14:00.262 INFO:teuthology.orchestra.run.smithi003.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T15:14:00.333 INFO:teuthology.orchestra.run.smithi003.stderr: 91.2% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T15:14:00.402 INFO:teuthology.orchestra.run.smithi019.stderr: 91.3% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T15:14:00.552 INFO:teuthology.orchestra.run.smithi003.stderr: 93.9% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T15:14:00.558 INFO:teuthology.orchestra.run.smithi019.stderr: 89.8% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T15:14:00.680 INFO:teuthology.orchestra.run.smithi019.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T15:14:00.849 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T15:14:01.042 INFO:teuthology.orchestra.run.smithi003.stderr: 91.1% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T15:14:01.357 INFO:teuthology.orchestra.run.smithi019.stderr: 90.0% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-13T15:14:03.293 INFO:teuthology.orchestra.run.smithi003.stderr: 89.4% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-13T15:14:26.686 INFO:teuthology.orchestra.run.smithi003.stderr: 91.9% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-13T15:14:31.296 INFO:teuthology.orchestra.run.smithi019.stderr: 92.1% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-13T15:14:54.251 INFO:teuthology.orchestra.run.smithi003.stderr: 90.1% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-13T15:38:42.693 INFO:teuthology.orchestra.run.smithi019.stderr: 93.6% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-13T15:44:52.255 INFO:teuthology.orchestra.run.smithi019.stderr: 92.5% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-13T15:46:13.503 INFO:teuthology.orchestra.run.smithi019.stderr: 92.9% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-13T15:47:12.321 INFO:teuthology.orchestra.run.smithi019.stderr: 92.2% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-13T15:47:12.342 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-13T15:47:12.342 INFO:teuthology.orchestra.run.smithi019.stderr:real 33m12.284s 2024-09-13T15:47:12.342 INFO:teuthology.orchestra.run.smithi019.stderr:user 11m34.942s 2024-09-13T15:47:12.342 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m33.534s 2024-09-13T15:49:41.394 INFO:teuthology.orchestra.run.smithi003.stderr: 93.2% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-13T15:53:04.563 INFO:teuthology.orchestra.run.smithi003.stderr: 93.0% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-13T15:54:06.335 INFO:teuthology.orchestra.run.smithi003.stderr: 92.5% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-13T15:54:43.361 INFO:teuthology.orchestra.run.smithi003.stderr: 92.4% -- replaced with /var/log/ceph/77a3f016-71da-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-13T15:54:43.431 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-09-13T15:54:43.432 INFO:teuthology.orchestra.run.smithi003.stderr:real 40m43.375s 2024-09-13T15:54:43.432 INFO:teuthology.orchestra.run.smithi003.stderr:user 13m55.822s 2024-09-13T15:54:43.432 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m39.852s 2024-09-13T15:54:43.433 INFO:tasks.cephadm:Archiving logs... 2024-09-13T15:54:43.434 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903558/remote/smithi003/log 2024-09-13T15:54:43.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T15:56:46.418 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903558/remote/smithi019/log 2024-09-13T15:56:46.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T15:58:28.020 INFO:tasks.cephadm:Removing cluster... 2024-09-13T15:58:28.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 77a3f016-71da-11ef-bceb-c7b262605968 --force 2024-09-13T15:58:30.011 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 77a3f016-71da-11ef-bceb-c7b262605968 --force 2024-09-13T15:58:31.585 INFO:tasks.cephadm:Removing cephadm ... 2024-09-13T15:58:31.585 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T15:58:31.605 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T15:58:31.622 INFO:tasks.cephadm:Teardown complete 2024-09-13T15:58:31.622 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T15:58:31.634 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T15:58:31.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T15:58:31.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T15:58:31.765 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T15:58:31.774 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T15:58:31.775 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:58:31.778 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:58:31.793 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:58:31.794 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:58:31.862 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:58:31.863 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-09-13T15:58:31.863 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv01.front.sepia.ceph.com 3 6 377 59 +230us[ +244us] +/- 41ms 2024-09-13T15:58:31.863 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 60 -575us[ -561us] +/- 58ms 2024-09-13T15:58:31.863 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 124 +370us[ +394us] +/- 84ms 2024-09-13T15:58:31.863 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:58:31.873 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:58:31.873 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-09-13T15:58:31.873 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv01.front.sepia.ceph.com 3 7 377 2 +221us[ +235us] +/- 42ms 2024-09-13T15:58:31.873 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 59 -599us[ -585us] +/- 58ms 2024-09-13T15:58:31.874 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 188 +354us[ +382us] +/- 83ms 2024-09-13T15:58:31.874 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:58:31.883 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T15:58:31.894 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T15:58:31.894 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T15:58:31.905 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T15:58:32.055 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T15:58:32.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T15:58:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:58:32.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T15:58:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:58:32.233 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T15:58:32.243 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T15:58:32.252 INFO:teuthology.task.internal:Duration was 6642.020124 seconds 2024-09-13T15:58:32.252 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T15:58:32.262 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T15:58:32.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T15:58:32.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T15:58:32.325 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:58:32.336 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:58:32.769 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T15:58:32.769 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-09-13T15:58:32.769 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T15:58:32.793 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-09-13T15:58:32.793 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T15:58:32.826 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T15:58:32.827 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T15:58:32.836 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T15:58:32.909 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T15:58:32.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T15:58:32.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T15:58:34.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T15:58:34.857 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T15:58:34.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T15:58:34.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T15:58:34.918 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T15:58:34.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:58:34.931 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:58:34.971 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-09-13T15:58:35.010 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-09-13T15:58:35.036 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:58:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:58:35.051 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:58:35.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:58:35.068 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T15:58:35.079 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T15:58:35.080 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903558/remote/smithi003 2024-09-13T15:58:35.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T15:58:35.167 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903558/remote/smithi019 2024-09-13T15:58:35.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T15:58:35.219 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T15:58:35.220 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T15:58:35.222 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T15:58:35.268 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T15:58:35.279 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T15:58:35.279 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T15:58:35.292 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T15:58:35.292 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T15:58:35.296 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T15:58:35.326 INFO:teuthology.orchestra.run.smithi019.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 15:58 /home/ubuntu/cephtest 2024-09-13T15:58:35.338 INFO:teuthology.orchestra.run.smithi003.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:58 /home/ubuntu/cephtest 2024-09-13T15:58:35.338 INFO:teuthology.orchestra.run.smithi003.stdout: 295263 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:09 /home/ubuntu/cephtest/mnt.0 2024-09-13T15:58:35.338 INFO:teuthology.orchestra.run.smithi003.stdout: 295265 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:09 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T15:58:35.338 INFO:teuthology.orchestra.run.smithi003.stdout: 6553603 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 15:09 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T15:58:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:58:35.339 INFO:teuthology.orchestra.run.smithi003.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-13T15:58:35.339 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T15:58:35.346 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T15:58:35.356 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T15:58:35.367 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi003 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T15:58:35.371 INFO:teuthology.run:Summary data: description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} duration: 6642.020124197006 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi003 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_akupczyk@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=47d64360dfd044f687078225323e4dec status: fail success: false 2024-09-13T15:58:35.371 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T15:58:35.504 INFO:teuthology.run:FAIL